Alexnet, which started the deep learning revolution, was loosely based on a network architecture(LENet) proposed by Yann Lecun in 1998. However, back then, we didn’t have the compute or the training data to train and produce the results like Alexnet. Alex used Nvidia GPUs for training, in fact he used two GPUs to train which split the network. Before that, GPUs were a toy for computer graphics and computer gamers. After Alexnet, GPUs were intensively used for training neural networks resulting in a multifold rise in revenue of Nvidia.
Rise of GPUs and NVidia:
In 2013, Google brain built a cluster of 2000 CPUs for deep learning costing 0.5 Million dollars. A similar setup can be created using just 3 GPUs with a cost of 33,000 dollars.
A CPU consists of a few cores which are optimized for sequential serial processing, on the other hand a GPU has massively parallel architecture consisting of thousands of smaller, more efficient cores designed for handling multiple tasks simultaneously.
So, for deep learning, the CPU uses a GPU to run the most computationally expensive part of the algorithm while simple tasks are run on the CPU. So, In order to implement any decent deep learning algorithm, you need to have access to a CPU with a decent NVidia GPU.
CUDA (Compute Unified Device Architecture):
CUDA is NVIDIA’s parallel computing platform based on C. The point of CUDA is to allow developers to write code that can run on massively parallel architectures. In a way, CUDA is a software development toolkit that includes libraries, bindings and various debugging, profiling, compiling tools that allow programs running on CPUs to utilize the GPU.
cuDNN:
NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library for deep learning that runs alongside CUDA. cuDNN provides highly optimized implementations for standard neural network routines and operations. Using Cudnn provides huge acceleration to Deep Learning training and running deep learning training with CUDA and cuDNN is almost a requirement for any real-world deep learning application.
Performance Gain: Systems with CUDA accelerated GPUs can be 5-50x faster than a system with just a CPU.
Various libraries and frameworks:
There are many dedicated libraries which makes the task of implementing Artificial Neural Networks simple and quick. Here is a quick summary of few of the most popular ones:
- Theano:
Theano was a Python framework developed at the University of Montreal and run by Yoshua Bengio for research and development into state of the art deep learning algorithms. It used to be one of the most popular deep learning libraries. The official support of Theano ceased in 2017.
- Torch:
Torch (also called Torch7) is a Lua based deep learning framework developed by Clement Farabet, Ronan Collobert and Koray Kavukcuoglu for research and development into deep learning algorithms. The torch is used and has been further developed by the Facebook AI lab.
- Tensorflow: Tensorflow is an open-source Deep Learning by Google. It draws its popularity from its distributed training support, scalable production deployment options, and support for various devices like Android. It’s the same infrastructure that runs Google’s machine learning infrastructure in production.
- Caffe:
Caffe is a Python deep learning library developed by Yangqing Jia at the University of Berkeley for supervised computer vision problems. It used to be the most popular deep learning library in use. It only supports convolutional neural networks.
- MXNet: built and supported by Amazon, MxNet is also supported by the Apache foundation.
- CNTK: CNTK framework is maintained and supported by Microsoft.
- Caffe2: Supported by Facebook, built on the original Caffe. It is designed with expression, speed, and modularity in mind.
- PyTorch: PyTorch is one of the most popular deep learning framework which is gaining popularity due to its simplicity and ease of use.
- DeepLearning4j(DL4J): DeepLearning4J is another deep Learning framework developed in Java by Adam Gibson.
“DL4J is a JVM-based, industry-focused, commercially supported, distributed deep-learning framework intended to solve problems involving massive amounts of data in a reasonable amount of time.”
As you can see, that almost every large technology company has its own framework. In fact, almost every year a new framework has risen to a new height, leading to a lot of pain and re-skilling required for deep learning practitioners.
The world of Deep Learning is very fragmented and evolving very fast. Look at this tweet by Karpathy:
Imagine the pain all of us have been enduring, of learning a new framework every year. As of now, it appears that Tensorflow is here to stay.
Keras:
François Chollet, who works at Google developed Keras as a wrapper on top of Theano for quick prototyping. Later this was expanded for Tensorflow as back-end. Later, Tensorflow has decided to adopt it and provide it as part of contrib folder in the Tensorflow code.
Keras is being hailed as the future of building neural networks. Here are some of the reasons for its popularity:
- Light-weight and quick: Keras is designed to remove boilerplate code. Few lines of keras code will achieve so much more than native Tensorflow code. You can easily design both CNN and RNNs and can run them on either GPU or CPU.
- Emerging possible winner: Keras is an API which runs on top of a back-end. This back-end could be either Tensorflow or Theano. Microsoft is also working to provide CNTK as a back-end to Keras.
Currently, Keras is one of the fastest growing libraries for deep learning. The power of being able to run the same code with different back-end is a great reason for choosing Keras. Imagine, you read a paper which seems to be doing something so interesting that you want to try with your own dataset. Let’s say you work with Tensorflow and don’t know much about Theano, then you will have to implement the paper in Tensorflow, which obviously will take longer. Now, If the code is written in Keras all you have to do is change the back-end to Tensorflow. This will turbocharge collaborations for the whole community.
Pytorch got very popular for its dynamic computational graph and efficient memory usage.While, In Tensorflow, the graph is static and you need to define the graph before running your model. Although, Tensorflow also introduced Eager execution to add the dynamic graph capability. Dynamic graph is very suitable for certain use-cases like working with text.
MxNet is more of a deep learning framework for R. Although, it supports multiple layers and keras has got mxnet as a backend.
To solve this problem Open Neural Network Exchange (ONNX) was announced in the end on 2017 which aims to solve the compatibility issues among frameworks. So, you can train a network in PyTorch and deploy it in Caffe2. It currently supports MXNet, Caffe2, Pytorch, CNTK(Read Amazon, Microsoft, and Facebook). So, that could be a good thing for the overall community. However, it’s still too early to know. I would love if Tensorflow joins the alliance. That will be a force to reckon with.
One of the most awesome and useful things in Tensorflow is Tensorboard visualization. In general, during the training, one has to have multiple runs to tune the hyperparameters or identify any potential data issues. Using Tensorboard makes it very easy to visualize and spot problems.
Tensorflow Serving is another reason why Tensorflow is an absolute darling of the industry. This specialized grpc server is the same infrastructure that Google uses to deploy its models in production so it’s robust and tested for scale. In Tensorflow Serving, the models can be hot-swapped without bringing the service down which can be a crucial reason for many businesses.
Pytorch is easy to learn and easy to code. For the lovers of oop programming, torch.nn.Module allows for creating reusable code which is very developer-friendly. Pytorch is great for rapid prototyping and small scale or academic projects.
TensorFlow code gets converted into a graph by Python which is then run by the TensorFlow execution engine.
In Tensorflow, the entire graph(with parameters) can be saved as a protocol buffer which can then be deployed to non-pythonic infrastructure like Java.
One downside with Tensorflow device management is that it hogs all the memory on all the available GPUs by default even if only one of them is being used.
Support for Theano library was discontinued in 2017. Theano and many other libraries like Torch are getting sidelined with the new frameworks. The most suitable framework that is supposed to stand the test of time is Tensorflow with the heavy backing of Google. I don’t feel the same confidence with Caffe2 because Facebook is not known for a community-friendly approach to open-source software. Another thing, that is definitely going to survive in Keras. So, in this course, we shall learn these two libraries as they are an optimal choice for a long career in AI.
Google’s TPUs:
Tensor Processing Units (TPUs) are Google’s own custom chips for running machine learning workloads written for its TensorFlow framework. Google has been using TPUs on their data center for more than a year now, Currently, TPUs are not available to be bought separately, but one can access them using Google’s cloud platform. Compared to GPUs, TPUs are designed for a high volume of low precision computation with lower power consumption.
Key People in Industry and Academicians:
There are certain people that you must know and follow:
- Geoffrey Hinton:
Considered the GodFather of AI, Geoffrey Hinton divides his time between Google and the University of Toronto. He is known for many key original contributions such as backpropagation and Boltzmann machines.
- Yann Lecun:
Currently at Facebook, Yann Lecun is known for his contributions to convolutional neural networks which are one of the most fundamental concepts in Deep Learning.
- Yoshua Bengio:
Bengio is known for his fundamental work in autoencoders, neural machine translation, and generative adversarial networks.
- Andrew NG:
Founder of Coursera and former head of AI research at Baidu, Andrew NG is one of the key proponents of AI.
Edge Computing:
Since deep learning-based algorithms are very slow on CPUs, it becomes a problem to run them on a device in the wild. Imagine trying to do facial recognition on the data captured by a camera, it seems like a waste of time and resources to send the whole video to a cloud based server running 24×7. Putting a device at the point of action which processes the data locally is called edge computing. There are only a handful of devices like Nvidia Jetson tx1 and tx2 which are considered to be good candidates for edge computing.
Artificial Super Intelligence?:
Depending on it’s capability, AI can be classified into two types:
Artificial Narrow Intelligence (ANI): In other words Weak AI, Artificial Narrow Intelligence is AI that specializes in one task. AlphaGo, the AI program that beat the world Go champion in 2017, is only good at Go. Ask it to identify cats from dogs, and it would fail. AI systems that are being built today are mostly Weak AI. There have been attempts to train them at multiple tasks but the results are not that encouraging.
Artificial General Intelligence (AGI): Human-Level AI, Artificial General Intelligence refers to a system that is as smart as a human in performing different tasks—a machine that can perform any intellectual task that a human being can. We see this kind of AI in our movies and books. We are yet to create any kind of general AI, in fact, we are quite far from creating any.
Artificial Superintelligence (ASI): Artificial Super Intelligence refers to an intellect that is much better than the best human brains in every task including reasoning, science, creativity and social skills. Artificial Superintelligence could be just a little smarter than a human or a million times smarter. Hinton has stated that ASI seems more than 50 years away, but he also warns that “there is not a good track record of less intelligent things controlling things of greater intelligence”.
Will we ever be able to build a superintelligence? If yes, will that mean end of the human race? All these questions make AI a topic of discussion at parties and the topic of various movies and books.
Reference:
https://developer.nvidia.com/cudnn
One model to learn then all https://arxiv.org/abs/1706.05137
Original post: https://cv-tricks.com/how-to/deep-learning-eco-system-for-beginners/
I’m the proprietor of JustCBD Store label (justcbdstore.com) and I’m presently trying to grow my wholesale side of company. It would be great if anybody at targetdomain is able to provide some guidance . I considered that the very best way to do this would be to connect to vape companies and cbd stores. I was really hoping if anybody could suggest a reliable web-site where I can purchase Vape Shop B2B Leads I am currently examining creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. On the fence which one would be the very best choice and would appreciate any assistance on this. Or would it be much simpler for me to scrape my own leads? Suggestions?
I am the proprietor of JustCBD Store company (justcbdstore.com) and am trying to develop my wholesale side of business. It would be great if someone at targetdomain share some guidance 🙂 I thought that the most effective way to do this would be to talk to vape stores and cbd retailers. I was really hoping if anybody at all could recommend a qualified site where I can purchase CBD Shops Business Data I am currently taking a look at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Not sure which one would be the very best solution and would appreciate any support on this. Or would it be easier for me to scrape my own leads? Ideas?
Greetings! Very useful advice in this particular article! It’s the little changes that make the greatest changes. Many thanks for sharing!
After looking at a few of the articles on your site, I truly appreciate your technique of blogging. I book-marked it to my bookmark webpage list and will be checking back soon. Take a look at my web site too and tell me how you feel.
You are so interesting! I do not believe I’ve read through anything like this before. So wonderful to find somebody with genuine thoughts on this topic. Seriously.. thank you for starting this up. This site is something that’s needed on the internet, someone with some originality!
Good post. I learn something new and challenging on sites I stumbleupon on a daily basis. It’s always useful to read articles from other writers and use something from other websites.
Hello there! This article could not be written any better! Looking through this article reminds me of my previous roommate! He constantly kept talking about this. I most certainly will send this post to him. Pretty sure he’ll have a great read. Many thanks for sharing!
I would like to thank you for the efforts you have put in penning this website. I really hope to check out the same high-grade blog posts by you later on as well. In truth, your creative writing abilities has inspired me to get my very own website now 😉
Excellent blog post. I definitely appreciate this website. Stick with it!
I’m impressed, I have to admit. Rarely do I come across a blog that’s both equally educative and entertaining, and without a doubt, you have hit the nail on the head. The problem is an issue that not enough folks are speaking intelligently about. Now i’m very happy I came across this in my search for something concerning this.
Spot on with this write-up, I seriously believe that this web site needs far more attention. I’ll probably be back again to read more, thanks for the advice!
Very nice blog post. I certainly appreciate this website. Keep writing!
You are so awesome! I do not suppose I have read anything like that before. So good to discover another person with original thoughts on this subject. Seriously.. thank you for starting this up. This website is one thing that is needed on the internet, someone with a little originality!
You need to be a part of a contest for one of the most useful blogs online. I will recommend this site!
I have to thank you for the efforts you have put in penning this site. I really hope to check out the same high-grade blog posts from you later on as well. In fact, your creative writing abilities has inspired me to get my very own blog now 😉
I want to to thank you for this very good read!! I absolutely loved every bit of it. I’ve got you saved as a favorite to check out new things you post…
An impressive share! I’ve just forwarded this onto a co-worker who was doing a little research on this. And he in fact ordered me lunch simply because I discovered it for him… lol. So let me reword this…. Thank YOU for the meal!! But yeah, thanx for spending the time to talk about this matter here on your internet site.
There’s definately a lot to learn about this issue. I like all the points you made.
I’m impressed, I have to admit. Rarely do I encounter a blog that’s equally educative and interesting, and let me tell you, you have hit the nail on the head. The issue is something too few people are speaking intelligently about. I am very happy I found this in my hunt for something relating to this.
Greetings, There’s no doubt that your website might be having browser compatibility problems. Whenever I take a look at your site in Safari, it looks fine but when opening in Internet Explorer, it’s got some overlapping issues. I simply wanted to give you a quick heads up! Aside from that, excellent website!
Nice post. I learn something new and challenging on sites I stumbleupon on a daily basis. It will always be interesting to read through articles from other authors and practice a little something from their websites.
Excellent site you have got here.. It’s hard to find good quality writing like yours nowadays. I seriously appreciate individuals like you! Take care!!
I’m amazed, I have to admit. Seldom do I come across a blog that’s both equally educative and entertaining, and without a doubt, you’ve hit the nail on the head. The problem is something which not enough folks are speaking intelligently about. I’m very happy that I stumbled across this in my search for something concerning this.
Can I just say what a comfort to uncover someone who really understands what they are talking about on the web. You certainly understand how to bring an issue to light and make it important. More and more people should read this and understand this side of the story. It’s surprising you are not more popular because you certainly have the gift.
That is a great tip particularly to those fresh to the blogosphere. Simple but very precise info… Thanks for sharing this one. A must read post!
I couldn’t refrain from commenting. Well written!
Oh my goodness! Amazing article dude! Thank you, However I am going through difficulties with your RSS. I don’t understand the reason why I can’t join it. Is there anybody having the same RSS problems? Anybody who knows the answer can you kindly respond? Thanks!!
This site was… how do you say it? Relevant!! Finally I have found something which helped me. Kudos!
Excellent post! We will be linking to this great post on our website. Keep up the good writing.
Good blog you have here.. It’s hard to find good quality writing like yours nowadays. I honestly appreciate individuals like you! Take care!!
Way cool! Some very valid points! I appreciate you writing this article and the rest of the site is very good.
After going over a number of the blog posts on your web site, I seriously like your technique of blogging. I added it to my bookmark site list and will be checking back soon. Please visit my web site as well and tell me what you think.
Having read this I believed it was really informative. I appreciate you finding the time and effort to put this informative article together. I once again find myself personally spending a lot of time both reading and leaving comments. But so what, it was still worth it!
Everyone loves it when folks come together and share ideas. Great website, stick with it!
Having read this I believed it was rather informative. I appreciate you finding the time and effort to put this content together. I once again find myself personally spending a lot of time both reading and commenting. But so what, it was still worth it!
This is a topic that is near to my heart… Many thanks! Where are your contact details though?
There is definately a lot to find out about this topic. I like all the points you have made.
I wanted to thank you for this excellent read!! I certainly loved every little bit of it. I’ve got you book marked to check out new things you post…
Hello there! I could have sworn I’ve been to this site before but after looking at some of the articles I realized it’s new to me. Nonetheless, I’m definitely pleased I discovered it and I’ll be book-marking it and checking back often!
Pretty! This was a really wonderful article. Many thanks for supplying this information.
Howdy! This blog post could not be written any better! Looking through this article reminds me of my previous roommate! He continually kept talking about this. I’ll send this article to him. Pretty sure he’s going to have a good read. I appreciate you for sharing!
You have made some good points there. I checked on the net for more information about the issue and found most people will go along with your views on this web site.
After looking into a few of the blog posts on your web site, I really appreciate your way of blogging. I saved it to my bookmark site list and will be checking back soon. Take a look at my web site as well and let me know your opinion.
An interesting discussion is worth comment. I believe that you ought to publish more on this subject matter, it might not be a taboo matter but typically people do not talk about such topics. To the next! Kind regards!!
You need to be a part of a contest for one of the finest websites on the internet. I’m going to recommend this web site!
Having read this I believed it was extremely informative. I appreciate you finding the time and effort to put this article together. I once again find myself spending a lot of time both reading and leaving comments. But so what, it was still worth it!
May I just say what a relief to find somebody that truly knows what they’re talking about over the internet. You certainly understand how to bring a problem to light and make it important. More and more people ought to read this and understand this side of your story. I was surprised that you aren’t more popular given that you certainly have the gift.
There is definately a great deal to know about this subject. I love all the points you’ve made.
I enjoy reading through an article that can make people think. Also, thanks for allowing me to comment!
Nice post. I learn something new and challenging on websites I stumbleupon on a daily basis. It’s always interesting to read content from other writers and practice something from their websites.
Right here is the right webpage for anyone who really wants to find out about this topic. You realize a whole lot its almost tough to argue with you (not that I really will need to…HaHa). You certainly put a brand new spin on a subject which has been written about for decades. Wonderful stuff, just great!
I must thank you for the efforts you’ve put in penning this blog. I’m hoping to check out the same high-grade content from you later on as well. In truth, your creative writing abilities has inspired me to get my own site now 😉
Wonderful article! We are linking to this particularly great content on our website. Keep up the good writing.
Aw, this was an extremely good post. Finding the time and actual effort to create a top notch article… but what can I say… I procrastinate a lot and don’t manage to get nearly anything done.
This is a topic that is near to my heart… Take care! Exactly where are your contact details though?
Saved as a favorite, I love your website!|
I’m impressed, I have to admit. Seldom do I come across a blog that’s equally educative and amusing, and let me tell you, you have hit the nail on the head. The issue is an issue that too few people are speaking intelligently about. I’m very happy I stumbled across this during my search for something regarding this.
Good info. Lucky me I came across your site by chance (stumbleupon). I have saved it for later!
Your style is unique in comparison to other folks I’ve read stuff from. Thanks for posting when you have the opportunity, Guess I’ll just bookmark this blog.
That is a very good tip particularly to those new to the blogosphere. Simple but very precise info… Many thanks for sharing this one. A must read post!
It is the best time to make some plans for the long run and it’s time to be happy. I have read this put up and if I could I desire to recommend you few fascinating issues or suggestions. Maybe you could write subsequent articles relating to this article. I desire to learn even more things about it!|
Can I just say what a relief to find a person that actually knows what they are discussing over the internet. You definitely know how to bring an issue to light and make it important. A lot more people really need to read this and understand this side of your story. I can’t believe you aren’t more popular given that you most certainly have the gift.
bookmarked!!, I like your site!
I love it whenever people come together and share opinions. Great website, stick with it!
Achieving your fitness goal does not need a certified personal trainer or an expensive gym memberships, especially when you have the budget and the space to consider practically every workout machine in the market.
I love reading through an article that will make men and women think. Also, thank you for permitting me to comment!
I like reading a post that will make men and women think. Also, thank you for permitting me to comment!
May I simply just say what a relief to find a person that really knows what they are talking about online. You certainly understand how to bring a problem to light and make it important. More people have to check this out and understand this side of your story. I can’t believe you’re not more popular since you surely possess the gift.
I have to thank you for the efforts you have put in writing this site. I really hope to check out the same high-grade blog posts from you in the future as well. In truth, your creative writing abilities has motivated me to get my own, personal site now 😉
Pretty! This has been an incredibly wonderful article. Thanks for supplying these details.
Hi, I do think this is an excellent web site. I stumbledupon it 😉 I will come back once again since I bookmarked it. Money and freedom is the best way to change, may you be rich and continue to help other people.
Heya i’m for the first time here. I came across this board and I find It really useful & it helped me out a lot. I hope to give something back and aid others like you aided me.|
I seriously love your website.. Great colors & theme. Did you build this web site yourself? Please reply back as I’m hoping to create my own personal blog and would love to know where you got this from or exactly what the theme is named. Thank you!
You made some good points there. I looked on the internet to learn more about the issue and found most individuals will go along with your views on this website.
You need to be a part of a contest for one of the greatest sites on the internet. I will recommend this website!
Spot on with this write-up, I honestly believe that this amazing site needs far more attention. I’ll probably be back again to see more, thanks for the advice!
Great info. Lucky me I found your website by accident (stumbleupon). I have saved as a favorite for later!
I like looking through a post that can make people think. Also, thank you for permitting me to comment!
Excellent article! We are linking to this particularly great content on our site. Keep up the good writing.
I absolutely love your website.. Great colors & theme. Did you make this web site yourself? Please reply back as I’m looking to create my own website and would like to learn where you got this from or just what the theme is named. Many thanks!
This site was… how do I say it? Relevant!! Finally I have found something which helped me. Appreciate it!
Howdy! I could have sworn I’ve been to this website before but after browsing through a few of the articles I realized it’s new to me. Nonetheless, I’m certainly delighted I stumbled upon it and I’ll be bookmarking it and checking back often!
Very good information. Lucky me I came across your website by chance (stumbleupon). I have saved as a favorite for later!
Very nice post. I absolutely love this site. Continue the good work!
http://wallflowers.moonfruit.com/
Thanks so much for the article post.Thanks Again. Will read on…