Zero to Hero: Guide to Object Detection using Deep Learning: Faster R-CNN,YOLO,SSD

In this post, I shall explain object detection and various algorithms like Faster R-CNN, YOLO, SSD. We shall start from beginners’ level and go till the state-of-the-art in object detection, understanding the intuition, approach and salient features of each method.

What is Image Classification?:

Image classification takes an image and predicts the object in an image. For example, when we built a cat-dog classifier, we took images of cat or dog and predicted their class:

What do you do if both cat and dog are present in the image:

What would our model predict? To solve this problem we can train a multi-label classifier which will predict both the classes(dog as well as cat). However, we still won’t know the location of cat or dog. The problem of identifying the location of an object(given the class) in an image is called localization. However, if the object class is not known, we have to not only determine the location but also predict the class of each object.

Predicting the location of the object along with the class is called object Detection. In place of predicting the class of object from an image, we now have to predict the class as well as a rectangle(called bounding box) containing that object. It takes 4 variables to uniquely identify a rectangle. So, for each instance of the object in the image, we shall predict following variables:

class_name, 

bounding_box_top_left_x_coordinate,

bounding_box_top_left_y_coordinate,

bounding_box_width,

bounding_box_height

Just like multi-label image classification problems, we can have multi-class object detection problem where we detect multiple kinds of objects in a single image:

In the following section, I will cover all the popular methodologies to train object detectors. Historically, there have been many approaches to object detection starting from Haar Cascades proposed by Viola and Jones in 2001. However, we shall be focussing on state-of-the-art methods all of which use neural networks and Deep Learning.

Object Detection is modeled as a classification problem where we take windows of fixed sizes from input image at all the possible locations feed these patches to an image classifier.

Demo of Sliding WIndow detector

Each window is fed to the classifier which predicts the class of the object in the window( or background if none is present). Hence, we know both the class and location of the objects in the image. Sounds simple! Well, there are a few more problems. How do you know the size of the window so that it always contains the image? Look at examples:

Small sized object

Big sized object. What size do you choose for your sliding window detector?

As you can see that the object can be of varying sizes. To solve this problem an image pyramid is created by scaling the image.Idea is that we resize the image at multiple scales and we count on the fact that our chosen window size will completely contain the object in one of these resized images. Most commonly, the image is downsampled(size is reduced) until certain condition typically a minimum size is reached. On each of these images, a fixed size window detector is run. It’s common to have as many as 64 levels on such pyramids. Now, all these windows are fed to a classifier to detect the object of interest. This will help us solve the problem of size and location.

 

 

There is one more problem, aspect ratio. A lot of objects can be present in various shapes like a sitting person will have a different aspect ratio than standing person or sleeping person. We shall cover this a little later in this post. There are various methods for object detection like RCNN, Faster-RCNN, SSD etc. Why do we have so many methods and what are the salient features of each of these? Let’s have a look:

1. Object Detection using Hog Features:

In a groundbreaking paper in the history of computer vision, Navneet Dalal and Bill Triggs introduced Histogram of Oriented Gradients(HOG) features in 2005. Hog features are computationally inexpensive and are good for many real-world problems. On each window obtained from running the sliding window on the pyramid, we calculate Hog Features which are fed to an SVM(Support vector machine) to create classifiers. We were able to run this in real time on videos for pedestrian detection, face detection, and so many other object detection use-cases.

2. Region-based Convolutional Neural Networks(R-CNN):

Since we had modeled object detection into a classification problem, success depends on the accuracy of classification. After the rise of deep learning, the obvious idea was to replace HOG based classifiers with a more accurate convolutional neural network based classifier. However, there was one problem. CNNs were too slow and computationally very expensive. It was impossible to run CNNs on so many patches generated by sliding window detector. R-CNN solves this problem by using an object proposal algorithm called Selective Search which reduces the number of bounding boxes that are fed to the classifier to close to 2000 region proposals. Selective search uses local cues like texture, intensity, color and/or a measure of insideness etc to generate all the possible locations of the object. Now, we can feed these boxes to our CNN based classifier. Remember, fully connected part of CNN takes a fixed sized input so, we resize(without preserving aspect ratio) all the generated boxes to a fixed size (224×224 for VGG) and feed to the CNN part. Hence, there are 3 important parts of R-CNN:

  1. Run Selective Search to generate probable objects.
  2. Feed these patches to CNN, followed by SVM to predict the class of each patch.
  3. Optimize patches by training bounding box regression separately.

3. Spatial Pyramid Pooling(SPP-net):

Still, RCNN was very slow. Because running CNN on 2000 region proposals generated by Selective search takes a lot of time. SPP-Net tried to fix this. With SPP-net, we calculate the CNN representation for entire image only once and can use that to calculate the CNN representation for each patch generated by Selective Search. This can be done by performing a pooling type of operation on JUST that section of the feature maps of last conv layer that corresponds to the region. The rectangular section of conv layer corresponding to a region can be calculated by projecting the region on conv layer by taking into account the downsampling happening in the intermediate layers(simply dividing the coordinates by 16 in case of VGG).

There was one more challenge: we need to generate the fixed size of input for the fully connected layers of the CNN so, SPP introduces one more trick. It uses spatial pooling after the last convolutional layer as opposed to traditionally used max-pooling. SPP layer divides a region of any arbitrary size into a constant number of bins and max pool is performed on each of the bins. Since the number of bins remains the same, a constant size vector is produced as demonstrated in the figure below.

However, there was one big drawback with SPP net, it was not trivial to perform back-propagation through spatial pooling layer. Hence, the network only fine-tuned the fully connected part of the network. SPP-Net paved the way for more popular Fast RCNN which we will see next.

4. Fast R-CNN:

Fast RCNN uses the ideas from SPP-net and RCNN and fixes the key problem in SPP-net i.e. they made it possible to train end-to-end. To propagate the gradients through spatial pooling,  It uses a simple back-propagation calculation which is very similar to max-pooling gradient calculation with the exception that pooling regions overlap and therefore a cell can have gradients pumping in from multiple regions.

              One more thing that Fast RCNN did that they added the bounding box regression to the neural network training itself. So, now the network had two heads, classification head, and bounding box regression head. This multitask objective is a salient feature of Fast-rcnn as it no longer requires training of the network independently for classification and localization. These two changes reduce the overall training time and increase the accuracy in comparison to SPP net because of the end to end learning of CNN.

5. Faster R-CNN:

So, what did Faster RCNN improve? Well, it’s faster. And How does it achieve that? Slowest part in Fast RCNN was Selective Search or Edge boxes. Faster RCNN replaces selective search with a very small convolutional network called Region Proposal Network to generate regions of Interests.

To handle the variations in aspect ratio and scale of objects, Faster R-CNN introduces the idea of anchor boxes. At each location, the original paper uses 3 kinds of anchor boxes for scale 128x 128, 256×256 and 512×512. Similarly, for aspect ratio, it uses three aspect ratios 1:1, 2:1 and 1:2. So, In total at each location, we have 9 boxes on which RPN predicts the probability of it being background or foreground. We apply bounding box regression to improve the anchor boxes at each location. So, RPN gives out bounding boxes of various sizes with the corresponding probabilities of each class. The varying sizes of bounding boxes can be passed further by apply Spatial Pooling just like Fast-RCNN. The remaining network is similar to Fast-RCNN. Faster-RCNN is 10 times faster than Fast-RCNN with similar accuracy of datasets like VOC-2007. That’s why Faster-RCNN has been one of the most accurate object detection algorithms. Here is a quick comparison between various versions of RCNN.

Regression-based object detectors:

So far, all the methods discussed handled detection as a classification problem by building a pipeline where first object proposals are generated and then these proposals are send to classification/regression heads. However, there are a few methods that pose detection as a regression problem. Two of the most popular ones are YOLO and SSD. These detectors are also called single shot detectors. Let’s have a look at them:

6. YOLO(You only Look Once):

For YOLO, detection is a simple regression problem which takes an input image and learns the class probabilities and bounding box coordinates. Sounds simple?

YOLO divides each image into a grid of S x S and each grid predicts N bounding boxes and confidence. The confidence reflects the accuracy of the bounding box and whether the bounding box actually contains an object(regardless of class). YOLO also predicts the classification score for each box for every class in training. You can combine both the classes to calculate the probability of each class being present in a predicted box.

So, total SxSxN boxes are predicted. However, most of these boxes have low confidence scores and if we set a threshold say 30% confidence, we can remove most of them as shown in the example below.

Notice that at runtime, we have run our image on CNN only once. Hence, YOLO is super fast and can be run real time. Another key difference is that YOLO sees the complete image at once as opposed to looking at only a generated region proposals in the previous methods. So, this contextual information helps in avoiding false positives. However, one limitation for YOLO is that it only predicts 1 type of class in one grid hence, it struggles with very small objects.

7. Single Shot Detector(SSD):

Single Shot Detector achieves a good balance between speed and accuracy. SSD runs a convolutional network on input image only once and calculates a feature map. Now, we run a small 3×3 sized convolutional kernel on this feature map to predict the bounding boxes and classification probability. SSD also uses anchor boxes at various aspect ratio similar to Faster-RCNN and learns the off-set rather than learning the box. In order to handle the scale, SSD predicts bounding boxes after multiple convolutional layers. Since each convolutional layer operates at a different scale, it is able to detect objects of various scales.

That’s a lot of algorithms. Which one should you use? Currently, Faster-RCNN is the choice if you are fanatic about the accuracy numbers. However, if you are strapped for computation(probably running it on Nvidia Jetsons), SSD is a better recommendation. Finally, if accuracy is not too much of a concern but you want to go super fast, YOLO will be the way to go. First of all a visual understanding of speed vs accuracy trade-off:

SSD seems to be a good choice as we are able to run it on a video and the accuracy trade-off is very little. However, it may not be that simple, look at this chart that compares the performance of SSD, YOLO, and Faster-RCNN on various sized objects. At large sizes, SSD seems to perform similarly to Faster-RCNN. However, look at the accuracy numbers when the object size is small, the gap widens.

YOLO vs SSD vs Faster-RCNN for various sizes

Choice of a right object detection method is crucial and depends on the problem you are trying to solve and the set-up. Object Detection is the backbone of many practical applications of computer vision such as autonomous cars, security and surveillance, and many industrial applications. Hopefully, this post gave you an intuition and understanding behind each of the popular algorithms for object detection.

122 comentários em “Zero to Hero: Guide to Object Detection using Deep Learning: Faster R-CNN,YOLO,SSD

  1. Wow, wonderful weblog format! How long have
    you ever been blogging for? you made running a
    blog look easy. The whole glance of your website is excellent, let alone the content material!

  2. When I originally commented I clicked the “Notify me when new comments are added” checkbox and now each time a comment is
    added I get four emails with the same comment.
    Is there any way you can remove people from that service?

    Bless you!

  3. Attractive section of content. I just stumbled upon your web site and in accession capital to assert that I get actually
    enjoyed account your blog posts. Anyway I’ll be subscribing
    to your augment and even I achievement you access consistently fast.

  4. What’s up, constantly i used to check website posts here in the
    early hours in the break of day, for the reason that i enjoy to gain knowledge of more and more.

  5. After going over a number of the blog posts on your web site, I truly appreciate your
    way of writing a blog. I added it to my bookmark site list and will be checking back
    soon. Please check out my website as well and let me know how you feel.

  6. Great beat ! I would like to apprentice at the same time as
    you amend your web site, how could i subscribe for a weblog site?
    The account aided me a acceptable deal. I have been a little bit acquainted of this
    your broadcast provided brilliant clear idea

  7. I believe that is among the most significant info for me. And i’m glad reading
    your article. However wanna commentary on some normal things, The site style is perfect, the articles
    is in point of fact great : D. Good process, cheers

  8. I’m the business owner of JustCBD company (justcbdstore.com) and am seeking to grow my wholesale side of company. I am hoping someone at targetdomain can help me ! I thought that the best way to do this would be to reach out to vape stores and cbd retailers. I was really hoping if anybody could recommend a qualified web-site where I can buy Vape Shop Business Mailing List I am currently looking at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Unsure which one would be the most suitable choice and would appreciate any guidance on this. Or would it be easier for me to scrape my own leads? Suggestions?

  9. I’m the owner of JustCBD Store label (justcbdstore.com) and I’m presently seeking to broaden my wholesale side of business. It would be great if anybody at targetdomain is able to provide some guidance ! I considered that the best way to accomplish this would be to connect to vape stores and cbd retail stores. I was hoping if anybody could suggest a dependable web site where I can buy Vape Shop B2B Marketing List I am presently taking a look at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Not sure which one would be the most ideal choice and would appreciate any advice on this. Or would it be easier for me to scrape my own leads? Ideas?

  10. Greetings! I know this is kinda off topic however ,
    I’d figured I’d ask. Would you be interested in exchanging links
    or maybe guest writing a blog post or vice-versa? My blog
    addresses a lot of the same topics as yours and I think we could greatly benefit from each other.
    If you are interested feel free to send me an e-mail. I look forward to hearing from you!
    Excellent blog by the way!

  11. Hi! I could have sworn I’ve been to this blog before but after checking through
    some of the post I realized it’s new to me. Nonetheless,
    I’m definitely glad I found it and I’ll be book-marking and checking back often!

  12. Hi there! This post couldn’t be written any better! Going through this article reminds me of my previous roommate! He constantly kept talking about this. I am going to forward this information to him. Pretty sure he will have a good read. Thanks for sharing!

  13. I’m amazed, I must say. Rarely do I come across a blog that’s both equally educative and engaging, and let me tell you, you have hit the nail on the head. The problem is an issue that too few folks are speaking intelligently about. I’m very happy I found this during my hunt for something relating to this.

  14. Hi, I do think this is an excellent website. I stumbledupon it 😉 I am going to come back yet again since i have book marked it. Money and freedom is the best way to change, may you be rich and continue to guide others.

  15. It’s the best time to make some plans for the
    future and it’s time to be happy. I have read this post and
    if I could I desire to suggest you few interesting things or
    tips. Maybe you can write next articles referring to this article.
    I wish to read even more things about it!

  16. I’m excited to discover this great site. I wanted to thank you for your time for this particularly fantastic read!! I definitely savored every little bit of it and I have you book-marked to check out new stuff in your site.

  17. You have made some decent points there. I checked on the net for more info about the issue and found most people will go along with your views on this website.

  18. Attractive part of content. I just stumbled upon your weblog and in accession capital
    to assert that I get in fact loved account your blog posts.
    Anyway I’ll be subscribing in your feeds and even I fulfillment you get entry to constantly quickly.

  19. When I originally commented I appear to have clicked on the -Notify me when new comments are added- checkbox and from now on each time a comment is added I recieve 4 emails with the exact same comment. Is there a way you can remove me from that service? Thanks!

  20. Hi there, just became aware of your blog through Google, and found that it’s truly informative.
    I’m gonna watch out for brussels. I’ll appreciate if you continue this in future.
    A lot of people will be benefited from your writing. Cheers!

  21. Hello there! I could have sworn I’ve visited your blog before but after looking at many of the articles I realized it’s new to me. Anyways, I’m definitely delighted I stumbled upon it and I’ll be book-marking it and checking back often!

  22. After going over a number of the blog articles on your site, I truly appreciate your way of writing a blog. I book marked it to my bookmark webpage list and will be checking back soon. Please check out my web site too and let me know how you feel.

  23. Next time I read a blog, Hopefully it won’t disappoint me just as much as this particular one. I mean, Yes, it was my choice to read, nonetheless I really thought you’d have something useful to talk about. All I hear is a bunch of complaining about something that you could fix if you weren’t too busy seeking attention.

  24. Excellent post. I was checking continuously this weblog and I am inspired!
    Very helpful information particularly the remaining part
    🙂 I deal with such info much. I used to be looking for this particular
    info for a long time. Thank you and good luck.
    adreamoftrains web hosting service

  25. Hi there! This article couldn’t be written much better! Reading through this post reminds me of my previous roommate! He continually kept talking about this. I will forward this article to him. Pretty sure he’s going to have a very good read. Thanks for sharing!

  26. An outstanding share! I’ve just forwarded this onto a colleague who was conducting a little homework on this. And he actually ordered me lunch because I found it for him… lol. So let me reword this…. Thank YOU for the meal!! But yeah, thanx for spending time to discuss this matter here on your web page.

  27. An interesting discussion is worth comment. I think that you should write more about this subject matter, it might not be a taboo matter but typically folks don’t discuss such issues. To the next! Best wishes!!

  28. A motivating discussion is definitely worth comment. I think that you need to publish more about this subject matter, it may not be a taboo matter but typically people don’t discuss such subjects. To the next! All the best!!

  29. Greetings, I do think your site could possibly be having web browser compatibility problems. When I take a look at your web site in Safari, it looks fine however, if opening in IE, it has some overlapping issues. I merely wanted to provide you with a quick heads up! Aside from that, wonderful blog!

  30. After going over a number of the blog posts on your web site, I honestly appreciate your way of writing a blog. I added it to my bookmark website list and will be checking back in the near future. Please visit my web site as well and let me know your opinion.

  31. Hi! I could have sworn I’ve been to your blog before but after browsing through many of the articles I realized it’s new to me. Anyways, I’m certainly pleased I discovered it and I’ll be bookmarking it and checking back regularly!

  32. hello there and thank you for your information –
    I have definitely picked up anything new from right here.
    I did however expertise some technical issues using this web site, since I experienced to reload the website many times previous to
    I could get it to load correctly. I had been wondering if your web
    host is OK? Not that I am complaining, but sluggish loading instances times will very
    frequently affect your placement in google and can damage your high-quality score if ads and marketing with Adwords.
    Well I’m adding this RSS to my email and
    could look out for much more of your respective fascinating content.

    Make sure you update this again soon.

  33. I’m more than happy to find this site. I need to to thank you for ones time for this particularly fantastic read!! I definitely savored every part of it and i also have you book-marked to look at new information on your site.

  34. Hi, Neat post. There’s a problem together with your site in internet explorer, could test this?
    IE nonetheless is the market chief and a big element of people will omit your wonderful writing because
    of this problem.

  35. May I simply just say what a comfort to discover somebody that genuinely understands what they’re talking about online. You definitely understand how to bring a problem to light and make it important. More and more people ought to check this out and understand this side of the story. I can’t believe you’re not more popular because you certainly have the gift.

  36. The next time I read a blog, Hopefully it does not disappoint me as much as this particular one. I mean, I know it was my choice to read, however I genuinely believed you would have something useful to say. All I hear is a bunch of moaning about something you can fix if you weren’t too busy seeking attention.

  37. A motivating discussion is definitely worth comment. I do believe that you should write more about this subject, it may not be a taboo subject but typically people do not discuss these topics. To the next! Best wishes!!

  38. After exploring a number of the blog posts on your web page, I truly appreciate your technique of blogging. I added it to my bookmark site list and will be checking back in the near future. Take a look at my web site as well and let me know your opinion.

  39. After exploring a handful of the articles on your site, I truly appreciate your technique of blogging. I book-marked it to my bookmark website list and will be checking back soon. Please visit my web site as well and tell me what you think.

  40. I’m amazed, I must say. Seldom do I encounter a blog that’s equally educative and amusing, and let me tell you, you have hit the nail on the head. The problem is an issue that too few people are speaking intelligently about. Now i’m very happy that I stumbled across this in my search for something relating to this.

  41. I’d like to thank you for the efforts you have put in penning this site. I am hoping to check out the same high-grade blog posts by you later on as well. In fact, your creative writing abilities has inspired me to get my own, personal site now 😉

  42. Good post. I learn something totally new and challenging on blogs I stumbleupon everyday. It’s always exciting to read through articles from other writers and practice something from their websites.

  43. Oh my goodness! Amazing article dude! Thank you, However I am having issues with your RSS. I don’t understand the reason why I am unable to subscribe to it. Is there anybody else having similar RSS issues? Anyone who knows the solution will you kindly respond? Thanx!!

  44. I’m amazed, I must say. Rarely do I encounter a blog that’s both educative and interesting, and let me tell you, you’ve hit the nail on the head. The problem is something which not enough people are speaking intelligently about. I am very happy that I stumbled across this in my search for something concerning this.

  45. I’m amazed, I must say. Rarely do I come across a blog that’s both educative and amusing, and without a doubt, you’ve hit the nail on the head. The problem is something that too few people are speaking intelligently about. Now i’m very happy I came across this in my search for something relating to this.

  46. Oh my goodness! Incredible article dude! Thank you so much, However I am experiencing troubles with your RSS. I don’t know the reason why I can’t subscribe to it. Is there anybody getting similar RSS problems? Anyone that knows the solution will you kindly respond? Thanx!!

  47. Hi, I do believe this is a great web site. I stumbledupon it 😉 I’m going to come back once again since i have bookmarked it. Money and freedom is the best way to change, may you be rich and continue to guide other people.

  48. Oh my goodness! Awesome article dude! Many thanks, However I am going through issues with your RSS. I don’t know why I cannot subscribe to it. Is there anybody else having the same RSS problems? Anyone who knows the solution can you kindly respond? Thanx!!

  49. Oh my goodness! Incredible article dude! Thank you, However I am experiencing issues with your RSS. I don’t understand why I cannot join it. Is there anybody having the same RSS issues? Anyone that knows the answer will you kindly respond? Thanx!!

  50. I seriously love your site.. Great colors & theme. Did you create this web site yourself? Please reply back as I’m attempting to create my own personal website and would love to know where you got this from or just what the theme is named. Cheers!

  51. Hi there! I could have sworn I’ve visited this website before but after going through a few of the posts I realized it’s new to me. Anyways, I’m certainly delighted I stumbled upon it and I’ll be book-marking it and checking back frequently!

  52. I was very happy to discover this website. I need to to thank you for your time due to this wonderful read!! I definitely really liked every part of it and i also have you book marked to see new stuff on your web site.

  53. Achieving your fitness goals does not have to require a certified personal trainer or an expensive gym membership, especially if you have the budget and the space to consider practically every workout machine on the market.

  54. Howdy! This post could not be written much better! Looking at this article reminds me of my previous roommate! He continually kept talking about this. I’ll send this information to him. Pretty sure he’s going to have a good read. I appreciate you for sharing!

  55. Hello, i think that i saw you visited my blog so i came to “return the favor”.I am trying to find things to improve
    my site!I suppose its ok to use a few of your ideas!!
    34pIoq5 cheap flights

  56. Hi, I do believe this is a great site. I stumbledupon it 😉 I’m going to return once again since I saved as a favorite it. Money and freedom is the best way to change, may you be rich and continue to guide other people.

  57. Hi would you mind letting me know which hosting company you’re using?
    I’ve loaded your blog in 3 different internet browsers and I must
    say this blog loads a lot faster then most. Can you suggest a good web hosting provider at a honest price?
    Kudos, I appreciate it!

  58. Can I just say what a comfort to discover someone who genuinely understands what they’re talking about over
    the internet. You certainly realize how to bring a problem to light
    and make it important. More people need to read this and understand this side of your story.
    I can’t believe you’re not more popular given that you definitely possess the gift.
    3aN8IMa cheap flights

  59. I’ll immediately grab your rss feed as I can’t find your email subscription link or e-newsletter service. Do you have any? Kindly let me understand in order that I may just subscribe. Thanks.|

  60. Next time I read a blog, I hope that it doesn’t disappoint me as much as this one. After all, Yes, it was my choice to read, however I genuinely believed you’d have something helpful to say. All I hear is a bunch of complaining about something you could possibly fix if you weren’t too busy seeking attention.

  61. Hello there! This blog post could not be written much better! Looking through this post reminds me of my previous roommate! He always kept talking about this. I’ll forward this article to him. Pretty sure he’s going to have a very good read. I appreciate you for sharing!

  62. An interesting discussion is definitely worth comment. There’s no doubt that that you ought to publish more on this subject matter, it might not be a taboo matter but typically people don’t speak about such issues. To the next! Many thanks!!

  63. Good blog! I really love how it is easy on my eyes and the data are well written. I’m wondering how I might be notified whenever a new post has been made. I have subscribed to your feed which must do the trick! Have a great day!

  64. Well, I’m glad to have read this to say the least. Never would have dove into this topic myself. Very interesting!

  65. Self-watering planters are perfect as you don’t need to concern yourself with your plants
    when you are on long trips or a vacation. You spend time and effort tending towards
    the seedlings whenever they commence to sprout leaves.

    The key to turning your hindering meal plan into a friendly assistant is usually to
    borrow a good through the typical diet’s worst enemy – junk food.

Leave a Reply

Your email address will not be published. Required fields are marked *