<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Saurabh Bagalkar on Medium]]></title>
        <description><![CDATA[Stories by Saurabh Bagalkar on Medium]]></description>
        <link>https://medium.com/@sbagalka?source=rss-e74591f5d3b0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*C3JSmvqpaaI1_3m3Bno8Ow.png</url>
            <title>Stories by Saurabh Bagalkar on Medium</title>
            <link>https://medium.com/@sbagalka?source=rss-e74591f5d3b0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:19:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sbagalka/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Single Object Tracking using the Siamese Family of Trackers - Part 1 - SiamFC]]></title>
            <link>https://medium.com/alegion/single-object-tracking-using-the-siamese-family-of-trackers-part-1-siamfc-43e6aabfe55a?source=rss-e74591f5d3b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/43e6aabfe55a</guid>
            <dc:creator><![CDATA[Saurabh Bagalkar]]></dc:creator>
            <pubDate>Thu, 26 Aug 2021 16:13:11 GMT</pubDate>
            <atom:updated>2021-08-26T17:50:46.922Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YbYmE4sKNCZNn1iLiTJmiw.jpeg" /></figure><h3>Introduction</h3><p>Object tracking is an inherently compelling computer vision task where the challenge is to associate target objects in consecutive video frames as well as localize them. An initial “seed” or an initial bounding box is generally provided at the start of the video sequence and the tracker is expected to track the region-of-interests(ROIs) with a certain level of accuracy.</p><p>With the advent of deep learning, especially Convolutional Neural Networks, there has been an exponential progress in the domain of image classification and subsequently object detection. Research in Object Tracking is rapidly advancing, albeit not on par as Object Detection, as it is benefiting from the data driven approaches and algorithmic thinking for understanding localization, employed generally in Object Detection tasks. There are 2 major tracking challenges -</p><ol><li>Single Object Tracking — comprising of short and long term tracking challenges and hosted yearly by VOT challenge</li><li>Multi Object Tracking — hosted yearly by MOT challenge</li></ol><p>This first article in the series, predominantly will focus on Single Object tracking, using SiamFC tracker which is the first in line under the Siamese family of trackers. Our next posts will take a deep dive to its successors SiamRPN and SiamRPN++</p><h3>Siamese Networks</h3><p>A Siamese Neural Network is a class of neural network architectures that contain 2 neural identical subnetworks running in tandem. These parallel subnetworks share the same weights and parameter space. Siamese Networks get their name from the Siamese co-joined twins and generally have a Y-shaped neural architecture indicating a comparative approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*d5sDTz9PQvvU_RsZRqa3bw.jpeg" /></figure><p>The unifying theme in all Siamese architectures is this- we have 2 input vectors that we wish to compare, so we pass both of them through the same subnetwork configuration to obtain a multi-dimensional embedding representation. This embeddings are then trained on a certain loss function( like L2 loss or triple loss) to measure semantic similarity between them.</p><h3><strong>SiamFC- Architecture</strong></h3><p>Traditionally the object tracking scene has been mostly dominated by kernel based tracking ( like KCF, mean-shift etc) and contour based tracking like <strong>Co</strong>nditional <strong>De</strong>nsity <strong>Pr</strong>opogation (Condensation). These algorithms deterministically learned the features online while clearly being data deprived to do any offline learning as most of them were proposed in the pre-deep learning era. This resulted in compromising localization accuracy which manifested poorly especially in instances of occlusion, change of camera angle and illumination and hence it was hard to predict the state space of the ROI even when combined with filtering approaches like Kalman and Particle filters.</p><p>The beauty of Siamese-styled tracking approach is that it can fully leverage the juxtaposition approach needed to localize an ROI by its inherent design which is the fundamental idea of tracking-</p><blockquote>Given an object and its location in current frame, find the location of the same object in the next frame</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oIIaa_o-I4WB_AdU-cKQMw.png" /><figcaption>SiamFC architecture</figcaption></figure><p>SiamFC uses 2 identical CNN’s to address an offline similarity learning problem and then this function is then evaluated online during inference phase. As shown in the above figure, we are essential training the network to locate an exemplar image, denoted by <strong>z</strong>, within the larger search image <strong>x</strong><em>. </em>In this example, the red and blue pixels in the score map contain the similarities for the corresponding sub-windows.The<em> </em>FC in the SiamFC stands for fully convolutional architecture with respect to the search image and so there is no restriction on the size of the test image.</p><p>Mathematically, the central idea here is to learn a function<strong><em> </em>𝑓<em>(z, x) </em></strong>that compares the exemplar image <strong>z</strong> with the candidate image <strong>x </strong>and outputs a high scalar valued score map in case they are similar and low score otherwise. In other words, we are essentially building a class-agnostic similarity scoring function between 2 image patches. <strong>𝑓<em>(z, x)</em></strong> can be viewed as a composite function 𝕘( 𝝋(z), 𝝋(x)) where 𝝋 is the identical mapping function that takes the variable sized input vector and creates an embedding of both the exemplar and candidate vectors. 𝕘 in this case can be considered as a similarity metric or just a distance metric.</p><h3>Training and Testing</h3><p>ImageNet 2015 Video dataset was used to train the model which contained more than 30 different classes for animals and vehicles with more than 4500 videos and more than 1 million annotated images. During training the exemplar image size is fixed to 127 x 127 while the candidate size is 255 x 255 pixels. Positive and negative training pair images are carefully curated (i.e they are extracted from the same video and are usually at most 𝕋 frames apart), before training and a logistic loss is used as the cost function.</p><blockquote>(y, v) = log(1 + exp(−yv))</blockquote><p>Here <em>v </em>is the score of the exemplar-candidate and <em>y</em> is the ground truth either +1 or -1. Stochastic Gradient Descent optimizer is applied on the loss function to find the best parameters for the model.</p><h3><strong>Conclusion</strong></h3><p><strong>SiamFC</strong>, released in 2016 and its improved version <a href="https://openaccess.thecvf.com/content_cvpr_2017/html/Valmadre_End-To-End_Representation_Learning_CVPR_2017_paper.html"><strong>SiamFC conv5 baseline</strong></a> won the VOT-17 real-time challenge and was the state of the art in 2016–2017. As we always aim for better results, there were still some fundamental flaws in its design which were hindering it from being a really stellar object tracking method. This quickly changed and we will be seeing it in the next post where we discuss SiamRPN</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43e6aabfe55a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alegion/single-object-tracking-using-the-siamese-family-of-trackers-part-1-siamfc-43e6aabfe55a">Single Object Tracking using the Siamese Family of Trackers - Part 1 - SiamFC</a> was originally published in <a href="https://medium.com/alegion">Alegion</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Faster R-CNN- Using Region Proposal Network for Object Detection]]></title>
            <link>https://medium.com/alegion/faster-r-cnn-using-region-proposal-network-for-object-detection-c113b6dd00a?source=rss-e74591f5d3b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c113b6dd00a</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[faster-r-cnn]]></category>
            <category><![CDATA[data-labeling]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Saurabh Bagalkar]]></dc:creator>
            <pubDate>Tue, 04 Feb 2020 19:01:01 GMT</pubDate>
            <atom:updated>2020-02-04T21:51:03.987Z</atom:updated>
            <content:encoded><![CDATA[<h3>Faster R-CNN: Using Region Proposal Network for Object Detection</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AwOBlidvstUqi5K___ZGxA.png" /></figure><h3><strong>Introduction</strong></h3><p>Object detection is a cornerstone of computer vision. It is connected to both image recognition and image segmentation. Where image recognition outputs a classification label for an identified object and image segmentation creates a pixel level understanding of objects in the scene, object detection locates objects within images or videos, allowing them to be tracked and counted. This allows for many of the most popular ML and AI applications such as face detection, autonomous vehicles, video surveillance, and anomaly detection.</p><p>One of the most popular object detection methods is the <a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-r-cnn-e88f85ea7c16">R-CNN</a> series, developed by <a href="https://arxiv.org/pdf/1311.2524.pdf">Ross Girshick et al</a> in 2014, improved upon with <a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-fast-r-cnn-85d52e3928a1">Fast R-CNN</a> and then finally with <a href="https://arxiv.org/abs/1506.01497">Faster R-CNN</a>. The differentiating approach that makes Faster R-CNN better and faster is the introduction of Region Proposal Network (RPN). RPN is a fully convolutional network, trained end-to-end, that simultaneously predicts object boundaries and object scores at each detection. With RPN being so important to Faster R-CNN, which continues to be one of the best object detection frameworks available to researchers, the bulk of this piece will focus on the RPN design and the concept of anchor boxes and non-maximum suppression.</p><h3>Need for Speed</h3><p><a href="https://arxiv.org/abs/1311.2524">R-CNN</a> was introduced in 2014 and gained a lot of interest in the computer vision community. The idea of R-CNN was to use a Selective Search (SS) approach to propose around 2000 Regions-Of-Interest (ROI), which were then fed into a Convolutional Neural Network (CNN) to extract features. These features were used to classify the images and their object boundaries using SVM (Support Vector Machines) and regression methods. For a more detailed explanation, see this <a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-r-cnn-e88f85ea7c16?source=friends_link&amp;sk=936f709e94684a351b32b63f2af37a6b">piece</a>. This was quickly followed by Fast R-CNN, a faster and better approach of object detection, in early 2015. Fast R-CNN used an ROI pooling approach, which shares the features across the whole image and uses a modified form of spatial pyramid pooling method to extract features in a computationally efficient way. For a more in-depth explanation, see our <a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-fast-r-cnn-85d52e3928a1?source=friends_link&amp;sk=67b53ba5da53028895b8a057e4175fcc">last piece</a>. The problem with Fast R-CNN is that it is still slow because it needs to perform SS which is computationally very slow. Although Fast R-CNN takes 0.32 seconds as opposed to 47 seconds at test time to do a prediction, it takes 2 seconds for generating 2000 ROI’s. This adds up to 2.3 seconds for each image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J1TjdECLsS4EdCbqSVAuOQ.png" /><figcaption>We still spend 2 seconds on each image with selective search. <a href="http://cs231n.stanford.edu/slides/2016/winter1516_lecture8.pdf">Source</a></figcaption></figure><p>This shortcoming led researchers to come up with Faster R-CNN where the test time per image is only 0.2 seconds with region proposals. This is due to the fact that this latest approach is a fully differentiable model using end-to-end training. Let’s see how it works in detail.</p><h3><strong>Faster R-CNN</strong></h3><p>A Faster R-CNN object detection network is composed of a feature extraction network which is typically a pretrained CNN, similar to what we had used for its predecessor. This is then followed by two subnetworks which are trainable. The first is a Region Proposal Network (RPN), which is, as its name suggests, used to generate object proposals and the second is used to predict the actual class of the object. So the primary differentiator for Faster R-CNN is the RPN which is inserted after the last convolutional layer. This is trained to produce region proposals directly without the need for any external mechanism like Selective Search. After this we use ROI pooling and an upstream classifier and bounding box regressor similar to Fast R-CNN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*NXWE7BHug0i-FQlHo5xa7w.png" /><figcaption>Unified Network of Faster R-CNN. <a href="https://arxiv.org/abs/1506.01497">Image Source</a></figcaption></figure><h3><strong>Architecture and Design</strong></h3><p>As the feature extraction, ROI pooling and classifier are the same as the <a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-fast-r-cnn-85d52e3928a1">previous versions</a>, we will focus the bulk of this piece to understand the RPN design and the concept of anchor boxes and non-maximum suppression. The rest of the details are the same as the previous versions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uQNculABX73lqJwESVHr5w.png" /><figcaption>The different layers of Faster R-CNN. <a href="https://tryolabs.com/blog/2018/01/18/faster-r-cnn-down-the-rabbit-hole-of-modern-object-detection/">Image source</a></figcaption></figure><p>Region Proposal Network</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*kA3KLTFaWzW23mxgosHs6w.jpeg" /><figcaption>RPN architecture. <a href="https://medium.com/@smallfishbigsea/faster-r-cnn-explained-864d4fb7e3f8">Image Source</a></figcaption></figure><p>The goal of RPN is to output a set of proposals, each of which has a score of its probability of being an object and also the class/label of the object. RPN can take any sized input to achieve this task. These proposals are further refined by feeding to 2 sibling fully connected layers-one for bounding box regression and the other for box classification i.e is the object foreground or background.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*Z4FFgjKMCNkPbkR0mccbyg.png" /><figcaption>There are 2k classification scores and 4k bbox coordinates generated by RPN where k is the maximum possible number of region proposals. <a href="https://arxiv.org/abs/1506.01497">Image source</a></figcaption></figure><p>The RPN, to generate the proposals, slide a small network over the output of the last layer of feature map. This network uses nxn spatial window as input from the feature map. Each sliding window is mapped to a lower dimensional feature.The position of the sliding window provides localization information with reference to the image while the regression provides finer localization information.</p><h4>Anchor Boxes</h4><p>Anchor box is the one of the most important concept in Faster R-CNN. These are responsible for providing a predefined set of bounding boxes of different sizes and ratios that are going to be used for reference when first predicting object locations for the RPN. These boxes are defined to capture the scale and aspect ratio of specific object classes you want to detect and are typically chosen based on object sizes in the training dataset. Anchor Boxes are typically centered at sliding window.</p><p>The original implementation uses 3 scales and 3 aspect ratios, which means k=9. If the final feature map from feature extraction layer has width W and height H , then the total number of anchors generated will be W*H*k.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*zpwhcEUy3kFPLFLrtPod3A.png" /><figcaption>Different scales and aspect ratios of anchor boxes. <a href="https://medium.com/@smallfishbigsea/faster-r-cnn-explained-864d4fb7e3f8">Image source</a></figcaption></figure><p><strong>Need of Anchor Boxes</strong></p><p>The main reason to use Anchor Boxes is so that we can evaluate all object predictions at once. They help speed up and improve efficiency for the detection portion of a deep learning neural network framework. Anchor boxes also help to detect multiple objects, objects of different scales, and overlapping objects without the need to scan an image with a sliding window that computes a separate prediction at every potential position as we saw in previous versions of R-CNN. This makes real time object detection possible.</p><p>The improvement in speed is made possible because Anchor boxes are translation invariant and so the same ones can be used at every location. As we mentioned before, the information from these anchor boxes is relayed to the regression and classification layer, where regression gives offsets from anchor boxes and classification gives the probability that each regressed anchor shows an object.</p><p><strong>How Anchor Boxes help in identifying an object ?</strong></p><p>Although Anchors take the final feature map as input, the final anchors refer to the original image. This is made possible because of the convolution correspondence property of CNN’s, thus, enabling extracted features to be associated back to their location in that image. For a down sampling ratio d , the feature map will have dimensions W/d * H/d . In other words, in an image, each anchor point, considering we have just one at each spatial location of feature map, will be separated by d spatial pixels. A value of 4 or 16 is common for d , which also corresponds to the <em>stride </em>between tiled anchor boxes. This is a tunable parameter in the configuration of Faster R-CNN. Making it too low or too high can give rise to localization errors. One way to mitigate this localization errors is to learn the offsets applied to each anchor box which is the goal of the regression layer we discussed above</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*TibsMbdUbUCButHSjhoYrQ.png" /><figcaption>Downsampling ratios of CNN feature maps. <a href="https://www.mathworks.com/help/vision/ug/anchor-boxes-for-object-detection.html">Image source</a></figcaption></figure><p>Anchor boxes at each spatial location, mark an object as <em>foreground</em> or <em>background </em>depending on its IOU threshold with the ground truth. All the anchors are places in a mini-batch and trained using softmax cross entropy to learn the classification loss and smooth L1 loss for regression. We use smooth L1 loss as regular L1 loss function is not differentiable at 0.</p><p><strong>Non-maximum suppression(NMS)</strong></p><p>NMS is a second stage of filtering to get rid of overlapping boxes as even after filtering by thresholding over the classes scores, we still end up a lot of overlapping boxes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CuqLjro26cHShpQVO1rgdQ.png" /><figcaption>NMS pseudocode.</figcaption></figure><p>Overview of NMS:</p><ol><li>Select the box that has the highest score.</li><li>Compute its overlap with all other boxes, and remove boxes that overlap it more than the IOU threshold</li><li>Go back to step 1 and iterate until there’s no more boxes with a lower score than the current selected box.</li></ol><p>This will remove all boxes that have a large overlap with the selected boxes. Only the “best” boxes remain. There are further improvements made to this method which is called <a href="https://arxiv.org/pdf/1704.04503.pdf">Soft-NMS</a></p><h3><strong>Conclusion</strong></h3><p>We saw the primary concepts of RPN and anchor boxes in this piece, which are the core ideas that make Faster R-CNN better and faster than its predecessor. We skipped over the Feature extraction and ROI pooling because these concepts are the same that were used in Fast R-CNN and have been explained in depth in our <a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-fast-r-cnn-85d52e3928a1?source=friends_link&amp;sk=67b53ba5da53028895b8a057e4175fcc">last article</a>. Faster R-CNN is still widely used today and remains one of the best object detection frameworks available to researchers. For full implementation and Tensorflow code, refer to this official <a href="https://github.com/ssbagalkar/tf-faster-rcnn">github repo.</a></p><p><strong>Learn More</strong></p><p>Interested in learning more about machine learning? Check out <a href="https://content.alegion.com/podcast">No BiAS</a>, a podcast about the emerging and ever-shifting terrain of artificial intelligence and machine learning hosted by <a href="https://medium.com/u/826e84c38047">Melody Travers</a> and <a href="https://medium.com/u/da8fb383140a">Saurabh Bagalkar</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c113b6dd00a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alegion/faster-r-cnn-using-region-proposal-network-for-object-detection-c113b6dd00a">Faster R-CNN- Using Region Proposal Network for Object Detection</a> was originally published in <a href="https://medium.com/alegion">Alegion</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Learning for Object Detection and Localization using Fast R-CNN]]></title>
            <link>https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-fast-r-cnn-85d52e3928a1?source=rss-e74591f5d3b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/85d52e3928a1</guid>
            <category><![CDATA[convolution-neural-net]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Saurabh Bagalkar]]></dc:creator>
            <pubDate>Thu, 17 Oct 2019 14:01:02 GMT</pubDate>
            <atom:updated>2019-11-18T22:24:02.442Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Deep Dive Into The Improved Fast R-CNN Approach</h3><h4>Deep Learning for Object Detection and Localization Part II</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FBuxJ5RfUn_vy_9kCU-k4Q.png" /></figure><h3><strong>Introduction</strong></h3><p>Deep Learning for Object Detection Part II — A Deep Dive Into Fast R-CNN is the second article in our Deep Learning for Object Detection series, which explores state-of-the-art, region based, object detection methods and their evolution over time. In this piece, we look at innovations that improve training and testing speed to overcome many of the <a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-r-cnn-e88f85ea7c16">drawbacks</a> of the R-CNN approach.</p><p>In <a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-r-cnn-e88f85ea7c16">Part I</a>, we saw how R-CNN, pioneered by <a href="https://arxiv.org/pdf/1311.2524.pdf">Ross Girshick</a> and his team, sparked new research interest in region based object detection. The following year, they released a new paper detailing how they suped up R-CNN.</p><p>We will take a look at how they improved upon R-CNN and what methodologies were employed to develop a deeper understanding of why this method is called Fast R-CNN. We have also provided sample TensorFlow code to help aid understanding of how the tensors are flowing between layers and how spatial pyramid pooling works in action.</p><h3><strong>R-CNN vs Fast R-CNN</strong></h3><p>The landscape of computer vision, and more specifically object detection, has changed dramatically since Convolutional Neural Networks (CNNs) were introduced in 2013. As you can see in the graph below, precision began to plateau at 40% until the advent of deep learning for object detection started to take off. Advancement in research coupled with accessibility to faster and more powerful hardware resulted in precision growing exponentially and current state of the art is around 80%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zw3ucH4FKZq51GQ5fFiLOg.png" /><figcaption>There was an meteoric rise in precision after CNN’s were used for object detection. <a href="https://tutorial.caffe.berkeleyvision.org/caffe-cvpr15-detection.pdf">Image Source</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*CP3X0CXeCIUki5NB4cjsTA.png" /><figcaption>R-CNN modules. <a href="https://ieeexplore.ieee.org/document/6909475">Image source</a></figcaption></figure><p>As we saw last time, R-CNN predominantly uses region proposals using a selective search (SS) approach to pre-compute the priors. The features between these regions are not shared, so we have to extract features individually for each region-of-interest (RoI) generated using a SS approach. This is not a particularly computationally effective approach and makes real-time object detection inefficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*GrqJeumA9QUSxyry6JLMaA.png" /><figcaption>Fast R-CNN modules <a href="https://arxiv.org/abs/1504.08083">Image source</a></figcaption></figure><p>Fast R-CNN, on the contrary, trains a deep VGG-16 network, 9x faster than R-CNN and is 213x faster at test time, achieving a higher <a href="https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173">mAP</a> on <a href="http://host.robots.ox.ac.uk/pascal/VOC/voc2012/">PASCAL VOC 2012.</a> How does it achieve this humongous gain in speed? Fast R-CNN evaluates the network and extracts features for the whole image once, instead of extracting features from each RoI, which are cropped and rescaled every time in R-CNN. It then uses the concept of RoI pooling, a special case of pyramid pooling used in <a href="https://arxiv.org/abs/1406.4729">SPPNet</a>, to give a feature vector of desired length. This feature vector is then used for classification and localization. This method is more effective than R-CNN, because the computations for overlapping regions are shared. Let us unpack the details of the process to gain a deeper understanding.</p><h3>Architectural Design of Fast R-CNN</h3><p><strong>Building blocks of Fast R-CNN:</strong></p><ol><li><a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-r-cnn-e88f85ea7c16"><em>Region proposal network</em></a></li><li><a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-r-cnn-e88f85ea7c16"><em>Feature extraction using CNN</em></a></li><li><em>RoI pooling layer — This is where the real magic of Fast R-CNN happens</em></li><li><a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-r-cnn-e88f85ea7c16"><em>Classification and Localization</em></a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*ATyBsAsDQNqT4GYKLrO81w.png" /><figcaption>Building blocks of Fast R-CNN <a href="https://www.mathworks.com/help/vision/ug/getting-started-with-r-cnn-fast-r-cnn-and-faster-r-cnn.html#d117e11449">Image Source</a></figcaption></figure><p>The <em>Region proposal network </em>and <em>Feature extraction</em> modules are very similar to what we have seen in R-CNN. But instead of passing each cropped and re-scaled RoI, the entire input image is passed through a feature extractor like VGG-16 to produce a convolutional feature map. The features (i.e convolutional feature map) are combined with <em>Region proposal network</em>, which uses a SS approach, to form a fixed-length feature vector in the RoI pooling layer. Each of these feature vectors is then passed along to classification and localization modules. The classification module classifies K+1 (1 for background) object classes using a softmax probability. The localization module outputs four real-valued numbers for each K object classes.</p><h4>Spatial Pyramid Pooling</h4><p>Let’s see what Fast R-CNN does differently that makes it much faster than its predecessor. To understand RoI pooling, it is vital to understanding how SPPNet does spatial pyramid pooling (SPP) to extract fixed length output vectors. SPP is heavily inspired by two widely used concepts in image processing and computer vision:</p><ul><li><a href="https://en.wikipedia.org/wiki/Pyramid_(image_processing)">Pyramid representation</a></li><li><a href="https://en.wikipedia.org/wiki/Bag-of-words_model_in_computer_vision">Bag-of-words</a></li></ul><p>Image pyramids are traditionally used in computer vision to generate filter-based decomposed representations of the original image to extract useful features at multiple scales. It is also used for storing compressed image representations. Based on this concept, let us take a look at the following figure which uses a 3 level SPP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*9C-HXo2-AfY-pF6bGwJeOQ.png" /><figcaption>A network structure with a spatial pyramid pooling layer. <a href="https://arxiv.org/abs/1406.4729">Image Source</a></figcaption></figure><p>Imagine we have a seven layer deep CNN. Let’s say the 5th convolutional layer conv_5 has a depth (number of filters) of 256, which means it has 256 individual feature maps stacked in z-axis (number of channels) or depth. Normally, this feature map would have to be flattened and connected to a fully-connected layer as <a href="http://cvml.ist.ac.at/courses/DLWT_W17/material/AlexNet.pdf">some previous CNN architectures did</a>. But what SPP does differently is to decompose this m x n x 256 (where m and n are height and width of the conv_5) into a fixed length/size 1-dimensional vector by concatenating all image decompositions at varying scale using a max-pooling operation.</p><p>The coarsest pyramid level (at the very bottom, indicated in white) uses a global pooling operation that covers the entire image. This creates a 256-d flattened vector. The middle layer (one indicated in green) pools 4 values from each feature map and hence we get 4 x 256-d flattened vector. The top most (blue) pools 16 values from each feature map and hence we end up with 16 x 256-d vector. So, in this example, we have a (256 + 4 x 256 + 16 x 256) 5376-dimensional flattened vector that will be fed to our fully connected layer. For a more in-depth understanding of the math, please refer to this <a href="https://github.com/Alegion/AlexNet-with-Spatial-Pyramid-Pooling">github repo</a>.</p><h4>Need for Spatial Pyramid Pooling</h4><p>But why do we have to do the spatial pyramid pooling in the first place? You see, CNNs, which have dense connection representations at deeper levels, traditionally require a fixed input size in accordance with the architecture used. For example, it is common practice to use 227 x 227 X 3 for AlexNet. So why do CNNs require a fixed image size? Let’s analyze this quote from the SPPNet paper:</p><blockquote>In fact, convolutional layers do not require a fixed image size and can generate feature maps of any sizes. On the other hand, the fully-connected layers need to have fixed size/length input by their definition. Hence, the fixed size constraint comes only from the fully-connected layers, which exist at a deeper stage of the network.</blockquote><p>The authors make a very important point that fully connected layers require a fixed image size. Why? As we know, backpropagation is always done with respect to weights and biases. In convolutional layers, these weights are nothing but the filters or feature maps. The number of feature maps is always independent of the image height and width and is decided <em>a priori</em>. Fully connected layers have a fixed length. If we have images of varying spatial dimensions while training, we cannot comply to a fixed flattened vector, because this will lead to dimension mismatch errors. To avoid this, we have to use fixed image dimensions. Resizing and cropping images is not always ideal because the recognition/classification accuracy can be compromised as a result of content and information loss or distortion.</p><p>The spatial bins that we use here (16, 4, 1) have sizes proportional to image size, so the number of bins is fixed regardless of image size. This means we get rid of the size constraint i.e no cropping and resizing of original images. This significantly increases efficiency because now we do not have to preprocess images in a fixed aspect ratio (i.e. fixed height and width) and will always get a fixed flattened vector regardless of what image size you pass in as input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z4MZlKHE0wpkTOk-OQ_U5A.png" /><figcaption>Spp method can take images of arbitrary size and generate a fixed-length feature vector. <a href="https://medium.com/coinmonks/review-sppnet-1st-runner-up-object-detection-2nd-runner-up-image-classification-in-ilsvrc-906da3753679">Image source</a></figcaption></figure><h4>Tensor Flow Code</h4><p>Let’s look at some tensorflow code to understand how the tensors are flowing between layers and how spatial pyramid pooling works in action. For simplicity of demonstration, I am using AlexNet architecture to illustrate how the tensors are spatially pooled by the introduction of spatial pooling layer. Here is the feedforward architecture of AlexNet that optionally uses spatial pyramid pooling operation. <strong>Note: This part does not include any kind of training as I am not back propagating my derivatives nor defining accuracy metrics.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/efc83f819bdad16feee0600797495ad3/href">https://medium.com/media/efc83f819bdad16feee0600797495ad3/href</a></iframe><h4>Implementation of Spatial Pyramid Pooling</h4><p>Here is the implementation of spatial pyramid pooling layer using tensorflow. Try experimenting with different image sizes to verify that no matter the image height and width (although technically for this architecture, the images should be at least 163x163), you will always get a fixed length vector that feeds into your fully connected network.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5409e7b743fae876b68c5e9cf865120/href">https://medium.com/media/c5409e7b743fae876b68c5e9cf865120/href</a></iframe><h4>Breakdown of Parameters</h4><p>Scroll through this CSV for a complete breakdown of the number of parameters used by AlexNet with SPP along with its architectural design.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fairtable.com%2Fembed%2FshruxJyrZqf1DCtmt&amp;url=https%3A%2F%2Fairtable.com%2FshruxJyrZqf1DCtmt&amp;image=https%3A%2F%2Fstatic.airtable.com%2Fimages%2Foembed%2Fairtable.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=airtable" width="800" height="533" frameborder="0" scrolling="no"><a href="https://medium.com/media/1bae6eef7d67eec14842f9fd247131d7/href">https://medium.com/media/1bae6eef7d67eec14842f9fd247131d7/href</a></iframe><h4><strong>ROI pooling layer</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FnDKNXBA0ZSSUczEOFbMFQ.png" /><figcaption>(Left) ROI pooling vs (Right) SPP. <a href="https://jhui.github.io/2017/03/15/Fast-R-CNN-and-Faster-R-CNN/">Image Source</a></figcaption></figure><p>Now, that we understand how SPP works, the only difference in RoI pooling layer is we do not generate separate pyramid layers for each scale, rather just warp them into one layer using all regions-of-interests. This will create one feature map of H x W (height x width), which are hyper-parameters that the user can tune. So imagine in the above spatial_pyramid_pooling.py code, we specify for example n (number of levels) = 1. Let’s say H X W is 7 x 7. So the output of RoI pooling layer will be a vector of length →(2000 * 7 * 7 * 256). The 256 comes from previous layer’s depth or convolutional feature maps, while the 2000 comes from the number of object proposals generated using SPP.</p><p>Here is an in-depth look at the original architecture i.e VGG-16 used in Fast R-CNN. <strong>Note: This is the original architecture, so its depth and spatial dimensions are different than what we used with AlexNet for demo purposes above.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0BJEv0i2OCptYL0PWZve7g.png" /><figcaption>How tensors flow between layers in original architecture. <a href="https://tutorial.caffe.berkeleyvision.org/caffe-cvpr15-detection.pdf">Image source</a></figcaption></figure><p>In the above image, the input to RoI pooling is 2000x5 tensor of RoI’s selected with SPP and a feature map of depth 512. The 512 is the depth size/number of filters of the previous convolutional layer. Why 5 for RoI representation? It represents [batch_size, x_min, y_min, x_max, y_max] of the RoI’s. These RoI’s are combined with the CNN feature maps (shown as depth of 512 in the above image), which means the RoI’s are scaled by a factor of k, where k is the total spatial downsampling that has occured over its journey from layer_1 to present layer (say from 1000 to 100, so the downsampling ratio is 10 on width of image). Then RoI’s are further downsampled by a factor of 1/H X 1/W and spatial pyramid pooling is performed on all 2000 RoI’s. This part was a bit confusing to me at first, but this snippet from Ross Girschick’s original Caffe implementation made things more clear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S3tE3xFaKk8aUFrqc-Byqw.png" /><figcaption><a href="https://github.com/rbgirshick">Original Github repo</a>. Spatial scale used in original code is 1/16.</figcaption></figure><p>Once we have a vector, or a tensor really, of dimensions [batch_size,width, height, depth] from RoI pooling, we flatten the vector and add a couple of fully connected layers to it. This flattened vector is then used for calculating both classification and bounding box regression loss.</p><h4>Conclusion</h4><p>Fast R-CNN achieves a giant leap in improving accuracy and reducing computation time for training and inference because of its smart use of RoI pooling layer enabling it to train the entire image without cropping multiple RoI’s and training them separately. Here, all parameters including the CNN architecture is trained together with a cross entropy function for the class classification and a smoothL1 loss function for the boundary box prediction. Besides this change, Fast R-CNN is very similar in its architecture to its predecessor.</p><p>As with all the architectures, Fast R-CNN is still not perfect and has its own limitations as it still requires region-proposals, which are non trainable. So essentially, it is not an end-to-end trainable architecture. In our next blog, we will look at how this was achieved by researchers who aptly called the next version Faster R-CNN.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85d52e3928a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-fast-r-cnn-85d52e3928a1">Deep Learning for Object Detection and Localization using Fast R-CNN</a> was originally published in <a href="https://medium.com/alegion">Alegion</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Learning for Object Detection and Localization using R-CNN]]></title>
            <link>https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-r-cnn-e88f85ea7c16?source=rss-e74591f5d3b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e88f85ea7c16</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Saurabh Bagalkar]]></dc:creator>
            <pubDate>Mon, 09 Sep 2019 16:11:25 GMT</pubDate>
            <atom:updated>2019-10-23T15:57:03.246Z</atom:updated>
            <content:encoded><![CDATA[<h4>A deep dive into R-CNN for object detection</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3SyzDFFYhAZqMAGNbTctuw.png" /></figure><h3>Introduction</h3><p>Object detection is at the forefront of computer vision research, primarily due to its plethora of practical applications in almost every field. With technology progressing so fast, it is important to develop a deep understanding of what these methods are and how they have evolved over time as we innovate beyond them. This is the first piece in a new series that will cover the concepts behind some very powerful state-of-the-art object detection techniques and localization methods that use region based object detectors. <a href="https://ps.is.tue.mpg.de/research_fields/semantic-scene-understanding">Scene understanding</a>, of which object detection is a major part, helps to automate a lot of computer vision tasks, such as pedestrian detection, vehicle detection and localization and traffic sign recognition among others. We will start with the R-CNN proposed by <a href="https://arxiv.org/pdf/1311.2524.pdf">Ross Girshick et al.</a> which is a precursor for understanding the more efficient and widely used successors like Fast R-CNN and Faster R-CNN and eventually Mask R-CNN which will be covered later in this series</p><h3><strong>What is R-CNN?</strong></h3><p>R-CNN is an object detection and localization method. It aims to “find” objects of interest in an image and draws bounding boxes around them while also categorizing its class. The name R-CNN stands for Regions with CNN (Convolutional Neural Networks) features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*E-8oQW8ZO-hHgTf6laWhhQ.png" /><figcaption>R-CNN attempts to categorize each object in an image with its associated bounding box. <a href="https://arxiv.org/abs/1311.2524">Image source</a></figcaption></figure><h3><strong>Motivation Behind R-CNN</strong></h3><p>Object Recognition has been traditionally done using <a href="https://en.wikipedia.org/wiki/Scale-invariant_feature_transform">SIFT</a> and <a href="https://en.wikipedia.org/wiki/Histogram_of_oriented_gradients">HOG</a> features. These methods, although very successful, breakdown when there are images that have other objects, or considerable noise around them. As we know, recognition in human visual cortex system is much more complicated than just identifying gradients and clustering them, which HOG tends to do. We need a more sophisticated, hierarchical and multistage process if we are going to get to human level visual recognition.</p><p>With the advent of Deep-Learning and specifically CNN’s, we have a good starting point to extract features from an image using a similar hierarchical architecture. As <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf">AlexNet</a> outperformed every other image recognition model out there in 2012, because of its clever use of dropouts and ReLU activations, it quickly became the de facto standard for image recognition. Then the burning question became, <em>can we somehow extend CNN classification results from ImageNet challenge to object detection results on PASCAL VOC challenge</em>? This idea gave rise to R-CNN or region proposal using CNN, which bridges the gap between image classification and object detection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yLisuzX7vdc1BO9kzSjOwA.png" /><figcaption>CNN’s classify images by extracting their high and low level features. <a href="https://www.mathworks.com/help/deeplearning/ug/introduction-to-convolutional-neural-networks.html">Image source</a></figcaption></figure><h3><strong>Why is Object Detection and Localization a non-trivial task ?</strong></h3><p>Localizing an object can be framed as a regression problem by itself but that is not terribly effective because it yields poor results and is very slow. Another approach is to use a <a href="https://www.pyimagesearch.com/2015/03/23/sliding-windows-for-object-detection-with-python-and-opencv/">sliding-window detector</a>, traditional for CNN’s, but as the layers and subsequent strides increase, resolution is lost. This makes precise localization challenging, not to mention the fact that it is too slow for many real world applications. Another issue with using the sliding-window approach is that if you need to capture different aspect ratios of different regions within an image, spatial location will vary greatly. This will also blow up the computational power.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*tWBYg9YZ-gCL2t0UNxFU6A.png" /><figcaption>The sliding window approach is very cumbersome, when used to localize cars in an image.<a href="http://jeffwen.com/2017/08/16/vehicle_detection"> Image source</a></figcaption></figure><h3><strong>R-CNN building blocks and architecture</strong></h3><p>To deal with the aforementioned localization issues that slow down not just the detection speed but also the accuracy, R-CNN’s were introduced which use a very clever approach to detection and localization.</p><p>R-CNN has the following main modules:</p><ol><li><em>Region Proposals</em>- For generating 2000 proposed bounding boxes</li><li><em>Feature extraction using CNN</em>- For image classification of each region</li><li><em>Classification and Localization</em></li></ol><ul><li><em>Class recognition using </em><a href="https://en.wikipedia.org/wiki/Support-vector_machine"><em>Linear SVM</em></a></li><li><em>Bounding-box regression for localization</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*CP3X0CXeCIUki5NB4cjsTA.png" /><figcaption>R-CNN modules. <a href="https://ieeexplore.ieee.org/document/6909475">Image source</a></figcaption></figure><p>Let’s look at each module in detail to understand how the architectural components interact with each other to give us the desired end result.</p><h4><strong>Region Proposals</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0AZidANlkEXOM1tLycnLpQ.png" /><figcaption>A clever way to select coherent regions within an image. <a href="https://www.koen.me/research/pub/uijlings-ijcv2013-draft.pdf">Image source</a></figcaption></figure><p>A smarter way to localize an object is to generate region proposals using the <em>concept of recognition using regions.</em> There are many ways of generating region proposals, but we will do it using the Selective Search method which, in a nutshell, clusters or groups similar regions and produces demarcated regions within an image. It does so using an iterative approach somewhat similar to <a href="https://en.wikipedia.org/wiki/Region_growing">region growing</a> until no further change is possible.</p><p>Selective Search uses these main steps to propose regions:</p><ol><li>From bottom up, find similarities between pixels in an image based on</li></ol><ul><li>Color</li><li>Texture</li><li>Size</li><li>Shape Compatibility</li></ul><p>2. Group the pixels together, which give a high similarity metric using a weighted sum of aforementioned similarities</p><p>3. Repeat until there are no more changes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*6qs7HAqdOAlsEpHJKSDRJg.png" /><figcaption>Selective Search uses a bottom up approach to propose similar regions. <a href="https://ivi.fnwi.uva.nl/isis/publications/bibtexbrowser.php?key=vandeSandeICCV2011&amp;bib=all.bib">Image source</a></figcaption></figure><p>At test time, R-CNN’s region proposal method using selective search generates 2000 category-independent region proposals. These region proposals will be filtered along the way to only account for the “best” region proposals. So the main task of this module is to propose a set of candidate regions for the next module i.e CNN feature extractor. The output of this module, which is a list of coordinates {x-coord, y-coord, width, height} is then fed to the feature extraction module. For a more in-depth understanding of how Selective Search works, please refer to this <a href="https://www.learnopencv.com/selective-search-for-object-detection-cpp-python/">link</a>.</p><h4><strong>Feature Extraction using CNN</strong></h4><p>Since the popularity of AlexNet proposed by <a href="https://papers.nips.cc/author/alex-krizhevsky-5653">Krizhevsky</a> et al, CNN’s have become hugely popular for feature extraction from images. This module extracts a 4096-dimensional feature vector for each region proposed by region proposals module. The overarching steps for feature extraction are as follows:</p><ul><li>Resize and warp the RGB image to 227 x 227 and perform mean subtraction as part of preprocessing. Before warping, there is some dilation of bounding box to add some context padding, which has shown better <a href="https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173">mAP</a> results.</li><li>Pass, i.e. forward propagate the individual resized regions through pre-trained AlexNet to get a 4096-dimensional feature vector of each region with each image. Finally a 1000 length classification layer is replaced by (N+1) length layer where N is number of classes and an extra one is for the background.</li></ul><p>All region proposals with <a href="https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/">IOU</a> ≥ 0.5 overlap with a ground-truth box is considered as positive for that box’s class and the rest as negatives. Essentially, the output for this module will be a 2000 x 4096 vector for each image.</p><h4><strong>Classification and Localization</strong></h4><p><em>Object classification is performed using a linear SVM model</em></p><p>Once we have a 4096 dimensional vector for each candidate object in each image, we can run a trained Linear SVM model to categorize that object in one of the many classes.</p><p>During training of SVM, a hard negative mining approach is used to compensate for training data that is too large to fit in memory. Hard negative mining basically makes sure:</p><ul><li>We have an approximately equal number of positive and negative examples. How? We only look for “hard” negatives and not all negatives. All negatives i.e the background, are in abundance in an image, whereas “hard” negatives comprise of objects which are occluded or difficult to classify.</li><li>The classifier learns during training which objects to focus on, as we explicitly capture a false positive and tell the classifier to consider it as a negative example for the next epoch of the training cycle, thus increasing the robustness of the classifier.</li></ul><p>In contrast to fine-tuning using CNN’S, where we use <a href="https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/">IOU</a> of at least 0.5, for training SVM’s we only take ground truth boxes as positive examples, and label proposals with less than 0.3 IOU as negatives, ignoring everything else. This approach while training has shown to improve the mAP score.</p><p><em>Bounding-box regression is used to precisely localize an object in an image</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KDHADNbab-fhbdP_ldiDng.png" /><figcaption>Transformation between predicted and ground truth coordinates. <a href="https://lilianweng.github.io/lil-log/2017/12/31/object-recognition-for-dummies-part-3.html">Image source</a></figcaption></figure><p>Selective search gives us candidate blobs around each object of interest along with its 4 coordinates. These bounding boxes might not be as tight as desired. So to resolve this issue, a function with learnable parameters <strong><em>P</em></strong> is used to map and transform the proposed box coordinates</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*-qt8Y6EpT_sRB49zvLkO1g.png" /></figure><p>to the ground truth coordinates <strong><em>G</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*KrZo-4uRTrvSK29xbyZRHA.png" /></figure><p>by imposing scale-invariant, between two centers, and log-space, on the width and height, transformations as shown here</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/1*hRTZ86gzebKAazyTIwxmTQ.png" /></figure><p>The regression targets <strong><em>t</em></strong> is defined as :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/1*kjXx-PygUt6QUu86hynmEg.png" /></figure><p>The standard regression model can solve this optimization problem using least squares objective</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*mLx_g_QeHNtum9jw9eN42Q.png" /></figure><p>where <strong>w</strong> is a vector of learnable model parameters which needs to be optimized.Please refer to the<a href="https://arxiv.org/abs/1311.2524?WT.mc_id=Blog_MachLearn_General_DI"> original paper</a> to understand the in-depth specifics of these transformations and mapping</p><h3><strong>Recap of R-CNN process</strong></h3><p>In a nutshell, R-CNN does the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4r0EM-ki03WFYRBbuUwSA.png" /><figcaption>R-CNN process</figcaption></figure><ol><li>Generates around 2000 class-independent region proposals for each image using Selective Search approach. These object proposals are different sizes, so they are warped into a standard 227 x 227 image, which is then fed to a pre-trained CNN architecture like AlexNet.</li><li>The CNN extracts a 4096 dimensional vector of each object in an image thus creating a 2000 x 4096 matrix.</li><li>This feature vector is passed onto a binary SVM, which is trained for each class separately, thus giving the classification of the object.</li><li>Precise localization is achieved using a least squares regression model, which minimizes the differences between the proposed coordinates and the ground truth coordinates by correcting the offset values.</li></ol><h3><strong>Why R-CNN needs improvement?</strong></h3><p>Although, R-CNN was a breakthrough paper at its time, it faced several drawbacks:</p><ol><li>It is computationally very costly to implement</li></ol><ul><li><em>For every region in each image, we have to calculate features using a forward pass in CNN architecture, which means there will be 2000 forward passes for each image.It takes 47 secs per image on a GPU</em></li></ul><p>2. Selective search is not a learnable model and hence can generate bad initial candidates.</p><p>3. There are 3 different models to use, which adds architectural and time complexity.</p><ul><li><em>CNN architecture for feature extraction, the SVM to recognize the object class, and learnable parameters for bounding box regressor, all of which are comprised of trainable parameters.</em></li></ul><p>To ameliorate these pitfalls of R-CNN, the authors improved the design of R-CNN with a new revamped approach called Fast R-CNN. We will take a look at it in our next piece, <a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-fast-r-cnn-85d52e3928a1">Deep Learning for Object Detection Part II</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e88f85ea7c16" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alegion/deep-learning-for-object-detection-and-localization-using-r-cnn-e88f85ea7c16">Deep Learning for Object Detection and Localization using R-CNN</a> was originally published in <a href="https://medium.com/alegion">Alegion</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>