<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shilpa Ananth on Medium]]></title>
        <description><![CDATA[Stories by Shilpa Ananth on Medium]]></description>
        <link>https://medium.com/@shilpa_47526?source=rss-55d4794e571b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5YalmtcXv3ANCejfWE59-A.jpeg</url>
            <title>Stories by Shilpa Ananth on Medium</title>
            <link>https://medium.com/@shilpa_47526?source=rss-55d4794e571b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:57:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shilpa_47526/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Faster R-CNN for object detection]]></title>
            <link>https://medium.com/data-science/faster-r-cnn-for-object-detection-a-technical-summary-474c5b857b46?source=rss-55d4794e571b------2</link>
            <guid isPermaLink="false">https://medium.com/p/474c5b857b46</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Shilpa Ananth]]></dc:creator>
            <pubDate>Fri, 09 Aug 2019 19:04:56 GMT</pubDate>
            <atom:updated>2024-08-16T08:22:24.699Z</atom:updated>
            <content:encoded><![CDATA[<h4>A technical paper summary</h4><p>The most widely used state of the art version of the R-CNN family — Faster R-CNN was first published in 2015. This article, the third and final one of a series to understand the fundamentals of current day object detection elaborates the technical details of the Faster R-CNN detection pipeline. For a review of its predecessors, check out these summaries: <a href="https://medium.com/@shilpa_47526/r-cnn-for-object-detection-a-technical-summary-9e7bfa8a557c">Regions with CNN (R-CNN)</a> and <a href="https://medium.com/@shilpa_47526/fast-r-cnn-for-object-detection-a-technical-summary-a0ff94faa022">Fast R-CNN</a>.</p><p>In the R-CNN family of papers, the evolution between versions was usually in terms of computational efficiency (integrating the different training stages), reduction in test time, and improvement in performance (mAP). These networks usually consist of — a) A region proposal algorithm to generate “bounding boxes” or locations of possible objects in the image; b) A feature generation stage to obtain features of these objects, usually using a CNN; c) A classification layer to predict which class this object belongs to; and d) A regression layer to make the coordinates of the object bounding box more precise.</p><p>The only stand-alone portion of the network left in Fast R-CNN was the region proposal algorithm. Both R-CNN and Fast R-CNN use CPU based region proposal algorithms, Eg- the Selective search algorithm which takes around 2 seconds per image and runs on CPU computation. The Faster R-CNN [3] paper fixes this by using another convolutional network (the RPN) to generate the region proposals. This not only brings down the region proposal time from 2s to 10ms per image but also allows the region proposal stage to share layers with the following detection stages, causing an overall improvement in feature representation. In the rest of the article, “Faster R-CNN” usually refers to a detection pipeline that uses the RPN as a region proposal algorithm, and <a href="https://medium.com/@shilpa_47526/fast-r-cnn-for-object-detection-a-technical-summary-a0ff94faa022">Fast R-CNN</a> as a detector network.</p><h3>Region Proposal Network (RPN)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S_-8lv4zP3W8IVfGP6_MHw.jpeg" /><figcaption>Figure 1: The architecture of the region proposal network or RPN</figcaption></figure><h4>Architecture</h4><ul><li>The region proposal network (RPN) starts with the input image being fed into the backbone convolutional neural network. The input image is first resized such that it’s shortest side is 600px with the longer side not exceeding 1000px.</li><li>The output features of the backbone network (indicated by H x W) are usually much smaller than the input image depending on the stride of the backbone network. For both the possible backbone networks used in the paper (VGG, ZF-Net) the network stride is 16. This means that two consecutive pixels in the backbone output features correspond to two points 16 pixels apart in the input image.</li><li>For every point in the output feature map, the network has to learn whether an object is present in the input image at its corresponding location and estimate its size. This is done by placing a set of “Anchors” on the input image for each location on the output feature map from the backbone network. These anchors indicate possible objects in various sizes and aspect ratios at this location. The figure below shows 9 possible anchors in 3 different aspect ratios and 3 different sizes placed on the input image for a point A on the output feature map. For the PASCAL challenge, the anchors used have 3 scales of box area 128², 256², 512² and 3 aspect ratios of 1:1, 1:2 and 2:1.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*hZF7pnTEhyOK8z25t_QZVw.jpeg" /><figcaption>Figure 2: The possible anchors in the input image in a location corresponding to point A in the feature map.</figcaption></figure><ul><li>As the network moves through each pixel in the output feature map, it has to check whether these <em>k</em> corresponding anchors spanning the input image actually contain objects, and refine these anchors’ coordinates to give bounding boxes as “Object proposals” or regions of interest.</li><li>First, a 3 x 3 convolution with 512 units is applied to the backbone feature map as shown in Figure 1, to give a 512-d feature map for every location. This is followed by two sibling layers: a 1 x 1 convolution layer with 18 units for object classification, and a 1 x 1 convolution with 36 units for bounding box regression.</li><li>The 18 units in the classification branch give an output of size (H, W, 18). This output is used to give probabilities of whether or not each point in the backbone feature map (size: H x W) <strong>contains an object</strong> within all 9 of the anchors at that point.</li><li>The 36 units in the regression branch give an output of size (H, W, 36). This output is used to give the 4 regression coefficients of each of the 9 anchors for every point in the backbone feature map (size: H x W). These regression coefficients are used to improve the coordinates of the anchors that contain objects.</li></ul><h4>Training and Loss functions</h4><ul><li>The output feature map consists of about 40 x 60 locations, corresponding to 40*60*9 ~ 20k anchors in total. At train time, all the anchors that cross the boundary are ignored so that they do not contribute to the loss. This leaves about 6k anchors per image.</li><li>An anchor is considered to be a “positive” sample if it satisfies either of the two conditions — a) The anchor has the highest IoU (Intersection over Union, a measure of overlap) with a groundtruth box; b) The anchor has an IoU greater than 0.7 with any groundtruth box. The same groundtruth box can cause multiple anchors to be assigned positive labels.</li><li>An anchor is labeled “negative” if its IoU with all groundtruth boxes is less than 0.3. The remaining anchors (neither positive nor negative) are disregarded for RPN training.</li><li>Each mini-batch for training the RPN comes from a single image. Sampling all the anchors from this image would bias the learning process toward negative samples, and so 128 positive and 128 negative samples are randomly selected to form the batch, padding with additional negative samples if there are an insufficient number of positives.</li><li>The training loss for the RPN is also a multi-task loss, given by:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*mq0jKvp4I1o1ecHBTzQLHA.png" /></figure><ul><li>Here <em>i</em> is the index of the anchor in the mini-batch. The classification loss <em>L𝒸ₗₛ(pᵢ, pᵢ*)</em> is the log loss over two classes (object vs not object). <em>pᵢ</em> is the output score from the classification branch for anchor<em> i</em>, and <em>pᵢ*</em> is the groundtruth label (1 or 0).</li><li>The regression loss <em>Lᵣₑ(tᵢ, tᵢ*)</em> is activated only if the anchor actually contains an object i.e., the groundtruth <em>pᵢ*</em> is 1. The term <em>tᵢ</em> is the output prediction of the regression layer and consists of 4 variables [tₓ, tᵧ, t<em>w</em>, tₕ]. The regression target tᵢ* is calculated as —</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*s6uMkokhzyGoMLSXLMm9Ug.png" /></figure><ul><li>Here x, y, w, and h correspond to the <em>(x, y)</em> coordinates of the box centre and the height <em>h</em> and width <em>w</em> of the box. xₐ, x* stand for the coordinates of the anchor box and its corresponding groundtruth bounding box.</li><li>Remember that all <em>k </em>(<em>= </em>9) of the anchor boxes have different regressors that do not share weights. So the regression loss for an anchor <em>i</em> is applied to its corresponding regressor (if it is a positive sample).</li><li>At test time, the learned regression output tᵢ can be applied to its corresponding anchor box (that is predicted positive), and the <em>x, y, w, h</em> parameters for the predicted object proposal bounding box can be back-calculated from —</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*8BnFj3-7f6h4xM-EUkeykw.png" /></figure><h4>Test time details</h4><p>At test time, the 20k anchors from each image go through a series of post-processing steps to send in the object proposal bounding boxes.</p><ul><li>The regression coefficients are applied to the anchors for precise localization. This gives precise bounding boxes.</li><li>All the boxes are arranged according to their <em>cls</em> scores. Then, a non-maximum suppression (NMS) is applied with a threshold of 0.7: That is, from the top down, all of the bounding boxes which have an IoU of greater than 0.7 with another bounding box are discarded. Thus the highest-scoring bounding box is retained for a group of overlapping boxes.</li><li>This gives about 2k proposals per image.</li><li>The cross-boundary bounding boxes are retained and clipped to image boundary.</li><li>While using these object proposals to train the Fast R-CNN detection pipeline, all 2k proposals from the RPN are used. At test time for Fast R-CNN detection, only the Top N proposals from the RPN are chosen.</li></ul><h3>Object detection: Faster R-CNN (RPN + Fast R-CNN)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*tTqg3W165itg-LVRFxHJfA.jpeg" /><figcaption>Figure 3: The RPN for region proposals and Fast R-CNN as a detector in the Faster R-CNN detection pipeline</figcaption></figure><p>The Faster R-CNN architecture consists of the RPN as a region proposal algorithm and the Fast R-CNN as a detector network.</p><h4>Fast R-CNN as a detector for Faster R-CNN</h4><p>The Fast R-CNN detector also consists of a CNN backbone, an ROI pooling layer and fully connected layers followed by two sibling branches for classification and bounding box regression as shown in Figure 3.</p><ul><li>The input image is first passed through the backbone CNN to get the feature map (Feature size: 60, 40, 512). Besides test time efficiency, another key reason using an RPN as a proposal generator makes sense is the advantages of <strong>weight sharing between the RPN backbone and the Fast R-CNN detector backbone</strong>.</li><li>Next, the bounding box proposals from the RPN are used to pool features from the backbone feature map. This is done by the ROI pooling layer. The ROI pooling layer, in essence, works by a) Taking the region corresponding to a proposal from the backbone feature map; b) Dividing this region into a fixed number of sub-windows; c) Performing max-pooling over these sub-windows to give a fixed size output. To understand the details of the ROI pooling layer and it’s advantages, read <a href="https://medium.com/@shilpa_47526/fast-r-cnn-for-object-detection-a-technical-summary-a0ff94faa022">Fast R-CNN</a>.</li><li>The output from the ROI pooling layer has a size of (N, 7, 7, 512) where N is the number of proposals from the region proposal algorithm. After passing them through two fully connected layers, the features are fed into the sibling classification and regression branches.</li><li>Note that these classification and detection branches are different from those of the RPN. Here the classification layer has C units for each of the classes in the detection task (including a catch-all background class). The features are passed through a softmax layer to get the classification scores — the probability of a proposal belonging to each class. The regression layer coefficients are used to improve the predicted bounding boxes. Here the regressor is size agnostic, (unlike the RPN) but is specific to each class. That is, all the classes have individual regressors with 4 parameters each corresponding to C*4 output units in the regression layer.</li><li>For more details on how the Faster R-CNN is trained and its loss functions refer to <a href="https://medium.com/@shilpa_47526/fast-r-cnn-for-object-detection-a-technical-summary-a0ff94faa022">Fast R-CNN</a>.</li></ul><h4>4 Step Alternating training</h4><p>In order to force the network to share the weights of the CNN backbone between the RPN and the detector, the authors use a 4 step training method:</p><p>a)The RPN is trained independently as described above. The backbone CNN for this task is initialized with weights from a network trained for an ImageNet classification task, and is then fine-tuned for the region proposal task.<br>b) The Fast R-CNN detector network is also trained independently. The backbone CNN for this task is initialized with weights from a network trained for an ImageNet classification task, and is then fine-tuned for the object detection task. The RPN weights are fixed and the proposals from the RPN are used to train the Faster R-CNN.<br>c) The RPN is now initialized with weights from this Faster R-CNN, and fine-tuned for the region proposal task. This time, weights in the common layers between the RPN and detector remain fixed, and only the layers unique to the RPN are fine-tuned. This is the final RPN.<br>d) Once again using the new RPN, the Fast R-CNN detector is fine-tuned. Again, only the layers unique to the detector network are fine-tuned and the common layer weights are fixed.</p><p>This gives a Faster R-CNN detection framework that has shared convolutional layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*hLM5gqM7l4fXBvXDjM6khw.png" /><figcaption>Figure 4: Results of the Faster R-CNN detection framework with a VGG backbone</figcaption></figure><h3>Results</h3><ul><li>In all the experiments on the PASCAL datasets, Fast R-CNN was chosen as a detector. The use of the RPN+ZF backbone as just a proposal network (without sharing weights with the detector) matched the performance of using “Selective Search” (SS) as a region proposal algorithm. This already gives us comparable results with a large decrease in detection time. RPN+VGG backbone as just a proposal network with unshared weights performs slightly better than the SS region proposal baseline. When using shared weights with the detector, both the ZF and VGG backbones in RPN surpassed the performance of the SS baseline. This along with a large number of other experiments validated the use of RPN as a region proposal method.</li><li>Detection with a VGG RPN takes 198ms compared to the 1.8 seconds of Selective Search.</li><li>The other experiments performed validate the use of NMS, and the use of separate classification and regression branches. The classification branch used to rank proposals by score seems to be an important factor in maintaining a reasonably high recall vs IoU overlap ratio even when the number of object proposals is reduced.</li><li>In ablation studies to observe the importance of scale and aspect ratios of anchor boxes, the authors find that using 3 scales with a single aspect ratio works almost as well as 3 scales and 3 aspect ratios. Depending on the task and the dataset, these ratios and scales can be modified. Using a single anchor at each location causes the mAP to drop considerably.</li></ul><p>This concludes the technical summary of the Faster R-CNN paper. Hope you enjoyed (understood)! Open to discussions or corrections in the comments below.</p><h3>References:</h3><p>[1] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, <a href="https://arxiv.org/pdf/1506.01497.pdf">Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks</a>, NIPS’15 Proceedings<br>[2] <a href="http://www.telesens.co/2018/03/11/object-detection-and-classification-using-r-cnns/">http://www.telesens.co/2018/03/11/object-detection-and-classification-using-r-cnns/</a><br>[3] <a href="https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/object_localization_and_detection.html">https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/object_localization_and_detection.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=474c5b857b46" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/faster-r-cnn-for-object-detection-a-technical-summary-474c5b857b46">Faster R-CNN for object detection</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fast R-CNN for Object detection]]></title>
            <link>https://medium.com/data-science/fast-r-cnn-for-object-detection-a-technical-summary-a0ff94faa022?source=rss-55d4794e571b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0ff94faa022</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Shilpa Ananth]]></dc:creator>
            <pubDate>Mon, 05 Aug 2019 19:20:12 GMT</pubDate>
            <atom:updated>2019-08-30T22:16:53.879Z</atom:updated>
            <content:encoded><![CDATA[<h3>Fast R-CNN for object detection</h3><h4>A technical paper summary</h4><p>In an attempt to understand the key concepts and milestones of object detection using deep learning, the previous article about <a href="https://medium.com/@shilpa_47526/r-cnn-for-object-detection-a-technical-summary-9e7bfa8a557c">Regions with CNNs</a> [1] highlighted the features and training details of each stage in one of the first ever detector networks with a deep learning backbone. This article digs into the details of its popular successor, Fast R-CNN. [2]</p><p>With the technological advancements and increase in computational power that we have seen since 2014 when the R-CNN paper was published, it is easy for us to now see the obvious pitfalls of R-CNNs the authors discovered back then:</p><ul><li><strong>Multi-stage, expensive training:</strong> The separate training processes required for all the stages of the network — fine-tuning a CNN on object proposals, learning an SVM to classify the feature vector of each proposal from the CNN and learning a bounding box regressor to fine-tune the object proposals (refer to <a href="https://medium.com/@shilpa_47526/r-cnn-for-object-detection-a-technical-summary-9e7bfa8a557c">Regions with CNNs</a> for more details) proves to be a burden in terms of time, computation and resources. For example, to train the SVM, we would need the features of thousands of possible region proposals to be written to the disk from the previous stage.</li><li><strong>Slow test time: </strong>Given this multi-stage pipeline, detection using a simple VGG network as the backbone CNN takes 47s/image.</li></ul><p>Another paper published in 2015, SPP-Nets or “Spatial Pyramid Pooling Networks”[3] solved one part of the problem. Its main contribution was a) the use of multi-scale pooling of image features to improve the network performance for classification and detection and also b) the use of arbitrary sized input images to train CNN’s despite the use of fully connected (FC) layers. In its detection pipeline, it introduced an important concept— extracting a single feature map from the entire image only once, and pooling the features of arbitrary sized “region proposals” for that image, from this single feature map. This allowed their network to run orders of magnitude faster than R-CNN. Instead of running a CNN on 2000 region proposals from an image individually like R-CNN, they have only one forward pass from which they can pool the features of all 2000 candidate proposals.</p><h3>Fast R-CNN Pipeline</h3><h4>Architecture details:</h4><p>To better understand how and why the Fast R-CNN improved efficiency and performance of R-CNN and SPP Networks, let’s first look into its architecture.</p><ul><li>The Fast R-CNN consists of a CNN (usually pre-trained on the ImageNet classification task) with its final pooling layer replaced by an “ROI pooling” layer and its final FC layer is replaced by two branches — a (K + 1) category softmax layer branch and a category-specific bounding box regression branch.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jYDMaYeH-TrcoofDqCdxug.jpeg" /><figcaption>Figure 1: The Fast R-CNN pipeline</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*qhN4EKjJO1hxKfpADOkcaw.png" /><figcaption>Figure 2: The ROI pooling layer as a special case of SPP layer</figcaption></figure><ul><li>The entire image is fed into the backbone CNN and the features from the last convolution layer are obtained. Depending on the backbone CNN used, the output feature maps are much smaller than the original image size. This depends on the stride of the backbone CNN, which is usually 16 in the case of a VGG backbone.</li><li>Meanwhile, the object proposal windows are obtained from a region proposal algorithm like selective search[4]. As explained in <a href="https://medium.com/@shilpa_47526/r-cnn-for-object-detection-a-technical-summary-9e7bfa8a557c">Regions with CNNs</a>, object proposals are rectangular regions on the image that signify the presence of an object.</li><li>The portion of the backbone feature map that belongs to this window is then fed into the ROI Pooling layer.</li><li>The ROI pooling layer is a special case of the spatial pyramid pooling (SPP) layer with just one pyramid level. The layer basically divides the features from the selected proposal windows (that come from the region proposal algorithm) into sub-windows of size <em>h/H</em> by <em>w/W </em>and performs a pooling operation in each of these sub-windows. This gives rise to fixed-size output features of size (H x W) irrespective of the input size. H and W are chosen such that the output is compatible with the network’s first fully-connected layer. The chosen values of H and W in the Fast R-CNN paper is 7. Like regular pooling, ROI pooling is carried out in every channel individually.</li><li>The output features from the ROI Pooling layer (N x 7 x 7 x 512 where N is the number of proposals) are then fed into the successive FC layers, and the softmax and BB-regression branches. The softmax classification branch produces probability values of each ROI belonging to K categories and one catch-all background category. The BB regression branch output is used to make the bounding boxes from the region proposal algorithm more precise.</li></ul><h4>Loss:</h4><p>The classification branch of the softmax layer gives probabilities for every ROI over (K +1) categories p = p₀, … pₖ. The classification loss <em>L</em>𝒸ₗₛ<em>(p,u)</em> is given by -log(pᵤ) which is the log loss for the true class u.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*noNvUyI_rMI7mh7WZYZ_0A.png" /><figcaption>Equation 1: smooth L1 loss for BB regression</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*e1JBzoj4wlajm7Ipzg5Sig.png" /><figcaption>Equation 2: Joint loss for multi-task training</figcaption></figure><p>The regression branch produces 4 bounding box regression offsets tᵏᵢ where <em>i = x, y, w, </em>and<em> h. (x, y)</em> stands for the top-left corner and <em>w</em> and <em>h</em> denote the height and width of the bounding box. The true bounding box regression targets for a class u are indicated by <em>v</em>ᵢ where<em> i = x, y, w, </em>and<em> h </em>when u≥1<em>.</em> The case where<em> u=0 </em>is ignored because the background classes have no groundtruth boxes. The regression loss used is a smooth L1 loss given in equation 1. The joint multi-task loss for each ROI is given by the combination of the two losses as shown in equation 2. Notice that the Fast R-CNN here has a combined learning scheme that fine-tunes the backbone CNN, and classifies and regresses the bounding box.</p><h4>Training:</h4><ul><li>During training, each mini-batch is constructed from N=2 images. The mini-batch consists of 64 ROIs from each image.</li><li>Like R-CNN, 25% of the ROIs are object proposals that have at least 0.5 IoU with a ground-truth bounding box of a foreground class. These would be positive for that particular class and would be labelled with the appropriate u=1…K.</li><li>The remaining ROIs are sampled from proposals that have an IoU with groundtruth bounding boxes between [0.1, 0.5). These ROIs are labelled as belonging to the class u = 0 (background class).</li><li>Until before the ROI pooling layer, the entire image goes through the CNN. This way, all the ROIs from the same image share computation and memory in the forward and backward passes through the CNN (before the ROI pooling layer).</li><li>The sampling strategy (sampling all the ROI in a batch from very few images) allows the entire module — feature generation, classification, and regression — to be trained together, unlike SPP Nets. In the SPP Nets most of the input ROIs come from different images, so the training process for detection only finetunes the fully connected layers after the feature generation because it is not feasible to update the weights before the SPP layer (some of the ROI’s could have very large receptive fields on the original image). The actual features still come from a pre-trained network that was trained for classification. This limits the accuracy of SPP Nets.</li><li>Note that the batch size of the network is very small for the CNN until the ROI pooling layer (batch size = 2), but is much larger (batch size = 128) for the following softmax and regression layers.</li></ul><h4>Other Notes:</h4><ul><li><strong>Backpropagation through ROI pooling layer:</strong> For each mini-batch ROI <em>r, </em>let the ROI pooling output unit <em>yᵣⱼ</em> be the output of max-pooling in it’s sub-window <em>R(r, j).</em> Then, the gradient is accumulated in an input unit (<em>xᵢ</em>) in <em>R(r, j) </em>if this position <strong><em>i</em></strong> is the argmax selected for <em>yᵣⱼ.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*P99SvHcojXFv2LtgyiDQYQ.jpeg" /></figure><ul><li>In the multi-scale pipeline of Fast R-CNN, input images are resized into a randomly sampled size at train time to introduce scale invariance. At test time, each image is fed to the network in multiple fixed scales. For each ROI, the features are pooled from only one of these scales, chosen such that the scaled candidate window has the number of pixels closest to 224 x 224. However, the authors find that the single scale pipeline performs almost as well with a much lower computing time cost. In the single-scale approach, all images at train and test time are resized to 600 (shortest side) with the upper cap on the longest side being 1000.</li><li>The large fully connected layers can be compressed with truncated SVD to make the network more efficient. Here a layer parameterized by W as its weight matrix can be factorized to reduce the parameter count by splitting it into two layers (ΣₜVᵀ and U with biases) without a non-linearity between them, where W ~ U ΣₜVᵀ.</li></ul><h3>Results</h3><ul><li>Fast R-CNN achieved state of the art performance at the time of its publication on VOC12, VOC10(with additional data) and VOC07.</li><li>Fast R-CNN processes images 45x faster than R-CNN at test time and 9x faster at train time. It also trains 2.7x faster and runs test images 7x faster than SPP-Net. On further using truncated SVD, the detection time of the network is reduced by more than 30% with just a 0.3 drop in mAP.</li><li>Other contributions of the paper show that for larger networks fine-tuning the conv layers as well (and not just the FC layers like SPPnets) is really important to achieve significantly better mAP. For smaller networks, this improvement might not be as large.</li><li>Multi-task training is not only easier but also improves performance because the tasks influence each other while training. Hence training the different stages of the network altogether improves it’s shared representative power (the backbone CNN)</li></ul><p>This concludes the technical summary of the Fast R-CNN paper. Hope you enjoyed (understood)! Open to discussions or corrections in the comments below.</p><p>Read about the Fast R-CNN’s successor and state of the art object detection network— Faster R-CNN <a href="https://medium.com/@shilpa_47526/faster-r-cnn-for-object-detection-a-technical-summary-474c5b857b46">here</a>.</p><h4>References:</h4><p>[1] Girshick, Ross et al. “Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation.” 2014 IEEE Conference on Computer Vision and Pattern Recognition (2014)<br>[2] Girshick, Ross. “Fast R-CNN.” 2015 IEEE International Conference on Computer Vision (ICCV) (2015)<br>[3] He, Kaiming et al. “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition.” Lecture Notes in Computer Science (2014)<br>[4] Uijlings, J. R. R. et al. “Selective Search for Object Recognition.” International Journal of Computer Vision 104.2 (2013)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0ff94faa022" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/fast-r-cnn-for-object-detection-a-technical-summary-a0ff94faa022">Fast R-CNN for Object detection</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[R-CNN for Object detection]]></title>
            <link>https://medium.com/data-science/r-cnn-for-object-detection-a-technical-summary-9e7bfa8a557c?source=rss-55d4794e571b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e7bfa8a557c</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Shilpa Ananth]]></dc:creator>
            <pubDate>Sat, 27 Apr 2019 02:09:12 GMT</pubDate>
            <atom:updated>2020-09-30T23:54:38.594Z</atom:updated>
            <content:encoded><![CDATA[<h3>R-CNN for object detection</h3><h4>A technical paper summary</h4><p>In the rapidly growing field of deep learning, this is part of a series to understand the publications that are the fundamentals for current day object detection.</p><p>The original paper “Rich feature hierarchies for accurate object detection and semantic segmentation” [1] elaborates one of the first breakthroughs of the use of CNNs in an object detection system called the ‘R-CNN’ or ‘Regions with CNN’ that had a much higher object detection performance than other popular methods at the time.</p><h3>Region proposals</h3><p>The first stage of the R-CNN pipeline is the generation of ‘region proposals’ or regions in an image that <em>could </em>belong to a particular object. The authors use the selective search algorithm. The selective search algorithm [2] works by generating sub-segmentations of the image that could belong to one object — based on color, texture, size and shape — and iteratively combining similar regions to form objects. This gives ‘object proposals’ of different scales. Note the R-CNN pipeline is agnostic to the region proposal algorithm. The authors use the selective search algorithm to <strong>generate 2000 category independent region proposals</strong> (usually indicated by rectangular regions or ‘bounding boxes’) <strong>for each individual image.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*g4izhFRwaDAi9vLB.png" /><figcaption>Figure 1: How the selective search algorithm iteratively obtains “region proposals”</figcaption></figure><h3>Stage 1: Feature extraction from Region Proposals</h3><p>At the end of this stage in the pipeline, the authors generate a 4096 dimensional feature vector from each of the 2000 region proposals for every image using a Convolutional Neural Network (CNN). The details of training this CNN are as given below.</p><p><strong>Supervised Pre-training: </strong>The CNN described by Krizhevsky et al [3], now popularly known as the “AlexNet” has 5 convolutional and 2 fully connected layers. The CNN is first trained on the ILSVRC2012 <em>classification</em> dataset for a 1000 way image classification task with a large number of images so that the convolution layers can learn basic image features.</p><p><strong>Domain-Specific Fine-Tuning: </strong>Now, the network needs to be fine-tuned to learn a) The visual features of the new types of images- distorted region proposals, and b) Specific target classes of the smaller dataset for the detection task. We fine-tune the classification network to identify the classes belonging to the detection task from the region proposals.</p><ul><li>The final 1000 way classification layer of the CNN from pre-training is replaced with a randomly initialized (N+1) way softmax classification layer for the N object classes and a general background class of the detection task.</li><li><strong>Inputs :</strong><em> </em>Each of the 2000 region proposals generated from every image (using the selective search algorithm) are converted into fixed inputs of size 227 x 227 by a simple warping, irrespective of the size or aspect ratio to enable their use in fine-tuning the CNN (we need fixed-sized inputs irrespective of the actual dimensions to feed to the CNN). An additional parameter, <em>p </em>is used to indicate the amount of possible dilation of the original bounding box to include some context from the area around it.</li><li><strong>Labels for training: </strong>The authors map each object proposal to the <strong>ground-truth instance with which it has maximum IoU </strong>overlap and label it as a <strong><em>positive</em></strong><em> </em>(for the matched ground-truth class) if the IoU is at least 0.5. The rest of the boxes are treated as the <strong><em>background</em></strong> class (negative for all classes).</li><li><strong>Training pipeline: </strong>The authors train the network using SGD with (1/10)th of the initial pre-training learning rate. In each iteration they sample 32 windows that are positive over all the classes and 96 windows that belong to the background class to form a mini-batch of 128, to ensure that there is enough representation from the positive classes during training.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xxm2czX19Fcxd-yLZvfodg.png" /><figcaption>Figure 2: The CNN training pipeline taking in a batch of region proposals and giving a classification label (at train time)</figcaption></figure><p><strong><em>The final output of Stage 1: </em></strong><em>After training, the final classification layer is removed and a</em><strong><em> </em></strong><em>4096 dimensional feature vector is obtained from the penultimate layer of the CNN for each of the 2000 region proposals (for every image). Refer to the image above.</em></p><h3>Stage 2: SVM for object classification</h3><p>This stage consists of learning an <strong>individual</strong> linear SVM (Support Vector Machine) classifier for each class, that detects the presence or absence of an object belonging to a particular class.</p><ul><li><strong>Inputs: </strong>The 4096-<em>d </em>feature vector for each region proposal.</li><li><strong>Labels for training: </strong>The features of all region proposals that have an IoU overlap of less than 0.3 with the ground truth bounding box are considered <em>negatives</em><strong><em> </em></strong>for that class during training. The <em>positives </em>for that class are simply the features from the ground truth bounding boxes itself. All other proposals (IoU overlap greater than 0.3, but not a ground truth bounding box) are <em>ignored for the purpose of training the SVM</em>.</li><li><strong>Test time inference for a single image: </strong>The class-specific dot products between the features and SVM weights are consolidated into a single matrix-matrix product for an image (Figure 3 below). That is, for every image, a 2000 x 4096 feature matrix is generated (the 4096-<em>d </em>feature from the CNN for all 2000 region proposals). The SVM weight matrix is 4096 x N where N is the number of classes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*cmgaF4K6A_Ql-AyqYH3ZXQ.png" /><figcaption>Figure 3: Final consolidated test time SVM computation</figcaption></figure><p><strong><em>The final output of stage 2: </em></strong><em>After training the SVM, the final output of stage 2 is a set of positive object proposals for each class, from the CNN features of 2000 region proposals (of every image).</em></p><h3>Stage 3: Bounding box regression</h3><p>In order to improve localization performance, the authors include a bounding-box regression step to learn corrections in the predicted bounding box location and size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PUJocoWsZIHZltbQeXsqaw.png" /><figcaption>Figure 4: The equations for bounding box regression stages explained below.</figcaption></figure><ul><li>Equation 1: The aim of this task is to learn a target <strong>transformation</strong> between our predicted proposal <em>P </em>and the target proposal <em>G</em>. The variables<em> x, y, w, </em>and<em> h</em> stand for the coordinates of the center <em>(x, y)</em> and the width <em>w </em>and height <em>h </em>of the proposal.</li><li>Equation 2: The <strong>ground-truth transformations</strong> that need to be learnt are shown in equation 2. The first two transformations specify a scale-invariant translation of the center of <em>P — x </em>and<em> y</em>, and the second two specify log space transformations of the width <em>w </em>and height <em>h</em>.</li><li>Equation 3: <em>d</em>ₖ<em>(P)</em> where ₖ can belong to <em>(x, y, h or w) </em>is the <strong>predicted transformation</strong>. Ĝ signifies the corrected predicted box calculated using the original predicted box <em>P </em>and the predicted transformation <em>d</em>ₖ<em>(P)</em>.</li><li>Equation 4: The predicted transformation <em>d</em>ₖ(P) is modeled as a linear function of the pool₅(shown in Figure 2) features — <em>Φ₅</em>. <br>Hence, <em>dₖ(P) = wₖᵀ Φ₅(P) </em>where<strong> <em>wₖ </em>is the vector of learnable model parameters. </strong>Note that Φ₅ is dependent on the actual image features. <strong><em>wₖ</em> is learnt by optimizing the regularized least-squares objective function shown in the second line of equation 4.</strong></li><li>Other notes: The algorithm only learns from a predicted box <em>P</em> if it is near at least one groundtruth box. Each predicted box <em>P </em>is mapped to its groundtruth by choosing the groundtruth box with which it has a maximum overlap (provided it has an IoU overlap of at least 0.6).<strong> A separate transformation is learnt for each class of the detection task.</strong></li></ul><p><strong><em>The final output of stage 3: </em></strong><em>For all the positive region proposals of each class predicted from the SVM, we have an accurate, corrected bounding box around the object.</em></p><h3>Other Contributions</h3><ul><li>At the time of it’s publication, the R-CNN achieved a mAP of 54% on PASCAL VOC 2010 and 31% on ILSVRC detection, much higher than its competing algorithms.</li><li>One of the major other contributions of the paper was conclusive evidence that supervised pre-training for a similar task boosted performance much more than unsupervised pre-training.</li><li>After a general analysis of the CNN, the authors found that without fine-tuning for the detection task, the network with the fully connected layers removed, was able to detect object proposals as well as the network with the fully connected layers attached — even though the former (pool₅) features are computed using only 6% of the network’s parameters. Post fine-tuning, the largest performance improvement occurred in the network including the FC layers. This showed that the convolutional layers contain more generalizable features than fully connected layers, which contain task-specific information.</li><li>The authors hypothesize that the need for a separate SVM for detection rather than using the fine-tuned CNN itself for classification comes from a combination of several factors including a) The fact that the definition of positive examples used in fine-tuning does not emphasize precise localization and b) The softmax classifier in fine-tuning was trained on randomly sampled negative examples rather than on the subset of “hard negatives” used for SVM training.</li></ul><p>This concludes the technical summary of the R-CNN paper. Hope you enjoyed (understood)! Open to discussions or corrections in the comments below.</p><p>Read about R-CNNs popular successors: <a href="https://medium.com/@shilpa_47526/fast-r-cnn-for-object-detection-a-technical-summary-a0ff94faa022">Fast R-CNN</a> and <a href="https://medium.com/@shilpa_47526/faster-r-cnn-for-object-detection-a-technical-summary-474c5b857b46">Faster R-CNN</a>.</p><h4>References:</h4><p>[1] Girshick, Ross et al. “Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation.” 2014 IEEE Conference on Computer Vision and Pattern Recognition (2014)<br>[2] Uijlings, J. R. R. et al. “Selective Search for Object Recognition.” International Journal of Computer Vision 104.2 (2013)<br>[3] Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton “<a href="https://www.semanticscholar.org/paper/ImageNet-Classification-with-Deep-Convolutional-Krizhevsky-Sutskever/6bca5ff6cfe3b473a1ae04c635404c3e4109fca3">ImageNet Classification with Deep Convolutional Neural Networks</a>”, Published in NIPS 2012</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e7bfa8a557c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/r-cnn-for-object-detection-a-technical-summary-9e7bfa8a557c">R-CNN for Object detection</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>