<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ethan Yanjia Li on Medium]]></title>
        <description><![CDATA[Stories by Ethan Yanjia Li on Medium]]></description>
        <link>https://medium.com/@ethanyanjiali?source=rss-60ef96b05d56------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KoN0OHLPIBgTzRjhn5fDaA.png</url>
            <title>Stories by Ethan Yanjia Li on Medium</title>
            <link>https://medium.com/@ethanyanjiali?source=rss-60ef96b05d56------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:41:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ethanyanjiali/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[12 Papers You Should Read to Understand Object Detection in the Deep Learning Era]]></title>
            <link>https://medium.com/data-science/12-papers-you-should-read-to-understand-object-detection-in-the-deep-learning-era-3390d4a28891?source=rss-60ef96b05d56------2</link>
            <guid isPermaLink="false">https://medium.com/p/3390d4a28891</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ethan Yanjia Li]]></dc:creator>
            <pubDate>Mon, 10 Aug 2020 02:11:17 GMT</pubDate>
            <atom:updated>2023-06-15T06:50:33.648Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qidM28kjq00OFfmqx34n5Q.jpeg" /><figcaption>Free picture from <a href="https://unsplash.com/photos/qcqmS0JG58Q">Unsplash</a>. Photography from <a href="https://unsplash.com/@joannakosinska">Joanna Kosinska</a> and edited by myself.</figcaption></figure><blockquote>We are a team of experts to help your business adopt AI solutions or build new AI products. Contact us at info@imaginaire.ai or visit our website <a href="https://www.imaginaire.ai/">https://www.imaginaire.ai</a></blockquote><h3>Foreword</h3><p>As the second article in the “Papers You Should Read” series, we are going to walk through both the history and some recent developments in a more difficult area of computer vision research: object detection. Before the deep learning era, hand-crafted features like HOG and feature pyramids are used pervasively to capture localization signals in an image. However, those methods usually can’t extend to generic object detection well, so most of the applications are limited to face or pedestrian detections. With the power of deep learning, we can train a network to learn which features to capture, as well as what coordinates to predict for an object. And this eventually led to a boom of applications based on visual perception, such as the commercial face recognition system and autonomous vehicle. In this article, I picked 12 must-read papers for newcomers who want to study object detection. Although the most challenging part of building an object detection system hides in the implementation details, reading these papers can still give you a good high-level understanding of where the ideas come from, and how would object detection evolve in the future.</p><p>As a prerequisite for reading this article, you need to know the basic idea of the convolution neural network and the common optimization method such as gradient descent with back-propagation. It’s also highly recommended to read my previous article “<a href="https://towardsdatascience.com/10-papers-you-should-read-to-understand-image-classification-in-the-deep-learning-era-4b9d792f45a7">10 Papers You Should Read to Understand Image Classification in the Deep Learning Era</a>” first because many cool ideas of object detection originate from a more fundamental image classification research.</p><h3>2013: OverFeat</h3><blockquote>OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*vsDePOcgyd56MNkG--nVTQ.png" /><figcaption>From “<a href="https://arxiv.org/abs/1312.6229"><em>OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks</em></a><em>”</em></figcaption></figure><p>Inspired by the early success of AlexNet in the 2012 ImageNet competition, where CNN-based feature extraction defeated all hand-crafted feature extractors, OverFeat quickly introduced CNN back into the object detection area as well. The idea is very straight forward: if we can classify one image using CNN, what about greedily scrolling through the whole image with different sizes of windows, and try to regress and classify them one-by-one using a CNN? This leverages the power of CNN for feature extraction and classification, and also bypassed the hard region proposal problem by pre-defined sliding windows. Also, since a nearby convolution kernel can share part of the computation result, it is not necessary to compute convolutions for the overlapping area, hence reducing cost a lot. OverFeat is a pioneer in the one-stage object detector. It tried to combine feature extraction, location regression, and region classification in the same CNN. Unfortunately, such a one-stage approach also suffers from relatively poorer accuracy due to less prior knowledge used. Thus, OverFeat failed to lead a hype for one-stage detector research, until a much more elegant solution coming out 2 years later.</p><h3>2013: R-CNN</h3><blockquote>Region-based Convolutional Networks for Accurate Object Detection and Segmentation</blockquote><p>Also proposed in 2013, R-CNN is a bit late compared with OverFeat. However, this region-based approach eventually led to a big wave of object detection research with its two-stage framework, i.e, region proposal stage, and region classification and refinement stage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*UL6mcGK8-MEYNwUvS7PASQ.png" /><figcaption>From “<a href="http://islab.ulsan.ac.kr/files/announcement/513/rcnn_pami.pdf"><em>Region-based Convolutional Networks for Accurate Object Detection and Segmentation</em></a>”</figcaption></figure><p>In the above diagram, R-CNN first extracts potential regions of interest from an input image by using a technique called selective search. Selective search doesn’t really try to understand the foreground object, instead, it groups similar pixels by relying on a heuristic: similar pixels usually belong to the same object. Therefore, the results of selective search have a very high probability to contain something meaningful. Next, R-CNN warps these region proposals into fixed-size images with some paddings, and feed these images into the second stage of the network for more fine-grained recognition. Unlike those old methods using selective search, R-CNN replaced HOG with a CNN to extract features from all region proposals in its second stage. One caveat of this approach is that many region proposals are not really a full object, so R-CNN needs to not only learn to classify the right classes, but also learn to reject the negative ones. To solve this problem, R-CNN treated all region proposals with a ≥ 0.5 IoU overlap with a ground-truth box as positive, and the rest as negatives.</p><p>Region proposal from selective search highly depends on the similarity assumption, so it can only provide a rough estimate of location. To further improve localization accuracy, R-CNN borrowed an idea from “Deep Neural Networks for Object Detection” (aka DetectorNet), and introduced an additional bounding box regression to predict the center coordinates, width and height of a box. This regressor is widely used in the future object detectors.</p><p>However, a two-stage detector like R-CNN suffers from two big issues: 1) It’s not fully convolutional because selective search is not E2E trainable. 2) region proposal stage is usually very slow compared with other one-stage detectors like OverFeat, and running on each region proposal separately makes it even slower. Later, we will see how R-CNN evolve over time to address these two issues.</p><h3>2015: Fast R-CNN</h3><blockquote>Fast R-CNN</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*43dfIcDNqJ8SIu4_KtU45g.png" /><figcaption>From “<a href="https://arxiv.org/abs/1504.08083">Fast R-CNN</a>”</figcaption></figure><p>A quick follow-up for R-CNN is to reduce the duplicate convolution over multiple region proposals. Since these region proposals all come from one image, it’s naturally to improve R-CNN by running CNN over the entire image once and share the computation among many region proposals. However, different region proposals have different sizes, which also result in different output feature map sizes if we are using the same CNN feature extractor. These feature maps with various sizes will prevent us from using fully connected layers for further classification and regression because the FC layer only works with a fixed size input.</p><p>Fortunately, a paper called “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition” has already solved the dynamic scale issue for FC layers. In SPPNet, a feature pyramid pooling is introduced between convolution layers and FC layers to create a bag-of-words style of the feature vector. This vector has a fixed size and encodes features from different scales, so our convolution layers can now take any size of images as input without worrying about the incompatibility of the FC layer. Inspired by this, Fast R-CNN proposed a similar layer call the ROI Pooling layer. This pooling layer downsamples feature maps with different sizes into a fixed-size vector. By doing so, we can now use the same FC layers for classification and box regression, no matter how large or small the ROI is.</p><p>With a shared feature extractor and the scale-invariant ROI pooling layer, Fast R-CNN can reach a similar localization accuracy but having 10~20x faster training and 100~200x faster inference. The near real-time inference and an easier E2E training protocol for the detection part make Fast R-CNN a popular choice in the industry as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*voDoJY11x5kHWSu4eNg8CA.png" /><figcaption>From “<a href="https://arxiv.org/abs/1506.02640"><em>You Only Look Once: Unified, Real-Time Object Detection</em></a><em>”</em></figcaption></figure><p>This dense prediction over the entire image can cause trouble in computation cost, so YOLO took the bottleneck structure from GooLeNet to avoid this issue. Another problem of YOLO is that two objects might fall into the same coarse grid cell, so it doesn’t work well with small objects such as a flock of birds. Despite lower accuracy, YOLO’s straightforward design and real-time inference ability makes one-stage object detection popular again in the research, and also a go-to solution for the industry.</p><h3>2015: Faster R-CNN</h3><blockquote>Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks</blockquote><p>As we introduced above, in early 2015, Ross Girshick proposed an improved version of R-CNN called Fast R-CNN by using a shared feature extractor for proposed regions. Just a few months later, Ross and his team came back with another improvement again. This new network Faster R-CNN is not only faster than previous versions but also marks a milestone for object detection with a deep learning method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*r5h0GQtV-MxL1XuoYEh3aQ.png" /><figcaption>From “<a href="https://arxiv.org/abs/1506.01497"><em>Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks</em></a><em>”</em></figcaption></figure><p>With Fast R-CNN, the only non-convolutional piece of the network is the selective search region proposal. As of 2015, researchers started to realize that the deep neural network is so magical, that it can learn anything given enough data. So, is it possible to also train a neural network to proposal regions, instead of relying on heuristic and hand-crafted approach like selective search? Faster R-CNN followed this direction and thinking, and successfully created the Region Proposal Network (RPN). To simply put, RPN is a CNN that takes an image as input and outputs a set of rectangular object proposals, each with an objectiveness score. The paper used VGG originally but other backbone networks such as ResNet become more widespread later. To generate region proposals, a 3x3 sliding window is applied over the CNN feature map output to generate 2 scores (foreground and background) and 4 coordinates each location. In practice, this sliding window is implemented with a 3x3 convolution kernel with a 1x1 convolution kernel.</p><p>Although the sliding window has a fixed size, our objects may appear on different scales. Therefore, Faster R-CNN introduced a technique called anchor box. Anchor boxes are pre-defined prior boxes with different aspect ratios and sizes but share the same central location. In Faster R-CNN there are k=9 anchors for each sliding window location, which covers 3 aspect ratios for 3 scales each. These repeated anchor boxes over different scales bring nice translation-invariance and scale-invariance features to the network while sharing outputs of the same feature map. Note that the bounding box regression will be computed from these anchor box instead of the whole image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O3RZR_Q1dSVkZN1Klxd90A.png" /><figcaption>From “<a href="https://arxiv.org/abs/1506.01497"><em>Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks</em></a><em>”</em></figcaption></figure><p>So far, we discussed the new Region Proposal Network to replace the old selective search region proposal. To make the final detection, Faster R-CNN uses the same detection head from Fast R-CNN to do classification and fine-grained localization. Do you remember that Fast R-CNN also uses a shared CNN feature extractor? Now that RPN itself is also a feature extraction CNN, we can just share it with detection head like the diagram above. This sharing design doesn’t bring some trouble though. If we train RPN and Fast R-CNN detector together, we will treat RPN proposals as a constant input of ROI pooling, and inevitably ignore the gradients of RPN’s bounding box proposals. One walk around is called alternative training where you train RPN and Fast R-CNN in turns. And later in a paper “Instance-aware semantic segmentation via multi-task network cascades”, we can see that the ROI pooling layer can also be made differentiable w.r.t. the box coordinates proposals.</p><h3>2015: YOLO v1</h3><blockquote>You Only Look Once: Uniﬁed, Real-Time Object Detection</blockquote><p>While the R-CNN series started a big hype over two-stage object detection in the research community, its complicated implementation brought many headaches for engineers who maintain it. Does object detection need to be so cumbersome? If we are willing to sacrifice a bit of accuracy, can we trade for much faster speed? With these questions, Joseph Redmon submitted a network called YOLO to arxiv.org only four days after Faster R-CNN’s submission and finally brought popularity back to one-stage object detection two years after OverFeat’s debut.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*XL2LR39yeZ0zSloWkSp-OQ.png" /><figcaption>From “<a href="https://arxiv.org/abs/1506.02640"><em>You Only Look Once: Unified, Real-Time Object Detection</em></a><em>”</em></figcaption></figure><p>Unlike R-CNN, YOLO decided to tackle region proposal and region classification together in the same CNN. In other words, it treats object detection as a regression problem, instead of a classification problem relying on region proposals. The general idea is to split the input into an SxS grid and having each cell directly regress the bounding box location and the confidence score if the object center falls into that cell. Because objects may have different sizes, there will be more than one bounding box regressor per cell. During training, the regressor with the highest IOU will be assigned to compare with the ground-truth label, so regressors at the same location will learn to handle different scales over time. In the meantime, each cell will also predict C class probabilities, conditioned on the grid cell containing an object (high confidence score). This approach is later described as dense predictions because YOLO tried to predict classes and bounding boxes for all possible locations in an image. In contrast, R-CNN relies on region proposals to filter out background regions, hence the final predictions are much more sparse.</p><h3>2015: SSD</h3><blockquote>SSD: Single Shot MultiBox Detector</blockquote><p>YOLO v1 demonstrated the potentials of one-stage detection, but the performance gap from two-stage detection is still noticeable. In YOLO v1, multiple objects could be assigned to the same grid cell. This was a big challenge when detecting small objects, and became a critical problem to solve in order to improve a one-stage detector’s performance to be on par with two-stage detectors. SSD is such a challenger and attacks this problem from three angles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*NMMvka77x8qE4IiIfUykgQ.png" /><figcaption>From “<a href="https://arxiv.org/abs/1512.02325"><em>SSD: Single Shot MultiBox Detector</em></a><em>”</em></figcaption></figure><p>First, the anchor box technique from Faster R-CNN can alleviate this problem. Objects in the same area usually come with different aspect ratios to be visible. Introducing anchor box not only increased the amount of object to detect for each cell, but also helped the network to better differentiate overlapping small objects with this aspect ratio assumption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*l_zDryfY3i1MsCW9YeTgRA.png" /><figcaption>From “<a href="https://arxiv.org/abs/1512.02325"><em>SSD: Single Shot MultiBox Detector</em></a><em>”</em></figcaption></figure><p>SSD went down on this road further by aggregating multi-scale features before detection. This is a very common approach to pick up fine-grained local features while preserving coarse global features in CNN. For example, FCN, the pioneer of CNN semantic segmentation, also merged features from multiple levels to refine the segmentation boundary. Besides, multi-scale feature aggregation can be easily performed on all common classification networks, so it’s very convenient to swap out the backbone with another network.</p><p>Finally, SSD leveraged a large amount of data augmentation, especially targeted to small objects. For example, images are randomly expanded to a much larger size before random cropping, which brings a zoom-out effect to the training data to simulate small objects. Also, large bounding boxes are usually easy to learn. To avoid these easy examples dominating the loss, SSD adopted a hard negative mining technique to pick examples with the highest loss for each anchor box.</p><h3>2016: FPN</h3><blockquote>Feature Pyramid Networks for Object Detection</blockquote><p>With the launch of Faster-RCNN, YOLO, and SSD in 2015, it seems like the general structure an object detector is determined. Researchers start to look at improving each individual parts of these networks. Feature Pyramid Networks is an attempt to improve the detection head by using features from different layers to form a feature pyramid. This feature pyramid idea isn’t very novel in computer vision research. Back then when features are still manually designed, feature pyramid is already a very effective way to recognize patterns at different scales. Using the Feature Pyramid in deep learning is also not a new idea: SSPNet, FCN, and SSD all demonstrated the benefit of aggregating multiple-layer features before classification. However, how to share the feature pyramid between RPN and the region-based detector is still yet to be determined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*MxBjStvJAkhgSjmE1i3xcw.png" /><figcaption>From “<a href="https://arxiv.org/abs/1612.03144"><em>Feature Pyramid Networks for Object Detection</em></a><em>”</em></figcaption></figure><p>First, to rebuild RPN with an FPN structure like the diagram above, we need to have a region proposal running on multiple different scales of feature output. Also, we only need 3 anchors with different aspect ratios per location now because objects with different sizes will be handle by different levels of the feature pyramid. Next, to use an FPN structure in the Fast R-CNN detector, we also need to adapt it to detect on multiple scales of feature maps as well. Since region proposals might have different scales too, we should use them in the corresponding level of FPN as well. In short, if Faster R-CNN is a pair of RPN and region-based detector running on one scale, FPN converts it into multiple parallel branches running on different scales and collects the final results from all branches in the end.</p><h3>2016: YOLO v2</h3><blockquote>YOLO9000: Better, Faster, Stronger</blockquote><p>While Kaiming He, Ross Girshick, and their team keep improving their two-stage R-CNN detectors, Joseph Redmon, on the other hand, was also busy improving his one-stage YOLO detector. The initial version of YOLO suffers from many shortcomings: predictions based on a coarse grid brought lower localization accuracy, two scale-agnostic regressors per grid cell also made it difficult to recognize small packed objects. Fortunately, we saw too many great innovations in 2015 in many computer vision areas. YOLO v2 just needs to find a way to integrate them all to become better, faster, and stronger. Here are some highlights of the modifications:</p><ul><li>YOLO v2 added Batch Normalization layers from a paper called “<em>Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*Hdc4b2oinSf0Ambn4sRAHg.png" /><figcaption>Anchor boxes From “<a href="https://arxiv.org/abs/1612.08242"><em>YOLO9000: Better, Faster, Stronger</em></a><em>”</em></figcaption></figure><ul><li>Just like SSD, YOLO v2 also introduced Faster R-CNN’s idea of anchor boxes for bounding box regression. But YOLO v2 did some customization for its anchor boxes. Instead of predicting offsets to anchor boxes, YOLOv2 constraints the object center regression <em>tx</em> and <em>ty</em> within the responsible grid cell to stabilize early training. Also, anchors sizes are determined by a K-means clustering of the target dataset to better align with object shapes.</li><li>A new backbone network called Darknet is used for feature extraction. This is inspired by “<em>Network in Network</em>” and GooLeNet’s bottleneck structure.</li><li>To improve the detection of small objects, YOLO v2 added a passthrough layer to merge features from an early layer. This part can be seen as a simplified version of SSD.</li><li>Last but not least, Joseph realized that input resolution is a silver bullet for small object detection. It not only doubled the input for the backbone to 448x448 from 224x224 but also invented a multi-scale training schema, which involves different input resolutions at different periods of training.</li></ul><p>Note that YOLO v2 also experimented with a version that’s trained on 9000 classes hierarchical datasets, which also represents an early trial of multi-label classification in an object detector.</p><h3>2017: RetinaNet</h3><blockquote>Focal Loss for Dense Object Detection</blockquote><p>To understand why one-stage detectors are usually not as good as two-stage detectors, RetinaNet investigated the foreground-background class imbalance issue from a one-stage detector’s dense predictions. Take YOLO as an example, it tried to predict classes and bounding boxes for all possible locations in the meantime, so most of the outputs are matched to negative class during training. SSD addressed this issue by online hard example mining. YOLO used an objectiveness score to implicitly train a foreground classifier in the early stage of training. RetinaNet thinks they both didn’t get the key to the problem, so it invented a new loss function called Focal Loss to help the network learn what’s important.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*QPJ7dfAZQkQWRtnCv4LNag.png" /><figcaption><em>From “</em><a href="https://arxiv.org/abs/1708.02002"><em>Focal Loss for Dense Object Detection</em></a><em>”</em></figcaption></figure><p>Focal Loss added a power γ (they call it focusing parameter) to Cross-Entropy loss. Naturally, as the confidence score becomes higher, the loss value will become much lower than a normal Cross-Entropy. The α parameter is used to balance such a focusing effect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yBvUhfzZCkXFFTYr1AgOXQ.png" /><figcaption>From “<a href="https://arxiv.org/abs/1708.02002"><em>Focal Loss for Dense Object Detection</em></a><em>”</em></figcaption></figure><p>This idea is so simple that even a primary school student can understand. So to further justify their work, they adapted the FPN model they previously proposed and created a new one-stage detector called RetinaNet. It is composed of a ResNet backbone, an FPN detection neck to channel features at different scales, and two subnets for classification and box regression as detection head. Similar to SSD and YOLO v3, RetinaNet uses anchor boxes to cover targets of various scales and aspect ratios.</p><p>A bit of a digression, RetinaNet used the COCO accuracy from a ResNeXT-101 and 800 input resolution variant to contrast YOLO v2, which only has a light-weighted Darknet-19 backbone and 448 input resolution. This insincerity shows the team’s emphasis on getting better benchmark results, rather than solving a practical issue like a speed-accuracy trade-off. And it might be part of the reason that RetinaNet didn’t take off after its release.</p><h3>2018: YOLO v3</h3><blockquote>YOLOv3: An Incremental Improvement</blockquote><p>YOLO v3 is the last version of the official YOLO series. Following YOLO v2’s tradition, YOLO v3 borrowed more ideas from previous research and got an incredible powerful one-stage detector like a monster. YOLO v3 balanced the speed, accuracy, and implementation complexity pretty well. And it got really popular in the industry because of its fast speed and simple components. If you are interested, I wrote a very detailed explanation of how YOLO v3 works in my previous article “<a href="https://towardsdatascience.com/dive-really-deep-into-yolo-v3-a-beginners-guide-9e3d2666280e">Dive Really Deep into YOLO v3: A Beginner’s Guide</a>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u28J2n3UkWAoRmMs.jpeg" /><figcaption>From “<a href="https://towardsdatascience.com/dive-really-deep-into-yolo-v3-a-beginners-guide-9e3d2666280e">Dive Really Deep into YOLO v3: A Beginner’s Guide</a>”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F8Gooa9G7pcgHpxn.jpeg" /><figcaption>From “<a href="https://towardsdatascience.com/dive-really-deep-into-yolo-v3-a-beginners-guide-9e3d2666280e">Dive Really Deep into YOLO v3: A Beginner’s Guide</a>”</figcaption></figure><p>Simply put, YOLO v3’s success comes from its more powerful backbone feature extractor and a RetinaNet-like detection head with an FPN neck. The new backbone network Darknet-53 leveraged ResNet’s skip connections to achieve an accuracy that’s on par with ResNet-50 but much faster. Also, YOLO v3 ditched v2’s pass through layers and fully embraced FPN’s multi-scale predictions design. Since then, YOLO v3 finally reversed people’s impression of its poor performance when dealing with small objects.</p><p>Besides, there are a few fun facts about YOLO v3. It dissed the COCO mAP 0.5:0.95 metric, and also demonstrated the uselessness of Focal Loss when using a conditioned dense prediction. The author Joseph even decided to quit the whole computer vision research a year later, because of his concern of military usage.</p><h3>2019: Objects As Points</h3><p>Although the image classification area becomes less active recently, object detection research is still far from mature. In 2018, a paper called “CornerNet: Detecting Objects as Paired Keypoints” provided a new perspective for detector training. Since preparing anchor box targets is a quite cumbersome job, is it really necessary to use them as a prior? This new trend of ditching anchor boxes is called “anchor-free” object detection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*XAVE-G1pLDJXeHpi.png" /><figcaption>From “<a href="https://arxiv.org/abs/1603.06937">Stacked Hourglass Networks for Human Pose Estimation</a>”</figcaption></figure><p>Inspired by the use of heat-map in the Hourglass network for human pose estimation, CornerNet uses a heat-map generated by box corners to supervise the bounding box regression. To learn more about how heat-map is used in Hourglass Network, you can read my previous article “<a href="https://towardsdatascience.com/human-pose-estimation-with-stacked-hourglass-network-and-tensorflow-c4e9f84fd3ce">Human Pose Estimation with Stacked Hourglass Network and TensorFlow</a>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JNN8c2sWzRSeEGrm1Eka5g.png" /><figcaption>From “<a href="https://arxiv.org/abs/1904.07850"><em>Objects as Points</em></a><em>”</em></figcaption></figure><p>Objects As Points, aka CenterNet, took a step further. It uses heat-map peaks to represent object centers, and the network will regress the box width and height directly from these box centers. Essentially, CenterNet is using every pixel as grid cells. With a Gaussian distributed heat-map, the training is also easier to converge compared with previous attempts which tried to regress bounding box size directly.</p><p>The elimination of anchor boxes also has another useful side effect. Previously, we rely on IOU ( such as &gt; 0.7) between the anchor box and the ground truth box to assign training targets. By doing so, a few neighboring anchors may get all assigned a positive target for the same object. And the network will learn to predict multiple positive boxes for the same object too. The common way to fix this issue is to use a technique called Non-maximum Suppression (NMS). It’s a greedy algorithm to filter out boxes that are too close together. Now that anchors are gone and we only have one peak per object in the heat-map, there’s no need to use NMS any more. Since NMS is sometimes hard to implement and slow to run, getting rid of NMS is a big benefit for the applications that run in various environments with limited resources.</p><h3>2019: EfficientDet</h3><blockquote>EfficientDet: Scalable and Efficient Object Detection</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_gyTvX3vozNXY4lpafJbJw.png" /><figcaption>From “<a href="https://arxiv.org/abs/1911.09070"><em>EfficientDet: Scalable and Efficient Object Detection</em></a><em>”</em></figcaption></figure><p>In the recent CVPR’20, EfficientDet showed us some more exciting development in the object detection area. FPN structure has been proved to be a powerful technique to improve the detection network’s performance for objects at different scales. Famous detection networks such as RetinaNet and YOLO v3 all adopted an FPN neck before box regression and classification. Later, NAS-FPN and PANet (please refer to Read More section) both demonstrated that a plain multi-layer FPN structure may benefit from more design optimization. EfficientDet continued exploring in this direction, eventually created a new neck called BiFPN. Basically, BiFPN features additional cross-layer connections to encourage feature aggregation back and forth. To justify the efficiency part of the network, this BiFPN also removed some less useful connections from the original PANet design. Another innovative improvement over the FPN structure is the weight feature fusion. BiFPN added additional learnable weights to feature aggregation so that the network can learn the importance of different branches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*6S0yo_rrL36yvpHjkoqyyg.png" /><figcaption>From “<a href="https://arxiv.org/abs/1911.09070"><em>EfficientDet: Scalable and Efficient Object Detection</em></a><em>”</em></figcaption></figure><p>Moreover, just like what we saw in the image classification network EfficientNet, EfficientDet also introduced a principled way to scale an object detection network. The φ parameter in the above formula controls both width (channels) and depth (layers) of both BiFPN neck and detection head.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kI_fMzk7oOnevknZ2bLj7g.png" /><figcaption>From “<a href="https://arxiv.org/abs/1911.09070"><em>EfficientDet: Scalable and Efficient Object Detection</em></a><em>”</em></figcaption></figure><p>This new parameter results in 8 different variants of EfficientDet from D0 to D7. A light-weighed D0 variant can achieve similar accuracy with YOLO v3 while having much fewer FLOPs. A heavy-loaded D7 variant with monstrous 1536x1536 input can even reach 53.7 AP on COCO that dwarfed all other contenders.</p><h3>Read More</h3><p>From R-CNN, YOLO to recent CenterNet and EfficientDet, we have witnessed most major innovations in the object detection research in the deep learning era. Aside from the above papers, I’ve also provided a list of additional papers for you to keep reading to get a deeper understanding. They either provided a different perspective for object detection or extended this area with more powerful features.</p><h4>2009: DPM</h4><blockquote>Object Detection with Discriminatively Trained Part Based Models</blockquote><p>By matching many HOG features for each deformable parts, DPM was one of the most efficient object detection models before the deep learning era. Take pedestrian detection as an example, it uses a star structure to recognize the general person pattern first, and then recognize parts with different sub-filters and calculate an overall score. Even today, the idea to recognize objects with deformable parts is still popular after we switch from HOG features to CNN features.</p><h4>2012: Selective Search</h4><blockquote>Selective Search for Object Recognition</blockquote><p>Like DPM, Selective Search is also not a product of the deep learning era. However, this method combined so many classical computer vision approaches together, and also used in the early R-CNN detector. The core idea of selective search is inspired by semantic segmentation where pixels are group by similarity. Selective Search uses different criteria of similarity such as color space and SIFT-based texture to iteratively merge similar areas together. And these merged area areas served as foreground predictions and followed by an SVM classifier for object recognition.</p><h4>2016: R-FCN</h4><blockquote>R-FCN: Object Detection via Region-based Fully Convolutional Networks</blockquote><p>Faster R-CNN finally combined RPN and ROI feature extraction and improved the speed a lot. However, for each region proposal, we still need fully connected layers to compute class and bounding box separately. If we have 300 ROIs, we need to repeat this by 300 hundred times, and this is also the origin of the major speed difference between one-stage and two-stage detector. R-FCN borrowed the idea from FCN for semantic segmentation, but instead of computing the class mask, R-FCN computes a positive sensitive score maps. This map will predict the probability of the appearance of the object at each location, and all locations will vote (average) to decide the final class and bounding box. Besides, R-FCN also used atrous convolution in its ResNet backbone, which is originally proposed in the DeepLab semantic segmentation network. To understand what is atrous convolution, please see my previous article “<a href="https://towardsdatascience.com/witnessing-the-progression-in-semantic-segmentation-deeplab-series-from-v1-to-v3-4f1dd0899e6e">Witnessing the Progression in Semantic Segmentation: DeepLab Series from V1 to V3+</a>”.</p><h4>2017: Soft-NMS</h4><blockquote>Improving Object Detection With One Line of Code</blockquote><p>Non-maximum suppression (NMS) is widely used in anchor-based object detection networks to reduce duplicate positive proposals that are close-by. More specifically, NMS iteratively eliminates candidate boxes if they have a high IOU with a more confident candidate box. This could lead to some unexpected behavior when two objects with the same class are indeed very close to each other. Soft-NMS made a small change to only scaling down the confidence score of the overlapped candidate boxes with a parameter. This scaling parameter gives us more control when tuning the localization performance, and also leads to a better precision when a high recall is also needed.</p><h4>2017: Cascade R-CNN</h4><blockquote>Cascade R-CNN: Delving into High Quality Object Detection</blockquote><p>While FPN exploring how to design a better R-CNN neck to use backbone features Cascade R-CNN investigated a redesign of R-CNN classification and regression head. The underlying assumption is simple yet insightful: the higher IOU criteria we use when preparing positive targets, the less false positive predictions the network will learn to make. However, we can’t simply increase such IOU threshold from commonly used 0.5 to more aggressive 0.7, because it could also lead to more overwhelming negative examples during training. Cascade R-CNN’s solution is to chain multiple detection head together, each will rely on the bounding box proposals from the previous detection head. Only the first detection head will use the original RPN proposals. This effectively simulated an increasing IOU threshold for latter heads.</p><h4>2017: Mask R-CNN</h4><blockquote>Mask R-CNN</blockquote><p>Mask R-CNN is not a typical object detection network. It was designed to solve a challenging instance segmentation task, i.e, creating a mask for each object in the scene. However, Mask R-CNN showed a great extension to the Faster R-CNN framework, and also in turn inspired object detection research. The main idea is to add a binary mask prediction branch after ROI pooling along with the existing bounding box and classification branches. Besides, to address the quantization error from the original ROI Pooling layer, Mask R-CNN also proposed a new ROI Align layer that uses bilinear image resampling under the hood. Unsurprisingly, both multi-task training (segmentation + detection) and the new ROI Align layer contribute to some improvement over the bounding box benchmark.</p><h4>2018: PANet</h4><blockquote>Path Aggregation Network for Instance Segmentation</blockquote><p>Instance segmentation has a close relationship with object detection, so often a new instance segmentation network could also benefit object detection research indirectly. PANet aims at boosting information flow in the FPN neck of Mask R-CNN by adding an additional bottom-up path after the original top-down path. To visualize this change, we have a ↑↓ structure in the original FPN neck, and PANet makes it more like a ↑↓↑ structure before pooling features from multiple layers. Also, instead of having separate pooling for each feature layer, PANet added an “adaptive feature pooling” layer after Mask R-CNN’s ROIAlign to merge (element-wise max of sum) multi-scale features.</p><h4>2019: NAS-FPN</h4><blockquote>NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection</blockquote><p>PANet’s success in adapting FPN structure drew attention from a group of NAS researchers. They used a similar reinforcement learning method from the image classification network NASNet and focused on searching the best combination of merging cells. Here, a merging cell is the basic build block of an FPN that merges any two input features layers into one output feature layer. The final results proved the idea that FPN could use further optimization, but the complex computer-searched structure made it too difficult for humans to understand.</p><h3>Conclusion</h3><p>Object detection is still an active research area. Although the general landscape of this field is well shaped by a two-stage detector like R-CNN and one-stage detector such as YOLO, our best detector is still far from saturating the benchmark metrics, and also misses many targets in complicated background. At the same time, Anchor-free detector like CenterNet showed us a promising future where object detection networks can become as simple as image classification networks. Other directions of object detection, such as few-shot recognition and NAS, are still at an early age, and we will see how it goes in the next few years. Nevertheless, as object detection technology becomes more mature, we need to be very cautious about its adoption by the military and police. A dystopia where Terminators hunt and shoot humans with a YOLO detector is the last thing we want to see in our life.</p><p><em>Originally published at </em><a href="https://yanjia.li/12-papers-you-should-read-to-understand-object-detection-in-the-deep-learning-era/"><em>http://yanjia.li</em></a><em> on Aug 9, 2020</em></p><h3>References</h3><ul><li>Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, Yann LeCun, <a href="https://arxiv.org/abs/1312.6229"><em>OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks</em></a></li><li>Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik, <a href="http://islab.ulsan.ac.kr/files/announcement/513/rcnn_pami.pdf"><em>Region-based Convolutional Networks for Accurate Object Detection and Segmentation</em></a></li><li>Ross Girshick, <a href="https://arxiv.org/abs/1504.08083">Fast R-CNN</a></li><li>Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, <a href="https://arxiv.org/abs/1506.02640"><em>You Only Look Once: Unified, Real-Time Object Detection</em></a></li><li>Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, <a href="https://arxiv.org/abs/1512.02325"><em>SSD: Single Shot MultiBox Detector</em></a></li><li>Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, <a href="https://arxiv.org/abs/1506.01497"><em>Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks</em></a></li><li>Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie, <a href="https://arxiv.org/abs/1612.03144"><em>Feature Pyramid Networks for Object Detection</em></a></li><li>Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár, <a href="https://arxiv.org/abs/1708.02002"><em>Focal Loss for Dense Object Detection</em></a></li><li>Joseph Redmon, Ali Farhadi,<em> </em><a href="https://arxiv.org/abs/1612.08242"><em>YOLO9000: Better, Faster, Stronger</em></a></li><li>Joseph Redmon, Ali Farhadi, <a href="https://arxiv.org/abs/1804.02767"><em>YOLOv3: An Incremental Improvement</em></a></li><li>Yanjia Li,<em> </em><a href="https://towardsdatascience.com/dive-really-deep-into-yolo-v3-a-beginners-guide-9e3d2666280e"><em>Dive Really Deep into YOLO v3: A Beginner’s Guide</em></a></li><li>Xingyi Zhou, Dequan Wang, Philipp Krähenbühl, <a href="https://arxiv.org/abs/1904.07850"><em>Objects as Points</em></a></li><li>Yanjia Li, <a href="https://towardsdatascience.com/human-pose-estimation-with-stacked-hourglass-network-and-tensorflow-c4e9f84fd3ce"><em>Human Pose Estimation with Stacked Hourglass Network and TensorFlow</em></a></li><li>Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick, <a href="https://arxiv.org/abs/1703.06870"><em>Mask R-CNN</em></a></li><li>Zhaowei Cai, Nuno Vasconcelos, <a href="https://arxiv.org/abs/1712.00726"><em>Cascade R-CNN: Delving into High Quality Object Detection</em></a></li><li>Mingxing Tan, Ruoming Pang, Quoc V. Le, <a href="https://arxiv.org/abs/1911.09070"><em>EfficientDet: Scalable and Efficient Object Detection</em></a></li><li>Jifeng Dai, Yi Li, Kaiming He, Jian Sun, <a href="https://arxiv.org/abs/1605.06409"><em>R-FCN: Object Detection via Region-based Fully Convolutional Networks</em></a></li><li>Yanjia Li, <a href="https://towardsdatascience.com/witnessing-the-progression-in-semantic-segmentation-deeplab-series-from-v1-to-v3-4f1dd0899e6e"><em>Witnessing the Progression in Semantic Segmentation: DeepLab Series from V1 to V3+</em></a></li><li>Golnaz Ghiasi, Tsung-Yi Lin, Ruoming Pang, Quoc V. Le, <a href="https://arxiv.org/abs/1904.07392"><em>NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection</em></a></li><li>Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, Jiaya Jia, <a href="https://arxiv.org/abs/1803.01534"><em>Path Aggregation Network for Instance Segmentation</em></a></li><li>Yanjia Li, <a href="https://towardsdatascience.com/10-papers-you-should-read-to-understand-image-classification-in-the-deep-learning-era-4b9d792f45a7"><em>10 Papers You Should Read to Understand Image Classification in the Deep Learning Era</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3390d4a28891" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/12-papers-you-should-read-to-understand-object-detection-in-the-deep-learning-era-3390d4a28891">12 Papers You Should Read to Understand Object Detection in the Deep Learning Era</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Papers You Should Read to Understand Image Classification in the Deep Learning Era]]></title>
            <link>https://medium.com/data-science/10-papers-you-should-read-to-understand-image-classification-in-the-deep-learning-era-4b9d792f45a7?source=rss-60ef96b05d56------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b9d792f45a7</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ethan Yanjia Li]]></dc:creator>
            <pubDate>Sat, 01 Aug 2020 16:59:21 GMT</pubDate>
            <atom:updated>2023-06-15T03:43:07.702Z</atom:updated>
            <content:encoded><![CDATA[<h4>A quick walkthrough of the best papers for image classification in a decade to jump start your learning of computer vision</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*CV81vQUQTq-ko_ER9gvqjg.png" /><figcaption>Picture by myself</figcaption></figure><blockquote>We are a team of experts to help your business adopt AI solutions or build new AI products. Contact us at info@imaginaire.ai or visit our website <a href="https://www.imaginaire.ai/">https://www.imaginaire.ai</a></blockquote><h3>Foreword</h3><p>Computer vision is a subject to convert images and videos into machine-understandable signals. With these signals, programmers can further control the behavior of the machine based on this high-level understanding. Among many computer vision tasks, image classification is one of the most fundamental ones. It not only can be used in lots of real products like Google Photo’s tagging and AI content moderation but also opens a door for lots of more advanced vision tasks, such as object detection and video understanding. Due to the rapid changes in this field since the breakthrough of Deep Learning, beginners often find it too overwhelming to learn. Unlike typical software engineering subjects, there are not many great books about image classification using DCNN, and the best way to understand this field is though reading academic papers. But what papers to read? Where do I start? In this article, I’m going to introduce 10 best papers for beginners to read. With these papers, we can see how this field evolve, and how researchers brought up new ideas based on previous research outcome. Nevertheless, it is still helpful for you to sort out the big picture even if you have already worked in this area for a while. So, let’s get started.</p><h3>1998: LeNet</h3><blockquote>Gradient-based Learning Applied to Document Recognition</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IwZfpJm8FDLQKugV3sTFEQ.png" /><figcaption>From “<a href="http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf"><em>Gradient-based Learning Applied to Document Recognition</em></a><em>”</em></figcaption></figure><p>Introduced in 1998, LeNet sets a foundation for future image classification research using the Convolution Neural Network. Many classical CNN techniques, such as pooling layers, fully connected layers, padding, and activation layers are used to extract features and make a classification. With a Mean Square Error loss function and 20 epochs of training, this network can achieve 99.05% accuracy on the MNIST test set. Even after 20 years, many state-of-the-art classification networks still follows this pattern in general.</p><h3>2012: AlexNet</h3><blockquote>ImageNet Classification with Deep Convolutional Neural Networks</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/1*B7WN6HRLfA5LWARy4cndOw.png" /><figcaption>From “<a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf"><em>ImageNet Classification with Deep Convolutional Neural Networks</em></a><em>”</em></figcaption></figure><p>Although LeNet achieved a great result and showed the potential of CNN, the development in this area stagnated for a decade due to limited computing power and the amount of the data. It looked like CNN can only solve some easy tasks such as digit recognition, but for more complex features like faces and objects, a HarrCascade or SIFT feature extractor with an SVM classifier was a more preferred approach.</p><p>However, in 2012 ImageNet Large Scale Visual Recognition Challenge, Alex Krizhevsky proposed a CNN-based solution for this challenge and drastically increased ImageNet test set top-5 accuracy from 73.8% to 84.7%. Their approach inherits the multi-layer CNN idea from LeNet, but increased the size of CNN a lot. As you can see from the diagram above, the input is now 224x224 compared with LeNet’s 32x32, also many Convolution kernels have 192 channels compared with LeNet’s 6. Although the design isn’t changed much, with hundreds of more times of parameters, the network’s ability to capture and represent complex features improved hundreds of times too. To train such as a big model, Alex used two GTX 580 GPU with 3GB RAM for each, which pioneered a trend of GPU training. Also, the use of ReLU non-linearity also helped to reduce computation cost.</p><p>In addition to bringing many more parameters for the network, it also explored the overfitting issue brought by a larger network by using a Dropout layer. Its Local Response Normalization method didn’t get too much popularity afterward but inspired other important normalization techniques such as BatchNorm to combat with gradient saturation issue. To sum up, AlexNet defined the de facto classification network framework for the next 10 years: a combination of Convolution, ReLu non-linear activation, MaxPooling, and Dense layer.</p><h3>2014: VGG</h3><blockquote>Very Deep Convolutional Networks for Large-Scale Image Recognition</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZlUvPwqNUMIjCxGH" /><figcaption>From Quora “<a href="https://www.quora.com/What-is-the-VGG-neural-network">https://www.quora.com/What-is-the-VGG-neural-network</a>”</figcaption></figure><p>With such a great success of using CNN for visual recognition, the entire research community blew up and all started to look into why this neural network works so well. For example, in “Visualizing and Understanding Convolutional Networks” from 2013, Matthew Zeiler discussed how CNN pick up features and visualized the intermediate representations. And suddenly everyone started to realize that CNN is the future of computer vision since 2014. Among all those immediate followers, the VGG network from Visual Geometry Group is the most eye-catching one. It got a remarkable result of 93.2% top-5 accuracy, and 76.3% top-1 accuracy on the ImageNet test set.</p><p>Following AlexNet’s design, the VGG network has two major updates: 1) VGG not only used a wider network like AlexNet but also deeper. VGG-19 has 19 convolution layers, compared with 5 from AlexNet. 2) VGG also demonstrated that a few small 3x3 convolution filters can replace a single 7x7 or even 11x11 filters from AlexNet, achieve better performance while reducing the computation cost. Because of this elegant design, VGG also became the back-bone network of many pioneering networks in other computer vision tasks, such as FCN for semantic segmentation, and Faster R-CNN for object detection.</p><p>With a deeper network, gradient vanishing from multi-layers back-propagation becomes a bigger problem. To deal with it, VGG also discussed the importance of pre-training and weight initialization. This problem limits researchers to keep adding more layers, otherwise, the network will be really hard to converge. But we will see a better solution for this after two years.</p><h3>2014: GoogLeNet</h3><blockquote>Going Deeper with Convolutions</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wl6ZtcuSuX4EDN1UgR1zJA.png" /><figcaption>From “<a href="https://arxiv.org/abs/1409.4842"><em>Going Deeper with Convolutions</em></a><em>”</em></figcaption></figure><p>VGG has a good looking and easy-to-understand structure, but its performance isn’t the best among all the finalists in ImageNet 2014 competitions. GoogLeNet, aka InceptionV1, won the final prize. Just like VGG, one of the main contributions of GoogLeNet is to push the limit of the network depth with a 22 layers structure. This demonstrated again that going deeper and wider is indeed the right direction to improve accuracy.</p><p>Unlike VGG, GoogLeNet tried to address the computation and gradient diminishing issues head-on, instead of proposing a workaround with better pre-trained schema and weights initialization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*9XeTFJRnV8aqblcayXNX9g.png" /><figcaption>Bottleneck Inception Module From “<a href="https://arxiv.org/abs/1409.4842"><em>Going Deeper with Convolutions</em></a><em>”</em></figcaption></figure><p>First, it explored the idea of asymmetric network design by using a module called Inception (see diagram above). Ideally, they would like to pursuit sparse convolution or dense layers to improve feature efficiency, but modern hardware design wasn’t tailored to this case. So they believed that a sparsity at the network topology level could also help the fusion of features while leveraging existing hardware capabilities.</p><p>Second, it attacks the high computation cost problem by borrowing an idea from a paper called “Network in Network”. Basically, a 1x1 convolution filter is introduced to reduce dimensions of features before going through heavy computing operation like a 5x5 convolution kernel. This structure is called “Bottleneck” later and widely used in many following networks. Similar to “Network in Network”, it also used an average pooling layer to replace the final fully connected layer to further reduce cost.</p><p>Third, to help gradients to flow to deeper layers, GoogLeNet also used supervision on some intermediate layer outputs or auxiliary output. This design isn’t quite popular later in the image classification network because of the complexity, but getting more popular in other areas of computer vision such as Hourglass network in pose estimation.</p><p>As a follow-up, this Google team wrote more papers for this Inception series. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift” stands for InceptionV2. “Rethinking the Inception Architecture for Computer Vision” in 2015 stands for InceptionV3. And “Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning” in 2015 stands for InceptionV4. Each paper added more improvement over the original Inception network and achieved a better result.</p><h3>2015: Batch Normalization</h3><blockquote>Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</blockquote><p>The inception network helped researchers to reach a superhuman accuracy on the ImageNet dataset. However, as a statistic learning method, CNN is very much constrained to the statistic nature of a specific training dataset. Therefore, to achieve better accuracy, we usually need to pre-calculate the mean and standard deviation of the entire dataset and use them to normalize our input first to ensure most of the layer inputs in the network are close, which translates to better activation responsiveness. This approximate approach is very cumbersome, and sometimes doesn’t work at all for a new network structure or a new dataset, so the deep learning model is still viewed as difficult to train. To address this problem, Sergey Ioffe and Chritian Szegedy, the guy who created GoogLeNet, decided to invent something smarter called Batch Normalization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*EipYYGuqW1U3JoE3lV6Qmg.png" /><figcaption>From “<a href="https://arxiv.org/abs/1502.03167"><em>Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</em></a><em>”</em></figcaption></figure><p>The idea of batch normalization is not hard: We can use the statistics of a series of mini-batch to approximate the statistics of the whole dataset, as long as we train for long enough time. Also, instead of manually calculating the statistics, we can introduce two more learnable parameters “scale” and “shift” to have the network learn how to normalize each layer by itself.</p><p>The above diagram showed the process of calculating batch normalization values. As we can see, we take the mean of the whole mini-batch and calculate the variance as well. Next, we can normalize the input with this mini-batch mean and variance. Finally, with a scale and a shift parameter, the network will learn to adapt the batch normalized result to best fit the following layers, usually ReLU. One caveat is that we don’t have mini-batch information during inference, so a workaround is to calculate a moving average mean and variance during training, and then use these moving averages in the inference path. This little innovation is so impactful, and all later networks start to use it right away.</p><h3>2015: ResNet</h3><blockquote>Deep Residual Learning for Image Recognition</blockquote><p>2015 may be the best year for computer vision in a decade, we’ve seen so many great ideas popping out not only in image classification but all sorts of computer vision tasks such as object detection, semantic segmentation, etc. The biggest advancement of the year 2015 belongs to a new network called ResNet, or residual networks, proposed by a group of Chinese researchers from Microsoft Research Asia.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*RQo6-z3eXwP_h0Zx4RC4PQ.png" /><figcaption>From “<a href="https://arxiv.org/abs/1512.03385"><em>Deep Residual Learning for Image Recognition</em></a><em>”</em></figcaption></figure><p>As we discussed earlier for VGG network, the biggest hurdle of getting even deeper is the gradient vanishing issue, i.e, derivatives become smaller and smaller when back-propagate through deeper layers, and eventually reaches a point that modern computer architecture can’t really represent meaningfully. GoogLeNet tried to attack this by using auxiliary supervision and asymmetric inception module, but it only alleviates the problem to a small extent. If we want to use 50 or even 100 layers, will there be a better way for the gradient to flow through the network? The answer from ResNet is to use a residual module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*aOZegcVAMfz_xvDmltEPuw.png" /><figcaption>Residual Module From “<a href="https://arxiv.org/abs/1512.03385"><em>Deep Residual Learning for Image Recognition</em></a><em>”</em></figcaption></figure><p>ResNet added an identity shortcut to the output, so that each residual module can’t at least predict whatever the input is, without getting lost in the wild. Even more important thing is, instead of hoping each layer fit directly to the desired feature mapping, the residual module tries to learn the difference between output and input, which makes the task much easier because the information gain needed is less. Imagine that you are learning mathematics, for each new problem, you are given a solution of a similar problem, so all you need to do is to extend this solution and try to make it work. This is much easier than thinking of a brand new solution for every problem you run into. Or as Newton said, we can stand on the shoulders of giants, and the identity input is that giant for the residual module.</p><p>In addition to identity mapping, ResNet also borrowed the bottleneck and Batch Normalization from Inception networks. Eventually, it managed to build a network with 152 convolution layers and achieved 80.72% top-1 accuracy on ImageNet. The residual approach also becomes a default option for many other networks later, such as Xception, Darknet, etc. Also, thanks to its simple and beautiful design, it’s still widely used in many production visual recognition systems nowadays.</p><p>There’re many more invariants came out by following the hype of the residual network. In “Identity Mappings in Deep Residual Networks”, the original author of ResNet tried to put activation before the residual module and achieved a better result, and this design is called ResNetV2 afterward. Also, in a 2016 paper “Aggregated Residual Transformations for Deep Neural Networks”, researchers proposed ResNeXt which added parallel branches for residual modules to aggregate outputs of different transforms.</p><h3>2016: Xception</h3><blockquote>Xception: Deep Learning with Depthwise Separable Convolutions</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VeyeZtvi2p2oeiUpqkINFQ.png" /><figcaption>From “<a href="https://arxiv.org/abs/1610.02357"><em>Xception: Deep Learning with Depthwise Separable Convolutions</em></a>”</figcaption></figure><p>With the release of ResNet, it looked like most of the low hanging fruits in the image classifier were grabbed already. Researchers started to think about what is the internal mechanism of the magic of CNN. Since cross-channel convolution usually introduces a ton of parameters, the Xception network chose to investigate this operation to understand a full picture of its effect.</p><p>Like its name, Xception originates from the Inception network. In the Inception module, multiple branches of different transformations are aggregated together to achieve a topology sparsity. But why this sparsity worked? The author of Xception, also the author of the Keras framework, extended this idea to an extreme case where one 3x3 convolution file correspond to one output channel before a final concatenation. In this case, these parallel convolution kernels actually form a new operation called depth-wise convolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*YMxxyofVlT1yF7s0QC18AQ.png" /><figcaption>From “<a href="https://medium.com/@zurister/depth-wise-convolution-and-depth-wise-separable-convolution-37346565d4ec">Depth-wise Convolution and Depth-wise Separable Convolution</a>”</figcaption></figure><p>As shown in the diagram above, unlike traditional convolution where all channels are included for one computation, depth-wise convolution only computes convolution for each channel separately and then concatenate the output together. This cuts the feature exchange among channels, but also reduces a lot of connections, hence result in a layer with fewer parameters. However, this operation will output the same number of channels as input (Or a smaller number of channels if you group two or more channels together). Therefore, once the channel outputs are merged, we need another regular 1x1 filter, or point-wise convolution, to increase or reduce the number of channels, just like regular convolution does.</p><p>This idea is not from Xception originally. It’s described in a paper called “Learning visual representations at scale” and also used occasionally in InceptionV2. Xception took a step further and replaced almost all convolutions with this new type. And the experiment result turned out to be pretty good. It surpasses ResNet and InceptionV3 and became a new SOTA method for image classification. This also proved that the mapping of cross-channel correlations and spatial correlations in CNN can be entirely decoupled. In addition, sharing the same virtue with ResNet, Xception has a simple and beautiful design as well, so its idea is used in lots of other following research such as MobileNet, DeepLabV3, etc.</p><h3>2017: MobileNet</h3><blockquote>MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Application</blockquote><p>Xception achieved 79% top-1 accuracy and 94.5% top-5 accuracy on ImageNet, but that’s only 0.8% and 0.4% improvement respectively compared with previous SOTA InceptionV3. The marginal gain of a new image classification network is becoming smaller, so researchers start to shift their focus into other areas. And MobileNet led a significant push of image classification in a resource constrained environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/1*CqDUZEVRsbapzK4Kk3lApw.png" /><figcaption>MobileNet module from “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Application”</figcaption></figure><p>Similar to Xception, MobileNet used a same depthwise separable convolution module as shown above and had an emphasis on the high efficiency and less parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*rhXmisulDC4qj2IJB8KFvA.png" /><figcaption>Parameters ratio from “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Application”</figcaption></figure><p>The numerator in the above formula is the total number of parameters required by a depthwise separable convolution. And the denominator is the total number of parameters of a similar regular convolution. Here D[K] is the size of convolution kernel, D[F] is the size of the feature map, M is the number of input channels, N is the number of output channels. Since we separated the calculation of channel and spatial feature, we can turn multiplication into an addition, which is a magnitude smaller. Even better, as we can see from this ratio, the larger the number of output channels is, the more calculation we saved from using this new convolution.</p><p>Another contribution from MobileNet is the width and resolution multiplier. MobileNet team wanted to find a canonical way to shrink model size for mobile devices, and the most intuitive way is to reduce the number of input and output channels, as well as the input image resolution. To control this behavior, a ratio alpha is multiplied with channels, and a ratio rho is multiplied with input resolution (which also affects feature map size). So the total number of parameters can be represented in the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*lMISgVmutXsfFLTa9Cslhw.png" /><figcaption>“MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Application”</figcaption></figure><p>Although this change seems naive in terms of innovation, it has great engineering value because it’s the first time researchers conclude a canonical approach to adjust network for different resource constraints. Also, it kind of summarized the ultimate solution of improving neural network: wider and high-res input leads to better accuracy, thinner and low-res input leads to poorer accuracy.</p><p>Later in 2018 and 2019, the MobiletNet team also released “MobileNetV2: Inverted Residuals and Linear Bottlenecks” and “Searching for MobileNetV3”. In MobileNetV2, an inverted residual bottleneck structure is used. And in MobileNetV3, it started to search the optimal architecture combination using Neural Architecture Search technology, which we will cover next.</p><h3>2017: NASNet</h3><blockquote>Learning Transferable Architectures for Scalable Image Recognition</blockquote><p>Just like image classification for a resource-constrained environment, neural architecture search is another field that emerged around 2017. With ResNet, Inception, and Xception, it seems like we reached an optimal network topology that humans can understand and design, but what if there’s a better and much more complex combination that far exceeds human imagination? A paper in 2016 called “Neural Architecture Search with Reinforcement Learning” proposed an idea to search the optimal combination within a pre-defined search space by using reinforcement learning. As we know, reinforcement learning is a method to find the best solution with a clear goal and reward for the search agent. However, limited by computing power, this paper only discussed the application in a small CIFAR dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*ORd9Ur9UMouQ6T7qCSfbbQ.png" /><figcaption>NASNet Search Space. “<a href="https://arxiv.org/abs/1707.07012"><em>Learning Transferable Architectures for Scalable Image Recognition</em></a>”</figcaption></figure><p>With the goal of finding an optimal structure for a large dataset like ImageNet, NASNet made a search space that is tailored for ImageNet. It hopes to design a special search space so that the searched result on CIFAR can also work on ImageNet well. First, NASNet assumes that common hand-crafted module in good networks like ResNet and Xception are still useful when searching. So instead of searching for random connection and operations, NASNet searches the combination of these modules that have been proved to be useful already on ImageNet. Second, the actual searching is still performed on the CIFAR dataset with 32x32 resolution, so NASNet only searches for modules that are not affected by the input size. In order to make the second point work, NASNet predefined two types of module templates: Reduction and Normal. Reduction cell could have reduced feature map compared with the input, and for Normal cell it would be the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*6to5up8E7XqZagMc-sRQGA.png" /><figcaption>From “<a href="https://arxiv.org/abs/1707.07012"><em>Learning Transferable Architectures for Scalable Image Recognition</em></a>”</figcaption></figure><p>Although NASNet has better metrics than manually design networks, it also suffers from a few drawbacks. The cost of searching for an optimal structure is very high, which is only affordable by big companies like Google and Facebook. Also, the final structure doesn’t make too much sense to humans, hence harder to maintain and improve in a production environment. Later in 2018, “MnasNet: Platform-Aware Neural Architecture Search for Mobile” further extends this NASNet idea by limit the search step with a pre-defined chained-blocks structure. Also, by defining a weight factor, mNASNet gave a more systematic way to search model given specific resource constraints, instead of just evaluating based on FLOPs.</p><h3>2019: EfficientNet</h3><blockquote>EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks</blockquote><p>In 2019, it looks like there are no exciting ideas for supervised image classification with CNN anymore. A drastic change in network structure usually only offers a little accuracy improvement. Even worse, when the same network applied to different datasets and tasks, previously claimed tricks don’t seem to work, which led to critiques that whether those improvements are just overfitting on the ImageNet dataset. On the other side, there’s one trick that never fails our expectation: using higher resolution input, adding more channels for convolution layers, and adding more layers. Although very brutal force, it seems like there’s a principled way to scale the network on demand. MobileNetV1 sort of suggested this in 2017, but the focus was shifted to a better network design later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*XXE2w4w8H07wC-cZL5ZjcQ.png" /><figcaption>From “<a href="https://arxiv.org/abs/1905.11946"><em>EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks</em></a><em>”</em></figcaption></figure><p>After NASNet and mNASNet, researchers realized that even with the help from a computer, a change in architecture doesn’t yield that much benefit. So they start to fall back to the scaling the network. EfficientNet is just built on top of this assumption. On one hand, it uses the optimal building block from mNASNet to make sure a good foundation to start with. On the other hand, it defined three parameters alpha, beta, and rho to control the depth, width, and resolution of the network correspondingly. By doing so, even without a large GPU pool to search for an optimal structure, engineers can still rely on these principled parameters to tune the network based on their different requirements. In the end, EfficientNet gave 8 different variants with different width, depth, and resolution ratios and got good performance for both small and big models. In other words, if you want high accuracy, go for a 600x600 and 66M parameters EfficientNet-B7. If you want low latency and smaller model, go for a 224x224 and 5.3M parameters EfficientNet-B0. Problem solved.</p><h3>Read More</h3><p>If you finish reading above 10 papers, you should have a pretty good grasp of the history of image classification with CNN. If you like to keep learning this area, I’ve also listed some other interesting papers to read. Although not included in the top-10 list, these papers are all famous in their own area and inspired many other researchers in the world.</p><h4>2014: SPPNet</h4><blockquote>Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition</blockquote><p>SPPNet borrows the idea of feature pyramid from traditional computer vision feature extraction. This pyramid forms a bag of words of features with different scales, so it is can adapt to different input sizes and get rid of the fixed-size fully connected layer. This idea also further inspired the ASPP module of DeepLab, and also FPN for object detection.</p><h4>2016: DenseNet</h4><blockquote>Densely Connected Convolutional Networks</blockquote><p>DenseNet from Cornell further extends the idea from ResNet. It not only provides skip connection between layers but also has skip connections from all previous layers.</p><h4>2017: SENet</h4><blockquote>Squeeze-and-Excitation Networks</blockquote><p>The Xception network demonstrated that cross-channel correlation doesn’t have much to do with spatial correlation. However, as the champion of the last ImageNet competition, SENet devised a Squeeze-and-Excitation block and told a different story. The SE block first squeezes all channels into fewer channels using global pooling, applies a fully connected transform, and then “excite” them back to the original number of channels using another fully connected layer. So essentially, the FC layer helped the network to learn attentions on the input feature map.</p><h4>2017: ShuffleNet</h4><blockquote>ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices</blockquote><p>Built on top of MobileNetV2’s inverted bottleneck module, ShuffleNet believes that point-wise convolution in depthwise separable convolution sacrifices accuracy in exchange for less computation. To compensate for this, ShuffleNet added an additional channel shuffling operation to make sure point-wise convolution will not always be applied to the same “point”. And in ShuffleNetV2, this channel shuffling mechanism is further extended to a ResNet identity mapping branch as well, so that part of the identity feature will also be used to shuffle.</p><h4>2018: Bag of Tricks</h4><blockquote>Bag of Tricks for Image Classification with Convolutional Neural Networks</blockquote><p>Bag of Tricks focuses on common tricks used in the image classification area. It serves as a good reference for engineers when they need to improve benchmark performance. Interestingly, these tricks such as mixup augmentation and cosine learning rate can sometimes achieve far better improvement than a new network architecture.</p><h3>Conclusion</h3><p>With the release of EfficientNet, it looks like the ImageNet classification benchmark comes to an end. With the existing deep learning approach, there will never be a day we can reach 99.999% accuracy on the ImageNet unless another paradigm shift happened. Hence, researchers are actively looking at some novel areas such as self-supervised or semi-supervised learning for large scale visual recognition. In the meantime, with existing methods, it became more a question for engineers and entrepreneurs to find the real-world application of this non-perfect technology. In the future, I will also write a survey to analyze those real-world computer vision applications powered by image classification, so please stay tuned! If you think there are also other important papers to read for image classification, please leave a comment below and let us know.</p><p><em>Originally published at </em><a href="https://yanjia.li/10-papers-you-should-read-to-understand-image-classification-in-the-deep-learning-era/"><em>http://yanjia.li</em></a><em> on July 31, 2020</em></p><h3>References</h3><ul><li>Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf"><em>Gradient-based Learning Applied to Document Recognition</em></a></li><li>Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf"><em>ImageNet Classification with Deep Convolutional Neural Networks</em></a></li><li>Karen Simonyan, Andrew Zisserman, <a href="https://arxiv.org/abs/1409.1556"><em>Very Deep Convolutional Networks for Large-Scale Image Recognition</em></a></li><li>Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich, <a href="https://arxiv.org/abs/1409.4842"><em>Going Deeper with Convolutions</em></a></li><li>Sergey Ioffe, Christian Szegedy, <a href="https://arxiv.org/abs/1502.03167"><em>Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</em></a></li><li>Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, <a href="https://arxiv.org/abs/1512.03385"><em>Deep Residual Learning for Image Recognition</em></a></li><li>François Chollet, <a href="https://arxiv.org/abs/1610.02357"><em>Xception: Deep Learning with Depthwise Separable Convolutions</em></a></li><li>Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam, <a href="https://arxiv.org/abs/1704.04861"><em>MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Application</em></a></li><li>Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le, <a href="https://arxiv.org/abs/1707.07012"><em>Learning Transferable Architectures for Scalable Image Recognition</em></a></li><li>Mingxing Tan, Quoc V. Le, <a href="https://arxiv.org/abs/1905.11946"><em>EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks</em></a></li><li>Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, <a href="https://arxiv.org/abs/1406.4729"><em>Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition</em></a></li><li>Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger, <a href="https://arxiv.org/abs/1608.06993"><em>Densely Connected Convolutional Networks</em></a></li><li>Jie Hu, Li Shen, Samuel Albanie, Gang Sun, Enhua Wu, <a href="https://arxiv.org/abs/1709.01507">Squeeze-and-Excitation Networks</a></li><li>Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun, <a href="https://arxiv.org/abs/1707.01083">ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices</a></li><li>Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, Mu Li, <a href="https://arxiv.org/abs/1812.01187">Bag of Tricks for Image Classification with Convolutional Neural Networks</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b9d792f45a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/10-papers-you-should-read-to-understand-image-classification-in-the-deep-learning-era-4b9d792f45a7">10 Papers You Should Read to Understand Image Classification in the Deep Learning Era</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Witnessing the Progression in Semantic Segmentation: DeepLab Series from V1 to V3+]]></title>
            <link>https://medium.com/data-science/witnessing-the-progression-in-semantic-segmentation-deeplab-series-from-v1-to-v3-4f1dd0899e6e?source=rss-60ef96b05d56------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f1dd0899e6e</guid>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Ethan Yanjia Li]]></dc:creator>
            <pubDate>Sat, 04 Jul 2020 15:52:55 GMT</pubDate>
            <atom:updated>2023-06-15T06:47:28.124Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dpc2M14JX1NC-9Vy.jpg" /><figcaption>Image from <a href="https://sthalles.github.io/">Thalles’ blog</a></figcaption></figure><blockquote>We are a team of experts to help your business adopt AI solutions or build new AI products. Contact us at info@imaginaire.ai or visit our website <a href="https://www.imaginaire.ai/">https://www.imaginaire.ai</a></blockquote><h3>Foreword</h3><p>Semantic segmentation is a computer vision task to segment a target object or area from an image. This is very useful in many industries such as filming and augmented reality. In the deep learning era, Convolutional Neural Network becomes the most efficient method for semantic segmentation. Instead of trying to understand the boundary of an object through visual signals like contrast and sharpness, a deep convolutional neural network converts this task into a classification problem: if we know the class of every pixel in the image, we will get the boundary of objects for free. Under this assumption, we can just modify the final layers of a typical CNN classification network and have it output H*W values to represent classes of each pixel, instead of just outputting 1 value to represent the class for the whole picture. The outputs have class ids at each pixel location and usually encoded in a PNG-like mask. This idea comes from a paper called Fully Convolutional Network for Semantic Segmentation (FCN), and suddenly every researcher started to follow the suit.</p><p>DeepLab series is one of the followers of this FCN idea. From 2015 to 2018, the DeepLab series published four iterations called V1, V2, V3, and V3+. DeepLab V1 sets the foundation of this series, V2, V3, and V3+ each brings some improvement over the previous version. These four iterations borrowed innovations from image classification in recent years to improve semantic segmentation and also inspired lots of other research works in this area. Going into 2020, since there are no more updates on this DeepLab series of networks, it’s a good time to summarize this work that witnessed the progression of Deep Convolutional Neural Network in semantic segmentation.</p><h3>DeepLab V1</h3><blockquote>Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs</blockquote><h4>Multi-scale Feature Aggregation</h4><p>Like I said before, semantic segmentation networks evolve from image classification networks. So usually there’s a backbone network that’s proven to be efficient in the image classification task. Back in 2014, the VGG network surpassed AlexNet by making the network deeper with 3x3 convolution kernels and was the State-of-the-art solution for image classification.</p><p>DeepLabV1 and FCN both use VGG-16 as the backbone network to extract features before some fine-grained classification over pixels. Unfortunately, the author of DeepLabV1 didn’t make it very clear how the network looks like in the paper, so we have to read the source code to have a full picture. To save you some time reading verbose <a href="http://www.cs.jhu.edu/~alanlab/ccvl/DeepLab-MSc/train.prototxt">Caffe prototxt</a>, I’ve made a diagram below to visualize this architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NBL-yixJb2qzTcAPTZGhBg.jpeg" /><figcaption>DeepLabV1 diagram by myself</figcaption></figure><p>DeepLabV1 proposed three variants in the paper: DeepLab, DeepLab-Msc, and DeepLab-Msc-LargeFOV. This is because the author wants to show the ablation of each component used in the DeepLab network. The diagram above shows the DeepLab-Msc variant. Although the author claims that simplicity is one of the virtues of this network, the best DeepLab-Msc version (multi-scale) is far more complicated than FCN. As we can see from the diagram, the green blocks on the left side are convolutional blocks from the VGG backbone. The yellow blocks in the middle are DeepLab pixel-wise classification layers. And the red blocks on the right side are some post-processing of output. Because semantic segmentation usually needs very fine-grained details and high-level global features in the meantime, merging features from multi-scale is a very common method to combat coarse classification prediction. DeepLab aggregates feature from many intermediate convolutional layers and the input, then interpolate the element-wise summed value back to original resolution as an output mask. Usually, the deeper the convolution layer is, the smaller scale of features it produces, so essentially, DeepLab is merging features of multiple scales.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UuiAq9tyZhKJmdCEwom7WQ.jpeg" /><figcaption>A detailed view of classification head (yellow blocks from the diagram before). Diagram by myself.</figcaption></figure><p>Let take a closer look at those classification heads that process multi-scale features. The diagram above shows the internal look of one classification block. This design is similar to FCN where we use a 3x3 kernel to condense features first, and then use two 1x1 kernels to predict final classes. Dropout is also introduced to combat overfitting.</p><p>However, using multi-scale features isn’t really a big innovation over FCN, and DeepLab needs something else to standout. Next, let’s talk about another “issue” of FCN that DeepLab tried to fix.</p><h4>Larger Feature Map</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GP4HNbfzQTVQ7KCdh69YvQ.png" /><figcaption>Stride 1 for pooling. Diagram by myself.</figcaption></figure><p>In the above diagram, FCN (left side) converts the last few layers of VGG into a classification head for segmentation. So instead of outputting 1 value and 4096 channels in the final fully-connected layer, FCN is outputting 7x7 mask with 4096 channels. However, the author of DeepLab (right side) thinks downsampling from an original 224x224 image to a 7x7 features is too aggressive and could lead to poor segmentation quality. Therefore, DeepLab sets the strides of MaxPooling layer of last VGG blocks to 1, so that the feature map can keep at 28x28 even for the last few layers.</p><p>This change ensure that we have features with bigger scale to predict on, but also introduced another problem: the computation is much more expensive now, and we also lost some global signals distilled from a 7x7 feature map. One way the author tried is to reduce number of channels for the last FC layers. Another way, which we will focus on, is the true innovation of DeepLab: Atrous Convolution.</p><h4>Atrous Convolution</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/0*czuPJm-Ttsbt28Xm" /><figcaption>From “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs”</figcaption></figure><p>The idea of atrous (hole) convolution is simple but effective: the regular convolution kernel combines values from a few neighboring pixels to calculate the new pixel value, like in (a) below. The downside of using such kernel in semantic segmentation is that more local relationships, instead of global relationships are considered when we extract features. To improve this, DeepLab V1 borrows an idea from a signal processing algorithm to introduce a stride inside the kernel window like (b) below. In this atrous version, we put holes in between each pixel we sample so that we can sample a wider range of input with the same kernel size. For example, when the atrous rate (dilation rate) is 2 and kernel size is 3x3, rather than taking pixels from a 3x3 area, we will take 9 pixels from a 5x5 area by skipping those pixels in between.</p><p>By using atrous convolution, our 3x3 kernel isn’t as expensive as before because we can use fewer kernels to cover a bigger area. Also, an atrous convolution over a 28x28 features map can bring a similar global signal from a regular convolution over a 7x7 features map. Moreover, if we increase the atrous rate, we can effectively use the same computation of a 3x3 kernel but achieving much larger field-of-view (FOV). This is also proven to be useful from the paper.</p><h4>Fully-connected CRF</h4><p>Finally, DeepLab V1 also uses a module called Fully-connected Conditional Random Field (CRF) to further polish the segmentation mask. CRF is a probabilistic model to predict values given the conditional input of other pixels around. To simply put, we are trying to learn a probability distribution to help us compute the probability of output pixels given the values of input pixels and their relationships. A fully-connect CRF uses all pixels of an image as conditional inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/0*jSPEamhJWruplv3C" /><figcaption>From “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs”</figcaption></figure><p>This is especially useful to polish the boundary of the inner body of an object mask based on pixel correlation. For example, if two pixels are very different in color (blue for the sky and black for the plane in the diagram below), it’s more likely that they belong to two different categories. And with a few rounds of iterations by comparing output and ground truth, the probability distribution we got can correctly reflect this phenomenon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0HWZmXK_Ks6aqTyd" /><figcaption>From “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs”</figcaption></figure><p>This CRF isn’t used any more in recent segmentation networks because 1) our segmentation network becomes much better nowadays 2) CRF itself is not end-to-end trainable with the network and runs very slow.</p><h3>DeepLab V2</h3><blockquote>DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs</blockquote><p>Now that you understand how DeepLabV1 works, it’s very easy to catch the idea of DeepLabV2. All the core ideas still remain the same, such as atrous convolution and CRF, so I will only talk about the incremental changes.</p><h4>ASPP</h4><p>Atrous Spatial Pyramid Pooling (ASPP) is a new module introduced in DeepLabV2. DeepLabV2 borrows the idea from “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition”. Spatial Pyramid is a very classic algorithm in Computer Vision even before the deep learning era. This is an important technique to achieve scale-invariant for many algorithms such as SIFT. SPP network introduced the spatial pyramid into convolutional networks, and DeepLabV2 created an atrous version of that SPP module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dVO_ppyUTfEnjBdWvJoq5A.png" /><figcaption>From “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs”</figcaption></figure><p>With an SPP module, our network can encode features from different scales into a fixed-size feature map. This is kind of like a bag-of-word approach, but instead of bagging features, you assign elements in a new feature vector for different scales. For example, you can have 1 feature value correspond to a global scale, 2 feature value correspond to large scale and 3 feature value corresponds to a small scale. Then you will get a feature map with a fixed-size of 6 values. In the real ASPP module, we have four different scales that defined by different atrous rates from 6 to 24.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5dJrfrDsqvxVmebpiZYrvQ.png" /><figcaption>DeepLab V2 ASPP. Diagram by myself</figcaption></figure><p>In FCN and DeepLabV1 as illustrated in the diagram above, our input is fixed to 224x224 in order to match the exact number of connections 28x28x1024 of the FC layer. Now that we have this SPP module, our network can handle input of various sizes without the need to changing the FC layers. Since this SPP module uses atrous convolution, it also inherits the advantages of a large field of view with small computation cost.</p><h4>New Multi-scale Structure and Backbone</h4><p>Another difference between V1 and V2 is the new multi-scale structure. Instead of classifying on a different scale of computed feature, <a href="http://liangchiehchen.com/projects/DeepLabv2_resnet.html">DeepLabV2 </a>runs three times over 1.0, 0.75, and 0.5 downscaled images in parallel to achieve multi-scale feature fusion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/0*dHj1nggCo1Pwfh7j" /><figcaption>DeepLab V2 Multi-scale. Diagram by myself.</figcaption></figure><p>It’s hard to say if this design is wise, but it indeed makes the inference and training really slow by running the same computation three times. But no doubt, this is very useful for the author to improve metrics on the PASCAL dataset.</p><p>To further improve the metrics and the converging speed, DeepLabV2 also starts to use ResNet as a backbone network, and ASPP modules are connected right after Conv5 blocks of ResNet. Note that the original ResNet blocks have strides 2 to downsample the features, but DeepLabV2 set the strides of Conv4 and Conv5 blocks to 1. This is due to the same concern of coarse features we’ve discussed in the DeepLabV1 chapter. To increase field-of-view, DeepLabV2 also replaces Conv4 with an atrous convolution kernel with rate 2 and replaces Conv5 with an atrous convolution kernel with rate 4. In practice, I found that only having Conv5 to use strides 1 and atrous rate 2 is more effective compared to the original proposal.</p><h3>DeepLab V3</h3><p>Rethinking Atrous Convolution for Semantic Image Segmentation</p><h4>Upgraded ASPP</h4><p>The use of atrous convolution and spatial pyramid pooling brings great success to DeepLabV1 and V2, so the author keeps exploring in this direction and made the V3 of this DeepLab series, with the focus on ASPP module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c6U-D_6okozzLS3tLY7LbA.png" /><figcaption>From “Rethinking Atrous Convolution for Semantic Image Segmentation”</figcaption></figure><p>DeepLabV3 still uses the ResNet backbone as illustrated in the above diagram. In V2, ASPP merges features from 4 different scales into one fixed-size feature. However, the nature of atrous convolution makes it hard for the network to pick up small local features like tiny edges, and the very big global features that take into consideration of all pixels.</p><p>To fuse more information into one, DeepLabV3 redesigned ASPP module to have a separate channel of global image pooling to include global features, and then concatenate the result feature vector with a 1x1 convolution over the ASPP input, to use fine-grained details. Another change to the ASPP module is to use BatchNormalization after convolution and before ReLU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a9bYrueP79FpKmekNeZWcQ.jpeg" /><figcaption>DeepLabV3 ASPP. Diagram by myself.</figcaption></figure><p>Unlike the diagram from the paper, the official implementation also added a 50% dropout to ASPP output. And an additional 3x3 convolution with 10% dropout before predicting final classes.</p><h4>Multi-grid, Multi-scale and Output Strides</h4><p>There’re also some small changes to the atrous ResNet backbone network as well. Previously, we have the same atrous rate for convolution layers within the same part of the network, such as Conv4 and Conv5. DeepLabV3 introduced a new hyper-parameter called Multi-grid (MG) to adjust the atrous rate. For example, a multi-grid value of {1, 2, 4} means we will multiple the atrous rate of three convolution layers in the same bottleneck block by 1, 2, and 4 respectively. If the base atrous rate is 2, this will lead to 2, 4, and 8 atrous rates for each layer. According to the experiments, this adjustment can better fuse features from a residual block perspective.</p><p>The way that DeepLabV3 uses multi-scale is also different from V2. The author realized the inefficiency of DeepLabV2 training and moved multi-scale to inference time completely. So when you train a network, it will only use the original image scale and only one branch of computation instead of 3. At inference time, the network will run 6 times over input images scaled by {0.5, 0.75, 1.0, 1.25, 1.5, 1.75} times and then average out the 6 outputs (MS). To get even better metrics on the PASCAL dataset, the author also flips the query image and runs inference on these 6 scales again. This is the part that many people dislike, the author is clearly only trying to get a higher score in benchmark instead of proposing a universal solution, which makes the whole paper less convincing.</p><p>With unlimited computing power from Google, the author also experiments different output strides (OS) of the network. OS = 8 means the network will have at most 8 times reduction in resolution, which translates to a 28x28 feature map when the input is 224x224. The key to controlling OS is the pooling strides in the residual blocks. When strides = 2 for Conv4 and Conv5, the OS is 8. And if we set strides to 2 for Conv4, we will get OS = 16 and a 14x14 feature map. The experiments proved that the lower the OS, the better the result will be, but also the more computation is needed.</p><h4>No More CRF</h4><p>With various tricks used in DeepLabV3 to improve benchmark metrics, the network comes to a point where the additional CRF step can’t improve the result anymore. So CRF is removed in V3 and the following version. This result is very understandable, with crazy multi-scale and flipping inference, and large feature map from small Output Stride, the occasional uncertainty of network prediction is mostly eliminated in ensemble results.</p><h3>DeepLab V3+</h3><blockquote>Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation</blockquote><p>In 2018, DeepLab announced its final version DeepLabV3+ as a minor improvement over V3.</p><h4>New Backbone Network</h4><p>The most impactful change from DeepLabV3 is the use of new backbone network Xception. By using depthwise separable convolution, Xception can achieve a much better result when using less computation power compared with ResNet. It also achieved many state-of-the-art metrics on benchmarks like ImageNet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oHOadJ-4sd27vwjj4P3FiQ.png" /><figcaption>From “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J02wHJy3Az9gmo0IrfyKTg.png" /><figcaption>From “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation”</figcaption></figure><p>DeepLabV3+ borrowed a modified version of Xception from MSRA’s Deformable convolutional networks and also created an atrous version of the depthwise separable convolution to attain feature resolution and field-of-view like it did to ResNet. As shown in the diagram © above, atrous separable convolution just change the depthwise conv part into a dilated version with holes in between. The new backbone is indeed very powerful. It drives up the metrics on PASCAL by 2% compared with a ResNet-101 backbone.</p><h4>Encoder-Decoder Architecture</h4><p>Encode-decoded is a very common design in semantic segmentation. FCN, U-Net, E-Net all adopt this idea: encoding image into a small feature vector first using a backbone network, and learn a decoder network to upsample the features by using ConvolutionTranspose layer. ConvolutionTranspose is the inverse of convolution, instead of condensing features from multiple locations into one, it upsamples the value from one location and fans out into a wider range of area. The goal of the decoder is to learn a transformation to recover lost information from the encoder stage. In DeepLabV1 and V2, however, the author criticized the usage of decoder because he thinks atrous convolution with a higher resolution feature can achieve a similar up-sampling effect without losing too much information. Even though the DeepLab series avoids using ConvolutionTranspose to decode feature, other network architectures using this decoder design actually got widely adopted because of their simple design. So in V3+, DeepLab finally incorporates decoder to further improve the metrics on common benchmark datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vv5kTpUf3nkU-dyl5jcruQ.png" /><figcaption>From “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation”</figcaption></figure><p>In the diagram above, the encoder is the original DeepLabV3 network. It consists of a backbone DCNN network and an ASPP module. With features from ASPP output and also an intermediate output of the backbone network, the decoder use convolution and bilinear upsampling to gradually increase output’s resolution by 8 times. Please note that the <a href="https://github.com/tensorflow/models/tree/master/research/deeplab">official implementation</a> in TensorFlow is slightly different from the paper. For example, there’re two 3x3 Conv layer and one 1x1 Conv layer before the final 4x upsampling.</p><p>Ironically, the image-level feature (global pooling) fusion in the ASPP module introduced in V3 paper actually degrades the performance on the Cityscapes dataset when used together with the decoder. It probably means saturation of feature fusion used in DeepLab and none of them are really essential. The network architecture itself may have already overfitted to the PASCAL dataset.</p><h3>Other Details</h3><p>So far, we’ve discussed the evolution of DeepLab network over time. In this section, I want to briefly cover some details of training other than network architecture.</p><h4>Loss Function</h4><p>In DeepLabV1, the loss function is a simple Cross-Entropy loss over classes. In V2, since three branches work on different scales of input, the loss became the sum of three Cross-Entropy losses. In V3 and V3+, this is reverted back to a simple Cross-Entropy because the idea of multi-scale has been used in inference instead of training.</p><h4>Learning Rate Policy</h4><p>In DeepLabV1, the learning rate policy is a simple step decay policy that starts from 1e-3. In DeepLabV2, the author found that a new “ploy” decay policy works better. The learning rate will be (1-iteration/max_iteration)^power, where power is 0.9. This gradual decay learning rate helps the network to find the optimal solution more steadily. And this policy has also been employed by V3 and V3+.</p><h4>Upsampling</h4><p>To calculate loss, DeepLabV1 and V2 downsized ground truth to make it the same size as the network output. This contrasted FCN where the result is upsampled to match the original input resolution. Since V3, the author found that this could lead to no back-propagation of some finer details from ground truth, so the ground truth is no longer downsized during training.</p><h3>Conclusion</h3><p>DeepLab series is one of the most important semantic segmentation networks so far. It started from astrous convolution, and gradually adding more features from other networks, such as SPP and Decoder, to make itself even stronger. Therefore, DeepLab indeed witnessed the progression of image classification and semantic segmentation. It became a summary of all many cool techniques researchers discovered in recent years.</p><p>However, since DeepLabV3, it also starts to use tons of tricks to improve benchmark metrics so it’s also become harder to reproduce the result. Among those tricks, ensemble predictions from multiple scales of the same image, and increasing the feature resolution are the most useful ones, so it’s still a useful insight for engineering. Hopefully in the next decade when we have much more cheap computing power, we will eventually understand the essence of DeepLab after stripping out all the boosting effects from these tricks.</p><h3>References</h3><p>Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L. Yuille, <a href="https://arxiv.org/abs/1412.7062">Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs</a></p><p>Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L. Yuille, <a href="https://arxiv.org/abs/1606.00915">DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs</a></p><p>Liang-Chieh Chen, George Papandreou, Florian Schroff, Hartwig Adam, <a href="https://arxiv.org/abs/1706.05587">Rethinking Atrous Convolution for Semantic Image Segmentation</a></p><p>Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam, <a href="https://arxiv.org/abs/1802.02611">Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f1dd0899e6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/witnessing-the-progression-in-semantic-segmentation-deeplab-series-from-v1-to-v3-4f1dd0899e6e">Witnessing the Progression in Semantic Segmentation: DeepLab Series from V1 to V3+</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Human Pose Estimation with Stacked Hourglass Network and TensorFlow]]></title>
            <link>https://medium.com/data-science/human-pose-estimation-with-stacked-hourglass-network-and-tensorflow-c4e9f84fd3ce?source=rss-60ef96b05d56------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4e9f84fd3ce</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[pose-estimation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Ethan Yanjia Li]]></dc:creator>
            <pubDate>Sat, 14 Mar 2020 22:23:06 GMT</pubDate>
            <atom:updated>2023-06-15T06:48:42.743Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understand the core design of the classic pose estimation network and learn to code it in TF 2</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wBlQ1Sqw4r6dGMhH4bterg.jpeg" /><figcaption>A Blue Hourglass Photo from <a href="http://www.deskcity.org/download/219470-0-0x0.html">Deskcity</a></figcaption></figure><blockquote>We are a team of experts to help your business adopt AI solutions or build new AI products. Contact us at info@imaginaire.ai or visit our website <a href="https://www.imaginaire.ai/">https://www.imaginaire.ai</a></blockquote><p>For full source code, please go to <a href="https://github.com/ethanyanjiali/deep-vision/tree/master/Hourglass/tensorflow">https://github.com/ethanyanjiali/deep-vision/tree/master/Hourglass/tensorflow</a>. I really appreciate your ⭐STAR⭐ that supports my efforts.</p><p>Human is good at making different poses. Human is good at understanding these poses too. This makes body language become such an essential part of our daily communication, work, and entertainment. Unfortunately, poses have so much variance, so it’s not an easy task for a computer to recognize a pose from a picture…until we have deep learning!</p><p>With a deep neural network, the computer can learn a generalized pattern of human poses, and predict joints location accordingly. The Stacked Hourglass Network is just such kind of network, and I’m going to show you how to use it to make a simple human pose estimation. Although first introduced in 2016, it’s still one of the most important networks in pose estimation area, and widely used in lots of applications. No matter if you want to build a software to track basketball player’s action, or make a body language classifier based on a person’s pose, this would be a handy hands-on tutorial for you.</p><h3>Network Architecture</h3><h4>Overview</h4><p>To simply put, Stacked Hourglass Network (HG) is a stack of hourglass modules. It got this name because the shape of each hourglass module closely resemble an hourglass, as we can see from the picture below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*3js3X9fQu5CrkLgmmMudbw.png" /><figcaption>Diagram From “<a href="https://arxiv.org/abs/1603.06937">Stacked Hourglass Networks for Human Pose Estimation</a>”</figcaption></figure><p>The idea behind stacking multiple HG (Hourglass) modules instead of forming a giant encoder and decoder network is that each HG module will produce a full heat-map for joint prediction. Thus, the latter HG module can learn from the joint predictions of the previous HG module.</p><p>Why would a heat-map help human pose estimation? This is a pretty common technique nowadays. Unlike facial keypoints, human pose data has lots of variances, which makes it hard to converge if we just simply regress the joint coordinates. Smart researchers come up with an idea to use heat-map to represent a joint location in an image. This preserves the location information, and then we just need to find the peak of the heat-map and use that as the joint location (plus some minor adjustment since heat-map is coarse). For a 256x256 input image, our heat-map will be 64x64.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*XVeffw6FVp40P0M6c79vFQ.png" /><figcaption>Pictures From “<a href="https://arxiv.org/abs/1603.06937">Stacked Hourglass Networks for Human Pose Estimation</a>”</figcaption></figure><p>In addition, we would also calculate the loss for each intermediate prediction, which helps us to supervise not only the final output but also all HG modules effectively. This is a brilliant design back then because pose estimation relies on the relationship among each area of the human body. For example, without seeing the location of the body, it’s hard to tell if an arm is left arm or right arm. By using a full prediction as the next modules’ input, we are forcing the network to pay attention to other joints while predicting a new join location.</p><h4>Hourglass Module</h4><p>So how does this HG (Hourglass) module itself look like? Let’s take a look at another diagram from the original paper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HnN9NeL3M4nm2JlBLWD6CA.png" /><figcaption>Diagram From “<a href="https://arxiv.org/abs/1603.06937">Stacked Hourglass Networks for Human Pose Estimation</a>”</figcaption></figure><p>In the diagram, each box is a residual block plus some additional operations like pooling. If you are not familiar with residual block and bottleneck structure, I’d recommend you to read some ResNet article first. In general, an HG module is an encoder and decoder architecture, where we downsample the features first, and then upsample the features to recover the info and form a heat-map. Each encoder layer would have a connection to its decoder counterpart, and we could stack as many as layers we want. In the implementation, we usually make some recursions and let this HG module to repeat itself.</p><p>I understand that it still seems too “convoluted” here, so it might be easier just to read the code. Here’s a piece of code copied from my<a href="https://github.com/ethanyanjiali/deep-vision/tree/master/Hourglass/tensorflow"> Stacked Hourglass implementation on Github deep-vision repo</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4dba9fdb7d468fda7ded35e27735a14/href">https://medium.com/media/a4dba9fdb7d468fda7ded35e27735a14/href</a></iframe><p>This module looks like an onion, and let’s start from the outmost layer first. up1 went through two bottleneck blocks and added together with up2. This represents two bigs boxes on the left and top, and also the right-most plus sign. The whole flow is up in the air, so we call it up channel. On line 17, there’s also a low channel. This low1 goes through some pooling and bottleneck block, then goes into another smaller Hourglass module! On the diagram, it’s the second layer of the big onion. And this is also why we are using recursion here. We keep repeating this HG module until layer 4, where you just have a single bottleneck instead of an HG module. And this final layer in the three tiny boxes in the middle of the diagram.</p><p>If you are familiar with some image classification networks, it’s clear that the author borrows the idea of skip connection very heavily. This repeating pattern connects the corresponding layers in the encoder and decoder together, instead of just having one flow of features. This not only helps the gradient to pass through but also lets the network consider features from different scales when decoding.</p><h4>Intermediate Supervision</h4><p>Now that we have an Hourglass module, and we know that the whole network consists of multiple modules like this, but how do we stack them together precisely? Here comes the final piece of the network: intermediate supervision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*cpvMEj57hPctvCujNmg3Iw.png" /><figcaption>Diagram From “<a href="https://arxiv.org/abs/1603.06937">Stacked Hourglass Networks for Human Pose Estimation</a>”</figcaption></figure><p>As you can see from the diagram above, when we produce something from the HG module, we split the output into two paths. The top path includes some more convolutions to further process the features and then go to the next HG module. The interesting thing happens at the bottom path. Here we use the output of that convolution layer as an intermediate heat-map result (blue box) and then calculate loss between this intermediate heat-map and the ground-truth heat-map. In other words, if we have 4 HG modules, we will need to calculate four losses in total: 3 for the intermediate result, and 1 for the final result.</p><h3>Prepare the Data</h3><h4>MPII Dataset</h4><p>Once we finished the code for the Stacked Hourglass network, it’s time for us to think about what kind of data we’d like to use to train this network. If you have your own dataset, that’s great. But here I’d like to mention an open dataset for those beginners who want to have something to train on first. And it’s called MPII Dataset (<a href="https://www.mpi-inf.mpg.de/home/">Max Planck Institute for Informatics</a>). You could find the download link <a href="http://human-pose.mpi-inf.mpg.de/#download">here</a>.</p><p>Although this dataset is mostly used for single person pose estimation, it does provide joints annotations for multiple people in the same image. For each person, it gives the coordinates for 16 joints, such as the left ankle or right shoulder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*pELSj-O7UvxjUdcBDQHxIA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*eM4OdLu0G8sUzd-WPMVXYQ.png" /><figcaption>Examples from MPII Dataset. Both image has been cropped.</figcaption></figure><p>However, the original dataset annotation is in Matlab format, which is really hard to use nowadays. I also tried to parse that mat file by myself, and it turns out that there’re so many errors or caveats that not mentioned from MPII official website. An alternative is to use a preprocessed JSON format annotations provided by Microsoft <a href="https://github.com/microsoft/human-pose-estimation.pytorch/blob/master/README.md#data-preparation">here</a>. The Google Drive link is <a href="https://drive.google.com/drive/folders/1En_VqmStnsXMdldXA6qpqEyDQulnmS3a">here</a>. After you download this JSON annotation, you would see a list with elements like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9777bba64c70df7e8dae72fc72a6639/href">https://medium.com/media/b9777bba64c70df7e8dae72fc72a6639/href</a></iframe><p>“joint_vis” indicates the visibility of a joint. In recent datasets, we usually need to differentiate occluded joints and visible joints. But in MPII, we only care about if the joint is in the view of the image: 1 -&gt; in the view, 0-&gt; out of the view. “joints” is a list of joint coordinates, and they follow the order of 0 — r ankle, 1 — r knee, 2 — r hip, 3 — l hip, 4 — l knee, 5 — l ankle, 6 — pelvis, 7 — thorax, 8 — upper neck, 9 — head top, 10 — r wrist, 11 — r elbow, 12 — r shoulder, 13 — l shoulder, 14 — l elbow, 15 — l wrist.</p><p>The downside of this JSON annotations is that it doesn’t have the head rectangle and the single person flag from the original file, which could pose a problem during evaluation. If you need the version I parsed with these info, please leave me a message in my deep-vision Github repository.</p><h4>Cropping</h4><p>The less clear part is “the scale” and “center”. Sometimes we could have more than one person in the image, so we need to crop out the one we are interested. Unlike the MSCOCO dataset, MPII didn’t give us the bounding box of the person. Instead, it gave us a center coordinate and a rough scale of the person. Both value is not accurate, but still represent the general location of a person in an image. Note that you’ll need to multiply “scale” by 200px to get the true height of a person. But how about the width? Unfortunately, the dataset didn’t really specify it. And the body may align somewhat horizontally, which makes the width way larger than height. One of the examples I saw before is the curling player crawling on the ground, and if you only use the height to crop, you could end up leaving his arms out.</p><p><strong>Naive way: </strong>After some experiments, here’s my proposal to crop the image:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f20d07168e5f696b85c67b8c0d3cfb8/href">https://medium.com/media/0f20d07168e5f696b85c67b8c0d3cfb8/href</a></iframe><p>In short, we filter out invisible joints first and calculate coordinates of the left-most, top-most, bottom-most, and right-most joint from 16 joints. These four coordinates give us a region where we could at least include all available joint annotations. Then I padded this region based on a proportion of this person’s height, which is also calculated from the “scale” field. Lastly, we need to make sure that this crop would not go out of the border.</p><p><strong>More robust way: </strong>If you need to improve the PCKh metrics of your model, the above transformation may not be the ideal one. Why? Because the untold trick of single person pose estimation is that it relies on the center extremely to determine relative location of joints. So you will need to somehow “normalize” all the bodies from the input images to same center and same scale to get the most of the model. My solution to this to use affine transform. By doing so you can define translation, rotation and scaling with a precise mind of where the center is. Since this is a more advanced topic to cover, please visit my deep-vision Github repository for more discussion there.</p><h4>Gaussian</h4><p>Another important thing to know about the ground-truth data is gaussian. When we curate the ground-truth heat-map, we don’t just assign 1 for the joint coordinates and assign 0 for all other pixels. This would make ground-truth too sparse to learn. If the model prediction is just a few pixels off, we should sort of encourage this behavior.</p><p>How do we model this encouragement in our loss function? If you took a probability class before, you might remember Gaussian distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*ewB8UW4Xsikb4ROVR1OT0w.jpeg" /><figcaption><a href="https://www.projectrhea.org/rhea/index.php/Edge_Detection_with_Gaussian_Blur">https://www.projectrhea.org/rhea/index.php/Edge_Detection_with_Gaussian_Blur</a></figcaption></figure><p>The center has the highest value and gradually decreasing values for the area around the center. This is exactly what we need. We would draw such a Gaussian pattern in our all-zero ground-truth canvas like the left figure below. And when you combine all 16 joints in one heat-map, it looks like the right figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*uVPV1KAx8dfhphRzc6E_5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*4Heq3B3PJAMdjMDOWmR20w.png" /><figcaption>Overlay drew by myself. Original image on the right comes from <a href="http://human-pose.mpi-inf.mpg.de/">MPII dataset</a>.</figcaption></figure><p>As you can see from the code below, we calculate the size of the patch first, when sigma is 1, the size would be 7, and the center would be (3,3). Then we generate a meshgrid to represent the coordinates of each cell in this patch. And finally, substitute them into the gaussian formula.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cef5c251f9dc4e8847aaeb9411a317d8/href">https://medium.com/media/cef5c251f9dc4e8847aaeb9411a317d8/href</a></iframe><p>Note that the final code to generate a Gaussian is more complicated than this because it needs to handle some border cases. For full code, please take a look at my repo here: <a href="https://github.com/ethanyanjiali/deep-vision/blob/master/Hourglass/tensorflow/preprocess.py#L91">https://github.com/ethanyanjiali/deep-vision/blob/master/Hourglass/tensorflow/preprocess.py#L91</a></p><h3>Loss Function</h3><p>Until now, we discussed the network architecture and also the data to use. With those, we can make a forward pass for some training data to get a feeling about output. But modern deep learning is about back-propagation and gradient descent, which requires us to calculate the loss between ground-truth and prediction. So let’s get on it.</p><p>Fortunately, the loss function for Stacked Hourglass is pretty simple. You just take Mean Square Error between two vectors, which could be done in one line of code (vanilla version). However, in reality, I found it’s still kind of hard for the model to converge, and it learned to cheat by predicting all zeros to reach a local optimal. My solution here (improved version) is to assign a bigger weight for foreground pixels (those gaussian we drew) and make it hard for the network to just ignore these non-zero values. I choose 82 here because there’re 82 times background pixels than foreground pixels for a 7x7 patch in 64x64 heat-map.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/179b84c02e27d38d962c31b6f66b5be1/href">https://medium.com/media/179b84c02e27d38d962c31b6f66b5be1/href</a></iframe><h3>Predictions</h3><p>So far, we’ve discussed the network, the data, and the optimization goal (loss). This should be sufficient for you to start your own training. Once we finished the training and get a model, it’s still not done yet. One shortcoming of using heat-map compared with regressing directly is the granularity. For example, with a 256x256 input, we are getting a 64x64 heat-map to represent key-point location. The four-times down-scale doesn’t seem quite bad. However, we usually first resize a bigger image, such as 720x480, into this 256x256 input. In this scenario, a 64x64 heat-map would be too coarse. Researchers came up with a way to alleviate this problem when high precision is required. Instead of just using the pixel with max value, we also take into consideration of the neighbor pixel with largest value. Since the neighbor pixel also has a high , it infers that the actual key-point location might be a bit towards the direction of neighbor pixel. Sounds familiar, right? It’s pretty much like our gradient descent, which also points to the optimal solution. However, from my experience, taking the position of heat-map peak should be enough for most of the applications. So it’s your call if you need to implement this extra correction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*KNj8aQw13fGa60SoEHKg2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*ATXQjBct8NoGaLdgivdmGg.png" /><figcaption>Key-points and Skeleton drew by myself. The person in the image is Lee Westwood from <a href="https://www.golf-monthly.co.uk/videos/golf-swing-tips/lee-westwood-stance-and-ball-position-tips">Golf Monthly</a></figcaption></figure><p>Above is the prediction example of our network. On the left side, it is all the joint locations. On the right side, it’s the skeleton, drew by linking those joints together. Although the result looks pretty decent, I have to admit that this is an easy example. In reality, there’re lots of twisted poses or occluded joints, which brings significant challenges to our network. For example, when there is only one foot in the image, the network could confuse itself by assign both left and right foot to the same location. How do we address this? Since this is a more a topic about improvement, I’ll leave it to you to think about it first and write another article to discuss in the future.</p><h3>Conclusion</h3><p>Congratulation, you reached the end of this tutorial. If you comprehend everything we discussed above, you should have a solid understanding of the theory and major challenges now. To start coding it in TensorFlow, I would suggest you to clone/fork my repo: <a href="https://github.com/ethanyanjiali/deep-vision/tree/master/Hourglass/tensorflow">https://github.com/ethanyanjiali/deep-vision/tree/master/Hourglass/tensorflow</a>, follow the instruction to prepare dataset, and give it a run. To make predictions, you could refer to a notebook in that repo called demo_hourglass_network. If you run into any problems, please leave a Github issue so that I could take a look. And again, if you like my article or my repo, please ⭐star⭐ my <a href="https://github.com/ethanyanjiali/deep-vision/">repo</a> and that will be the biggest support for me.</p><h3>References</h3><ul><li>Alejandro Newell, Kaiyu Yang, Jia Deng, <a href="https://arxiv.org/abs/1603.06937">Stacked Hourglass Networks for Human Pose Estimation</a>, European Conference on Computer Vision (ECCV), 2016</li><li>Sudharshan Chandra Babu, <a href="https://nanonets.com/blog/human-pose-estimation-2d-guide/">A 2019 guide to Human Pose Estimation with Deep Learning</a>, Nanonets</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4e9f84fd3ce" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/human-pose-estimation-with-stacked-hourglass-network-and-tensorflow-c4e9f84fd3ce">Human Pose Estimation with Stacked Hourglass Network and TensorFlow</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dive Really Deep into YOLO v3: A Beginner’s Guide]]></title>
            <link>https://medium.com/data-science/dive-really-deep-into-yolo-v3-a-beginners-guide-9e3d2666280e?source=rss-60ef96b05d56------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e3d2666280e</guid>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[yolov3]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[tensorflow2]]></category>
            <dc:creator><![CDATA[Ethan Yanjia Li]]></dc:creator>
            <pubDate>Mon, 30 Dec 2019 22:10:54 GMT</pubDate>
            <atom:updated>2023-06-15T06:49:25.139Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V9N0HzCQaK7Co3jQlw9q0w.png" /><figcaption>Screenshot from a video made by <a href="https://www.youtube.com/channel/UClVWrKrmoeNM7A9nkEMeYzA">Joseph Redmon</a> on <a href="https://www.youtube.com/watch?v=MPU2HistivI">Youtube</a></figcaption></figure><blockquote>We are a team of experts to help your business adopt AI solutions or build new AI products. Contact us at info@imaginaire.ai or visit our website <a href="https://www.imaginaire.ai/">https://www.imaginaire.ai</a></blockquote><p>For full source code, please go to <a href="https://github.com/ethanyanjiali/deep-vision/tree/master/YOLO/tensorflow">https://github.com/ethanyanjiali/deep-vision/tree/master/YOLO/tensorflow</a>. I really appreciate your STAR that supports my efforts.</p><h3>Foreword</h3><p>When a self-driving car runs on a road, how does it know where are other vehicles in the camera image? When an AI radiologist reading an X-ray, how does it know where the lesion (abnormal tissue) is? Today, I will walk through this fascinating algorithm, which can identify the category of the given image, and also locate the region of interest. There’s plenty of algorithms introduced in recent years to address object detection in a deep learning approach, such as R-CNN, Faster-RCNN, and Single Shot Detector. Among those, I’m most interested in a model called YOLO — You Only Look Once. This model attracts me so much, not only because of its funny name, but also some practical design that truly makes sense for me. In 2018, this latest V3 of this model had been released, and it achieved many new State of the Art performance. Because I’ve programmed some GANs and image classification networks before, and also Joseph Redmon described it in the paper in a really easy-going way, I thought this detector would just be another stack of CNN and FC layers that just works well magically.</p><p>But I was wrong.</p><p>Perhaps it’s because I’m just dumber than usual engineers, I found it really difficult for me to translate this model from the paper to actual code. And even when I managed to do that in a couple of weeks (I gave up once put it away for a few weeks), I found it even more difficult for me to make it work. There’re so quite a few blogs, GitHub repos about YOLO V3, but most of them just gave a very high-level overview of the architecture, and somehow they just succeed. Even worse, the paper itself is too chill that it fails to provide many crucial details of implementation, and I have to read the author’s original C implementation (when is the last time did I write C? Maybe at college?) to confirm some of my guesses. When there’s a bug, I usually have no idea why it would occur. Then I end up manually debugging it step by step and calculating those formulas with my little calculator.</p><p>Fortunately, I didn’t give up this time and finally made it work. But in the meantime, I also felt really strongly that there should be a more thorough guide out there on the internet to help dumb people like me to understand every detail of this system. After all, if one single detail is wrong, the whole system would go south quickly. And I’m sure that if I don’t write these down, I would forget all these in few weeks too. So, here I am, presenting you this “Dive Really Deep into YOLO V3: A Beginner’s Guide”. I hope you’ll like it.</p><h3>Prerequisite</h3><p>Before getting into the network itself, I’ll need to clarify with some prerequisites first. As a reader, you are expected to:</p><p>1. Understand the basics of Convolutional Neural Network and Deep Learning<br>2. Understand the idea of object detection task<br>3. Have curiosity about how the algorithm works internally</p><p>If you need help on first two items, there’re plenty of excellent resources like <a href="https://www.udacity.com/course/computer-vision-nanodegree--nd891">Udacity Computer Vision Nanodegree</a>, <a href="https://www.coursera.org/specializations/deep-learning">Cousera Deep Learning Specialization</a> and <a href="http://cs231n.stanford.edu/">Stanford CS231n</a><br>If you just want to build something to detect some object with your custom dataset quickly, check out this <a href="https://github.com/tensorflow/models/tree/master/research/object_detection">Tensorflow Object Detection API</a></p><h3>YOLO V3</h3><p>YOLO V3 is an improvement over previous YOLO detection networks. Compared to prior versions, it features multi-scale detection, stronger feature extractor network, and some changes in the loss function. As a result, this network can now detect many more targets from big to small. And, of course, just like other single-shot detectors, YOLO V3 also runs quite fast and makes real-time inference possible on GPU devices. Well, as a beginner to object detection, you might not have a clear image of what do they mean here. But you will gradually understand them later in my post. For now, just remember that YOLO V3 is one of the best models in terms of real-time object detection as of 2019.</p><h4><strong>Network Architecture</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hULeMGlxnjjvuf7Fx7kuaA.jpeg" /><figcaption>Diagram by myself</figcaption></figure><p>First of all, let’s talk about how this network look like at a high-level diagram (Although, the network architecture is the least time-consuming part of implementation). The whole system can be divided into two major components: Feature Extractor and Detector; both are multi-scale. When a new image comes in, it goes through the feature extractor first so that we can obtain feature embeddings at three (or more) different scales. Then, these features are feed into three (or more) branches of the detector to get bounding boxes and class information.</p><p><strong>Darknet-53</strong></p><p>The feature extractor YOLO V3 uses is called Darknet-53. You might be familiar with the previous Darknet version from YOLO V1, where there’re only 19 layers. But that was like a few years ago, and the image classification network has progressed a lot from merely deep stacks of layers. ResNet brought the idea of skip connections to help the activations to propagate through deeper layers without gradient diminishing. Darknet-53 borrows this idea and successfully extends the network from 19 to 53 layers, as we can see from the following diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*a_hU7H_Av2YiUEFaui635w.jpeg" /><figcaption>Diagram from <a href="https://arxiv.org/abs/1804.02767">YOLOv3: An Incremental Improvement</a></figcaption></figure><p>This is very easy to understand. Consider layers in each rectangle as a residual block. The whole network is a chain of multiple blocks with some strides 2 Conv layers in between to reduce dimension. Inside the block, there’s just a bottleneck structure (1x1 followed by 3x3) plus a skip connection. If the goal is to do multi-class classification as ImageNet does, an average pooling and a 1000 ways fully connected layers plus softmax activation will be added.</p><p>However, in the case of object detection, we won’t include this classification head. Instead, we are going to append a “detection” head to this feature extractor. And since YOLO V3 is designed to be a multi-scaled detector, we also need features from multiple scales. Therefore, features from last three residual blocks are all used in the later detection. In the diagram below, I’m assuming the input is 416x416, so three scale vectors would be 52x52, 26x26, and 13x13. Please note that if the input size is different, the output size will differ too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SzyNALdsE9pDCpCvtqH7ZQ.jpeg" /><figcaption>Diagram by myself</figcaption></figure><p><strong>Multi-scale Detector</strong></p><p>Once we have three features vectors, we can now feed them into the detector. But how should we structure this detector? Unfortunately, the author didn’t bother to explain this part this his paper. But we could still take a look at the source code he published on Github. Through this <a href="https://github.com/pjreddie/darknet/blob/master/cfg/yolov3.cfg#L549">config file</a>, multiple 1x1 and 3x3 Conv layers are used before a final 1x1 Conv layer to form the final output. For medium and small scale, it also concatenates features from the previous scale. By doing so, small scale detection can also benefit from the result of large scale detection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kVFx54oUhBWzUdHzbFDsiw.jpeg" /><figcaption>Diagram by myself</figcaption></figure><p>Assuming the input image is (416, 416, 3), the final output of the detectors will be in shape of <em>[(52, 52, 3, (4 + 1 + num_classes)), (26, 26, 3, (4 + 1 + num_classes)), (13, 13, 3, (4 + 1 + num_classes))]. </em>The three items in the list represent detections for three scales. But what do the cells in this <em>52x52x3x(4+1+num_classes)</em> matrix mean? Good questions. This brings us to the most important notion in pre-2019 object detection algorithm: anchor box (prior box).</p><p><strong>Anchor Box</strong></p><p>The goal of object detection is to get a bounding box and its class. Bounding box usually represents in a normalized xmin, ymin, xmax, ymax format. For example, 0.5 xmin and 0.5 ymin mean the top left corner of the box is in the middle of the image. Intuitively, if we want to get a numeric value like 0.5, we are facing a regression problem. We may as well just have the network predict for values and use Mean Square Error to compare with the ground truth. However, due to the large variance of scale and aspect ratio of boxes, researchers found that it’s really hard for the network to converge if we just use this “brute force” way to get a bounding box. Hence, in Faster-RCNN paper, the idea of an anchor box is proposed.</p><p>Anchor box is a prior box that could have different pre-defined aspect ratios. These aspect ratios are determined before training by running K-means on the entire dataset. But where does the box anchor to? We need to introduce a new notion called the grid. In the “ancient” year of 2013, algorithms detect objects by using a window to slide through the entire image and running image classification on each window. However, this is so inefficient that researchers proposed to use Conv net to calculate the whole image all in once (technically, only when your run convolution kernels in parallel.) Since the convolution outputs a square matrix of feature values (like 13x13, 26x26, and 52x52 in YOLO), we define this matrix as a “grid” and assign anchor boxes to each cell of the grid. In other words, anchor boxes anchor to the grid cells, and they share the same centroid. And once we defined those anchors, we can determine how much does the ground truth box overlap with the anchor box and pick the one with the best IOU and couple them together. I guess you can also claim that the ground truth box anchors to this anchor box. In our later training, instead of predicting coordinates from the wild west, we can now predict offsets to these bounding boxes. This works because our ground truth box should look like the anchor box we pick, and only subtle adjustment is needed, whhich gives us a great head start in training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yQFsd4vMDWVWikme3-Mb8g.jpeg" /><figcaption>Diagram by myself</figcaption></figure><p>In YOLO v3, we have three anchor boxes per grid cell. And we have three scales of grids. Therefore, we will have 52x52x3, 26x26x3 and 13x13x3 anchor boxes for each scale. For each anchor box, we need to predict 3 things:</p><p>1. The location offset against the anchor box: <em>tx, ty, tw, th</em>. This has 4 values.<br>2. The objectness score to indicate if this box contains an object. This has 1 value.<br>3. The class probabilities to tell us which class this box belongs to. This has num_classes values.</p><p>In total, we are predicting <em>4 + 1 + num_classes </em>values for one anchor box, and that’s why our network outputs a matrix in shape of <em>52x52x3x(4+1+num_classes)</em> as I mentioned before. <em>tx, ty, tw, th</em> isn’t the real coordinates of the bounding box. It’s just the relative offsets compared with a particular anchor box. I’ll explain these three predictions more in the Loss Function section after.</p><p>Anchor box not only makes the detector implementation much harder and much error-prone, but also introduced an extra step before training if you want the best result. So, personally, I hate it very much and feel like this anchor box idea is more a hack than a real solution. In 2018 and 2019, researchers start to question the need for anchor box. Papers like CornerNet, Object as Points, and FCOS all discussed the possibility of training an object detector from scratch without the help of an anchor box.</p><h4>Loss Function</h4><p>With the final detection output, we can calculate the loss against the ground truth labels now. The loss function consists of four parts (or five, if you split noobj and obj): centroid (xy) loss, width and height (wh) loss, objectness (obj and noobj) loss and classification loss. When putting together, the formula is like this:</p><pre>Loss = Lambda_Coord * Sum(Mean_Square_Error((tx, ty), (tx’, ty’) * obj_mask)<br> + Lambda_Coord * Sum(Mean_Square_Error((tw, th), (tw’, th’) * obj_mask)<br> + Sum(Binary_Cross_Entropy(obj, obj’) * obj_mask) + Lambda_Noobj * Sum(Binary_Cross_Entropy(obj, obj’) * (1 -obj_mask) * ignore_mask)<br> + Sum(Binary_Cross_Entropy(class, class’))</pre><p>It looks intimidating but let me break them down and explain one by one.</p><pre>xy_loss = Lambda_Coord * Sum(Mean_Square_Error((tx, ty), (tx’, ty’)) * obj_mask)</pre><p>The first part is the loss for bounding box centroid. <em>tx</em> and <em>ty</em> is the relative centroid location from the ground truth. <em>tx’</em> and <em>ty’</em> is the centroid prediction from the detector directly. The smaller this loss is, the closer the centroids of prediction and ground truth are. Since this is a regression problem, we use mean square error here. Besides, if there’s no object from the ground truth for certain cells, we don’t need to include the loss of that cell into the final loss. Therefore we also multiple by <em>obj_mask</em> here. <em>obj_mask </em>is either 1 or 0, which indicates if there’s an object or not. In fact, we could just use <em>obj</em> as <em>obj_mask</em>, <em>obj</em> is the objectness score that I will cover later. One thing to note is that we need to do some calculation on ground truth to get this <em>tx</em> and <em>ty</em>. So, let’s see how to get this value first. As the author says in the paper:</p><pre>bx = sigmoid(tx) + Cx<br>by = sigmoid(ty) + Cy</pre><p>Here <em>bx</em> and <em>by</em> are the absolute values that we usually use as centroid location. For example, <em>bx = 0.5, by = 0.5</em> means that the centroid of this box is the center of the entire image. However, since we are going to compute centroid off the anchor, our network is actually predicting centroid relative the top-left corner of the grid cell. Why grid cell? Because each anchor box is bounded to a grid cell, they share the same centroid. So the difference to grid cell can represent the difference to anchor box. In the formula above, <em>sigmoid(tx)</em> and <em>sigmoid(ty)</em> are the centroid location relative to the grid cell. For instance, <em>sigmoid(tx) = 0.5 and sigmoid(ty) = 0.5</em> means the centroid is the center of the current grid cell (but not the entire image). <em>Cx</em> and <em>Cy</em> represents the absolute location of the top-left corner of the current grid cell. So if the grid cell is the one in the SECOND row and SECOND column of a grid 13x13, then <em>Cx = 1 and Cy = 1</em>. And if we add this grid cell location with relative centroid location, we will have the absolute centroid location <em>bx = 0.5 + 1</em> and <em>by = 0.5 + 1</em>. Certainly, the author won’t bother to tell you that you also need to normalize this by dividing by the grid size, so the true <em>bx</em> would be <em>1.5/13 = 0.115</em>. Ok, now that we understand the above formula, we just need to invert it so that we can get <em>tx</em> from <em>bx </em>in order to translate our original ground truth into the target label. Lastly, <em>Lambda_Coord</em> is the weight that Joe introduced in YOLO v1 paper. This is to put more emphasis on localization instead of classification. The value he suggested is 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*a7seKlMjTN4yXvmTcTuVSw.png" /><figcaption>Diagram from <a href="https://arxiv.org/abs/1804.02767">YOLOv3: An Incremental Improvement</a></figcaption></figure><pre>wh_loss = Lambda_Coord * Sum(Mean_Square_Error((tw, th), (tw’, th’)) * obj_mask)</pre><p>The next one is the width and height loss. Again, the author says:</p><pre>bw = exp(tw) * pw<br>bh = exp(th) * ph</pre><p>Here <em>bw</em> and <em>bh</em> are still the absolute width and height to the whole image. <em>pw</em> and <em>ph</em> are the width and height of the prior box (aka. anchor box, why there’re so many names). We take <em>e^(tw)</em> here because <em>tw</em> could be a negative number, but width won’t be negative in real world. So this <em>exp()</em> will make it positive. And we multiply by prior box width <em>pw</em> and <em>ph</em> because the prediction <em>exp(tw)</em> is based off the anchor box. So this multiplication gives us real width. Same thing for height. Similarly, we can inverse the formula above to translate <em>bw</em> and <em>bh</em> to <em>tx</em> and <em>th</em> when we calculate the loss.</p><pre>obj_loss = Sum(Binary_Cross_Entropy(obj, obj’) * obj_mask)</pre><pre>noobj_loss = Lambda_Noobj * Sum(Binary_Cross_Entropy(obj, obj’) * (1 — obj_mask) * ignore_mask)</pre><p>The third and fourth items are objectness and non-objectness score loss. Objectness indicates how likely is there an object in the current cell. Unlike YOLO v2, we will use binary cross-entropy instead of mean square error here. In the ground truth, objectness is always 1 for the cell that contains an object, and 0 for the cell that doesn’t contain any object. By measuring this <em>obj_loss</em>, we can gradually teach the network to detect a region of interest. In the meantime, we don’t want the network to cheat by proposing objects everywhere. Hence, we need <em>noobj_loss</em> to penalize those false positive proposals. We get false positives by masking prediciton with <em>1-obj_mask</em>. The `<em>ignore_mask</em>` is used to make sure we only penalize when the current box doesn’t have much overlap with the ground truth box. If there is, we tend to be softer because it’s actually quite close to the answer. As we can see from the paper, “If the bounding box prior is not the best but does overlap a ground truth object by more than some threshold we ignore the prediction.” Since there are way too many <em>noobj</em> than <em>obj</em> in our ground truth, we also need this <em>Lambda_Noobj = 0.5</em> to make sure the network won’t be dominated by cells that don’t have objects.</p><pre>class_loss = Sum(Binary_Cross_Entropy(class, class’) * obj_mask)</pre><p>The last loss is classification loss. If there’re 80 classes in total, the <em>class</em> and <em>class’ </em>will be the one-hot encoding vector that has 80 values. In YOLO v3, it’s changed to do multi-label classification instead of multi-class classification. Why? Because some dataset may contains labels that are hierarchical or related, eg <em>woman</em> and <em>person</em>. So each output cell could have more than 1 class to be true. Correspondingly, we also apply binary cross-entropy for each class one by one and sum them up because they are not mutually exclusive. And like we did to other losses, we also multiply by this <em>obj_mask</em> so that we only count those cells that have a ground truth object.</p><p>To fully understand how this loss works, I suggest you manually walk through them with a real network prediction and ground truth. Calculating the loss by your calculator (or <em>tf.math</em>) can really help you to catch all the nitty-gritty details. And I did that by myself, which helped me find lots of bugs. After all, the devil is in the detail.</p><h3>Implementation</h3><p>If I stop writing here, my post will just be like another “YOLO v3 Review” somewhere on the web. Once you digest the general idea of YOLO v3 from the previous section, we are now ready to go explore the remaining 90% of our YOLO v3 journey: Implementation.</p><h4>Framework</h4><p>At the end of September, Google finally released TensorFlow 2.0.0. This is a fascinating milestone for TF. Nevertheless, new design doesn’t necessarily mean less pain for developers. I’ve been playing around TF 2 since very early of 2019 because I always wanted to write TensorFlow code in the way I did for PyTorch. If it’s not because of TensorFlow’s powerful production suite like TF Serving, TF lite, and TF Board, etc., I guess many developers will not choose TF for new projects. Hence, if you don’t have a strong demand for production deployment, I would suggest you implement YOLO v3 in PyTorch or even MXNet. However, if you made your mind to stick with TensorFlow, please continue reading.</p><p>TensorFlow 2 officially made eager mode a first-tier citizen. To put it simply, instead of using TensorFlow specific APIs to calculate in a graph, you can now leverage native Python code to run the graph in a dynamic mode. No more graph compilation and much easier debugging and control flow. In the case where performance is more important, a handy <em>tf.function</em> decorator is also provided to help compile the code into a static graph. But, the reality is, eager mode and <em>tf.function</em> are still buggy or not well documented sometimes, which makes your life even harder in a complicated system like YOLO v3. Also, Keras model isn’t quite flexible, while the custom training loop is still quite experimental. Therefore, the best strategy for you to write YOLO v3 in TF 2 is to start with a minimum working template first, and gradually add more logic to this shell. By doing so, we can fail early and fix the bug before it hides too deeply in a giant nested graph.</p><h4>Dataset</h4><p>Aside from the framework to choose, the most important thing for successful training is the dataset. In the paper, the author used MSCOCO dataset to validate his idea. Indeed, this is a great dataset, and we should aim for a good accuracy on this benchmark dataset for our model. However, a big dataset like this could also hide some bugs in your code. For example, if the loss is not dropping, how do you know if it just needs more time to converge, or your loss function is wrong? Even with GPU, the training is still not fast enough for you to quickly iterate and fix things. Therefore, I recommend you to build a development set which contains tens of images to make sure your code looks “working” first. Another option is to use VOC 2007 dataset, which only has 2500 training images. To use MSCOCO or VOC2007 dataset and create TF Records, you could refer to my helper scripts here: <a href="https://github.com/ethanyanjiali/deep-vision/tree/master/Datasets/MSCOCO/">MSCOCO</a>, <a href="https://github.com/ethanyanjiali/deep-vision/tree/master/Datasets/VOC2007/">VOC2007</a></p><h4>Preprocessing</h4><p>Preprocessing stands for the operations to translate raw data into a proper input format of the network. For the image classification task, we usually just need to resize the image, and one-hot encode the label. But things are a bit more complicated for YOLO v3. Remember I said the output of the network is like <em>52x52x3x(4+1+num_classes)</em> and has three different scales? Since we need to calculate the delta between ground truth and prediction, we also need to format our ground truth into such a matrix first.</p><p>For each ground truth bounding box, we need to pick the best scale and anchor for it. For example, a tiny kite in the sky should be in the small scale (52x52). And if the kite is more like a square in the image, we should also pick the most square-shaped anchor in that scale. In YOLO v3, the author provides 9 anchors for 3 scales. All we need to do is to choose the one that matches our ground truth box the most. When I implement this, I thought I need the coordinates of the anchor box as well to calculate IOU. In fact, you don’t need to. Since we just want to know which anchor fits our ground truth box best, we can just assume all anchors and the ground truth box share the same centroid. And with this assumption, the degree of matching would be the overlapping area, which can be calculated by <em>min width * min height</em>.</p><p>During the transformation, one could also add some data augmentation to increase the variety of training set virtually. For example, typical augmentation includes random flipping, random cropping, and random translating. However, these augmentations won’t block you from training a working detector, so I won’t cover much about this advanced topic.</p><h4>Training</h4><p>After all these discussions, you finally have a chance to run “<em>python train.py”</em> and start your model training. And this is also when you meet most of your bugs. You could refer to my training script <a href="https://github.com/ethanyanjiali/deep-vision/tree/master/YOLO/tensorflow/train.py">here</a> when you are blocked. Meanwhile, I want to provide some tips that are helpful for my own training.</p><p><strong>NaN Loss</strong></p><ol><li>Check your learning rate and make sure it’s not too high to explode your gradient.</li><li>Check for 0 in binary cross-entropy because ln(0) is not a number. You can clip the value from (epsilon, 1 — epsilon).</li><li>Find an example and walk through your loss step by step. Find out which part of your loss goes to NaN. For example, if width/height loss went to NaN, it could be because the way you calculate from <em>tw</em> to <em>bw</em> is wrong.</li></ol><p><strong>Loss remains high</strong></p><ol><li>Try to increase your learning rate to see if it can drop faster. Mine starts at 0.01. But I’ve seen 1e-4 and 1e-5 works too.</li><li>Visualize your preprocessed ground truth to see if it makes sense. One problem I had before is that my output grid is in [y][x] instead of [x][y], but my ground truth is reversed.</li><li>Again, manually walk through your loss with a real example. I had a mistake of calculating cross-entropy between objectness and class probabilities.</li><li>My loss also remains around 40 after 50 epochs of MSCOCO. However, the result isn’t that bad.</li><li>Double-check the coordinates format throughout your code. YOLO requires <em>xywh</em> (centroid x, centroid y, width and height), but most of dataset comes as <em>x1y1x2y2</em> (xmin, ymin, xmax, ymax).</li><li>Double-check your network architecture. Don’t get misled by the diagram from a post called “A Closer Look at YOLOv3 — CyberAILab”.</li><li><em>tf.keras.losses.binary_crossentropy</em> isn’t the sum of binary cross-entropy you need.</li></ol><p><strong>Loss is low, but the prediction is off</strong></p><ol><li>Adjusting <em>lambda_coord</em> or <em>lambda_noobj</em> to the loss based on your observation.</li><li>If you are traininig on your own dataset, and the dataset is relative small (&lt; 30k images), you should intialize weights from a COCO pretrained model first.</li><li>Double-check your non max suppression code and adjust some threshold (I’ll talk about NMS later).</li><li>Make sure your <em>obj_mask</em> in the loss function isn’t mistakenly taking out necessary elements.</li><li>Again and again, your loss function. When calculating loss, it uses relative xywh in a cell (also called <em>tx, ty, tw, th</em>). When calculating ignore mask and IOU, it uses absolute xywh in the whole image, though. Don’t mix them up.</li></ol><p><strong>Loss os low, but there’s no prediction</strong></p><ol><li>If you are using a custom dataset, please check the distribution of your ground truth boxes first. The amount and quality of the boxes could really affect what the network learn (or cheat) to do.</li><li>Predict on your training set to see if your model can overfit on the training set at least.</li></ol><p><strong>Multi-GPU training</strong></p><p>Since the object detection network has so many parameters to train, it’s always better to have more computing power. However, TensorFlow 2.0 doesn’t have great support over multi-GPU training so far. To do that in TF, you’ll need to pick a training strategy like MirroredStrategy, as I did <a href="https://github.com/ethanyanjiali/deep-vision/blob/master/YOLO/tensorflow/train.py#L281">here</a>. Then wrap your dataset loader into a distributed version too. One caveat for distributed training is that the loss coming out of each batch should be divided by the global batch size because we are going to `reduce_sum` over all GPU results. For example, if the local batch size is 8, and there’re 8 GPUs, your batch loss should divide a global batch size of 64. Once you summed up losses from all replica, the final result will be the average loss of a single example.</p><h4><strong>Postprocessing</strong></h4><p>The final component in this detection system is a post-processor. Usually, postprocessing is just about trivial things like replacing machine-readable class id with human-readable class text. In object detection, though, we have one more crucial step to do to get final human-readable results. This is called non maximum suppression.</p><p>Let’s recall our objectness loss. When is false proposal has great overlap with ground truth, we won’t penalize it with <em>noobj_loss</em>. This encourages the network to predict close results so that we can train it more easily. Also, although not used in YOLO, when the sliding window approach is used, multiple windows could predict the same object. In order to eliminate these duplicate results, smart researchers designed an algorithm called non maximum supression (NMS).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Me9PqpjFGoL03G5l7qU6NA.png" /><figcaption>Photo by <a href="https://medium.com/@pythonlessons0">Python Lessons</a> from <a href="https://medium.com/analytics-vidhya/yolo-v3-theory-explained-33100f6d193">Analytics Vidhya</a></figcaption></figure><p>The idea of NMS is quite simple. Find out the detection box with the best confidence first, add it to the final result, and then eliminates all other boxes which have IOU over a certain threshold with this best box. Next, you choose another box with the best confidence in the remaining boxes and do the same thing over and over until nothing is left. In the code, since TensorFlow needs explicit shape most of the time, we will usually define a max number of detection and stop early if that number is reached. In YOLO v3, our classification is not mutually exclusive anymore, and one detection could have more than one true class. However, some existing NMS code doesn’t take that into consideration, so be careful when you use them.</p><h3>Conclusion</h3><p>YOLO v3 is a masterpiece in the rising era of Artificial Intelligence, and also an excellent summary of Convolution Neural Network techniques and tricks in the 2010s. Although there’re many turn-key solutions like Detectron out there to simplify the process of making a detector, a hands-on experience in coding such sophisticated detector is really a great learning opportunity for machine learning engineers because merely reading the paper is far from enough. Like Ray Dalio said about his philosophy:</p><blockquote>Pain plus reflection equals progress.</blockquote><p>I hope my article could be a lighthouse in your painful journey of implementing YOLO v3, and perhaps you can also share the delightful progress with us later. If you like my article or my source code of YOLO v3, please ⭐star⭐ my <a href="https://github.com/ethanyanjiali/deep-vision/">repo</a> and that will be the biggest support for me.</p><h3>References</h3><ul><li>逍遥王可爱, <a href="https://zhuanlan.zhihu.com/p/49995236">史上最详细的Yolov3边框预测分析</a>, 知乎专栏</li><li>Zihao Zhang, <a href="https://github.com/zzh8829/yolov3-tf2">YoloV3 Implemented in TensorFlow 2.0</a>, Github</li><li>Yun Yang, <a href="https://github.com/YunYang1994/TensorFlow2.0-Examples/tree/master/4-Object_Detection/YOLOV3">TensorFlow2.x-YOLOv3</a>, Github</li><li>Joseph Redmon, <a href="https://pjreddie.com/media/files/papers/YOLOv3.pdf">YOLOv3: An Incremental Improvement</a></li><li>Joseph Redmon, Ali Farhadi, <a href="https://arxiv.org/abs/1612.08242">YOLO9000: Better, Faster, Stronger</a></li><li>Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, <a href="https://arxiv.org/abs/1506.02640">You Only Look Once: Unified, Real-Time Object Detection</a>, 2016 IEEE Conference on Computer Vision and Pattern Recognition</li><li>Ayoosh Kathuria, <a href="https://towardsdatascience.com/yolo-v3-object-detection-53fb7d3bfe6b">What’s new in YOLO v3?</a>, Towards Data Science</li><li>Python Lessons, <a href="https://medium.com/analytics-vidhya/yolo-v3-theory-explained-33100f6d193">YOLO v3 theory explained</a>, Analytics Vidhya</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e3d2666280e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/dive-really-deep-into-yolo-v3-a-beginners-guide-9e3d2666280e">Dive Really Deep into YOLO v3: A Beginner’s Guide</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gender Swap and CycleGAN in TensorFlow 2.0]]></title>
            <link>https://ethanyanjiali.medium.com/gender-swap-and-cyclegan-in-tensorflow-2-0-359fe74ab7ff?source=rss-60ef96b05d56------2</link>
            <guid isPermaLink="false">https://medium.com/p/359fe74ab7ff</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ethan Yanjia Li]]></dc:creator>
            <pubDate>Thu, 30 May 2019 10:59:47 GMT</pubDate>
            <atom:updated>2023-06-15T06:49:52.486Z</atom:updated>
            <content:encoded><![CDATA[<h3>Gender Swap and CycleGAN in TensorFlow 2</h3><h4>Learn to use a Generative Adversarial Network to build the most popular Snapchat lens.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*-EkKEbbnFhDszaU9" /><figcaption>Image from “<a href="https://www.classicfm.com/discover-music/snapchat-gender-swapping-filter-composers/">www.classicfm.com</a>”</figcaption></figure><blockquote>We are a team of experts to help your business adopt AI solutions or build new AI products. Contact us at info@imaginaire.ai or visit our website <a href="https://www.imaginaire.ai/">https://www.imaginaire.ai</a></blockquote><p>To see the full source code of this CycleGAN, please visit <a href="https://github.com/ethanyanjiali/deep-vision/tree/master/CycleGAN/tensorflow">here</a>.</p><h3>Background</h3><p>Recently, the gender swap lens from Snapchat becomes very popular on the internet. There’re many buzzwords about Generative Adversarial Networks since 2016 but this is the first time that ordinary people get to experience the power of GANs. What’s more extraordinary about this lens is its great real-time performance which make it just like looking into a magic mirror. Although we can’t know the exact algorithm behind this virus lens, it’s most likely a CycleGAN which is introduced in 2017 by Jun-Yan, Taesung, Phillip and Alexei in this <a href="https://arxiv.org/abs/1703.10593">paper</a>. And in this article, I’m going to show you how to implement a gender swap effect with TensorFlow 2.0 just like Snapchat does.</p><h3>Generative Adversarial Network</h3><p>First of all, let’s quickly go over the basics of Generative Adversarial Network (GAN) to help readers who are not familiar with it. In some scenario, we want to generate an image that belongs to a particular domain. For example, we’d like to draw a random interior design photo, and we ask the computer to generate such an image. In order to do so, we need a mathematical representation of the interior design domain space. Assume there’s a function F, and a random input number x. We want y = F(x) to always be very close to our target domain Y. However, this target domain is in a very high dimensional space so that no human-being can figure out explicit rules to define it. A GAN is such kind of a network, by playing a minimax game between two AI agent, it can eventually find out an approximate representation F of our target domain Y.</p><p>So how does GAN accomplish it? The trick here is to break down the problem into two parts: 1. We need a generator to keep making new images out of some random seed 2. We need a discriminator to give feedback for the generator about how good the generated image is. The generator here is just like a young artist who has no idea how to paint but want to fake some masterpiece, and the discriminator is a judge who can tell what’s wrong in the new paint. The judge doesn’t need to know how to paint by himself. However, as long as he’s good at telling the difference between a good one and a bad one, our young painter can benefit from his feedback for sure. So we use Deep Learning to build a good judge and use it to train a good painter in the meantime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AIKL4r6EiybUnjfr.jpeg" /><figcaption>Diagram by myself</figcaption></figure><p>To train a good judge, we need to feed both the authentic image and the generated image to our discriminator together. Since we know which is authentic and which is fake beforehand, the discriminator can update its weights by comparing its decision and the truth. For the generator, it takes a look at the decision from the discriminator. If the discriminator seems more agreeable with the fake image, it indicates that the generator is heading in the right direction, and vice versa. The tricky part is that we can’t just train a judge without a painter, or a painter without a judge. They learn from each other and try to beat each other by playing this minimax game. If we train the discriminator too much without training the generator, the discriminator will become too dominant, and the generator won’t ever have a chance to improve because every move is a losing move.</p><p>Eventually, both generator and discriminator will be really good at their job. And now we can take the generator out to perform the painting task independently. However, what on earth does this GAN has anything to do with gender swap? In fact, generating a face with a different gender is just like generate an image from a random seed number. The only difference is that we will now take the original face image as input instead of that random seed. Let’s say we want to convert a male face to a female face. We are looking for a function F, by taking a male face x, can output a value y that&#39;s very close to the real female version y_0 of that face.</p><p>The GAN approach sounds clear. But we haven’t discussed one curial caveat in the above approach. To train the discriminator, we need both true image and false image. We get the false image y from the generator, but where do we get the true image for that specific male face? We can&#39;t just use a random female face here because we want to preserve some common trait when we swap the gender and a face from a different person would ruin that. But it&#39;s also really hard to get the paired training data as well. You could go out to find those real twin brothers and sisters and take pictures of them. Or you can ask a professional dresser to &#39;turn&#39; a man into a woman. Both are very expensive. So is there a way for the model to learn the most important facial difference between man and woman from an unpaired dataset?</p><h3>CycleGAN</h3><p>Fortunately, researchers discovered ways to utilize unpair training data. One of the most famous models is called CycleGAN. The main idea behind CycleGAN is that, instead of using paired data to train the discriminator, we can form a cycle, where two generators work together to convert the image back and forth. More specifically, generator A2B first generates an image from domain A to domain B, and then generator B2A uses that as input to generate another image from domain B to domain A. We then set a goal to make sure the second image (reconstructed image) looks as close as the first input. For example, if a generator A2B first converts a horse to a zebra, then generator B2A should convert that zebra back to a horse, and the reconstructed horse image should look identical to the original one. In this way, the generator will learn to not generate some trivial changes, but only those critical differences between the two domains. Otherwise, it probably won’t be able to convert it back. With this goal setup, we can now use unpaired images as training data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L2yr-9pPZU6_ACYu.png" /><figcaption>From “<a href="https://arxiv.org/abs/1703.10593">Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks</a>”</figcaption></figure><p>In reality, we need two cycles here. Since we are training generator A2B and generator B2A together, we have to make sure both generators is improving over time; otherwise, it will still have a problem to reconstruct a good image. Moreover, as we discussed above, improving a generator means we need to improve the discriminator in the meantime. In the cycle A2B2A (A -&gt; B -&gt; A), we use discriminator A to decide if the reconstructed image is in domain A. Thus, discriminator A will be trained. Likewise, we also need a cycle B2A2B so that discriminator B can be trained as well. If both discriminator A and discriminator B are well trained, it means our generator A2B and B2A can improve too!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pE1L27S39XYH2yzE.jpeg" /><figcaption>From “<a href="https://arxiv.org/abs/1703.10593">Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks</a>”</figcaption></figure><p>There’s another great article <a href="https://hardikbansal.github.io/CycleGANBlog/">here</a> for CycleGAN for further reading. Now that you get the main idea of this network, let’s dive deep into some details.</p><h3>Optimizer</h3><p>It’s recommended <a href="https://github.com/soumith/ganhacks">here</a> that Adam is the best optimizer for GAN training. It may comes from the delicate nature of GAN, and Adam can adapt to a complex gradient territory well. The learning rate remains at 0.0002 for the first 100 epochs, then linearly decay to 0 in the next 100 epochs. Here total_batches is the number of mini-batches for each epoch because our learning rate scheduler only considers each mini batch as a step.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60b76a0bb4cfba909126de6a819d5018/href">https://medium.com/media/60b76a0bb4cfba909126de6a819d5018/href</a></iframe><h3>Generator</h3><h4>Network Structure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rkZLj8HDayu8tMiU.jpeg" /><figcaption>Diagram by myself</figcaption></figure><p>CycleGAN uses a regular generator structure. It first encodes the input image into a feature matrix by applying 2D convolutions. This is used to extract valuable feature information from local or global.</p><p>Then, six or nine layers of ResNet blocks are used to transform the features from the encoder into the features in the target domain. As we know, the skip connection in ResNet block helps the network to memorize the gradients from previous layers, which makes sure the deeper layers can still learn something. If you are not familiar with ResNet, please refer to this <a href="https://arxiv.org/abs/1512.03385">paper</a>.</p><p>Finally, a few layers of deconvolution is used as a decoder. The decoder converts the features from the target domain into an actual image from the target domain by upsampling.</p><p>Unlike the idea from VGG and Inception network, it’s recommended for a GAN to use a larger convolution kernel size like 7X7 so that it can pick up broader information instead of just focusing on details. It makes sense because when we reconstruct an image, it’s not only the details matters but also the overall pattern. Also, reflection padding is used here to improve the quality around the image border.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0b9f9d3ece1bdd28d0f788ff5c18117/href">https://medium.com/media/e0b9f9d3ece1bdd28d0f788ff5c18117/href</a></iframe><h4>Loss Functions</h4><p>There’re three types of loss we care about here:</p><ul><li>To calculate the GAN loss, we measure the L2 distance (MSE) between the generated image and the truth image.</li><li>To calculate the cyclic loss, we measure the L1 distance (MAE) between the reconstructed image from the cycle and the truth image</li><li>To calculate the identity loss, we measure the L1 distance (MAE) between the identity image and the truth image</li></ul><p>GAN loss is the typical loss we use the GANs, and I won’t discuss much here. The interesting parts are cyclic loss and identity loss. The cyclic loss measures how good the reconstructed image is, which helps both generators to catch the essential style difference between the two domains. The identity loss is optional, but it helps to avoid the generator to make unnecessary changes. The way it works is that, by applying generator A2B to a real B image, it shouldn’t make any changes as it’s already the desired outcome. According to the author, this mitigates some weird issues like background color change.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0af4873d8268056731f18f51507d707b/href">https://medium.com/media/0af4873d8268056731f18f51507d707b/href</a></iframe><p>To combine all losses, we also need to assign some weights for each loss so indicate the importance. In the paper, the author proposed two Lambda parameters, which 10x the cycle loss and 5x the identity loss. Note the usage of the GradientTape here, we record the gradient and apply gradient descent for both generators together. Here, real_a is the truth image from domain A, real_b is the truth image from domain B. fake_a2b is the generated image from domain A to domain B. and fake_b2a is the generative image from domain B to domain A.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a64d06f19016c4aaf202271d0cb299a/href">https://medium.com/media/4a64d06f19016c4aaf202271d0cb299a/href</a></iframe><h3>Discriminator</h3><h4>Network Structure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bj6fpJA7cQiboRT8.jpeg" /><figcaption>Diagram by myself</figcaption></figure><p>Similar to other GANs, the discriminator consists of some 2d convolution layers to extract features from the generated image. However, to help the generator to generate a high-resolution image, CycleGAN uses a technique called PatchGAN to created more fine-grained decision matrix instead of one decision value. Each value in this 32×32 decision matrix maps to a patch of the generated image, and indicate how real this patch is.</p><p>In fact, we don’t crop a patch of the input image during implementation. We just need to use a final convolution layer to do the job for us. Essentially, the convolution layer performs like cropping a patch.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b6201b8e9ee05f746a7038d8ac9034e3/href">https://medium.com/media/b6201b8e9ee05f746a7038d8ac9034e3/href</a></iframe><h4>Loss Function</h4><p>The loss functions for discriminators are much more straightforward. Just like typical GANs, we tell the discriminator to treat truth image as real, and the generated image as fake. So we have two losses for each discriminator loss_real and loss_fake, both have an equal effect on the final loss. In calc_gan_loss, we are comparing two matrices. Usually, the output of a discriminator is just one value between 0 and 1. However, as we mentioned above, we use a technique called PatchGAN, so the discriminator will produce one decision for each patch, which forms a 32×32 decision matrix.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02c5cc10baa9b3b3ef3becd26921ee3e/href">https://medium.com/media/02c5cc10baa9b3b3ef3becd26921ee3e/href</a></iframe><h3>Training</h3><p>Now that we have defined both models and loss functions, we can put them together and start training. By default, the eager mode is enabled in TensorFlow 2.0, so we don’t have to make the graph. However, if you are a careful person, you might found that both discriminator and generator training functions are decorated with a tf.function decorator. This is the new way introduced by TensorFlow 2.0 to replace the old tf.Session(). With this decorator, all operations within will be converted into a graph. Hence, the performance could be much better compared with the default eager mode. To learn more about tf.function, please refer to <a href="https://pgaleone.eu/tensorflow/tf.function/2019/03/21/dissecting-tf-function-part-1/">this article</a>.</p><p>One thing to mention is that, instead of feeding the generated image to the discriminator directly, we are actually using an image pool here. Each time, the image pool will randomly decide to give the discriminator a newly generated image, or a generated image from past steps. The benefit of doing this is that the discriminator can learn from other cases and sort of having a memory about the hacks the generator uses. Unfortunately, we can’t use this random image pool in graph mode at the moment, so we need to put them back to CPU when selecting a random image from the pool. This indeed introduces some cost.</p><p>The model illustrated in this article is trained on my own GTX 2080 home computer, so it’s a bit slow. On a V100 16G GPU and 64G RAM instance, though, you should be able to set the mini batch size to 4, and the trainer can process one epoch of 260 mini batches in 3 minutes for the horse2zebra dataset. So it takes about 10 hours to train a horse2zebra model fully. If you reduce the image resolution and some network parameters correspondingly, the training could be faster. The final generator is about 44mb each.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca3cf94ea00c661e84107d269a72e470/href">https://medium.com/media/ca3cf94ea00c661e84107d269a72e470/href</a></iframe><p>To see the full training script, please go visit my repo <a href="https://github.com/ethanyanjiali/deep-vision/tree/master/GANs/CycleGAN">here</a>.</p><p>Let’s see some inference results on a few datasets. Among those, horse2zebra and monet2photo is the original dataset from the paper. And the CelebA dataset is from <a href="http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html">here</a>.</p><h3>Results</h3><p>Let’s see some generated image results from our model. On the left side it’s the original image, on the right side it’s the generated one.</p><h4>horse2zebra</h4><p>Horse -&gt; Zebra</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*9edtuy_hbM3vwVby.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*iBA31BCXdqnJRMei.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*cqRkhW90oBKz87kS.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*EJQjEudZ-jZuI0a8.jpeg" /><figcaption>Images generated by myself</figcaption></figure><p>Zebra -&gt; Horse</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*kIPAxuXEsfnFFTPU.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*LsgND4HQkmzh159B.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*z4429EPlR4iJBzDv.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*86hmm1M-2AXN2PJP.jpeg" /><figcaption>Images generated by myself</figcaption></figure><h4>monet2photo</h4><p>Monet -&gt; Photo</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*8oFVIHToEXcrSS7a.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*Mkx4_rLos0ldnwPg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*-ZYrCWPvAorMHqF9.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*Hw5CEaTmSed1KHXW.jpeg" /><figcaption>Images generated by myself</figcaption></figure><p>Photo -&gt; Monet</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*_ZE8hTS5a8bmED36.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*P9Mq4nYNcFyU70j8.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*jFrWlPmG_ZO4gRxd.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*ZVdHlSArv3Yt1DdJ.jpeg" /><figcaption>Images generated by myself</figcaption></figure><h4>CelebA</h4><p>Male -&gt; Female</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*B59cPP8CIGUOkz6Y.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*2iUIK2t2Hx6g7Pg-.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*u8pmf4uTRM6C38Ap.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*z4dQpCLGS7yoPjRz.jpeg" /><figcaption>Images generated by myself</figcaption></figure><p>Female -&gt; Male</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*XqG9atik93wenNpR.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*e7sxBYPIGlIydPCH.jpeg" /><figcaption>Images generated by myself</figcaption></figure><h3>Gender Swap</h3><p>We successfully mapped a male face to a female face, but to use it in a production environment, we need to pipeline to orchestrate lots of other steps together. One naive way to do so is:</p><ol><li>Run face detection to find a bounding box and keypoints for the most dominant face in the picture.</li><li>Extend the bounding box a little bit bigger to match the training dataset distribution.</li><li>Crop the picture with this extended bounding box and run CycleGAN over it.</li><li>Patch the generated image back to the original picture</li><li>Overlay some hair, eyeliner, and beards on top of the new face picture based on the keypoints we had from the last step</li></ol><p>Here’s a <a href="https://hackernoon.com/gender-and-race-change-on-your-selfie-with-neural-nets-9a9a1c9c5c16">great article</a> that explains how this pipeline works in details.</p><h3>Questions</h3><p>Lastly, I want to throw out some questions I have. I don’t know the answers yet, but I hope those who have experiences of building similar products can share their opinions in the comments below.</p><ul><li>The CycleGAN model turns out to be 44mb, with quantization it could become 12mb but still too large. What are the effective methods to make them usable on those mobile and embedded devices?</li><li>The output image resolution isn’t great and lost much of sharpness. How to generate a bigger image such as 1024×1024 without blowing up the model size? Will a super-resolution model help in this case?</li><li>How do we know if a model is thoroughly trained and converged? The loss isn’t a good metric here, but we also don’t know what’s the “best” output. How to measure the similarity between the two styles?</li></ul><h3>References</h3><ol><li><a href="https://hardikbansal.github.io/CycleGANBlog/">Understanding and Implementing CycleGAN in TensorFlow</a></li><li>Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros,<a href="https://arxiv.org/abs/1703.10593"> Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks</a></li><li>Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, <a href="https://arxiv.org/abs/1611.07004">Image-to-Image Translation with Conditional Adversarial Networks</a></li><li><a href="https://hackernoon.com/gender-and-race-change-on-your-selfie-with-neural-nets-9a9a1c9c5c16">Gender and Race Change on Your Selfie with Neural Nets</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=359fe74ab7ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>