<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arthur Ouaknine on Medium]]></title>
        <description><![CDATA[Stories by Arthur Ouaknine on Medium]]></description>
        <link>https://medium.com/@arthur_ouaknine?source=rss-9690d2dcec12------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*aeACCyh-gRb76rWD-vxHxg.jpeg</url>
            <title>Stories by Arthur Ouaknine on Medium</title>
            <link>https://medium.com/@arthur_ouaknine?source=rss-9690d2dcec12------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 00:21:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arthur_ouaknine/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Review of Deep Learning Algorithms for Image Semantic Segmentation]]></title>
            <link>https://medium.com/@arthur_ouaknine/review-of-deep-learning-algorithms-for-image-semantic-segmentation-509a600f7b57?source=rss-9690d2dcec12------2</link>
            <guid isPermaLink="false">https://medium.com/p/509a600f7b57</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[image-classification]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Arthur Ouaknine]]></dc:creator>
            <pubDate>Tue, 11 Dec 2018 11:21:55 GMT</pubDate>
            <atom:updated>2020-10-13T13:47:03.276Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*O6mEzgZITQA80xHI7UJrUQ.png" /><figcaption>Examples of the COCO dataset for stuff segmentation. Souce: <a href="http://cocodataset.org/">http://cocodataset.org/</a></figcaption></figure><p>Deep learning algorithms have solved several computer vision tasks with an increasing level of difficulty. In my previous blog posts, I have detailled the well kwown ones: <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-image-classification-5fdbca4a05e2">image classification</a> and <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852">object detection</a>. The image semantic segmentation challenge consists in classifying each pixel of an image (or just several ones) into an instance, each instance (or category) corresponding to an object or a part of the image (road, sky, …). This task is part of the concept of scene understanding: how a deep learning model can better learn the global context of a visual content ?</p><p>The object detection task has exceeded the image classification task in term of complexity. It consists in creating bounding boxes around the objects contained in an image and classify each one of them. Most of the object detection models use anchor boxes and proposals to detect bounding box around objects. Unfortunately, just a few models take into account the entire context of an image but they only classify a small part of the information. Thus, they can’t provide a full comprehension of a scene.</p><p>In order to understand a scene, each visual information has to be associated to an entity while considering the spatial information. Several other challenges have emerged to really understand the actions in a image or a video: keypoint detection, action recognition, video captioning, visual question answering and so on. A better comprehension of the environment will help in many fields. For example, an autonomous car needs to delimitate the roadsides with a high precision in order to move by itself. In robotics, production machines should understand how to grab, turn and put together two different pieces requiring to delimitate the exact shape of the object.</p><p>In this blog post, architecture of a few previous state-of-the-art models on image semantic segmentation challenges are detailed. Note that researchers test their algorithms using different datasets (PASCAL VOC, PASCAL Context, COCO, Cityscapes) which are different between the years and use different metrics of evaluation. Thus the cited performances cannot be directly compared <em>per se</em>. Moreover, the results depend on the pretrained top network (the backbone), the results published in this post correspond to the best scores published in each paper with respect to their test dataset.</p><h3>Datasets and Metrics</h3><h4>PASCAL Visual Object Classes (PASCAL VOC)</h4><p>The PASCAL VOC dataset (2012) is well-known an commonly used for object detection and segmentation. More than 11k images compose the train and validation datasets while 10k images are dedicated to the test dataset.</p><p>The segmentation challenge is evaluated using the <a href="https://www.tensorflow.org/api_docs/python/tf/metrics/mean_iou">mean Intersection over Union (mIoU)</a> metric. The Intersection over Union (IoU) is a metric also used in object detection to evaluate the relevance of the predicted locations. The IoU is the ratio between the area of overlap and the area of union between the ground truth and the predicted areas. The mIoU is the average between the IoU of the segmented objects over all the images of the test dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*nIz4PPDt1603WEOEyg7xsQ.png" /><figcaption>Examples of the 2012 PASCAL VOC dataset for image segmentation. Source: <a href="http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html">http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html</a></figcaption></figure><h4>PASCAL-Context</h4><p>The PASCAL-Context dataset (2014) is an extension of the 2010 PASCAL VOC dataset. It contains around 10k images for training, 10k for validation and 10k for testing. The specificity of this new release is that the entire scene is segmented providing more than 400 categories. Note that the images have been annotated during three months by six in-house annotators.</p><p>The official evaluation metric of the PASCAL-Context challenge is the mIoU. Several other metrics are published by researches as the pixel Accuracy (pixAcc). Here, the performances will be compared only with the mIoU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nb81B38WgOcQVxLIRzsrFg.png" /><figcaption>Example of the PASCAL-Context dataset. Source: <a href="https://cs.stanford.edu/~roozbeh/pascal-context/*">https://cs.stanford.edu/~roozbeh/pascal-context/</a></figcaption></figure><h4>Common Objects in COntext (COCO)</h4><p>There are two COCO challenges (in 2017 and 2018) for image semantic segmentation (“object detection” and “stuff segmentation”). The “object detection” task consists in segmenting and categorizing objects into 80 categories. The “stuff segmentation” task uses data with large segmented part of the images (sky, wall, grass), they contain almost the entire visual information. In this blog post, only the results of the “object detection” task will be compared because too few of the quoted research papers have published results on the “stuff segmentation” task.</p><p>The COCO dataset for object segmentation is composed of more than 200k images with over 500k object instance segmented. It contains a training dataset, a validation dataset, a test dataset for reseachers (test-dev) and a test dataset for the challenge (test-challenge). The annotations of both test datasets are not available. These datasets contain 80 categories and only the corresponding objects are segmented. This challenge uses the same metrics than the object detection challenge: the Average Precision (AP) and the Average Recall (AR) both using the Intersection over Union (IoU).</p><p>Details about IoU and AP metrics are available in my <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852">previous blog post</a>. Such as the AP, the Average Recall is computed using multiple IoU with a specific range of overlapping values. For a fixed IoU, the objects with the corresponding test / ground truth overlapping are kept. Then the Recall metric is computed for the detected objects. The final AR metric is the average of the computed Recalls for all the IoU range values. Basically the AP and the AR metrics for segmentation works the same way with object detection excepting that the IoU is computed pixel-wise with a non rectangular shape for semantic segmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*ltungS3ZQwFEMbs7lDuI6A.png" /><figcaption>Example of the COCO dataset for object segmentation. Source: <a href="http://cocodataset.org/*">http://cocodataset.org/</a></figcaption></figure><h4>Cityscapes</h4><p>The Cityscapes dataset has been released in 2016 and consists in complex segmented urban scenes from 50 cities. It is composed of 23.5k images for training and validation (fine and coarse annotations) and 1.5 images for testing (only fine annotation). The images are fully segmented such as the PASCAL-Context dataset with 29 classes (within 8 super categories: flat, human, vehicle, construction, object, nature, sky, void). It is often used to evaluate semantic segmentation models because of its complexity. It is also well known for its similarity with real urban scenes for autonomous driving applications. The performances of semantic segmentation models are computed using the mIoU metric such as the PASCAL datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sa7Q_BFc087-ISgnlAsk4w.png" /><figcaption>Examples of the Cityscapes dataset. Top: coarse annotations. Bottom: fine annotation. Source: <a href="https://www.cityscapes-dataset.com/*">https://www.cityscapes-dataset.com/</a></figcaption></figure><h3>Fully Convolutional Network (FCN)</h3><p><a href="https://arxiv.org/pdf/1411.4038.pdf">J. Long et al. (2015)</a> have been the firsts to develop an Fully Convolutional Network (FCN) (containing only convolutional layers) trained end-to-end for image segmentation.</p><p>The FCN takes an image with an arbitrary size and produces a segmented image with the same size. The authors start by modifying well-known architectures (AlexNet, VGG16, GoogLeNet) to have a non fixed size input while replacing all the fully connected layers by convolutional layers. Since the network produces several feature maps with small sizes and dense representations, an upsampling is necessary to create an output with the same size than the input. Basically, it consists in a convolutional layer with a stride inferior to 1. It is commonly called <strong>deconvolution</strong> because it creates an output with a larger size than the input. This way, the network is trained using a pixel-wise loss. Moreover they have added skip connections in the network to combine high level feature map representations with more specific and dense ones at the top of the network.</p><p>The authors have reached a 62.2% mIoU score on the 2012 PASCAL VOC segmentation challenge using pretrained models on the 2012 ImageNet dataset. For the 2012 PASCAL VOC object detection challenge, the benchmark model called Faster R-CNN has reached 78.8% mIoU. Even if we can’t directly compare the two results (different models, different datasets and different challenges), it seems that the semantic segmentation task is more difficult to solve than the object detection task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*pXYT1g8aWtadPBVVv9LuuA.png" /><figcaption>Architecture of the FCN. Note that the skip connections are not drawn here. Souce: <a href="https://arxiv.org/pdf/1411.4038.pdf">J. Long et al. (2015)</a></figcaption></figure><h3>ParseNet</h3><p><a href="https://arxiv.org/pdf/1506.04579.pdf">W. Liu et al. (2015)</a> have published a paper explaining improvements of the FCN model of <a href="https://arxiv.org/pdf/1411.4038.pdf">J. Long et al. (2015)</a>. According to the authors, the FCN model loses the global context of the image in its deep layers by specializing the generated feature maps. The ParseNet is an end-to-end convolutional network predicting values for all the pixels at the same time and it avoids taking regions as input to keep the global information. The authors use a module taking feature maps as input. The first step uses a model to generate feature maps which are reduced into a single global feature vector with a pooling layer. This context vector is normalised using the <a href="https://en.wikipedia.org/wiki/Norm_(mathematics)">L2 Euclidian Norm</a> and it is unpooled (the output is an expanded version of the input) to produce new feature maps with the same sizes than the inital ones. The second step normalises the entire initial feature maps using the L2 Euclidian Norm. The last step concatenates the feature maps generated by the two previous steps. The normalisation is helpful to scale the concatenated feature maps values and it leads to better performances. Basically, the ParseNet is a FCN with this module replacing convolutional layers. It has obtained a 40.4% mIoU score on the PASCAL-Context challenge and a 69.8% mIoU score on the 2012 PASCAL VOC segmentation challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*4LWdZErZ-ocldECLEB42og.png" /><figcaption>Comparison between the segmentation of the FCN and the ParseNet and architecture of the ParseNet module. Source: <a href="https://arxiv.org/pdf/1506.04579.pdf)">W. Liu et al. (2015)</a></figcaption></figure><h3>Convolutional and Deconvolutional Networks</h3><p><a href="https://arxiv.org/pdf/1505.04366.pdf">H. Noh et al. (2015)</a> have released an end-to-end model composed of two linked parts. The first part is a convolutional network with a VGG16 architecture. It takes as input an instance proposal, for example a bounding box generated by an object detection model. The proposal is processed and transformed by a convolutional network to generate a vector of features. The second part is a <strong>deconvolutional network</strong> taking the vector of features as input and generating a map of pixel-wise probabilities belonging to each class. The deconvolutional network uses <strong>unpooling</strong> targeting the maxium activations to keep the location of the information in the maps. The second network also uses deconvolution associating a single input to multiple feature maps. The deconvolution expands feature maps while keeping the information dense.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*6jJcV8_1J-s7eYl3jnXWYA.png" /><figcaption>Comparison of the convolutional network layers (pooling and convolution) with the deconvolutional network layers (unpooling and deconvolution). Source: <a href="https://arxiv.org/pdf/1505.04366.pdf">H. Noh et al. (2015)</a></figcaption></figure><p>The authors have analysed deconvolution feature maps and they have noted that the low-level ones are specific to the shape while the higher-level ones help to classify the proposal. Finally, when all the proposals of an image are processed by the entire network, the maps are concatenated to obtain the fully segmented image. This network has obtained a 72.5% mIoU on the 2012 PASCAL VOC segmentation challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*XPg1eIVN2i4x4Nrqw0gPhA.png" /><figcaption>Architecture of the full network. The convolution network is based on the VGG16 architecture. The deconvolution network uses unpooling and deconvolution layers. Source: <a href="https://arxiv.org/pdf/1505.04366.pdf">H. Noh et al. (2015)</a></figcaption></figure><h3>U-Net</h3><p><a href="https://arxiv.org/pdf/1505.04597.pdf">O. Ronneberger et al. (2015)</a> have extended the FCN of <a href="https://arxiv.org/pdf/1411.4038.pdf">J. Long et al. (2015)</a> for biological microscopy images. The authors have created a network called U-net composed in two parts: a contracting part to compute features and a expanding part to spatially localise patterns in the image. The <strong>downsampling</strong> or contracting part has a FCN-like archicture extracting features with 3x3 convolutions. The <strong>upsampling</strong> or expanding part uses up-convolution (or deconvolution) reducing the number of feature maps while increasing their height and width. Cropped feature maps from the downsampling part of the network are copied within the upsampling part to avoid loosing pattern information. Finally, a 1x1 convolution processes the feature maps to generate a segmentation map and thus categorise each pixel of the input image. Since then, the U-net architecture has been widely extended in recent works (FPN, PSPNet, DeepLabv3 and so on). Note that it doesn’t use any fully-connected layer. As consequencies, the number of parameters of the model is reduced and it can be trained with a small labelled dataset (using appropriate data augmentation). For example, the authors have used a public dataset with 30 images for training during their experiments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*mi0MoA6CaqTdiYDXMPbhMQ.png" /><figcaption>Architecture of the U-net for a given input image. The blue boxes correspond to feature maps blocks with their denoted shapes. The white boxes correspond to the copied and cropped feature maps. Source: <a href="https://arxiv.org/pdf/1505.04597.pdf">O. Ronneberger et al. (2015)</a></figcaption></figure><h3>Feature Pyramid Network (FPN)</h3><p>The Feature Pyramid Network (FPN) has been developped by <a href="https://arxiv.org/pdf/1612.03144.pdf">T.-Y. Lin et al (2016)</a> and it is used in object detection or image segmentation frameworks. Its architecture is composed of a bottom-up pathway, a top-down pathway and lateral connections in order to join low-resolution and high-resolution features. The <strong>bottom-up pathway</strong> takes an image with an arbitrary size as input. It is processed with convolutional layers and downsampled by pooling layers. Note that each bunch of feature maps with the same size is called a <strong>stage</strong>, the outputs of the last layer of each stage are the features used for the <strong>pyramid level</strong>. The <strong>top-down pathway</strong> consists in upsampling the last feature maps with unpooling while enhancing them with feature maps from the same stage of the bottom-up pathway using <strong>lateral connections</strong>. These connections consist in merging the feature maps of the bottom-up pathway processed with a 1x1 convolution (to reduce their dimensions) with the feature maps of the top-down pathway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*R7r8K9Ke6QOuKyzLbwQHnA.png" /><figcaption>Detail of a top-down block process with the lateral connection and the sum of the feature maps. Source: <a href="https://arxiv.org/pdf/1612.03144.pdf">T.-Y. Lin et al (2016)</a></figcaption></figure><p>The concatenated feature maps are then processed by a 3x3 convolution to produce the output of the stage. Finally, each stage of the top-down pathway generates a prediction to detect an object. For image segmentation, the authors uses two Multi-Layer Perceptrons (MLP) to generate two masks with different size over the objets. It works similarly to Region Proposal Networks with anchor boxes (R-CNN <a href="http://islab.ulsan.ac.kr/files/announcement/513/rcnn_pami.pdf">R. Girshick et al. (2014)</a>, Fast R-CNN <a href="https://arxiv.org/pdf/1504.08083.pdf">R. Girshick et al. (2015)</a>, Faster R-CNN <a href="https://arxiv.org/pdf/1506.01497.pdf">S. Ren et al. (2016)</a> and so on). This method is efficient because it better propagates low information into the network. The FPN based on DeepMask (<a href="https://arxiv.org/pdf/1506.06204.pdf">P. 0. Pinheiro et al. (2015)</a>) and SharpMask (<a href="https://arxiv.org/pdf/1603.08695.pdf">P. 0. Pinheiro et al. (2016)</a>) frameworks achieved a 48.1% Average Recall (AR) score on the 2016 COCO segmentation challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*vUHHpEPy2Os1szpb5wDyXg.png" /><figcaption>Comparison of architectures. (a): The image is scaled with several sizes and each one is processed with convolutions to provide predictions which is computationally expansive. (b): The image has a single scale processed by a CNN with convolution an pooling layers. © Each step of the CNN is used to provide a prediction. (d) Architecture of the FPN with the bottom-up part of the left and the top-down part on the right. Source: <a href="https://arxiv.org/pdf/1612.03144.pdf">T.-Y. Lin et al (2016)</a></figcaption></figure><h3>Pyramid Scene Parsing Network (PSPNet)</h3><p><a href="https://arxiv.org/pdf/1612.01105.pdf">H. Zhao et al. (2016)</a> have developped the Pyramid Scene Parsing Network (PSPNet) to better learn the global context representation of a scene. Patterns are extracted from the input image using a feature extractor (ResNet <a href="https://arxiv.org/pdf/1512.03385.pdf">K. He et al. (2015)</a>) with a <strong>dilated network strategy</strong>¹. The feature maps feed a <strong>Pyramid Pooling Module</strong> to distinguish patterns with different scales. They are pooled with four different scales each one corresponding to a pyramid level and processed by a 1x1 convolutional layer to reduce their dimensions. This way each pyramid level analyses sub-regions of the image with different location. The outputs of the pyramid levels are upsampled and concatenated to the inital feature maps to finally contain the local and the global context information. Then, they are processed by a convolutional layer to generate the pixel-wise predictions. The best PSPNet with a pretrained ResNet (using the COCO dataset) has reached a 85.4% mIoU score on the 2012 PASCAL VOC segmentation challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*Tp9s_0BrmxeIHIyQbg_vLw.png" /><figcaption>PSPNet architecture. The input image (a) is processed by a CNN to generate feature maps (b). They feed a Pyramid Pooling Module © and a final convolutional layer generates the pixel-wise predictions. Source: <a href="https://arxiv.org/pdf/1612.01105.pdf">H. Zhao et al. (2016)</a></figcaption></figure><h3>Mask R-CNN</h3><p><a href="https://arxiv.org/pdf/1703.06870.pdf">K. He et al. (2017)</a> have released the Mask R-CNN model beating all previous benchmarks on many COCO challenges². I have already provided details about Mask R-CNN for object detection in my <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852">previous blog post</a>. As a reminder, the Faster R-CNN (<a href="https://arxiv.org/pdf/1506.01497.pdf">S. Ren et al. (2015)</a>) architecture for object detection uses a <strong>Region Proposal Network</strong> (RPN) to propose bounding box candidates. The RPN extracts <strong>Region of Interest</strong> (RoI) and a <strong>RoIPool</strong> layer computes features from these proposals in order to infer the bounding box cordinates and the class of the object. The Mask R-CNN is a Faster R-CNN with 3 output branches: the first one computes the bounding box coordinates, the second one computes the associated class and the last one computes the binary mask³ to segment the object. The binary mask has a fixed size and it is generated by a FCN for a given RoI. It also uses a <strong>RoIAlign</strong> layer instead of a RoIPool to avoid misalignments due to the quantization of the RoI coordinates. The particularity of the Mask R-CNN model is its <strong>multi-task loss</strong> combining the losses of the bounding box coordinates, the predicted class and the segmentation mask. The model tries to solve complementary tasks leading to better performances on each individual task. The best Mask R-CNN uses a ResNeXt (<a href="https://arxiv.org/pdf/1611.05431.pdf">S. Xie et al. (2016)</a>) to extract features and a FPN architecture. It has obtained a 37.1% AP score on the 2016 COCO segmentation challenge and a 41.8% AP score on the 2017 COCO segmentation challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*IPB0s9fCiXlGpM_S6Y8rWQ.png" /><figcaption>Mask R-CNN achitecture. The first layer is a RPN extracting the RoI. The second layer processes the RoI to generate feature maps. They are directly used to compute the bounding box coordinates and the predicted class. The feature maps are also processed by an FCN (third layer) to generate the binary mask. Source: <a href="https://arxiv.org/pdf/1703.06870.pdf">K. He et al. (2017)</a></figcaption></figure><h3>DeepLab, DeepLabv3 and DeepLabv3+</h3><h4>DeepLab</h4><p>Inspired by the FPN model of <a href="https://arxiv.org/pdf/1612.03144.pdf">T.-Y. Lin et al (2016)</a>, <a href="https://arxiv.org/pdf/1606.00915.pdf">L.-C. Chen et al. (2017)</a> have released DeepLab combining atrous convolution, spatial pyramid pooling and fully connected CRFs. The model presented in this paper is also called the DeepLabv2 because it is an adjustment of the initial DeepLab model (details about the inital one will not be provided to avoid redundancy). According to the authors, consecutive max-pooling and striding reduces the resolution of the feature maps in deep neural networks. They have introduced the <strong>atrous convolution</strong> which is basically the dilated convolution of <a href="https://arxiv.org/pdf/1612.01105.pdf">H. Zhao et al. (2016)</a>. It consists of filters targeting sparse pixels with a fixed rate. For example, if the rate is equal to 2, the filter targets one pixel over two in the input; if the rate equal to 1, the atrous convolution is a basic convolution. Atrous convolution permits to capture multiple scale of objects. When it is used without max-poolling, it increases the resolution of the final output without increasing the number of weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*3UoUqTo_QjrcszzF_LRLfA.png" /><figcaption>Extraction patterns comparison between standard convolution on a low resolution input (top) and atrous convolution with a rate of 2 on a high resolution input (bottom). Source: <a href="https://arxiv.org/pdf/1606.00915.pdf">L.-C. Chen et al. (2017)</a></figcaption></figure><p>The <strong>Atrous Spatial Pyramid Pooling</strong> consists in applying several atrous convolution of the same input with different rate to detect spatial patterns. The features maps are processed in separate branches and concatenated using bilinear interpolation to recovert the original size of the input. The output feeds a fully connected Conditional Random Field (CRF) (<a href="https://arxiv.org/pdf/1210.5644.pdf">Krähenbühl and V. Koltun (2012)</a>) computing edges between the features and long terme dependencies to produce the semantic segmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*iKGyx6XR2EmIutDp1Uy5bw.png" /><figcaption>Atrous Spatial Pyramid Pooling (ASPP) exploiting multiple scale of objects to classify the pixel in the center. Source: <a href="https://arxiv.org/pdf/1606.00915.pdf">L.-C. Chen et al. (2017)</a></figcaption></figure><p>The best DeepLab using a ResNet-101 as backbone has reached a 79.7% mIoU score on the 2012 PASCAL VOC challenge, a 45.7% mIoU score on the PASCAL-Context challenge and a 70.4% mIoU score on the Cityscapes challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*fNCnDjexDIJsDdmvf217ew.png" /><figcaption>DeepLab framework. Source: <a href="https://arxiv.org/pdf/1606.00915.pdf">L.-C. Chen et al. (2017)</a></figcaption></figure><h4>DeepLabv3</h4><p><a href="https://arxiv.org/pdf/1706.05587.pdf">L.-C. Chen et al. (2017)</a> have revisited the DeepLab framework to create DeepLabv3 combining cascaded and parallel modules of atrous convolutions. The authors have modified the ResNet architecture to keep high resolution feature maps in deep blocks using atrous convolutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*JtVdwr11eTQnqukZ1YeQnA.png" /><figcaption>Cascaded modules in the ResNet architecture. Source: <a href="https://arxiv.org/pdf/1706.05587.pdf">L.-C. Chen et al. (2017)</a></figcaption></figure><p>The parallel atrous convolution modules are grouped in the Atrous Spatial Pyramid Pooling (ASPP). A 1x1 convolution and batch normalisation are added in the ASPP. All the outputs are concatenated and processed by another 1x1 convolution to create the final output with logits for each pixel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*R7tiLxyeHYHMXTGJIanZiA.png" /><figcaption>Atrous Spatial Pyramid Pooling in the Deeplabv3 framework. Source: <a href="https://arxiv.org/pdf/1706.05587.pdf">L.-C. Chen et al. (2017)</a></figcaption></figure><p>The best DeepLabv3 model with a ResNet-101 pretrained on ImageNet and JFT-300M datasets has reached 86.9% mIoU score in the 2012 PASCAL VOC challenge. It also achieved a 81.3% mIoU score on the Cityscapes challenge with a model only trained with the associated training dataset.</p><h4>DeepLabv3+</h4><p><a href="https://arxiv.org/pdf/1802.02611.pdf">L.-C. Chen et al. (2018)</a> have finally released the Deeplabv3+ framework using an encoder-decoder structure. The authors have introduced the <strong>atrous separable convolution</strong> composed of a depthwise convolution (spatial convolution for each channel of the input) and pointwise convolution (1x1 convolution with the depthwise convolution as input).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*c5rVU1iEJ0jk69DEPyB1-Q.png" /><figcaption>Combinaison of Depthwise convolution (a) and Pointwise convolution (b) to create Atrous Separable Convolution (with a rate of 2). Source: <a href="https://arxiv.org/pdf/1802.02611.pdf">L.-C. Chen et al. (2018)</a></figcaption></figure><p>They have used the DeepLabv3 framework as encoder. The most performant model has a modified Xception (<a href="https://arxiv.org/abs/1610.02357">F. Chollet (2017)</a>) backbone with more layers, atrous depthwise separable convolutions instead of max pooling and batch normalization. The outputs of the ASPP are processed by a 1x1 convolution and upsampled by a factor of 4. The outputs of the encoder backbone CNN are also processed by another 1x1 convolution and concatenated to the previous ones. The feature maps feed two 3x3 convolutional layers and the outputs are upsampled by a factor of 4 to create the final segmented image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*09azMblllhMGJgZuDeJrzw.png" /><figcaption>DeepLabv3+ framework: an encoder with a backbone CNN and an ASPP produces feature representations to feed a decoder with 3x3 convolutions producing the final predicted image. Source: <a href="https://arxiv.org/pdf/1802.02611.pdf">L.-C. Chen et al. (2018)</a></figcaption></figure><p>The best DeepLabv3+ pretrained on the COCO and the JFT datasets has obtained a 89.0% mIoU score on the 2012 PASCAL VOC challenge. The model trained on the Cityscapes dataset has reached a 82.1% mIoU score for the associated challenge.</p><h3>Path Aggregation Network (PANet)</h3><p><a href="https://arxiv.org/pdf/1803.01534.pdf">S. Liu et al. (2018)</a> have recently released the Path Aggregation Network (PANet). This network is based on the Mask R-CNN and the FPN frameworks while enhancing information propagation. The feature extractor of the network uses a FPN architecture with a new <strong>augmented bottom-up pathway</strong> improving the propagation of low-layer features. Each stage of this third pathway takes as input the feature maps of the previous stage and processes them with a 3x3 convolutional layer. The output is added to the same stage feature maps of the top-down pathway using lateral connection and these feature maps feed the next stage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/1*WmN0JUMwo2WaQnorPQ2zIg.png" /><figcaption>Lateral connection between the top-down pathway and the augmented bottom-up pathway. Source: <a href="https://arxiv.org/pdf/1803.01534.pdf">S. Liu et al. (2018)</a></figcaption></figure><p>The feature maps of the augmented bottom-up pathway are pooled with a RoIAlign layer to extract proposals from all level features. An <strong>adaptative feature pooling</strong> layer processes the features maps of each stage with a fully connected layer and concatenate all the outputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*6qnH-e2HH-gYBaV_vBqzxA.png" /><figcaption>Adatative feature pooling layer. Source: <a href="https://arxiv.org/pdf/1803.01534.pdf">S. Liu et al. (2018)</a></figcaption></figure><p>The output of the adaptative feature pooling layer feeds three branches similarly to the Mask R-CNN. The two first branches uses a fully connected layer to generate the predictions of the bounding box coordinates and the associated object class. The third branch process the RoI with a FCN to <strong>predict a binary pixel-wise mask</strong> for the detected object. The authors have added a path processing the output of a convolutional layer of the FCN with a fully connected layer to improve the localisation of the predicted pixels. Finally the output of the parallel path is reshaped and concatenated to the output of the FCN generating the binary mask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*yHoAq_qBXoJeIDU_fSgtAw.png" /><figcaption>Branch of the PANet predicting the binary mask using a FCN and a new path with a fully connected layer. Source: <a href="https://arxiv.org/pdf/1803.01534.pdf)*">https://arxiv.org/pdf/1803.01534.pdf</a></figcaption></figure><p>The PANet has achieved 42.0% AP score on the 2016 COCO segmentation challenge using a ResNeXt as feature extractor. They also performed the 2017 COCO segmentation challenge with an 46.7% AP score using a ensemble of seven feature extractors: ResNet (<a href="https://arxiv.org/pdf/1512.03385.pdf">K. He et al. (2015)</a>, ResNeXt (<a href="https://arxiv.org/pdf/1611.05431.pdf">S. Xie et al. (2016)</a>) and SENet (<a href="https://arxiv.org/pdf/1709.01507.pdf">J. Hu et al.(2017)</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*hxQWJjNxADU6VlQ5xnSGDw.png" /><figcaption>PANet Achitecture. (a): Feature extractor using the FPN achitecture. (b): The new augmented bottom-up pathway added to the FPN architecture. ©: The adaptative feature pooling layer. (d): The two branches predicting the bounding box coordinated and the object class. (e): The branch predicting the binary mask of the object. The dashed lines correspond to links between low-level and high level patterns, the red one is in the FPN and consists in more than 100 layers, the green one is a shortcut in the PANet consisting of less than 10 layers. Source: <a href="https://arxiv.org/pdf/1803.01534.pdf">S. Liu et al. (2018)</a></figcaption></figure><h3>Context Encoding Network (EncNet)</h3><p><a href="https://arxiv.org/pdf/1803.08904.pdf">H. Zhang et al. (2018)</a> have created a Context Encoding Network (EncNet) capturing global information in an image to improve scene segmentation. The model starts by using a basic feature extractor (ResNet) and feeds the feature maps into a <strong>Context Encoding Module</strong> inspired from the Encoding Layer of <a href="https://arxiv.org/pdf/1612.02844.pdf">H. Zhang et al. (2016)</a>. Basically, it learns visual centers and smoothing factors to create an embedding taking into account the contextual information while highlighting class-dependant feature maps. On top of the module, scaling factors for the contextual information are learnt with a feature maps attention layer (fully connected layer). In parallel, a <strong>Semantic Encoding Loss</strong> (SE-Loss) corresponding to a binary cross-entropy loss regularizes the training of the module by detecting presence of object classes (unlike the pixel-wise loss). The outputs of the Context Encoding Module are reshaped and processed by a dilated convolution strategy while minimizing two SE-losses and a final pixel-wise loss. The best EncNet has reached 52.6% mIoU and 81.2% pixAcc scores on the PASCAL-Context challenge. It has also achieved a 85.9% mIoU score on the 2012 PASCAL VOC segmentation challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/1*akweXy62Lay-Qqx7UpiUPA.png" /><figcaption>Dilated convolution strategy. In blue the convolutional filter with D the dilatation rate. The SE-losses (Semantic Encoding Loss) are applied after the third and the fourth stages to detect object classes. A final Seg-loss (pixel-wise loss) is applied to improve the segmentation. Source: <a href="https://arxiv.org/pdf/1803.08904.pdf">H. Zhang et al. (2018)</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*TCsoZHK71wSXy5MbT8JGDw.png" /><figcaption>Architecture of the EncNet. A feature extractor generates feature maps took as input of a Context Encoding Module. The module is trained with regularisation using the Semantic Encoding Loss. The outputs of the module are processed by a dilated convolution strategy to produce the final segmention. Source: [<a href="https://arxiv.org/pdf/1803.08904.pdf">H. Zhang et al. (2018)</a></figcaption></figure><h3>Conclusion</h3><p>Image semantic segmentation is a challenge recently takled by end-to-end deep neural networks. One of the main issue between all the architectures is to take into account the global visual context of the input to improve the prediction of the segmentation. The state-of-the-art models use architectures trying to link different part of the image in order to understand the relations between the objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*MsVAhHvNuFARowPZRiGiJQ.png" /><figcaption>Overview of the scores of the models over the 2012 PASCAL VOC dataset (mIoU), the PASCAL-Context dataset (mIoU), the 2016 / 2017 COCO datasets (AP and AR) and the Cityscapes dataset (mIoU)</figcaption></figure><p>The pixel-wise prediction over an entire image allows a better comprehension of the environement with a high precision. Scene understanding is also approached with keypoint detection, action recognition, video captioning or visual question answering. To my opinion, the segmentation task combined with these other issues using multi-task loss should help to outperform the global context understanding of a scene.</p><p>Finally, I would like to thanks <a href="https://ldocao.wordpress.com/">Long Do Cao</a> for helping me with all my posts, you should check his profile if you’re looking for a great senior data scientist ;).</p><p>¹: The dilated convolutional layer has been released by [F. Yu and V. Koltun (2015)](<a href="https://arxiv.org/pdf/1511.07122.pdf">https://arxiv.org/pdf/1511.07122.pdf</a>). It is a convolutional layer with expanded filter (the neurons of the filter are no more side-by-side). A dilatation rate fixes the gap between two neurons in term of pixel. More details are provided in the DeepLab section.</p><p>²: Object detection, object segmentation and keypoint detection.</p><p>³: The Mask R-CNN model compute a binary mask for an object for a predicted class (instance-first strategy) instead of classifying each pixel into a category (segmentation-first strategy).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=509a600f7b57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Learning Model Compression for Image Analysis: Methods and Architectures]]></title>
            <link>https://medium.com/zylapp/deep-learning-model-compression-for-image-analysis-methods-and-architectures-398f82b0c06f?source=rss-9690d2dcec12------2</link>
            <guid isPermaLink="false">https://medium.com/p/398f82b0c06f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[model-compression]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <dc:creator><![CDATA[Arthur Ouaknine]]></dc:creator>
            <pubDate>Tue, 06 Mar 2018 12:27:45 GMT</pubDate>
            <atom:updated>2018-03-06T12:27:45.532Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vmExhJp3KmfloDHEgyd03w.jpeg" /><figcaption>Source: <a href="https://www.youtube.com/watch?v=B0cKJL6EQBg">https://www.youtube.com/watch?v=B0cKJL6EQBg</a></figcaption></figure><p>In my previous blog posts, I have reviewed well-known and recent deep learning algorithms for <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-image-classification-5fdbca4a05e2">image classification</a> and <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852">object detection</a>. At <a href="https://comet.li/modelreduction">Zyl</a>, we aim to embed deep learning models directly within user’s smartphones to guarantee their privacy. This induces multiple constraints for these models known to be heavy and greedy in energy. Fortunately, at <a href="https://comet.li/modelreduction">Zyl</a> the embedded models are only used for inference¹. Nevertheless, the computational cost is high even for inference and requires a lot of energy.</p><p>This blog post describes theoretical methods to reduce model size. Size reduction for deep learning models is an active field of research. Those methods are truly performant, but the specific type of machine learning models used involves extremely deep and complex architectures (<a href="http://arxiv.org/abs/1409.1556">Simonyan et al. (2014)</a>, <a href="http://arxiv.org/abs/1512.03385">He et al. (2015)</a>, <a href="http://arxiv.org/abs/1602.07261">Szegedy et al. (2016)</a>). How can we simply transform a deep model into a lighter one without decreasing drastically its performances ? Moreover, does it exist specialized architectures to build light models while achieving state-of-the-art performances ? Note that researchers test their algorithms using different datasets. Thus the cited accuracies cannot be directly compared <em>per se</em>.</p><h3>Deep Compression</h3><p><a href="http://arxiv.org/abs/1510.00149">Han et al. (2015)</a> have developed a method called <strong>Deep Compression</strong> to reduce the size of a deep learning model. Their experiments have empirically shown that the deep compression have no significant impact on the performances of the model while reducing its size by a factor from 35 to 49. The process consists in 3 independent and ingenious steps realized consecutively: network pruning, trained quantization and Huffman encoding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p7k3DwpnSMP8FLgf.png" /><figcaption><em>Pipeline of the Deep Compression with the cumulated factors of size reduction. Source: </em><a href="http://arxiv.org/abs/1510.00149"><em>Han et al. (2015)</em></a></figcaption></figure><h4>Network Pruning</h4><p>As initiated in <a href="http://arxiv.org/abs/1510.00149">Han et al. (2015)</a>, <strong>network pruning</strong> consists in fine tuning the weights of a pretrained network while keeping only the relevant ones. After training a network using a standard method, the small weights (in absolute value) below a threshold are removed. The original dense network is transformed into a sparse one preserving the most important connections. The model is retrained while fixing the removed weights to 0. The pruning process is applied again until the remaining weights have high values. <a href="http://arxiv.org/abs/1510.00149">Han et al. (2015)</a> have experimented this method reducing the number of parameters by <strong>9 times and 13 times</strong> for the AlexNet (<a href="http://dl.acm.org/citation.cfm?id=2999134.2999257">Krizhevsky et al. (2012)</a>) and the VGG16 (<a href="http://arxiv.org/abs/1409.1556">Simonyan et al. (2014)</a>) models respectively.</p><h4>Trained Quantization</h4><p>As explain in the pipeline of <a href="http://arxiv.org/abs/1510.00149">Han et al. (2015)</a>, the <strong>quantization</strong> method is applied over the weights of the model after its pruning but the two methods can be applied independently. A clustering is realized over the 32-bits weights (floats) and the centroids of each cluster are stored as floats in a “code book”. The original weights are replaced with 2-bits units indexes (integers) of the code book corresponding to the centroid of their cluster. Thus multiple weights have the same values. This is called <strong>Weight Sharing</strong> because the model have to store only the common value of multiple weights one time and create links to use them. Like the network pruning method, the quantization is applied as a loop by retraining the network and realizing gradient updates in clusters mentioned above. After the updates, the centroids are stored in 32-bits float to preserve precision. According to <a href="http://arxiv.org/abs/1510.00149">Han et al. (2015)</a> experiments, the quantization method should be able to reduce the number of weights by <strong>18 times</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/0*bgt-bLk252dMbLxq.png" /><figcaption><em>Shared weights (centroids) after clustering from quantization (top) and centroids fine tuning (bottom). Source: </em><a href="http://arxiv.org/abs/1510.00149"><em>Han et al. (2015)</em></a></figcaption></figure><h4>Huffman Encoding</h4><p>The previous step provides weights with a discrete distribution from the indexes of the centroids. The <strong>Huffman code</strong> is a well-known method for lossless data compression. It stores discrete values by occurrence and each value is associated to a common symbol (number, letter, …). The higher the frequency of a character, the fewer the storage on bits. This method reduces considerably the storage requirements using the sparse matrix indices. According to the experiments of <a href="http://arxiv.org/abs/1510.00149">Han et al. (2015)</a>, the Huffman encoding reduces the global model size by <strong>8 times</strong> to <strong>18 times</strong> while keeping the performances unchanged.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-W1HOgyQAvUpltqn.png" /><figcaption><em>Distribution of the weights after quantization (left) and indices of sparse matrix after Huffman encoding (right). Source: </em><a href="http://arxiv.org/abs/1510.00149"><em>Han et al. (2015)</em></a></figcaption></figure><h3>Weight Quantization Methods</h3><p>The quantization method of Deep Compression uses clustering to compute shared values as new weights. Simpler weight quantization methods exist reducing the size of the model and its energy consumption while preserving performances.</p><h4>Binarized Neural Networks</h4><p><a href="https://arxiv.org/abs/1602.02830">Courbariaux et al. (2016)</a> have released a method to train Binarized Neural Networks (BNN) replacing weight and activation values by binary values. The authors have binarized them in a deterministic way using a simple <a href="https://en.wikipedia.org/wiki/Sign_function">sign function</a>. They have also detailed a stochastic way using a <a href="https://arxiv.org/pdf/1603.00391.pdf">hard sigmoid function</a> computing probabilities to assign 1 and -1 values. The deterministic binarization is mostly used while the stochastic one appears sometimes for activations and only at training time.</p><p>Gradients are computed with float values because stochastic gradient descent (SGD) algorithm is sensitive to small variations. While the derivative of the sign function is almost 0 everywhere, they use an estimator of its partial derivative. It is based on a dummy variable cancelling high weight or activation values when the function is non differentiable. During the forward pass, the weights of the hidden layers are mapped into a [-1, 1] space and quantized using the sign function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/0*is7M_NxOOkux-3in.png" /><figcaption><em>Example of binary weight filter. Source: </em><a href="https://arxiv.org/abs/1602.02830"><em>Courbariaux et al. (2016)</em></a></figcaption></figure><p>Storing weights as binary values on 2-bits instead of 32-bits dramatically reduces the memory size of the model. Moreover, It decreases the accesses to the memory and arithmetic operations are replaced with bit-wise operations. This way, the computational time for training and testing is reduced as well as the energy consumption useful to embed deep learning models.</p><h4>Trained Ternary Quantization</h4><p><a href="https://arxiv.org/abs/1612.01064">Zhu et al. (2017)</a> propose a method called Trained Ternary Quantization (TTQ) reducing weight precision without significantly decreasing the accuracy. This method transforms weights to ternary values each one stored on 2-bits. The weights are quantized into three values specific to each layer: zero to cancel useless connections, a positive value and a negative value.</p><p>The authors have been inspired by the work of <a href="https://arxiv.org/abs/1605.04711">Li et al. (2016)</a>. The full-precision weights are scaled into a [-1, 1] space. Then the weights are quantized into three values (1, -1 and 0) using a threshold shared between all the layers. Finally, for each layer, the binary values (1 and -1) are trained independently with float precision using gradient descent. The values are stored in a codebook as introduced by <a href="http://arxiv.org/abs/1510.00149">Han et al. (2015)</a> with indexes learnt during the update of the full-precision weights. During testing, the initial full-precision weights are replaced by the learnt ternary weights values rounded to be stored on 2-bits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XtnGFl1_SbNe4mKy.png" /><figcaption><em>Pipeline of the Trained Ternary Quantization method. Source: </em><a href="https://arxiv.org/abs/1612.01064"><em>Zhu et al. (2017)</em></a></figcaption></figure><p>According the authors’ implementations, ResNets with TTQ method have reached better performances than the full-precision ones. This method seems to improve generalization of deeper models avoiding overfitting due to high number of parameters. They also trained a customised AlexNet² with TTQ method using the 2012 ImageNet dataset. Slightly better results than the full-precision implementation are obtained and the model size has been reduced by 16 times. The final model have sparse weights reducing time and energy consumptions for inference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lNmMCduX8n15QN8I.png" /><figcaption><em>Visualization of kernels from Ternary AlexNet trained from Imagenet. Grey, black and white colors correspond to zero, negative and positive weights respectively. Source: </em><a href="https://arxiv.org/abs/1612.01064"><em>Zhu et al. (2017)</em></a></figcaption></figure><h3>Specific Model Architectures</h3><p>Even if methods for model reduction are efficient, architectures of deep neural networks can also be optimized. In this section, recent architectures for small neural networks are developed. This type of models are known to have performances close to the state-of-the-art while having fewer number of parameters to train and infering faster.</p><h4>SqueezeNet</h4><p><a href="http://arxiv.org/abs/1602.07360">Iandola et al. (2016)</a> have been inspired from <a href="http://arxiv.org/abs/1409.4842">Szegedy et al. (2014)</a> and their “inception module” in the GoogLeNet model detailed in my <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-image-classification-5fdbca4a05e2">previous blog post</a>. They have designed a new block called <strong>Fire module</strong> w.r.t. three strategies. The convolutional layers use 1x1 filters instead of several 3x3 filters to reduce the number of parameters by <strong>9 times</strong> for each layer. The Fire module uses fewer filters than classical architecture inducing fewer number of input channels for each layer and thus less parameters. The downsampling using pooling layers is applied lately in the network (close to the output) to increase accuracy without being greedy in the number of parameters used. Formally, the Fire module is a <strong>squeeze</strong> convolutional layer followed by an <strong>expand</strong> convolutional layer. The squeeze layer contains convolutional layers with 1x1 filters with the same input and producing stacked outputs. In the same way, the expand layer contains convolutional layers with 1x1 and 3x3 filters. In a Fire module, the number of filters in the squeeze layer has to be inferior to the total number of filters in the expand layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/0*l5D0_3oYJ_DEjpRc.png" /><figcaption><em>Example of Fire module. The squeeze layer has three 1x1 conv filters. The expand layer has four 1x1 conv filters and four 3x3 conv filters. Source: </em><a href="http://arxiv.org/abs/1602.07360"><em>Iandola et al. (2016)</em></a></figcaption></figure><p>The SqueezeNet model has a specific architecture using convolutional layers, pooling layers and Fire modules. A convolutional layer takes an image as input followed by 10 Fire modules and a final convolution layer before the softmax layer. In their experiments, the authors have reached the same performances as the AlexNet model (240 MB) (<a href="http://dl.acm.org/citation.cfm?id=2999134.2999257">Krizhevsky et al. (2012)</a>) over the ImageNet challenge³ while reducing the number of parameters by <strong>50 times</strong> (4.8 MB model). After applying the Deep Compression of <a href="http://arxiv.org/abs/1510.00149">Han et al. (2015)</a> detailed above, they have also obtained the same performances than the AlexNet with <strong>510 times</strong> less parameters⁴ (0.47 MB model).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/0*4ZbtnDZ2iTP1oMKp.png" /><figcaption><em>SqueezeNet architectures. Left: classic SqueezeNet architecture (57.5% top-1 accuracy). Right: SqueezeNet with residual connection (60.4% top-1 accuracy). Source: </em><a href="http://arxiv.org/abs/1602.07360"><em>Iandola et al. (2016)</em></a></figcaption></figure><h4>MobileNet V1</h4><p><a href="http://arxiv.org/abs/1704.04861">Howard et al. (2017)</a> have developed a group of small models called MobileNets for vision applications embedded on mobile. They have built models which are a trade-offs between speed and accuracy to match different purpose.</p><p>The MobileNets architecture is base on the <strong>depth-wise separable</strong> convolution similar to factorized convolution. First, a classic convolution with a filter with a fixed size (3x3 for example) is applied over all the input producing the same number of feature maps, it is the <strong>depth-wise</strong> convolution. Then convolutions with 1x1 filters are used over each feature map, it is the <strong>point-wise</strong> convolution. Factorizing initial convolutions with 1x1 filters reduces dramatically the number of parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/0*QZMjghOwp2MPu0BL.png" /><figcaption><em>Example of depth-wise separable filter construct with a depth-wise convolution and a 1x1 point-wise convolution where </em><em>M is the depth of the input, </em><em>N the number of feature maps and </em><em>Dk the width and height size of a square feature map. Source: </em><a href="http://arxiv.org/abs/1704.04861"><em>Howard et al. (2017)</em></a></figcaption></figure><p>The first layer of a MobileNet model is a classic convolutional layer taking an image as input. Then 13 depth-wise separable convolutions (2 layers each) follow each other. An average pooling layer links the feature maps and a fully connected layer with a softmax compute the predictions. All in all, a MobileNet has 28 layers each one followed by a batch normalization and a ReLU nonlinearity except the fully connected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/0*WPio8781uAzsHaiA.png" /><figcaption><em>Left: classic convolutional layer with batch normalization and ReLU nonlinearity. Right: depth-wise separable layer with batch normalization and ReLU nonlinearity after each depth-wise and point-wise convolutions. Source: </em><a href="http://arxiv.org/abs/1704.04861"><em>Howard et al. (2017)</em></a></figcaption></figure><p>MobileNets have variable speed and accuracy depending on two hyper-parameters. The <strong>width multiplier</strong> is a value between 0 and 1 thinning each layer by reducing the number of input and output channels. The <strong>resolution multiplier</strong> is an integer value corresponding to the size of the input image and the internal representations to reduce the computational cost. With the highest values of width and resolution multipliers, the MobileNet model has 4.2 million parameters and <a href="http://arxiv.org/abs/1704.04861">Howard et al. (2017)</a> have published a top-1 score of 70.6% over the ImageNet challenge. With a width multiplier of 0.25[⁵] and a fixed resolution multiplier of 1, the top-1 accuracy is 50.6% with 0.5 million parameters and a model 14 times faster than the previous one. If the width multiplier is fixed and the resolution multiplier is decreased, the number of parameters doesn’t change but the models are faster for a slight decrease of the accuracy. A trade-off between these two hyperparameters can be found for different applications and their expectations.</p><h4>SEP-Net</h4><p><a href="http://arxiv.org/abs/1706.03912">Li et al. (2017)</a> have focused their work on both side: model compression method and block architecture for small CNNs.</p><p>They propose a new approach called <strong>pattern binarization</strong> to binarize specific weights instead of quantize or binarize all of them. Only kxk (k&gt;1) filters of convolutional layers used for pattern extraction are binarized. Note that 1x1 filters are used for data transformation and they need to keep information as much as possible. Thus the method is the following: they train their model the usual way, then they binarize the kxk convolutional filters and finally they fine tune the model with binary values for kxk filters and floating points for 1x1 filters using the same dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*WSmKSLW1014if5sO.png" /><figcaption><em>Example of binarized 3x3 convolutional filter. Source: </em><a href="http://arxiv.org/abs/1706.03912"><em>Li et al. (2017)</em></a></figcaption></figure><p>To design a new block for small CNNs, the authors have developed a <strong>Pattern Residual Block</strong> (PRB). It is composed of a convolutional layer with 1x1 filters and a convolutional layer with 3x3 filters executed in parallel. The feature maps produced by the two layers are summed together. Moreover they use group-wise convolution, splitting the input feature maps of a convolutional features in N (N=4 in their implementation). The convolutions are applied on each of this subgroup of input data. This method reduces the number of parameter by <strong>N times</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tM6_JOTCtRo5ScSw.png" /><figcaption><em>Left: Pattern Residual Block, Right: Group Wise Convolution. Source: </em><a href="http://arxiv.org/abs/1706.03912"><em>Li et al. (2017)</em></a></figcaption></figure><p>The <strong>SEP-Net module</strong> consists of a convolutional layer with 1x1 filters followed by two PRBs and a final convolutional layer with 1x1 filters. The first layer is a dimension reduction layer while the last one is a dimension recovery layer. This way, a residual connection is constructed between the input of the block and the output of the last layer. The SEP-Net model alternates SEP-Net module and convolutional layer to feed a fully connected layer with a softmax for classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tKh4cKWM_bK94IBh.png" /><figcaption><em>Comparison between well-known architecture of blocks. Source: </em><a href="http://arxiv.org/abs/1706.03912"><em>Li et al. (2017)</em></a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x1rpBRXjUNBwYwuT.png" /><figcaption><em>Architecture of the SEP-Net model. Source: </em><a href="http://arxiv.org/abs/1706.03912"><em>Li et al. (2017)</em></a></figcaption></figure><p><a href="http://arxiv.org/abs/1706.03912">Li et al. (2017)</a> have highlighted the relevance of the pattern binarization by reducing the number of parameters of the GoogLeNet model by almost <strong>2 times</strong>. It has only decreased the top-1 accuracy by 3% over the 2012 ImageNet challenge. The SEP-Net model has a top-1 accuracy of 65.8% close to the reduced GoogLeNet for a 5.2 MB model with 1.3 million parameters. According to their implementation, the SqueezeNet model (1.2 million parameters, 4.8 MB) has a top-1 accuracy of 60.4% and the MobileNet (1.3 million parameters, 5.2 MB) has an accuracy of 63.7% which made the SEP-Net the state-of-the-art of small CNNs.</p><h4>MobileNet V2</h4><p><a href="https://arxiv.org/abs/1801.04381">Sandler et al. (2018)</a> have recently released a new version of the MobileNet model introduced by <a href="http://arxiv.org/abs/1704.04861">Howard et al. (2017)</a>. In their new architecture block, they have also used depth wise separable convolutions (detailed above) with 3x3 filters. Linear transformations are inserted between convolution blocks. It correponds to 1x1 convolutions over the depth of the input without activation function. They have called it <strong>linear bottleneck</strong> to prevent non-linear transformations to dammage information. This type of layers reduces the depth of the input feature maps. Another type of layer called <strong>expansion convolution</strong> are used to increased depth of feature maps from a linear bottleneck. In this layer, The ratio between the input and the output depths is called the <strong>expansion ratio</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vgVBXS3J-yBBR5L2.png" /><figcaption><em>Depth wise separable convolutions with linear bottleneck and expansion layer. The diagonally hatched layers do not contain non-linear transformation and depth of the blocks correspond to the relative number of channels. Source: </em><a href="https://arxiv.org/abs/1801.04381"><em>Sandler et al. (2018)</em></a></figcaption></figure><p>The linear bottleneck contains aggregated information from linear transformations. It is used as an input of an expansion convolution followed by depth wise separable convolutions. Finally, the outputs feed a new linear bottleneck producing a tensor with the same dimensions than the input. This way, the input and the output bottlenecks can be summed with residual connections allowing backpropagation in multiple layers during gradient descent. In this architecture, bottlenecks are connected with each other instead of the large depth layers, that is why the authors called it “<strong>Inverted residual block</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*QEKXZ6bSLKbHwfun.png" /><figcaption><em>Comparison between residual and inversted residual blocks. The diagonally hatched layers do not contain non-linear transformation and depth of the blocks correspond to the relative number of channels. Source: </em><a href="https://arxiv.org/abs/1801.04381"><em>Sandler et al. (2018)</em></a></figcaption></figure><p>The MobileNetV2 is composed of a fully connected layer followed by 19 inverted residual blocks with a fixed expansion ratio. Width and resolution multipliers of the initial MobileNet model also control respectively the number of feature maps and the resolution of the input in the new architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/0*uaJ6h0_n_uEYc8zV.png" /><figcaption><em>MobileNetV2 layers. Source: </em><a href="https://arxiv.org/abs/1801.04381"><em>Sandler et al. (2018)</em></a></figcaption></figure><p>This model has been trained with different datasets to solve several tasks. It has reached better performances than the MobileNetV1 on the 2012 ImageNet dataset for <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-image-classification-5fdbca4a05e2">image classification</a> with less parameters. <a href="https://arxiv.org/abs/1801.04381">Sandler et al. (2018)</a> have also used a SSDLite model for <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852">object detection</a> replacing convolutions with separable convolutions. According to the authors, they have combined SSDLite and MobileNetV2 models and trained them with the COCO dataset. They have achieved better performances than the YOLOv2 model for real-time object detection while reducing the number of parameters by 10 times. The same way, they have adapted an existing model for image segmentation, combined it with MobileNetV2 and train them over the 2012 PASCAL VOC dataset. They have obtained worse performances than a benchmark model but they have reduced the number of parameters by almost 30 times⁶ allowing embedding on mobile.</p><h3>Conclusion</h3><p>To conclude, model reduction is essential for deep learning embedded on mobile. We could use either deep compression methods, specific architectures with a low number of parameters, or both. Both methods create small models with high accuracy potential. However, a trade-off is made between lightweight and performances, and the good balance will depend upon the application.</p><p>If you want to embed deep learning on mobile, I guess you should have a look at your challenges regarding app weight, models performances, business constraints, etc… and make a decision. At Zyl, for exemple, we needed to quickly release our smart search feature on iOS while maintaining an app weight under 150MB. Our issue was that the embedded model made the app weight +300MB — but fortunately we found a secret sauce to maintain high accuracy. It’s up to you to balance all your company/project constraints to find the right method.</p><p>¹: Although, we might consider later training a model for each individual.</p><p>²: The first and the last layers keep full-precision weights while the others have ternary parameters.</p><p>³: <a href="http://arxiv.org/abs/1602.07360">Iandola et al. (2016)</a> have published a top-1 score of 57.5%.</p><p>⁴: The final model of 0.47 MB has around 0.4 million parameters.</p><p>⁵: The number of feature maps is divided by 4 times.</p><p>⁶: The considered MobileNet V2 for image segmentation has reached a mIOU score of 75.32% for a 2.11MB model while the benchmark ResNet-101 has a mIOU score of 82.70% for a 58.16MB model. Note that better MobileNet V2 performances are presented but for heavier models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=398f82b0c06f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zylapp/deep-learning-model-compression-for-image-analysis-methods-and-architectures-398f82b0c06f">Deep Learning Model Compression for Image Analysis: Methods and Architectures</a> was originally published in <a href="https://medium.com/zylapp">Zyl Story</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Review of Deep Learning Algorithms for Object Detection]]></title>
            <link>https://medium.com/zylapp/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852?source=rss-9690d2dcec12------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1f3d437b852</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <dc:creator><![CDATA[Arthur Ouaknine]]></dc:creator>
            <pubDate>Mon, 05 Feb 2018 13:47:12 GMT</pubDate>
            <atom:updated>2018-02-05T13:47:12.719Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hz6t-tokG1niaUfmcysusw.jpeg" /><figcaption>Comparison between image classification, object detection and instance segmentation.</figcaption></figure><h3>Why object detection instead of image classification?</h3><p>Image classification models detailed in my <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-image-classification-5fdbca4a05e2">previous blog post</a> classify images into a single category, usually corresponding to the most salient object. However at Zyl we are developing features that bring back old memories buried in your smartphone. Photos taken with mobile phones are usually complex and contain multiple objects. This being said, assigning a label with image classification models can become tricky and uncertain. Object detection models are therefore more appropriate to identify multiple relevant objects in a single image. The second significant advantage of object detection models versus image classification ones is that localization of the objects is provided. This wasn’t a relevant criteria for Zyl and features such as photo albums creation or duplicate detection but in other use cases it can be interesting. Think about autonomous cars or caption generation of images.</p><p>In this blog post I will review the state-of-the art of object detection models. I will provide details about the evolution of the architectures of the most accurate object detection models from 2012 up to today. One of my analysis criteria will be on their speed at inference allowing real-time analysis. Note that researchers test their algorithms using different datasets (PASCAL VOC, COCO, ImageNet) which are different between the years. Thus the cited accuracies cannot be directly compared <em>per se</em>.</p><h3>Datasets and Performance Metric</h3><p>Several datasets have been released for object detection challenges. Researchers publish results of their algorithms applied to these challenges. Specific performance metrics have been developed to take into account the spatial position of the detected object and the accuracy of the predicted categories.</p><h4>Datasets</h4><p>The <a href="http://host.robots.ox.ac.uk/pascal/VOC/">PASCAL Visual Object Classification (PASCAL VOC)</a> dataset is a well-known dataset for object detection, classification, segmentation of objects and so on. There are 8 different challenges spanning from 2005 to 2012, each of them having its own specificities. There are around 10 000 images for training and validation containing bounding boxes with objects. Although, the PASCAL VOC dataset contains only 20 categories, it is still considered as a reference dataset in the object detection problem.</p><p>ImageNet has released an object detection dataset since 2013 with bounding boxes. The training dataset is composed of around 500 000 images only for training and 200 categories. It is rarely used because the size of the dataset requires an important computational power for training. Also, the high number of classes complicates the object recognition task. A comparison between the 2014 ImageNet dataset and the 2012 PASCAL VOC dataset is available <a href="http://image-net.org/challenges/LSVRC/2014/">here</a>.</p><p>On the other hand, the <a href="http://cocodataset.org/#home">Common Objects in COntext (COCO)</a> dataset is developed by Microsoft and detailed by <a href="https://arxiv.org/pdf/1405.0312.pdf">T.-Y.Lin and al. (2015)</a>. This dataset is used for multiple challenges: caption generation, object detection, key point detection and object segmentation. We focus on the COCO object detection challenge consisting in localizing the objects in an image with bounding boxes and categorizing each one of them between 80 categories. The dataset changes each year but usually is composed of more than 120 000 images for training and validation, and more than 40 000 images for testing. The test dataset has been recently cut into the test-dev dataset for researchers and the test-challenge dataset for competitors. Both associated labeled data are not publicly available to avoid overfitting on the test dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RKkLHZ2BXkQ_Kf3ZZVBuRQ.png" /><figcaption><em>Examples of segmented objects from the 2015 COCO dataset. Source: </em><a href="https://arxiv.org/pdf/1405.0312.pdf"><em>T.-Y.Lin and al. (2015)</em></a></figcaption></figure><h4>Performance Metric</h4><p>The object detection challenge is, at the same time, a regression and a classification task. First of all, to assess the spatial precision we need to remove the boxes with low confidence (usually, the model outputs many more boxes than actual objects). Then, we use the <strong>Intersection over Union (IoU)</strong> area, a value between 0 and 1. It corresponds to the overlapping area between the predicted box and the ground-truth box. The higher the IoU, the better the predicted location of the box for a given object. Usually, we keep all bounding box candidates with an IoU greater than some threshold.</p><p>In binary classification, the Average Precision (AP) metric is a summary of the precision-recall curve, details are provided <a href="https://en.wikipedia.org/w/index.php?title=Information_retrieval&amp;oldid=793358396#Average_precision">here</a>. The commonly used metric used for object detection challenges is called the <strong>mean Average Precision (mAP)</strong>. It is simply the mean of the Average Precisions computed over all the classes of the challenge. The mAP metric avoids to have extreme specialization in few classes and thus weak performances in others.</p><p>The mAP score is usually computed for a fixed IoU but a high number of bounding boxes can increase the number of candidate boxes. The COCO challenge has developed an official metric to avoid an over generation of boxes. It computes a mean of the mAP scores for variable IoU values in order to penalize high number of bounding boxes with wrong classifications.</p><h3>Region-based Convolutional Network (R-CNN)</h3><p>The first models intuitively begin with the region search and then perform the classification. In R-CNN, the <strong>selective search</strong> method developed by <a href="http://www.huppelen.nl/publications/selectiveSearchDraft.pdf">J.R.R. Uijlings and al. (2012)</a> is an alternative to exhaustive search in an image to capture object location. It initializes small regions in an image and merges them with a hierarchical grouping. Thus the final group is a box containing the entire image. The detected regions are merged according to a variety of color spaces and similarity metrics. The output is a few number of region proposals which could contain an object by merging small regions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*JCTKABvEOLBTZsfT85Q2AQ.png" /><figcaption><em>Selective Search application, top: visualisation of the segmention results of the algorithm, down: visualisation of the region proposals of the algorithm. Source: </em><a href="http://www.huppelen.nl/publications/selectiveSearchDraft.pdf"><em>J.R.R. Uijlings and al. (2012)</em></a></figcaption></figure><p>The R-CNN model <a href="http://islab.ulsan.ac.kr/files/announcement/513/rcnn_pami.pdf">(R. Girshick et al., 2014)</a> combines the <strong>selective search</strong> method to detect region proposals and deep learning to find out the object in these regions. Each region proposal is resized to match the input of a CNN from which we extract a 4096-dimension vector of features. The features vector is fed into multiple classifiers to produce probabilities to belong to each class. Each one of these classes has a SVM classifier trained to infer a probability to detect this object for a given vector of features. This vector also feeds a linear regressor to adapt the shapes of the bounding box for a region proposal and thus reduce localization errors.</p><p>The CNN model described by the authors is trained on the 2012 ImageNet dataset of the original challenge of image classification. It is fine-tuned using the region proposals corresponding to an IoU greater than 0.5 with the ground-truth boxes. Two versions are produced, one version is using the 2012 PASCAL VOC dataset and the other the 2013 ImageNet dataset with bounding boxes. The SVM classifiers are also trained for each class of each dataset.</p><p>The best R-CNNs models have achieved a 62.4% mAP score over the PASCAL VOC 2012 test dataset (22.0 points increase w.r.t. the second best result on the leader board) and a 31.4% mAP score over the 2013 ImageNet dataset (7.1 points increase w.r.t. the second best result on the leader board).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*RUjYe8yqo7nKAG2lNd2mbw.png" /><figcaption><em>Region-based Convolution Network (R-CNN). Each region proposal feeds a CNN to extract a features vector, possible objects are detected using multiple SVM classifiers and a linear regressor modifies the coordinates of the bounding box. Source: </em><a href="https://towardsdatascience.com/deep-learning-for-object-detection-a-comprehensive-review-73930816d8d9"><em>J. Xu’s Blog</em></a></figcaption></figure><h3>Fast Region-based Convolutional Network (Fast R-CNN)</h3><p>The purpose of the Fast Region-based Convolutional Network (Fast R-CNN) developed by <a href="https://arxiv.org/pdf/1504.08083.pdf">R. Girshick (2015)</a> is to reduce the time consumption related to the high number of models necessary to analyse all region proposals.</p><p>A main CNN with multiple convolutional layers is taking the entire image as input instead of using a CNN for each region proposals (R-CNN). <strong>Region of Interests</strong> (RoIs) are detected with the selective search method applied on the produced feature maps. Formally, the feature maps size is reduced using a RoI pooling layer to get valid Region of Interests with fixed heigh and width as hyperparameters. Each RoI layer feeds fully-connected layers¹ creating a features vector. The vector is used to predict the observed object with a softmax classifier and to adapt bounding box localizations with a linear regressor.</p><p>The best Fast R-CNNs have reached mAp scores of 70.0% for the 2007 PASCAL VOC test dataset, 68.8% for the 2010 PASCAL VOC test dataset and 68.4% for the 2012 PASCAL VOC test dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*U95lm-Jkwkpy3p8X6IOKlQ.png" /><figcaption><em>The entire image feeds a CNN model to detect RoI on the feature maps. Each region is separated using a RoI pooling layer and it feeds fully-connected layers. This vector is used by a softmax classifier to detect the object and by a linear regressor to modify the coordinates of the bounding box. Source: </em><a href="https://towardsdatascience.com/deep-learning-for-object-detection-a-comprehensive-review-73930816d8d9"><em>J. Xu’s Blog</em></a></figcaption></figure><h3>Faster Region-based Convolutional Network (Faster R-CNN)</h3><p>Region proposals detected with the selective search method were still necessary in the previous model, which is computationally expensive. <a href="https://arxiv.org/pdf/1506.01497.pdf">S. Ren and al. (2016)</a> have introduced <strong>Region Proposal Network</strong> (RPN) to directly generate region proposals, predict bounding boxes and detect objects. The Faster Region-based Convolutional Network (Faster R-CNN) is a combination between the RPN and the Fast R-CNN model.</p><p>A CNN model takes as input the entire image and produces feature maps. A window of size 3x3 slides all the feature maps and outputs a features vector linked to two fully-connected layers, one for box-regression and one for box-classification. Multiple region proposals are predicted by the fully-connected layers. A maximum of k regions is fixed thus the output of the box-regression layer has a size of 4k (coordinates of the boxes, their height and width) and the output of the box-classification layer a size of 2k (“objectness” scores to detect an object or not in the box). The k region proposals detected by the sliding window are called anchors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*a8c0DGOsEEAtxBafLO1Zbw.png" /><figcaption><em>Detecting the anchor boxes for a single 3x3 window. Source: </em><a href="https://arxiv.org/pdf/1506.01497.pdf"><em>S. Ren and al. (2016)</em></a></figcaption></figure><p>When the anchor boxes are detected, they are selected by applying a threshold over the “objectness” score to keep only the relevant boxes. These anchor boxes and the feature maps computed by the initial CNN model feeds a Fast R-CNN model.</p><p>Faster R-CNN uses RPN to avoid the selective search method, it accelerates the training and testing processes, and improve the performances. The RPN uses a pre-trained model over the ImageNet dataset for classification and it is fine-tuned on the PASCAL VOC dataset. Then the generated region proposals with anchor boxes are used to train the Fast R-CNN. This process is iterative.</p><p>The best Faster R-CNNs have obtained mAP scores of 78.8% over the 2007 PASCAL VOC test dataset and 75.9% over the 2012 PASCAL VOC test dataset. They have been trained with PASCAL VOC and COCO datasets. One of these models² is 34 times faster than the Fast R-CNN using the selective search method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*HlLHTPDyaJZHLnSl.png" /><figcaption><em>The entire image feeds a CNN model to produce anchor boxes as region proposals with a confidence to contain an object. A Fast R-CNN is used taking as inputs the feature maps and the region proposals. For each box, it produces probabilities to detect each object and correction over the location of the box. Source: </em><a href="https://towardsdatascience.com/deep-learning-for-object-detection-a-comprehensive-review-73930816d8d9"><em>J. Xu’s Blog</em></a></figcaption></figure><h3>Region-based Fully Convolutional Network (R-FCN)</h3><p>Fast and Faster R-CNN methodologies consist in detecting region proposals and recognize an object in each region. The Region-based Fully Convolutional Network (R-FCN) released by <a href="https://arxiv.org/pdf/1605.06409.pdf">J. Dai and al. (2016)</a> is a model with only convolutional layers³ allowing complete backpropagation for training and inference. The authors have merged the two basic steps in a single model to take into account simultaneously the object detection (location invariant) and its position (location variant).</p><p>A <a href="https://arxiv.org/pdf/1512.03385.pdf">ResNet-101</a> model takes the initial image as input. The last layer outputs feature maps, each one is specialized in the detection of a category at some location. For example, one feature map is specialized in the detection of a cat, another one in a banana and so on. Such feature maps are called <strong>position-sensitive score maps</strong> because they take into account the spatial localization of a particular object. It consists of k*k*(C+1) score maps where k is the size of the score map, and C the number of classes. All these maps form the score bank. Basically, we create patches that can recognize part of an object. For example, for k=3, we can recognize 3x3 parts of an object.</p><p>In parallel, we need to run a RPN to generate Region of Interest (RoI). Finally, we cut each RoI in bins and we check them against the score bank. If enough of these parts are activated, then the patch vote ‘yes’, I recognized the object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JFtFIzpDhb3KsN1jran6yA.png" /><figcaption><em>The input image feeds a ResNet model to produce feature maps. A RPN model detects the Region of Interests and a score is computed for each region to determine the most likely object if there is one. Source: </em><a href="https://arxiv.org/pdf/1605.06409.pdf"><em>J. Dai and al. (2016)</em></a></figcaption></figure><p><a href="https://arxiv.org/pdf/1605.06409.pdf">J. Dai and al. (2016)</a> have detailed an example displayed below. The figures show the reaction of a R-FCN model specialized in detecting a person. For a RoI in the center of the image (Figure 3), the subregions in the feature maps are specific to the patterns associated to a person. Thus they vote for ‘yes, there is a person at this location’. In the Figure 4, the RoI is shifted to the right and it is no longer centred on the person. The subregions in the feature maps do not agree on the person detection, thus they vote ‘no, there is no person at this location’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*492ImpC37vSxbGupMbpYnQ.png" /><figcaption><em>Source: </em><a href="https://arxiv.org/pdf/1605.06409.pdf"><em>J. Dai and al. (2016)</em></a></figcaption></figure><p>The best R-FCNs have reached mAP scores of 83.6% for the 2007 PASCAL VOC test dataset and 82.0%, they have been trained with the 2007, 2012 PASCAL VOC datasets and the COCO dataset. Over the test-dev dataset of the 2015 COCO challenge, they have had a score of 53.2% for an IoU = 0.5 and a score of 31.5% for the official mAP metric. The authors noticed that the R-FCN is 2.5–20 times faster than the Faster R-CNN counterpart.</p><h3>You Only Look Once (YOLO)</h3><p>The YOLO model <a href="https://arxiv.org/pdf/1506.02640.pdf">(J. Redmon et al., 2016)</a>) directly predicts bounding boxes and class probabilities with a single network in a single evaluation. The simplicity of the YOLO model allows real-time predictions.</p><p>Initially, the model takes an image as input. It divides it into an SxS grid. Each cell of this grid predicts B bounding boxes with a confidence score. This confidence is simply the probability to detect the object multiply by the IoU between the predicted and the ground truth boxes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*n09xW-miKM_0M62a8VsVjw.png" /><figcaption><em>Example of application. The input image is divided into an SxS grid, B bounding boxes are predicted (regression) and a class is predicted among C classes (classification) over the most confident ones. Source: </em><a href="https://arxiv.org/pdf/1506.02640.pdf"><em>J. Redmon and al. (2016)</em></a></figcaption></figure><p>The CNN used is inspired by the <a href="https://arxiv.org/pdf/1409.4842.pdf">GoogLeNet</a> model introducing the inception modules. The network has 24 convolutional layers followed by 2 fully-connected layers. Reduction layers with 1x1 filters⁴ followed by 3x3 convolutional layers replace the initial inception modules. The Fast YOLO model is a lighter version with only 9 convolutional layers and fewer number of filters. Most of the convolutional layers are pretrained using the ImageNet dataset with classification. Four convolutional layers followed by two fully-connected layers are added to the previous network and it is entirely retrained with the 2007 and 2012 PASCAL VOC datasets.</p><p>The final layer outputs a S*S*(C+B*5) tensor corresponding to the predictions for each cell of the grid. C is the number of estimated probabilities for each class. B is the fixed number of anchor boxes per cell, each of these boxes being related to 4 coordinates (coordinates of the center of the box, width and height) and a confidence value.</p><p>With the previous models, the predicted bounding boxes often contained an object. The YOLO model however predicts a high number of bounding boxes. Thus there are a lot of bounding boxes without any object. The <strong>Non-Maximum Suppression</strong> (NMS) method is applied at the end of the network. It consists in merging highly-overlapping bounding boxes of a same object into a single one. The authors noticed that there are still few false positive detected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBVa18sgJ2_L9Azxngibbw.png" /><figcaption><em>YOLO architecture: it is composed of 24 convolutional layers and 2 fully-connected layers. Source: </em><a href="https://arxiv.org/pdf/1506.02640.pdf"><em>J. Redmon and al. (2016)</em></a></figcaption></figure><p>The YOLO model has a 63.7% mAP score over the 2007 PASCAL VOC dataset and a 57.9% mAP score over the 2012 PASCAL VOC dataset. The Fast YOLO model has lower scores but they have both real time performances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*TVBjict_R-4XYytl5ALaCg.png" /><figcaption>Real Time Systems on PASCAL VOC 2007. Comparison of speeds and performances for models trained with the 2007 and 2012 PASCAL VOC datasets. The published results correspond to the implementations of <a href="https://arxiv.org/pdf/1506.02640.pdf">J. Redmon and al. (2016)</a>.</figcaption></figure><h3>Single-Shot Detector (SSD)</h3><p>Similarly to the YOLO model, <a href="https://arxiv.org/pdf/1512.02325.pdf">W. Liu et al. (2016)</a> have developed a Single-Shot Detector (SSD) to predict all at once the bounding boxes and the class probabilities with a end-to-end CNN architecture.</p><p>The model takes an image as input which passes through multiple convolutional layers with different sizes of filter (10x10, 5x5 and 3x3). Feature maps from convolutional layers at different position of the network are used to predict the bounding boxes. They are processed by a specific convolutional layers with 3x3 filters called extra feature layers to produce a set of bounding boxes similar to the anchor boxes of the Fast R-CNN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f0p4it3vSVV_qeTJq5Jv1Q.png" /><figcaption><em>Comparison between the SSD and the YOLO architectures. The SSD model uses extra feature layers from different feature maps of the network in order to increase the number of relevant bounding boxes. Source: </em><a href="https://arxiv.org/pdf/1512.02325.pdf"><em>W. Liu and al. (2016)</em></a></figcaption></figure><p>Each box has 4 parameters: the coordinates of the center, the width and the height. At the same time, it produces a vector of probabilities corresponding to the confidence over each class of object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9juuB8HOBnoNqvEruiCT2A.png" /><figcaption><em>SSD Framework. (a) The model takes an image and its ground truth bounding boxes. Small sets of boxes with different aspect ratios are fixed by the different feature map ((b) and ©). During training, the boxes localization are modified to best match the ground truth. Source: </em><a href="https://arxiv.org/pdf/1512.02325.pdf"><em>W. Liu and al. (2016)</em></a></figcaption></figure><p>The Non-Maximum Suppression method is also used at the end of the SSD model to keep the most relevant bounding boxes. The <strong>Hard Negative Mining</strong> (HNM) is then used because a lot of negative boxes are still predicted. It consists in selecting only a subpart of these boxes during the training. The boxes are ordered by confidence and the top is selected depending on the ratio between the negative and the positive which is at most 1/3.</p><p><a href="https://arxiv.org/pdf/1512.02325.pdf">W. Liu et al. (2016)</a> distinguish the SSD300 model (the architecture is detailed on the figure above) and the SSD512 model which is the SSD300 with an extra convolutional layer for prediction to improve performances. The best SSDs models are trained with the 2007, 2012 PASCAL VOC datasets and the 2015 COCO dataset with data augmentation. They have obtained mAP scores of 83.2% over the 2007 PASCAL VOC test dataset and 82.2% over the 2012 PASCAL VOC test dataset. Over the test-dev dataset of the 2015 COCO challenge, they have had a score of 48.5% for an IoU = 0.5, 30.3% for an IoU = 0.75 and 31.5% for the official mAP metric.</p><h3>YOLO9000 and YOLOv2</h3><p><a href="https://arxiv.org/pdf/1612.08242.pdf">J. Redmon and A. Farhadi (2016)</a> have released a new model called YOLO9000 capable of detecting more than 9000 object categories while running in almost real time according to the authors. They also provide improvements over the initial YOLO model to improve its performances without decreasing its speed (around 10 images per second on a recent mobile according to our implementation).</p><h4>YOLOv2</h4><p>The YOLOv2 model is focused on improving accuracy while still being a fast detector. Batch normalization is added to prevent overfitting without using dropout. Higher resolution images are accepted as input: the YOLO model uses 448x448 images while the YOLOv2 uses 608x608 images, thus enabling the detection of potentially smaller objects.</p><p>The final fully-connected layer of the YOLO model predicting the coordinates of the bounding boxes has been removed to use anchor boxes in the same way as Faster R-CNN. The input image is reduced to a grid of cells, each one containing 5 anchor boxes. YOLOv2 uses 19*19*5=1805 anchor boxes by image instead of 98 boxes for the YOLO model. YOLOv2 predicts correction of the anchor box relative to the location of the grid cell (the range is between 0 and 1) and selects the boxes according to their confidence as the SSD model. The dimensions of the anchor boxes has been fixed using k-means on the training set of bounding boxes.</p><p>It uses a ResNet-like architecture to stack high and low resolution feature maps to detect smaller objects. The “Darknet-19” is composed of 19 convolutional layers with 3x3 and 1x1 filters, groups of convolutional layers are followed by maxpooling layers to reduce the output dimension. A final 1x1 convolutional layer outputs 5 boxes per cell of the grid with 5 coordinates and 20 probabilities each (the 20 classes of the PASCAL VOC dataset).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*0tJvHdI1tt3xBACl7KB9cQ.png" /><figcaption><em>YOLOv2 architecture. Source: </em><a href="https://arxiv.org/pdf/1612.08242.pdf"><em>J. Redmon and A. Farhadi (2016)</em></a></figcaption></figure><p>The YOLOv2 model trained with the 2007 and 2012 PASCAL VOC dataset has a 78.6% mAP score over the 2007 PASCAL VOC test dataset with a FPS value of 40. The model trained with the 2015 COCO dataset have mAP scores over the test-dev dataset of 44.0% for an IoU = 0.5, 19.2% for an IoU = 0.75 and 21.6% for the official mAP metric.</p><h4>YOLO9000</h4><p>The authors have combined the ImageNet dataset with the COCO dataset in order to have a model capable of detecting precise objects or animal breed. The ImageNet dataset for classification contains 1000 categories and the 2015 COCO dataset only 80 categories. The ImageNet classes are based on the WordNet lexicon developed by the Princeton University⁵ which is composed of more than 20 000 words. <a href="https://arxiv.org/pdf/1612.08242.pdf">J. Redmon and A. Farhadi (2016)</a> detail a method to construct a tree version of the WordNet. A softmax is applied on a group of labels with the same hyponym when the model predicts on an image. Thus the final probability associated to a label is computed with posterior probabilities in the tree. When the authors extend the concept to the entire WordNet lexicon excluding under-represented categories, they obtain more than 9 000 categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GwF8cFlvUvOCq7ZMUtvMJg.png" /><figcaption><em>Prediction on ImageNet vs WordTree. Source: </em><a href="https://arxiv.org/pdf/1612.08242.pdf"><em>J. Redmon and A. Farhadi (2016)</em></a></figcaption></figure><p>A combination between the COCO and the ImageNet datasets is used to train a YOLOv2-like architecture with 3 prior convolution layers instead of 5 to limit the output size. The model is evaluated on the ImageNet dataset for the detection task with around 200 labels. Only 44 labels are shared between the training and the testing dataset so the results are somewhat irrelevant. It gets a 19.7% mAP score overall the test dataset.</p><h3>Neural Architecture Search Net (NASNet)</h3><p>The Neural Architecture Search <a href="https://arxiv.org/pdf/1611.01578.pdf">(B. Zoph and Q.V. Le, 2017)</a> is detailed in my <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-image-classification-5fdbca4a05e2">previous post</a>. It consists in learning the architecture of a model to optimize the number of layers while improving the accuracy over a given dataset. <a href="https://arxiv.org/pdf/1707.07012.pdf">B. Zoph et al. (2017)</a> have reached higher performances with lighter model than previous works over the 2012 ImageNet classification challenge.</p><p>The authors have applied this method to spatial object detection. The NASNet network has an architecture learned from the CIFAR-10 dataset and is trained with the 2012 ImageNet dataset. This model is used for feature maps generation and is stacked into the Faster R-CNN pipeline. Then the entire pipeline is retrained with the COCO dataset.</p><p>The best NASNet models for object recognition have obtained a 43.1% mAP score over the test-dev dataset of the COCO challenge with an IoU = 0.5. The lighter version of the NASNet optimized for mobile have a 29.6% mAP score over the same dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*u8w4Ocl_1zF8Nu3zHRuILg.png" /><figcaption><em>Example of object detection results. Comparison of Faster R-CNN pipelines one is using </em><a href="http://arxiv.org/abs/1602.07261"><em>Inception-ResNet</em></a><em> as feature maps generator (top) and the other the NASNet model (bottom). Source: </em><a href="https://arxiv.org/pdf/1707.07012.pdf"><em>B. Zoph and al. (2017)</em></a></figcaption></figure><h3>Mask Region-based Convolutional Network (Mask R-CNN)</h3><p>Another extension of the Faster R-CNN model has been released by <a href="https://arxiv.org/pdf/1703.06870.pdf">K. He and al. (2017)</a> adding a parallel branch to the bounding box detection in order to predict object mask. The mask of an object is its segmentation by pixel in an image. This model outperforms the state-of-the-art in the four COCO challenges: the instance segmentation, the bounding box detection, the object detection and the key point detection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IX55uRz8s-E79AvNhpsBmg.png" /><figcaption><em>Examples of Mask R-CNN application on the COCO test dataset. The model detects each object of an image, its localization and its precise segmentation by pixel. Source: </em><a href="https://arxiv.org/pdf/1703.06870.pdf"><em>K. He and al. (2017)</em></a></figcaption></figure><p>The Mask Region-based Convolutional Network (Mask R-CNN) uses the Faster R-CNN pipeline with three output branches for each candidate object: a class label, a bounding box offset and the object mask. It uses Region Proposal Network (RPN) to generate bounding box proposals and produces the three outputs at the same time for each Region of Interest (RoI).</p><p>The initial RoIPool layer used in the Faster R-CNN is replaced by a <strong>RoIAlign</strong> layer. It removes the quantization of the coordinates of the original RoI and computes the exact values of the locations. The RoIAlign layer provides scale-equivariance and translation-equivariance with the region proposals.</p><p>The model takes an image as input and feeds a <a href="https://arxiv.org/pdf/1611.05431.pdf">ResNeXt</a> network with 101 layers. This model looks like a ResNet but each residual block is cut into lighter transformations which are aggregated to add sparsity in the block. The model detects RoIs which are processed using a RoIAlign layer. One branch of the network is linked to a fully-connected layer to compute the coordinates of the bounding boxes and the probabilities associated to the objects. The other branch is linked to two convolutional layers, the last one computes the mask of the detected object.</p><p>Three loss functions associated to each task to solve are summed. This sum is minimized and produces great performances because solving the segmentation task improve the localization and thus the classification.</p><p>The Mask R-CNN have reached mAP scores of 62.3% for an IoU = 0.5, 43.4% for an IoU = 0.7 and 39.8% for the official metric over the 2016 COCO test-dev dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cje-Cm-RO_1hCe1YFxpd1Q.png" /><figcaption><em>Mask R-CNN framework for instance segmentation. Source: </em><a href="https://arxiv.org/pdf/1703.06870.pdf"><em>K. He and al. (2017)</em></a></figcaption></figure><h3>Conclusion</h3><p>Through the years, object detection models tend to infer localization and classification all at once to have an entirely differentiable network. Thus it can be trained from head to tail with backpropagation. Moreover, A trade-off between high performance and realtime prediction capability is made between the recent models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kcp1fWeJ3cD5pRvLXi9w5A.png" /><figcaption>Overview of the mAP scores on the 2007, 2010, 2012 PASCAL VOC dataset and 2015, 2016 COCO datasets.</figcaption></figure><p>The models presented in this blog post are either accurate or fast for inference. However, they all have complex and heavy architectures. For example, the YOLOv2 model is around 200MB and the best NASNet around 400MB. Reduction of size while keeping the same performances is an active field of research to embed deep learning models into mobile devices. Some details will be provided in an upcoming post.</p><p>¹: The entire architecture is inspired from the VGG16 model, thus it has 13 convolutional layers and 3 fully-connected layers.</p><p>²: The fastest Faster R-CNN has an architecture inspired by the ZFNet model introduced by <a href="https://arxiv.org/pdf/1311.2901.pdf">M.D. Zeiler and R. Fergus (2013)</a>. The commonly used Faster R-CNN has an architecture similar to the VGG16 model and it is 10 times faster than the Fast R-CNN.</p><p>³: Except the last layer which is a fully connected layer.</p><p>⁴: It reduces the features space from the previous layers.</p><p>⁵: Details are provided in the <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-image-classification-5fdbca4a05e2">previous blog post</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1f3d437b852" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zylapp/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852">Review of Deep Learning Algorithms for Object Detection</a> was originally published in <a href="https://medium.com/zylapp">Zyl Story</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Review of Deep Learning Algorithms for Image Classification]]></title>
            <link>https://medium.com/zylapp/review-of-deep-learning-algorithms-for-image-classification-5fdbca4a05e2?source=rss-9690d2dcec12------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fdbca4a05e2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Arthur Ouaknine]]></dc:creator>
            <pubDate>Tue, 16 Jan 2018 14:41:26 GMT</pubDate>
            <atom:updated>2018-01-16T14:41:26.977Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*v64BvvzbeJvhSen1X7IPNA.jpeg" /></figure><h3>Why do we need image classification?</h3><p>In the <a href="https://medium.com/comet-app/computer-vision-in-mobile-phones-d6b3d237dcee">previous post</a>, we praised the advantages of embedded deep learning algorithms into mobile phones. While applications are numerous, we will focus on computer vision algorithms, the heart of <a href="https://zyl.ai/">Zyl</a>. One of the most popular task of such algorithms is image classification, i.e. telling which object appears on a picture. Indeed mobile phones host a diverse and rich photo gallery which then become a personal database difficult to manage especially to recover specific events. Users should be able to have a memory and find the associated images in the most efficient way. A first intuitive approach would be to type in a word corresponding to the content of the image. Searching images with words is, from a machine learning point of view, a classification problem with a high number of classes.</p><p>The purpose of this post is to provide a review of the state-of-the-art of image classification algorithms based on the most popular labelled dataset, ImageNet. We will describe some of the innovative architectures which lead to significant improvements.<br>Note that researchers test their algorithms using different datasets (a new ImageNet dataset is released as a new challenge with different images each year). Thus the cited accuracies cannot be directly compared<em> per se</em>.</p><h3>The ImageNet challenge</h3><p>The ImageNet database is the outcome of a collaboration between Stanford University and Princeton University and has become a reference in the field of computer vision. It contains around fourteen millions images originally labeled with Synsets¹ of the WordNet lexicon tree. The original challenge<br>consisted in a simple classification task, each image belonging to a single<br>category among one thousand, from specific breed of dog to precise<br>type of food. Although the original challenge is still on-going, it has further evolved to a multi-classification task with bounding boxes around each individual object. This second challenge will not be covered in this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*RYTTd-jftnd1oP5PtU1StQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*vt1f7nNj9njnouhqzmJcIA.jpeg" /><figcaption>Example of images in the 2012 ImageNet dataset. Left: Carbonara. Right: English Foxhound. Source: <a href="https://github.com/cometapp/engine/blob/master/content/article2/www.image-net.org"><em>ImageNet</em></a></figcaption></figure><h3>The advent of deep learning</h3><p>The ImageNet challenge has been traditionally tackled with image analysis algorithms such as SIFT with mitigated results until the late 90s. However, a gap in performance has been brought by using neural networks. Inspired by <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf">Y. Lecun et al. (1998)</a>, the first deep learning model published by <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks">A. Krizhevsky et al. (2012)</a> drew attention to the public by getting a top-5 error rate of 15.3% outperforming the previous best one with an accuracy of 26.2% using a SIFT model. This famous model, the so-called “AlexNet” is what can be considered today as a simple architecture with five consecutive convolutional filters, max-pool layers and three fully-connected layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/0*K65DI0nToCsufUE-.png" /><figcaption><em>LeNet-5 architecture for digit recognition. Source: </em><a href="http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf"><em>Y. Lecun et al. (1998)</em></a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/0*KSJcPbT_EOjdEucl.png" /><figcaption><em>AlexNet architecture for training with 2 GPUs. Source: </em><a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks"><em>A. Krizhevsky et al. (2012)</em></a></figcaption></figure><h3>Going deeper</h3><p>Since the 2012 milestone, researchers have tried to go deeper in the sequences of convolutional layers. In 2014, <a href="https://arxiv.org/abs/1409.1556.pdf">K. Simonyan &amp; A. Zisserman (2015)</a> released the VGG16 model, composed of sixteen convolutional layers, multiple max-pool layers and three final fully-connected layers. One of its specificities is to chain multiple convolutional layers with ReLU activation functions creating nonlinear transformations. Indeed, introducing nonlinearities allow models to learn more complex patterns. Moreover they introduced 3x3 filters for each convolution (as opposed to 11x11 filters for the AlexNet model) and noticed they could recognized the same patterns than larger filters while decreasing the number of parameters to train. These transformations reached 7.3% top-5 error rate on the 2014 ImageNet challenge reducing by a factor of two the error of the AlexNet model.</p><h3>Inception modules</h3><p>This same year, <a href="https://arxiv.org/abs/1312.4400.pdf">M. Lin et al. (2014)</a> have developed the concept of “inception modules”. Original convolutional layer uses linear transformations with a nonlinear activation function. However, training multiple convolutional layers simultaneously and stack their feature maps linked with a multi-layer perceptron also produces a nonlinear transformation. This idea has been exploited by <a href="https://arxiv.org/abs/1409.4842">C. Szegedy et al. (2014)</a> who proposed a deeper network called GoogLeNet (aka Inception V1) with 22 layers using such “inception modules” for a total of over 50 convolution layers. Each module is composed of 1x1, 3x3, 5x5 convolution layers and a 3x3 max-pool layer to increase sparsity in the model and obtain different type of patterns. The feature maps produced are then concatenated and analyzed by the next inception module. The GoogLeNet model has a 6.7% error rate over the 2014 ImageNet challenge which is somewhat lower than the VGG16 but astonishingly smaller (55 MB vs 490 MB). This gap is mainly due to the presence of the three large fully-connected layers in the VGG architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/0*T8YJzo1-9UHxrHfj.png" /><figcaption><em>Inception module. Source: </em><a href="https://arxiv.org/abs/1409.4842"><em>C. Szegedy et al. (2014)</em></a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bYkCKdn8DPiDxM4F8hRWyg.png" /><figcaption><em>GoogLeNet architecture. Source: </em><a href="https://arxiv.org/abs/1409.4842"><em>C. Szegedy et al. (2014)</em></a></figcaption></figure><p>In 2015, <a href="https://arxiv.org/abs/1512.00567">C. Szegedy et al. (2015)</a> developed the Inception V2 model, mostly inspired by the first version. The authors have however changed the 5x5 filter in the inception modules by two 3x3 filters, a 3x3 convolution and a 3x1 fully-connected slided over the first one. This method called convolution factorization decreases the number of parameters in each inception module, thus reducing the computational cost. This model reached a top-5 error rate of 5.6% on the 2012 ImageNet challenge.</p><p>Going further, <a href="https://arxiv.org/abs/1512.00567">C. Szegedy et al. (2015)</a> have fine-tuned the batch-normalization and used a higher resolution input, the famous Inception V3 model. They reduced the strides of the first two layers and removed a max-pool layer to analyze images with higher precision. They finally reached a top-5 error rate of 3.58% over the 2012 ImageNet challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/0*_TDDzJep17eXboRt.png" /><figcaption><em>Inception module factorization after a nxn convolution. Source: </em><a href="https://arxiv.org/abs/1512.00567"><em>C. Szegedy et al. (2015)</em></a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/0*Vt5OFUXNKH28eNBl.png" /><figcaption><em>Inception module factorization application replacing 5x5 convolution by two 3x3 convolutions. Source: </em><a href="https://arxiv.org/abs/1512.00567"><em>C. Szegedy et al. (2015)</em></a></figcaption></figure><h3>Residual learning</h3><p>The main common trend in convolutional neural network models is their increasing depth. <a href="http://arxiv.org/abs/1512.03385">K. He et al. (2015)</a> noticed however, that the increasing depth involves an increasing error rate, not due to overfitting but to the difficulties to train and optimize an extremely deep models. “Residual Learning” has been introduced to create a connection between the output of one or multiple convolutional layers and their original input with an identity mapping. In other words, the model is trying to learn a residual function which keeps most of the information and produces only slight changes. Consequently, patterns from the input image can be learned in deeper layers. Moreover, this method doesn’t add any additional parameter and doesn’t increase the computational complexity of the model. This model, dubbed “ResNet”, is composed of 152 convolutional layers with 3x3 filters using residual learning by block of two layers. Although it got a top-5 error rate of 4.49% over the 2012 ImageNet challenge (less than the Inception V3), the ResNet model has won the 2015 challenge with a top-5 error rate of 3.57%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/0*I8JR3SBYCY97zDLB.png" /><figcaption><em>Residual learning block architecture. Source: </em><a href="http://arxiv.org/abs/1512.03385"><em>K. He et al. (2015)</em></a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/0*7Xt5Sn5zplLepHkH.png" /><figcaption><em>ResNet architecture. Source: </em><a href="http://arxiv.org/abs/1512.03385"><em>K. He et al. (2015)</em></a></figcaption></figure><h3>The Inception-ResNet</h3><p>One year after the success of the ResNet model, <a href="http://arxiv.org/abs/1602.07261">C. Szegedy et al. (2016)</a> combined inception modules (to increase sparsity) and residual blocks (to learn deeper layers), building residual inception blocks. The inception modules have been improved to fine-tune the layer sizes and to detect more specific patterns. This model doesn’t use batch-normalization before the first traditional layers of the network to increase even more the number of inception blocks. The resulting Inception V4 (Inception-ResNet)² model can be trained faster and outperforms all other models over the 2012 ImageNet challenge with a top-5 error rate of 3.08%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*0Dn1vwgB5H_e_hmeX4YdvA.png" /><figcaption>Architecture of an Inception-resnet-A module. Source: <a href="http://arxiv.org/abs/1602.07261"><em>C. Szegedy et al. (2016)</em></a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/0*RPpX8agxkM4xOevN.png" /><figcaption><em>Inception-ResNet architecture using customized Inception-ResNet modules. Source: </em><a href="http://arxiv.org/abs/1602.07261"><em>C. Szegedy et al. (2016)</em></a></figcaption></figure><h3>Squeeze and Excitation</h3><p>Every day, new blocks to improve performance and speed up training are proposed. For example, the “Squeeze-and-Excitation” module <a href="https://arxiv.org/abs/1709.01507">(J. Hu, 2017)</a> uses an architecture combining multiple fully-connected layers, inception modules and residual blocks. One of its main advantages is the low number of parameters (thus reducing computational cost) while retaining a top-5 error rate of 2.25%, promoting him winner of the 2017 ImageNet challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/166/0*JG_31nsSpElcZM_M.png" /><figcaption><em>Squeeze-Excitation-ResNet module. Source: </em><a href="https://arxiv.org/abs/1709.01507"><em>J. Hu (2017)</em></a></figcaption></figure><h3>Neural Architecture Search</h3><p>Google Brain researchers <a href="https://arxiv.org/abs/1611.01578">(B. Zoph and Q.V. Le, 2017)</a> have released a new concept called Neural Architecture Search (NAS). Basically, it is used as a cell in a Recurrent Neural Network to learn its own architecture using reinforcement learning. For a given range of operations and hyperparameters, multiple sequences are realized to maximize the accuracy as a signal reward for a given dataset. The objective is to learn the best sequence of operations (given a maximal depth) to get an optimized architecture. A CNN architecture learned with NAS have reached the state-of-the-art test error rate on the CIFAR-10 dataset.</p><p>Using this previous work, <a href="https://arxiv.org/abs/1707.07012">B. Zoph et al. (2017)</a> have created a model with an architecture block learned using NAS on the CIFRA-10 dataset to perform the ImageNet challenge. These blocks are duplicated and stacked with their own parameters to create the “NASNet” model. The ImageNet dataset is too large to be used for the NAS method but the authors have succeeded to create lighter and speeder block architectures than <a href="http://arxiv.org/abs/1602.07261">C. Szegedy et al. (2016)</a>. The parameters of the NASNet model are trained using the ImageNet. The model achieved a top-5 error rate of 3.8% over the ImageNet 2012 challenge. A smaller version of less than 50 MB is also released with a lower error rate than any other equivalently-sized model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/0*E2Vw6v2pNRRlHN3Y.png" /><figcaption><em>Architecture of the best convolutional modules learnt with NAS computing the next hidden state using the past one as input. Left: the Normal Cell is the module creating the feature maps. Right: the Reduction Cell is the module reducing the size of the feature maps by a factor of two (it replaces max-pooling layers). Source: </em><a href="https://arxiv.org/abs/1707.07012"><em>B. Zoph et al. (2017)</em></a></figcaption></figure><h3>Progressive Neural Architecture Search</h3><p><a href="https://arxiv.org/abs/1712.00559">C. Liu et al.(2017)</a> have recently released a method called Progressive Neural Architecture Search (PNAS) to reduce the search space of the algorithm of <a href="https://arxiv.org/abs/1707.07012">B. Zoph et al. (2017)</a> while keeping competitive results.</p><p>Inspired from the previous work, they have reduced the number of available functions and the combination operators. They have removed the Reinforcement Learning part to create a progressive search. They first start by construct all the possible simple structures with a single function. Each one of them is stacked with an operator to another structure and a learnt function predicts their score to quantify the profitability of learning it. These scores are ordered and the K best ones are selected to be stacked with another structure. The process is iterative until the maximum number of block is reached.</p><p>The PNAS model have achieved the same performances than the NAS model developed by <a href="https://arxiv.org/abs/1707.07012">B. Zoph et al. (2017)</a> while reducing by factor 5 the training time. This method is promising for deep learning because new intuitive architectures are difficult to find by researchers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/0*ONkFROMoU3trk8pf.png" /><figcaption><em>Cell structure of the best PNAS model. Source: </em><a href="https://arxiv.org/abs/1712.00559"><em>C. Liu et al.(2017)</em></a></figcaption></figure><h3>Conclusion</h3><p>This post described the milestones reached in deep learning for the image classification problem and more specifically about the ImageNet challenge. However, it is not an exhaustive list of all the existing models. New ones appear every day, as a reminder that image classification is a very active field of research in computer vision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*O9Rv_0CQb1czdu87jaKO6w.png" /><figcaption>Overview of the top-5 error rates on the 2012, 2014, 2015 and 2017 ImageNet challenges.</figcaption></figure><p>Most of them however requires, with a size of hundred of megabytes, a significant computational cost due to the large number of operation involved, even in inference mode. This constitutes a real matter of concern for deep learning models embedded on mobile devices. Optimization of architectures and weights storage in inference also constitutes an active field of research, which will be addressed in a future post.</p><p>¹: Synonym sets</p><p>²: <a href="http://arxiv.org/abs/1602.07261">C. Szegedy et al. (2016)</a> developed a pure (i.e. without residual block) Inception V4 and an Inception-ResNet V2 model which uses inception modules and residual blocks. The aforementioned Inception V4 is the Inception-ResNet V2 providing the best performances.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fdbca4a05e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zylapp/review-of-deep-learning-algorithms-for-image-classification-5fdbca4a05e2">Review of Deep Learning Algorithms for Image Classification</a> was originally published in <a href="https://medium.com/zylapp">Zyl Story</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>