<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Beeren Sahu on Medium]]></title>
        <description><![CDATA[Stories by Beeren Sahu on Medium]]></description>
        <link>https://medium.com/@beeren.sahu?source=rss-fd635964bb9d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nYM7Q6eR0JaR3h0URxrM6A.jpeg</url>
            <title>Stories by Beeren Sahu on Medium</title>
            <link>https://medium.com/@beeren.sahu?source=rss-fd635964bb9d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:02:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@beeren.sahu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An Evolution in Single Image Super Resolution using Deep Learning]]></title>
            <link>https://medium.com/data-science/an-evolution-in-single-image-super-resolution-using-deep-learning-66f0adfb2d6b?source=rss-fd635964bb9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/66f0adfb2d6b</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[sub-pixel-convolution]]></category>
            <category><![CDATA[srgan-srcnn-srresnet]]></category>
            <category><![CDATA[perceptual-loss]]></category>
            <category><![CDATA[super-resolution]]></category>
            <dc:creator><![CDATA[Beeren Sahu]]></dc:creator>
            <pubDate>Tue, 03 Dec 2019 16:24:48 GMT</pubDate>
            <atom:updated>2021-09-20T14:48:20.745Z</atom:updated>
            <content:encoded><![CDATA[<h4>From classical interpolation to deep learning methods with Generative Adversarial Networks</h4><p>Reconstructing a high resolution photo-realistic image from its counterpart low resolution image has been a long challenging task in the fraternity of computer vision. This task becomes even more difficult when all you have is a single low resolution image as input to recreate its high resolution image.</p><p>What is super resolution? The estimation of a <strong>high-resolution (HR)</strong> image from <strong>a single low-resolution (LR)</strong> counterpart is referred to as <strong>super-resolution (SR)</strong>. In other words, LR is a <strong>single image</strong> input, HR is the ground truth, and SR is the predicted high resolution image. When applying ML/DL solutions, the LR images are generally the down-sampled HR image with some blurring and noise added to them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4tVNlats95iVJUbpQvHeaQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@robinmathlener?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Robin Mathlener</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Generation #0 of Evolution: Interpolation</h3><p>To start with, a very early solution was the method of interpolation in image processing. Here, the low resolution image is resized by a factor of 2x or 4x using some interpolation method like nearest-neighbor, bilinear or bicubic method of interpolation.</p><blockquote>“Interpolation works by using known data to estimate values at unknown points. Image interpolation works in two directions, and tries to achieve a best approximation of a pixel’s intensity based on the values at surrounding pixels.” — <a href="https://sisu.ut.ee/imageprocessing/book/3">ebook on Digital Image Processing from University of Tartu</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*XSu3LkO1usO3Jwe-wfwl3w.png" /><figcaption>Figure: Effect of interpolation (<a href="https://www.youtube.com/watch?v=lmUxbRY7H2I">source</a>)</figcaption></figure><p>As from above illustration it is very clear that is resultant image is blurred and unrealistic.</p><h3>Generation #1 of Evolution: SRCNN</h3><p>With the success of fully convolutional neural network (FCNN) in solving semantic segmentation, it popularity in other fields of computer vision spread rapidly. FCNN is a CNN without any dense connections (fully connected layer) at the rear of it. Every CNN has two main functional block, i) Feature extractor and ii) Classifier. The dense connections at the rear of CNN is the classifier whose task is to map the extracted features to class probabilities. I consider FCNN as a basic design practice in DL to generate/predict an output map from an input image. Output map can be semantic segmentation map, style transfer image or even super resolution image. In other words, FCNN is a image-to-image mapping engine. One such preliminary application of FCNN in super resolution is <a href="https://arxiv.org/abs/1501.00092">SRCNN</a>.</p><p>In SRCNN, first the image is upsampled using bicubic interpolation and then feed to a simple FCNN. It is important to note that there is no pooling operations involved. Thus, resulting in an output of the same spatial size as that of the upsampled input image. Finally, we compute the MSE loss between the target HR image with the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/0*kMPkTap02DWj2_jy" /><figcaption>Figure: SRCNN Model (<a href="https://www.youtube.com/watch?v=lmUxbRY7H2I">source</a>)</figcaption></figure><h3>Generation #2 of Evolution: SRResNet and Sub-pixel convolution for upsampling</h3><p>With some success in super resolution from single image using SRCNN, inspired others to bring further improvement the architecture. As it is known that <a href="https://arxiv.org/abs/1512.03385">ResNet</a> (CNN with skip connections) are better than conventional CNNs. <a href="https://arxiv.org/abs/1609.04802">SRResNets</a> replaced simple convolution blocks with residual blocks. As a result accuracy increased significantly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*7MQt50mjhQuqPpby" /><figcaption>Figure: SRResNet Model (<a href="https://www.youtube.com/watch?v=lmUxbRY7H2I">source</a>)</figcaption></figure><p>Many deep learning models also incorporated transposed convolution for upsampling. Bilinear and bicubic upsampling are not learnable, that means, it can only be used before or after the deep learning architecture and not in between. Other advantages of learnable upsampling is its speed and accuracy.</p><p>But as one may have observed the upsampling operation as implemented above with strided convolution gradients adds zero values to upscale the image, that have to be later filled in with meaningful values. Maybe even worse, these zero values have no gradient information that can be back-propagated through.</p><blockquote>“To cope with that problem, <a href="https://arxiv.org/abs/1609.05158">Shi et. al</a> proposed what we argue to be one the most useful recent convnet tricks (at least in my opinion as a generative model researcher!) They proposed a subpixel convolutional neural network layer for upscaling. This layer essentially uses regular convolutional layers followed by a specific type of image reshaping called a phase shift. In other words, instead of putting zeros in between pixels and having to do extra computation, they calculate more convolutions in lower resolution and resize the resulting map into an upscaled image. This way, no meaningless zeros are necessary.” — [<a href="https://github.com/atriumlts/subpixel">https://github.com/atriumlts/subpixel</a>]</blockquote><p>Image reshaping using phase shift is also called “<strong>pixel shuffle</strong>”, which rearranges the elements of H × W × C · r² tensor to form rH × rW × C tensor as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/0*-82Ps97CPgtUVMua" /><figcaption>Figure: Sub-pixel convolution operation (<a href="https://arxiv.org/abs/1609.05158">source</a>)</figcaption></figure><h3>Generation #3 of Evolution: Perceptual Loss</h3><p>The main disadvantage of using MSE or MSE type error methods as loss function in applications like super resolution is that it is computed pixel-wise. That is, it only measures the change between two corresponding pixels in the predicted and the target images. This encourages finding pixel-wise averages of plausible solutions which are typically overly-smooth and thus have poor perceptual quality. This argument also holds good for not using only PSNR for quality index as it is also computed pixel-wise. Therefore, I would advise not to check PSNR alone while comparing performance of any two method in such tasks.</p><p><a href="https://arxiv.org/abs/1603.08155">Perceptual loss</a> is computed by comparing two images based on high-level representations from pre-trained CNN model. The function is used to compare high level differences, like content and style discrepancies, between images.</p><p>In other words, both target and predicted input are passed through a pre-trained network and compute the euclidean distance between the two resulting feature maps (at the same stage). The perceptual loss function works by summing all the squared errors between all the pixels and taking the mean. This is in contrast to a per-pixel loss function which sums all the absolute errors between pixels.</p><h3>Generation #4 of Evolution: SRGAN</h3><p>Generative adversarial networks (GANs) provide a powerful framework for generating plausible-looking natural images with high perceptual quality. The GAN procedure encourages the reconstructions to move towards regions of the search space with high probability of containing photo-realistic images and thus closer to the natural image manifold.</p><p><a href="https://arxiv.org/abs/1609.04802">SRGAN</a> is a GAN based network, where the generator (G) learns to generates SR images from LR images as close as possible to HR. The discriminator (D) learns to distinguish generated SR images from real images. The G takes advantage of ResNet and sub-pixel convolution for upsampling. It also combines perceptual loss with generative or adversarial loss for the computation of its loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fMB6MEKKRJayrK58" /><figcaption>Figure: Architecture of Generator and Discriminator Network in SRGAN. (<a href="https://arxiv.org/abs/1609.04802">source</a>)</figcaption></figure><p><strong>Loss</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/0*pj5P-iTobQcJMDg4" /><figcaption>Equation of modified perceptual loss in SRGAN. (<a href="https://arxiv.org/abs/1609.04802">source</a>)</figcaption></figure><h3>Conclusion</h3><p>Studying the evolution of estimating single image super resolution using deep learning, it is evident that ResNet based GAN that combines perceptual loss with generative loss and applying sub-pixel convolution for upsampling can generate better photo realistic super resolved images.</p><h3>Notable Credit</h3><p>I give credit to <a href="https://pydata.org/warsaw2018/speaker/profile/115/"><strong>Katarzyna Kańska</strong></a> for her great inspiring presentation video on <em>“Single Image Super Resolution”</em>: Youtube Video — <a href="https://www.youtube.com/watch?v=lmUxbRY7H2I">Can you enhance that? Single Image Super Resolution — Katarzyna Kańska</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66f0adfb2d6b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-evolution-in-single-image-super-resolution-using-deep-learning-66f0adfb2d6b">An Evolution in Single Image Super Resolution using Deep Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Evolution of Deeplab for Semantic Segmentation]]></title>
            <link>https://medium.com/data-science/the-evolution-of-deeplab-for-semantic-segmentation-95082b025571?source=rss-fd635964bb9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/95082b025571</guid>
            <category><![CDATA[deeplab]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Beeren Sahu]]></dc:creator>
            <pubDate>Fri, 12 Jul 2019 09:07:53 GMT</pubDate>
            <atom:updated>2019-08-01T03:41:07.595Z</atom:updated>
            <content:encoded><![CDATA[<h4>From classical image segmentation methods through Deep learning based semantic segmentation to the Deeplab and its variant.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-JZc9fZmI6M2hUuoUHkQMg.jpeg" /></figure><p>In computer vision, a simple image segmentation is the process of partitioning a digital image into multiple segments (sets of pixels). Image segmentation is a long standing computer Vision problem. However, semantic segmentation is the technique of segmenting image with “understanding” of image in pixel level. In other words, semantic segmentation is analysis and classification of each pixel into multiple classes (labels).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*9xzz3l4hhYaprUTTb1G5lA.jpeg" /><figcaption>Figure 1: a) Image segmentation using Thesholding b) Semantic Segmantation</figcaption></figure><p>There are many applications of semantic segmentation that include autonomous driving, human-machine interaction, computational photography, image search engines, and augmented reality to name a few.</p><h3>A Brief History of Semantic Segmentation</h3><p>Quite a few algorithms have been designed to solve this task, such as the Watershed algorithm, Image thresholding, K-means clustering, Graph partitioning methods, etc. The simplest method would be the thresholding method. In this method a gray-scale image is converted into a binary image based on a threshold value. In spite of many traditional image processing techniques however the deep learning methods has been the game changer.</p><p>In order to properly understand how semantic segmentation is tackled by modern deep learning architectures, it is important to know that it is not an isolated field. Rather it is a natural step in the progression from coarse to fine inference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gXwtxTzO_gzEzm50OLcN9w.png" /><figcaption>Figure: Coarse to fine inference</figcaption></figure><p>The origin could be located at classification, which consists of making a prediction for a whole input, i.e., predicting the object in an image. The next step towards fine-grained inference is localization or detection, providing not only the classes but also additional information regarding the spatial location of those classes. A semantic segmentation can be seen as a <strong>dense-prediction<em> </em></strong>task. In dense prediction, the objective is to generate an output map of the same size as that of the input image. Now, it is obvious that semantic segmentation is the natural step to achieve fine-grained inference. Its goal is to make dense predictions inferring labels for every pixel. This way, each pixel is labeled with the class of its enclosing object or region. Further improvements can be made, such as <strong>instance segmentation</strong> (separate labels for different instances of the same class).</p><p>A general semantic segmentation architecture can be broadly thought of as an <strong>encoder</strong> network followed by a <strong>decoder</strong> network:</p><ul><li>The <strong>encoder</strong> is usually is a pre-trained classification network like VGG/ResNet followed by a decoder network.</li><li>The task of the <strong>decoder</strong> is to semantically project the discriminative features (lower resolution) learnt by the encoder onto the pixel space (higher resolution) to get a dense classification.</li></ul><p>One of the very early Deep Convolutional Neural Networks (DCNNs) used for semantic segmentation is Fully Convolutional network (FCN). The FCN network pipeline is an extension of the classical CNN. The main idea is to make the classical CNN take as input arbitrary-sized images. The restriction of CNNs to accept and produce labels only for specific sized inputs comes from the fully-connected layers which are fixed. Contrary to them, FCNs only have convolutional and pooling layers which give them the ability to make predictions on arbitrary-sized inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*DicJgZPx9bHud0xIp0lFRw.png" /><figcaption>Figure: FCN Architecture [ref: <a href="https://arxiv.org/abs/1411.4038">https://arxiv.org/abs/1411.4038</a>]</figcaption></figure><p>One issue in this specific FCN is that by propagating through several alternated convolutional and pooling layers, the resolution of the output feature maps is down sampled. Therefore, the direct predictions of FCN are typically in low resolution, resulting in relatively fuzzy object boundaries. A variety of more advanced FCN-based approaches have been proposed to address this issue, including <a href="https://arxiv.org/pdf/1511.00561.pdf">SegNet</a>, <a href="https://arxiv.org/abs/1505.04597">UNet</a>, <a href="https://arxiv.org/pdf/1412.7062.pdf">DeepLab</a>, and Dilated Convolutions.</p><p>In the following section we will discuss the Deeplab for semantic segmentation and its evolution.</p><h3>The Evolution of Deeplab</h3><p>DeepLab is a state-of-the-art semantic segmentation model designed and open-sourced by Google. The dense prediction is achieved by simply up-sampling the output of the last convolution layer and computing pixel-wise loss. The Deeplab applies <strong>atrous convolution</strong> for up-sample.</p><p><strong><em>Atrous Convolution</em></strong></p><p>The repeated combination of max-pooling and striding at consecutive layers in DCCN reduces significantly the spatial resolution of the resulting feature maps. One solution is to use deconvolution layers to up-sample the resulting map. However, it requires additional memory and time. Atrous convolution offers a simple yet powerful alternative to using deconvolutional. Atrous convolution allows to effectively enlarge the field of view of filters without increasing the number of parameters or the amount of computation.</p><p>Atrous convolution as a shorthand for convolution with up-sampled filters. Filter up-sampling amounts to inserting holes (‘trous’ in French) between nonzero filter taps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*s29kfYkjWGMaS_Vs" /><figcaption>Figure: (Top) regular convolution followed by up-sampling, (Bottom) atrous convolution resulting denser feature map. [ref: <a href="https://arxiv.org/abs/1606.00915">https://arxiv.org/abs/1606.00915</a>]</figcaption></figure><p>Mathematically, atrous convolution <em>y[i]</em> for a one-dimensional signals <em>x[i]</em> with a filter <em>w[k]</em> of length <em>K</em> and stride rate <em>r</em> is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/0*dhHcccZ1L2hAcQjL" /></figure><h4>Deeplab (Deeplabv1)</h4><p>Success of Deeplabv1 in the task of semantic segmentation is due to some advancements added to the previous state of the art models, specifically to the FCN model. These advancements addresses the following two challenges.</p><p><strong><em>Challenge 1: reduced feature resolution</em></strong></p><p>Due to multiple pooling and down-sampling (‘stride’) in DCNN, there is a significant reduction in spatial resolution. They remove the down-sampling operator from the last few max-pooling layers of DCNNs and instead up-sample the filters (atrous) in subsequent convolutional layers, resulting in feature maps computed at a higher sampling rate.</p><p><strong><em>Challenge 2: reduced localization accuracy due to DCNN invariance</em></strong></p><p>In order to capture fine details by employing a fully connected Conditional Random Field (CRF). The CRF potentials incorporate smoothness terms that maximize label agreement between similar pixels, and can integrate more elaborate terms that model contextual relationships between object classes. Following figure illustrates the improvement of segmentation map after few iterations of CRF.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/0*Rygrm-EYNDVKLuNJ" /><figcaption>Figure: [Top] Score map (input before softmax function), [Bottom] belief map (output map after mean field iterations) [ref: <a href="https://arxiv.org/abs/1606.00915">https://arxiv.org/abs/1606.00915</a>]</figcaption></figure><p>The Deeplabv1 model takes the images as input and passes through usual DCCN layers followed by one or two atrous layer and results in a coarse score map. This map is then up-sampled to the original size of the image, using bi-linear interpolation. Finally, to improve the segmentation result fully connected CRF is applied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/0*-w5U-g3JK68VZCbm" /><figcaption>Figure: Deeplabv1 flowchart [ref: <a href="https://arxiv.org/abs/1606.00915">https://arxiv.org/abs/1606.00915</a>]</figcaption></figure><h4>Deeplabv2</h4><p>To further improve the performance of the Deeplabv1 architecture, the next challenge is existence of object at multiple scales.</p><p><strong><em>Challenge: existence of objects at multiple scales</em></strong></p><p>To represent the object in multiple scales, a standard way to deal with this is to present to the DCNN re-scaled versions of the same image and then aggregate the feature or score maps.</p><p><em>Solution</em>: Use of <strong>Atrous Spatial Pyramid Pooling (ASPP)</strong>. The idea is to apply multiple atrous convolution with different sampling rates to the input feature map, and fuse together. As objects of the same class can have different scales in the image, ASPP helps to account for different object scales which can improve the accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*bP_ztEeZ7jv-ONmK" /><figcaption>Figure: Atrous Spatial Pyramid Pooling (ASPP). [ref: <a href="https://arxiv.org/abs/1606.00915">https://arxiv.org/abs/1606.00915</a>]</figcaption></figure><h4>Deeplabv3</h4><p>The former networks are able to encode multi-scale contextual information by probing the incoming features with filters or pooling operations (atrous convolution) at multiple rates and multiple effective fields-of-view (ASPP). The next challenge was to capture sharper object boundaries by gradually recovering the spatial information.</p><p><strong>Challenge: capture sharper object boundaries</strong></p><p>Deeplabv3 architecture adopts a novel encoder-decoder with atrous separable convolution to address the above issue. The encoder-decoder model is able to obtain sharp object boundaries. The general encoder-decoder networks have been successfully applied to many computer vision tasks, including object detection, human pose estimation, and also semantic segmentation.</p><p>Typically, the encoder-decoder networks contain:</p><ul><li>An encoder module that gradually reduces the feature maps and captures higher semantic information.</li><li>A decoder module that gradually recovers the spatial information.</li></ul><p>In addition to the above encoder-decoder network, it also applies <strong>depth-wise separable convolution </strong>to increases computational efficiency. This is achieved by factorizing a standard convolution into a depth-wise convolution followed by a point-wise convolution (i.e., 1 × 1 convolution). Specifically, the depth-wise convolution performs a spatial convolution independently for each input channel, while the point-wise convolution is employed to combine the output from the depth-wise convolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/0*Tp5wuSFEaO4U9hPp" /><figcaption>Figure: 3×3 Depth-wise separable convolution for atrous convolution. [ref: <a href="https://arxiv.org/abs/1802.02611">https://arxiv.org/abs/1802.02611</a>]</figcaption></figure><h4>Deeplabv3+</h4><p>This, extends DeepLabv3 by adding a simple yet effective decoder module to further refine the segmentation results especially along object boundaries.</p><p>Encoder: Compared to Deeplabv3, it uses <a href="https://arxiv.org/abs/1703.06211">Aligned Xception</a> instead of <a href="http://www.arxiv.org/abs/1512.03385">ResNet-101</a> as its main feature extractor (encoder), but with a significant modification. All max pooling operations are replaced by depth-wise separable convolution.</p><p>Decoder: The encoder is based on an <strong>output stride </strong>of 16, i.e. the input image is down-sampled by a factor of 16. So, instead of using bilinear up-sampling with a factor of 16, the encoded features are first up-sampled by a factor of 4 and concatenated with corresponding low level features from the encoder module having the same spatial dimensions. Before concatenating, 1 x 1 convolutions are applied on the low level features to reduce the number of channels. After concatenation, a few 3 x 3 convolutions are applied and the features are up-sampled by a factor of 4. This gives the output of the same size as that of the input image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*nnPrlrQAt0bDzIWF" /><figcaption>Figure: Deeplabv3+ model. [ref: <a href="https://arxiv.org/abs/1802.02611">https://arxiv.org/abs/1802.02611</a>]</figcaption></figure><p>Hope you enjoyed reading!!!</p><p><em>Cover Photo by </em><a href="https://unsplash.com/@scw1217?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Suzanne D. Williams</em></a><em> on </em><a href="https://unsplash.com/search/photos/evolution?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Unsplash</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95082b025571" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-evolution-of-deeplab-for-semantic-segmentation-95082b025571">The Evolution of Deeplab for Semantic Segmentation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semantic Segmentation of Garment for Image Creation with Deep Learning]]></title>
            <link>https://medium.com/@beeren.sahu/semantic-segmentation-of-garment-for-image-creation-with-deep-learning-c385bbb457b1?source=rss-fd635964bb9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c385bbb457b1</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artficial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[image-processing]]></category>
            <dc:creator><![CDATA[Beeren Sahu]]></dc:creator>
            <pubDate>Thu, 27 Sep 2018 10:34:03 GMT</pubDate>
            <atom:updated>2018-09-27T10:34:03.876Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using DeepLab Convolutional Neural Network and transfer learning for the Semantic Segmentation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*t-esACzKQuQDGXgnRNotDQ.jpeg" /></figure><h3>What is DeepLab?</h3><p><a href="https://arxiv.org/abs/1706.05587">DeepLab</a> is a recent and one of the most promising technique for <strong>semantic image segmentation</strong> with Deep Learning. Semantic segmentation is understanding an image at the pixel level and assigning a label to every pixel in an image such that pixels with the same label share certain characteristics.</p><p><strong>Application of semantic segmentation:</strong></p><ul><li>Face detection</li><li>Pedestrian detection</li><li>Lane detection</li><li>Locate tumors in medical images</li></ul><p>Most of the successful semantic segmentation systems developed in the previous decade relied on hand-crafted features combined with flat classifiers, such as Random Forests or Support Vector Machines. Over the past few years, the breakthroughs of Deep Learning in image classification were quickly transferred to the semantic segmentation task like <a href="https://arxiv.org/abs/1411.4038">Fully Convolutional Networks</a>, <a href="https://arxiv.org/abs/1511.00561">SegNet</a>, <a href="https://arxiv.org/abs/1611.06612">RefineNet</a> and <a href="https://arxiv.org/abs/1612.01105">PSPNet</a>.</p><p><strong>However, DeepLab overcame two challenges in the application of Deep Learning to semantic image segmentation:</strong></p><ol><li>Reduced feature resolution</li><li>Existence of objects at multiple scales.</li></ol><p>The main contribution of DeepLab is <em>“</em><a href="https://arxiv.org/pdf/1606.00915.pdf"><em>atrous convolution</em></a><em>”</em>, popularly known as <em>dilated convolution</em>. Atrous convolution helps in enhancing the feature resolution and increases the detailed spatial information.</p><p>To handle the problem of segmenting objects at multiple scales, it employs atrous convolution in cascade or in parallel to capture multi-scale context by adopting multiple atrous rates, a method known as <em>Atrous Spatial Pyramid Pooling (ASPP)</em>.</p><p>The above features make DeepLab an ideal choice for semantic segmentation for our proposed approach.</p><h3>Garment Segmentation Approach</h3><p>The given task is to extract the garment from the mannequin in an image. To address this problem we have used semantic segmentation to identify the garment region in the image. The overall process is depicted in the following illustration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ApwqW3vTq32krEpW2NWOdw.jpeg" /><figcaption>Garment segmentation using trained DeepLab model.</figcaption></figure><p>For the semantic segmentation task we have identified DeepLab CNN model to be appropriate. For training and testing purpose, we have created our own private garment segmentation data-set. It consists of images of various garments and their corresponding segmented ground truth images as described in the image above. These ground truth images are manually annotated by professionals. We have used only two labels for the annotation. 1&#39;s representing garment pixels and 0&#39;s representing the background pixels. The following figure demonstrates the abstract level training process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUoI-T9B2pIdHcju5LYxnA.jpeg" /><figcaption>Training DeepLab on TensorFlow for the garment data-set.</figcaption></figure><p>To speedup the overall training process we take the advantage of the transfer learning. Transfer learning is a machine learning technique where a pre-trained model is used as the starting point for training the new model. We have used the the DeepLab <a href="https://www.tensorflow.org/extend/tool_developers/#freezing">frozen graph</a>, trained on <a href="http://host.robots.ox.ac.uk/pascal/VOC/">PASCAL VOC 2012 data-set</a>.</p><p>The following code snippet, demonstrates transfer learning using DeepLab code. Here, <strong>TF_CHECKPOINT</strong> specifies the location of pre-trained model.</p><pre><strong>TF_CHECKPOINT=&quot;${INIT_FOLDER}/deeplabv3_pascal_train_aug/model.ckpt&quot;</strong></pre><pre>python &quot;${WORK_DIR}&quot;/train.py \<br>...<br>  <strong>--tf_initial_checkpoint=&quot;${TF_CHECKPOINT}&quot;</strong> \<br>...</pre><p>For the complete guide to TensorFlow implementation of DeepLab and how to use it for your own dataset visit the following post.</p><p><a href="https://medium.freecodecamp.org/how-to-use-deeplab-in-tensorflow-for-object-segmentation-using-deep-learning-a5777290ab6b">How to use DeepLab in TensorFlow for object segmentation using Deep Learning</a></p><p>Once the DeepLab model is trained on our garment data set, it is now capable of segmenting the garment in a given image. Finally, this segmented output can be used as a mask for extracting garment region from the mannequin.</p><p>In the coming months, I will be sharing more of my experiences with Images &amp; Deep Learning. Stay tuned and don’t forget to spare some claps if you like this article. It will encourage me immensely.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c385bbb457b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use DeepLab in TensorFlow for object segmentation using Deep Learning]]></title>
            <link>https://medium.com/free-code-camp/how-to-use-deeplab-in-tensorflow-for-object-segmentation-using-deep-learning-a5777290ab6b?source=rss-fd635964bb9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5777290ab6b</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Beeren Sahu]]></dc:creator>
            <pubDate>Mon, 24 Sep 2018 20:20:58 GMT</pubDate>
            <atom:updated>2018-09-24T20:20:58.565Z</atom:updated>
            <content:encoded><![CDATA[<h4>Modifying the DeepLab code to train on your own dataset for object segmentation in images</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mfz-HW5TIBU0AvprtApydQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/FmD8tIkf8bo?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Nick Karvounis</a> on <a href="https://unsplash.com/search/photos/images?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>I work as a Research Scientist at <a href="http://www.flixstock.com/">FlixStock</a>, focusing on Deep Learning solutions to generate and/or edit images. We identify coherent regions belonging to various objects in an image using Semantic Segmentation.</p><p><a href="https://arxiv.org/abs/1706.05587">DeepLab</a> is an ideal solution for Semantic Segmentation. The code is available in TensorFlow.</p><p>In this article, I will be sharing how we can train a DeepLab semantic segmentation model for our own data-set in TensorFlow. But before we begin…</p><h3>What is DeepLab?</h3><p><a href="https://arxiv.org/abs/1706.05587">DeepLab</a> is one of the most promising techniques for <strong>semantic image segmentation</strong> with Deep Learning. Semantic segmentation is understanding an image at the pixel level, then assigning a label to every pixel in an image such that pixels with the same label share certain characteristics.</p><h3>Installation</h3><p>DeepLab implementation in TensorFlow is available on GitHub <a href="https://github.com/tensorflow/models/tree/master/research/deeplab">here</a>.</p><h3>Preparing Dataset</h3><p>Before you create your own dataset and train DeepLab, you should be very clear about what you want to want to do with it. Here are the two scenarios:</p><ul><li>Training the model from scratch: you are free to have any number of classes of objects (number of labels) for segmentation. This needs a very long time for training.</li><li>Use the pre-trained model: you are free to have any number of classes of objects for segmentation. Use the pre-trained model and only update your classifier weights with transfer learning. This will take far less time for training compared to the prior scenario.</li></ul><p>Let us name your new dataset as “PQR”. Create a new folder “PQR” as: tensorflow/models/research/deeplab/datasets/PQR.</p><p>To start, all you need is input images and their pre-segmented images as ground-truth for training. Input images need to be color images and the segmented images need to be color indexed images. Refer to the PASCAL dataset.</p><p>Create a folder named “dataset” inside “PQR”. It should have the following directory structure:</p><pre>+ <strong>dataset</strong><br>    <em>-JPEGImages</em><br>    <em>-SegmentationClass</em><br>    <em>-ImageSets</em><br>+ <strong>tfrecord</strong></pre><h4>JPEGImages</h4><p>It contains all the input color images in *.jpg format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*M5PBchudNjWPqxPP.jpg" /><figcaption>A sample input image from PASCAL VOC dataset</figcaption></figure><h4>SegmentationClass</h4><p>This folder contains all the semantic segmentation annotations images for each of the color input images, which is the ground truth for the semantic segmentation.</p><p>These images should be color indexed. Each color index represents a unique class (with unique color) known as a color map.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/0*OjQiFBSrKsYnZzGS." /><figcaption>Sample Color Map [source: <a href="https://github.com/DrSleep/tensorflow-deeplab-resnet">https://github.com/DrSleep/tensorflow-deeplab-resnet</a>]</figcaption></figure><p><strong>Note:</strong> Files in the “SegmentationClass” folder should have the same name as in the “JPEGImage” folder for corresponding image-segmentation file pair.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*L3AEyxEId0-95rRq.png" /><figcaption>A sample semantic segmentation ground truth image from PASCAL VOC dataset</figcaption></figure><h4>ImageSets</h4><p>This folder contains:</p><ul><li>train.txt: list of image names for the training set</li><li>val.txt: list of image names for the validation set</li><li>trainval.txt: list of image names for training + validation set</li></ul><p>Sample *.txt file looks something like this:</p><pre>pqr_000032<br>pqr_000039<br>pqr_000063<br>pqr_000068<br>pqr_000121</pre><h4>Remove the color-map in the ground truth annotations</h4><p>If your segmentation annotation images are RGB images instead of color indexed images. Here is a Python script that will be of help.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3127d5d4f81c7fcb8a3163786b4dc8d9/href">https://medium.com/media/3127d5d4f81c7fcb8a3163786b4dc8d9/href</a></iframe><p>Here, the palette defines the “RGB:LABEL” pair. In this sample code (0,0,0):0 is background and (255,0,0):1 is the foreground class. Note, the new_label_dir is the location where the raw segmentation data is stored.</p><p>Next, the task is to convert the image dataset to a TensorFlow record. Make a new copy of the script file./dataset/download_and_convert_voc2012.sh as ./dataset/convert_pqr.sh. Below is the modified script.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a9b6e8d32ff76b59566ca24e40469ec/href">https://medium.com/media/5a9b6e8d32ff76b59566ca24e40469ec/href</a></iframe><p>The converted dataset will be saved at ./deeplab/datasets/PQR/tfrecord</p><h4>Defining the dataset description</h4><p>Open the file <strong>segmentation_dataset.py </strong>present in the<strong> research/deeplab/datasets/ </strong>folder.<strong> </strong>Add the following code segment defining the description for your PQR dataset.</p><pre><strong>_PQR_SEG_INFORMATION</strong> = DatasetDescriptor(<br>    splits_to_sizes={<br>        <strong>&#39;train&#39;: 11111</strong>, <em># number of file in the train folder</em><br>        &#39;trainval&#39;: 22222,<br>        &#39;val&#39;: 11111,<br>    },<br>    <strong>num_classes=2</strong>, <em># number of classes in your dataset</em><br>    <strong>ignore_label=255</strong>, <em># white edges that will be ignored to be class</em><br>)</pre><p>Make the following changes as shown bellow:</p><pre>_DATASETS_INFORMATION = {<br>    &#39;cityscapes&#39;: _CITYSCAPES_INFORMATION,<br>    &#39;pascal_voc_seg&#39;: _PASCAL_VOC_SEG_INFORMATION,<br>    &#39;ade20k&#39;: _ADE20K_INFORMATION,<br>    <strong>&#39;pqr&#39;: _PQR_SEG_INFORMATION</strong><br>}</pre><h3>Training</h3><p>In order to train the model on your dataset, you need to run the train.py file in the <strong>research/deeplab/ </strong>folder. So, we have written a script file train-pqr.sh to do the task for you.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/371ef8f8622f41e2bcce521285f967c9/href">https://medium.com/media/371ef8f8622f41e2bcce521285f967c9/href</a></iframe><p>Here, we have used xception_65 for your local training. You can specify the number of training iterations to the variable NUM_ITERATIONS. and set “ — tf_initial_checkpoint” to the location where you have downloaded or pre-trained the model *.ckpt. After training, the final trained model can be found in the TRAIN_LOGDIR directory.</p><p>Finally, run the above script from the …/research/deeplab directory.</p><pre># sh ./train-pqr.sh</pre><p>Voilà! You have successfully trained DeepLab on your dataset.</p><p>In the coming months, I will be sharing more of my experiences with Images &amp; Deep Learning. Stay tuned and don’t forget to spare some claps if you like this article. It will encourage me immensely.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5777290ab6b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/free-code-camp/how-to-use-deeplab-in-tensorflow-for-object-segmentation-using-deep-learning-a5777290ab6b">How to use DeepLab in TensorFlow for object segmentation using Deep Learning</a> was originally published in <a href="https://medium.com/free-code-camp">We’ve moved to freeCodeCamp.org/news</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>