<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by De Jun Huang on Medium]]></title>
        <description><![CDATA[Stories by De Jun Huang on Medium]]></description>
        <link>https://medium.com/@dejunhuang?source=rss-fc04c8eeae27------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0uutbJYaouHJHLUrzasWrA.jpeg</url>
            <title>Stories by De Jun Huang on Medium</title>
            <link>https://medium.com/@dejunhuang?source=rss-fc04c8eeae27------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:43:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dejunhuang/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Learning Day 71: Image captioning]]></title>
            <link>https://medium.com/dejunhuang/learning-day-71-image-captioning-b4c66b439575?source=rss-fc04c8eeae27------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4c66b439575</guid>
            <category><![CDATA[dejunhuang]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[daily-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[De Jun Huang]]></dc:creator>
            <pubDate>Mon, 28 Jun 2021 17:36:01 GMT</pubDate>
            <atom:updated>2021-06-28T17:38:26.379Z</atom:updated>
            <content:encoded><![CDATA[<h3>Image captioning</h3><ul><li>Vision → Language</li><li>The combination of CV and NLP</li></ul><h4>Traditional method</h4><ol><li>Image content → keywords/tags</li><li>Keywords → sentences</li></ol><p>Advantages:</p><ul><li>Directly use existing knowledge of CV and NLP</li><li>Able to filter noise</li><li>Modular</li></ul><p>Disadvantage</p><ul><li>If first step is wrong (CV), second step will be wrong (NLP)</li></ul><h4>Deep learning end-to-end method</h4><p><strong>CNN + RNN</strong></p><ul><li>A encoder-decoder structure</li><li>CNN as an encoder to extract the 2nd last layer of features map before softmax</li><li>RNN/LSTM as a decoder to take the feature map as input (the first input only)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*U4jr7BaoxDTdjllT.png" /><figcaption>An illustration of encoder-decoder structure of image captioning models (<a href="https://arxiv.org/abs/1807.08133">ref</a>)</figcaption></figure><ul><li>Neural Image Captioning(NIC)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*HXVzGHiUa4S7vlDwOkpxAw.png" /><figcaption>An example of image captioning model architecture (<a href="https://arxiv.org/abs/1411.4555">ref</a>)</figcaption></figure><p><strong>CNN + RNN + Attention mechanism</strong></p><ul><li>Attention mechanism is only implemented <strong>after</strong> encoder</li><li>Output from CNN (encoder) will be divided ton multiple cell states (C1, C2 C3) for LSTM (decoder) to generate outputs (Y1, Y2, Y3) accordingly</li><li>The cell states are generated from the weighted hₜ, with different weights represent the attention weightage</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*EKxq-Q2NRox93fpWSkSdyg.png" /><figcaption>Encoder + Decoder with attention mechanism</figcaption></figure><h4>Evaluation metrics for image captioning</h4><ul><li>Meteor</li><li>CIDEr</li><li>BLEU@N</li></ul><h3>Thoughts about this topic</h3><ul><li>RNN has been briefly studied in Day <a href="https://medium.com/dejunhuang/learning-day-25-rnn-concept-and-in-pytorch-fcbd51c3dfdf">25</a>–<a href="https://medium.com/dejunhuang/learning-day-30-imdb-comment-classification-with-lstm-in-pytorch-20c493e0effe">30</a></li><li>I did not understand this topic, image captioning, very well, especially the attention mechanism. (Does attention mechanism require additional marking on the data? If not, how does the model determine which part should be focused)</li><li>Miss out some details such as beam search</li><li>Will study this topic again when I focus on NLP learning later on</li></ul><h3>Reference</h3><p>Unless otherwise stated, all presented contents are summarised from this <a href="https://www.chinahadoop.cn/course/2038">course</a> for personal learning purposes</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4c66b439575" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dejunhuang/learning-day-71-image-captioning-b4c66b439575">Learning Day 71: Image captioning</a> was originally published in <a href="https://medium.com/dejunhuang">dejunhuang</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Day 70: 3D U-Net with 3D convolution layers, V-Net, DenseNet, FC-DenseNet]]></title>
            <link>https://medium.com/dejunhuang/learning-day-70-3d-u-net-with-3d-convolution-layers-v-net-densenet-fc-densenet-a263e018a5ac?source=rss-fc04c8eeae27------2</link>
            <guid isPermaLink="false">https://medium.com/p/a263e018a5ac</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dejunhuang]]></category>
            <category><![CDATA[daily-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[De Jun Huang]]></dc:creator>
            <pubDate>Fri, 25 Jun 2021 19:34:52 GMT</pubDate>
            <atom:updated>2021-06-25T19:36:25.615Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><ul><li>Continue from <a href="https://medium.com/dejunhuang/learning-day-69-image-segmentation-for-biomedical-applications-u-net-3b06eb8b8b9b">Day 69</a> regarding U-Net, one problem with biomedical imaging is that different slices at different locations would appear very differently, thus U-Net is not able to identify the target region</li><li>What if we directly feed the 3D data into a U-Net?</li></ul><h3>3D U-Net</h3><ul><li>It employs 3D convolution layers to take in 3-dimension imaging data</li><li>Besides 3D conv layers, the principle of 3D U-Net is very similar to U-Net</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*dy6q2D3M_ubQjeGlL1jNhw.png" /><figcaption>3D U-Net architecture (<a href="https://arxiv.org/abs/1606.06650">ref</a>)</figcaption></figure><h4>3D convolution</h4><ul><li>It is very similar to 2D convolution, just that we need to imagine 1 more dimension</li><li>A 2D convolution operation</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OAwIDVBg78iwIrpCheFdrg.png" /><figcaption>An illustration of a 2D convolution operation</figcaption></figure><ul><li>A 3D convolution operation with a 3D input of 1 input channel</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8AgYLKmugrN6eJItsmYmvQ.png" /><figcaption>Single-channel 3D input with a 3D filter</figcaption></figure><ul><li>A 3D convolution operation with 3D input of input channel, c = 3. In this case, the input is actually 4-dimension. The filter is 5-dimension.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8N-Km77soPLcciPUxl0X_A.png" /><figcaption>3-channel 3D input (4-dimension) with 3 x 3-channel 3D filter (5-dimension)</figcaption></figure><h3>V-Net</h3><ul><li>Added ResNet-like shortcut</li><li>Same loss function (DICE coefficient)</li><li>Include “down” conv layer (conv with stride=2)</li><li>Make use of 1x1x1 conv layer prior to softmax for classification between background and foreground</li><li>PReLU activation where the gradient at y&lt;0, a, is learnable</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/0*VGUU9k4noctoUhFQ.png" /><figcaption>ReLU (left), Leaky ReLU (middle) and PReLU (right) (<a href="https://medium.com/@shoray.goel/prelu-activation-e294bb21fefa">ref</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iiQz973vstSub37BchGPDQ.png" /><figcaption>V-Net architecture (<a href="https://arxiv.org/abs/1606.04797">ref</a>)</figcaption></figure><h3>DenseNet</h3><ul><li>Focus on fully utilising the feature maps</li><li>Reduce the no. of parameters</li><li>Not prone to overfitting even for small dataset</li><li>There are extra connections between all layers. I.e. Current layer’s input includes outputs from all previous layer</li><li>Eg. Input X0 →Output H1, Input X0 &amp;X1 →Output H2</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*RdPsufmHLj6UfGHFVTes4A.png" /><figcaption>An illustration of a dense block (<a href="https://arxiv.org/abs/1608.06993">ref</a>)</figcaption></figure><h3>FC-DenseNet</h3><ul><li>The architecture is still similar to U-Net</li><li>Just replace the original conv blocks withe dense block</li><li>Skip connections exist only at the scaling down side (left), not at the scaling up side (right) of the network</li><li>Skip connections are to avoid feature loss</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*sQZEXaOzv5QybQQucOSxvg.png" /><figcaption>An illustration of FC-DenseNet architecture (<a href="https://arxiv.org/abs/1611.09326">ref</a>)</figcaption></figure><h3>Reference</h3><p>Unless otherwise stated, all presented contents are summarised from this <a href="https://www.chinahadoop.cn/course/2038">course</a> for personal learning purposes</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a263e018a5ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dejunhuang/learning-day-70-3d-u-net-with-3d-convolution-layers-v-net-densenet-fc-densenet-a263e018a5ac">Learning Day 70: 3D U-Net with 3D convolution layers, V-Net, DenseNet, FC-DenseNet</a> was originally published in <a href="https://medium.com/dejunhuang">dejunhuang</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Day 69: Image segmentation for biomedical applications — U-Net]]></title>
            <link>https://medium.com/dejunhuang/learning-day-69-image-segmentation-for-biomedical-applications-u-net-3b06eb8b8b9b?source=rss-fc04c8eeae27------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b06eb8b8b9b</guid>
            <category><![CDATA[daily-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dejunhuang]]></category>
            <dc:creator><![CDATA[De Jun Huang]]></dc:creator>
            <pubDate>Thu, 24 Jun 2021 18:33:05 GMT</pubDate>
            <atom:updated>2021-06-24T18:38:41.547Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Day 69: Image segmentation for biomedical applications — U-Net</h3><h3>U-Net</h3><ul><li>It is a special type of FCN which contains only convolution layers</li><li>It gradually reduces the volume of feature maps (reduce size, increase depth; overall volume is decreasing) to a small feature map, then gradually increase the volume of feature maps again</li><li>At the volume increasing stage (2nd half of U-Net), It employs upsampling techniques such as deconvolution mentioned in <a href="https://medium.com/dejunhuang/learning-day-67-semantic-segmentation-1-fcn-8c377b46e3a8">Day 67</a>.</li><li>It also has shortcuts from the left to right feature maps without going through the entire U-Net</li><li>In each convolution block, it employs the structure of ResNet block</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PEM5APS4weBzF_Z2.png" /><figcaption>An example of U-Net architecture (<a href="https://towardsdatascience.com/unet-line-by-line-explanation-9b191c76baf5">ref</a>)</figcaption></figure><h4>Loss function — Dice</h4><ul><li>It is a measurement of overlap between two samples</li><li>0≤Dice≤1</li><li>Developed for binary data</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/199/1*XpgJ5ZaL8zmey8orsQUjzg.png" /><figcaption>Dice coefficient (<a href="https://www.jeremyjordan.me/semantic-segmentation/">ref</a>)</figcaption></figure><ul><li>It is very similar to IoU mentioned in <a href="https://medium.com/dejunhuang/learning-day-62-object-detection-r-cnn-3cf7aac3e594">Day 62</a>. IoU is also called Jaccard index/similarity coefficient:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*jN81Ccu7DAJEiryuwsp6vg.png" /><figcaption>Jaccard index (<a href="https://en.wikipedia.org/wiki/Jaccard_index">ref</a>)</figcaption></figure><ul><li>The difference between Dice and Jaccard coefficients was explained <a href="https://towardsdatascience.com/metrics-to-evaluate-your-semantic-segmentation-model-6bcb99639aa2">here</a></li></ul><h3>Data pre-processing in U-Net</h3><ul><li>These techniques are suitable for biomedical image segmentation or other similar applications</li></ul><h4>Curvature driven flow for image denoising</h4><ul><li>Biomedical images tend to contain noises</li><li>Gaussian blur is not fantastic in this application as it usually blurs the boundary</li><li>Curvature driven flow is able to preserve sharp boundaries with smoothing occurring only within a region (<a href="https://simpleitk.org/doxygen/latest/html/classitk_1_1simple_1_1CurvatureFlowImageFilter.html#details">ref</a>)</li><li>It is based on a theory that the brightness of objects should be smooth and the iso-brightness contours should have small curvature. If noise exists, irregular iso-brightness contours emerge resulting in high curvature at those regions</li></ul><h4>Elastic deformation for data augmentation (Besides the conventional ones)</h4><ul><li>It generates a form of distorted data as illustrated below</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*jrXBzC4HO-AtSI2rQZtaJA.png" /><figcaption>Elastic deformation example from original paper (<a href="https://www.semanticscholar.org/paper/Best-practices-for-convolutional-neural-networks-to-Simard-Steinkraus/5562a56da3a96dae82add7de705e2bd841eb00fc">ref</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/0*F7UM2Vz3g_rYeSK6.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/0*Lwr5lSmzK1rE35p2.png" /><figcaption>Before (left) and after (right) elastic deformation (<a href="https://www.programmersought.com/article/2384798517/">ref</a>)</figcaption></figure><ul><li>This method was used to suit the context of biomedical image: since the imaging of internal organs may distort at different measurement postures</li></ul><h3>Reference</h3><p>All presented contents without specific references are summarised from the course <a href="https://www.chinahadoop.cn/course/2038">link1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b06eb8b8b9b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dejunhuang/learning-day-69-image-segmentation-for-biomedical-applications-u-net-3b06eb8b8b9b">Learning Day 69: Image segmentation for biomedical applications — U-Net</a> was originally published in <a href="https://medium.com/dejunhuang">dejunhuang</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Day 68: Semantic segmentation 2 — DeepLab]]></title>
            <link>https://medium.com/dejunhuang/learning-day-68-semantic-segmentation-2-deeplab-cd029b3843cc?source=rss-fc04c8eeae27------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd029b3843cc</guid>
            <category><![CDATA[daily-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dejunhuang]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[De Jun Huang]]></dc:creator>
            <pubDate>Wed, 23 Jun 2021 16:22:57 GMT</pubDate>
            <atom:updated>2021-07-05T04:17:47.570Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Day 68: Semantic segmentation 2 — DeepLab, atrous/dilated convolution</h3><h3>Background</h3><ul><li>For FCN in <a href="https://medium.com/dejunhuang/learning-day-67-semantic-segmentation-1-fcn-8c377b46e3a8">Day 67</a>, it still suffers from the problem of big-step upsampling from small feature maps to the final output.</li><li>DeepLab aims to solve this problem to make the object boundary more accurate</li></ul><h3>DeepLab v1</h3><ul><li>CNN + CRF</li><li>Use Atrous/Dilated convolution at the deeper layers in CNN</li></ul><h4>Atrous/Dilated convolution</h4><ul><li>As compared to using a 3x3 filter, holes are inserted in-between the filter to make it cover an area of 5x5</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/0*nSuxxsNT5PIYcQoD.gif" /><figcaption>Atrous/Dilated convolution using a dilated 3x3 filter (effective coverage is 5x5) (<a href="https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d">ref</a>)</figcaption></figure><ul><li>With the similar amount of weights, the field of view is bigger</li><li>Use it with the appropriate stride to replace upsampling deconvolution layer. The resultant feature map has more details by the below comparison.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*BkplN1Qi993VMRf-1Vjm7g.png" /><figcaption>Compare standard convolution with down- and up-sampling (top) with atrous convolution (bottom) (<a href="https://arxiv.org/abs/1606.00915">ref</a>)</figcaption></figure><ul><li>A concept called <strong>dilation rate</strong>. The amount of holes inserted in-between=rate-1. Eg. Rate=2, no. of holes to be inserted=1. However, it cannot be too big. If rate ≥the input size, it is similar to doing convolution with 1x1 filter.</li></ul><h4>CRF (Conditional Random Field)</h4><ul><li>Take the rough segmentation results from CNN and refine the boundary using fully connected CRF</li><li><strong><em>I don’t quite understand the theory of CRF</em></strong></li></ul><h3>DeepLab v2</h3><ul><li>The base model can be VGG16 or ResNet101</li><li>Introduce Atrous Spatial Pyramid Pooling (ASPP)</li></ul><h4>Atrous Spatial Pyramid Pooling (ASPP)</h4><ul><li>Use different dilation rate to capture features at different scales</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*OkGcIk4zYBibRgew84DQfg.png" /><figcaption>ASPP (<a href="https://arxiv.org/abs/1606.00915">ref</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*AZg4KmRbsTOlz4dFjWyEHg.png" /><figcaption>An illustration of ASPP at Conv6 layer (right after Pool5) (<a href="https://arxiv.org/abs/1606.00915">ref</a>)</figcaption></figure><h3>DeepLab v3</h3><ul><li>More universal as it can use any CNN structure as a backbone</li><li>Use batch-norm in ASPP</li><li>No CRF</li><li>Use “series” and “parallel” connections of atrous convolution layers</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*lsPjbX_mnoDwxZUaEyoe_Q.png" /><figcaption>Deep CNN with atrous convolution in “series” connection or in cascade (<a href="https://arxiv.org/abs/1706.05587">ref</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dn6bSXBABTAlog6w8z_jQQ.png" /><figcaption>ASPP in “parallel” connection (<a href="https://arxiv.org/abs/1706.05587">ref</a>)</figcaption></figure><h3>DeepLab v3+</h3><ul><li>Expanded on v3</li><li>added a encoder-decoder structure to conserve boundary information</li><li>The original DeepLab v3 is used as the encoder to apply atrous convolution at multiple scales</li><li>Decoder gets the low-level features from the backbone model and concat with output from encoder after some conv layers and upsampling</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*csTZle9IBTxE5HPN1eDo8w.png" /><figcaption>Encoder-decoder structure in DeepLab v3+ (<a href="https://arxiv.org/abs/1802.02611">ref</a>)</figcaption></figure><h3>Reference</h3><p><a href="https://www.chinahadoop.cn/course/2038">link1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd029b3843cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dejunhuang/learning-day-68-semantic-segmentation-2-deeplab-cd029b3843cc">Learning Day 68: Semantic segmentation 2 — DeepLab</a> was originally published in <a href="https://medium.com/dejunhuang">dejunhuang</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Day 67: Semantic segmentation 1 — FCN]]></title>
            <link>https://medium.com/dejunhuang/learning-day-67-semantic-segmentation-1-fcn-8c377b46e3a8?source=rss-fc04c8eeae27------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c377b46e3a8</guid>
            <category><![CDATA[daily-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dejunhuang]]></category>
            <dc:creator><![CDATA[De Jun Huang]]></dc:creator>
            <pubDate>Tue, 22 Jun 2021 16:35:12 GMT</pubDate>
            <atom:updated>2021-06-24T17:54:58.912Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Day 67: Semantic segmentation 1 — FCN; Deconvolution</h3><h3>Image segmentation</h3><ul><li>In <a href="https://medium.com/dejunhuang/learning-day-47-computer-vision-before-deep-learning-205a3d79979b">Day 47</a>, image segmentation with conventional CV has been mentioned.</li><li>It can be done based on features like colour, grayscale, texture and shape</li></ul><h3>Semantic segmentation</h3><ul><li>Understand and recognize the content of image on a pixel-level</li><li>A pixel-wise classification based on the semantic information</li><li>INPUT: images</li><li>OUTPUT: pixel-wise label with the same output size as input</li><li>It can be applied in robotics, scene understanding, autonomous driving and medical diagnostics</li></ul><h4>Before and after deep learning</h4><ul><li>Before: manually extracted features + <a href="https://en.wikipedia.org/wiki/Conditional_random_field">CRF </a>(Conditional Random Field)</li><li>After: improve CNN to Fully Convolutional Networks (FCN)</li></ul><h3>FCN</h3><ul><li>FCN has been mentioned in <a href="https://medium.com/dejunhuang/learning-day-65-object-detection-4-r-fcn-db29d0320701">day 65</a> for object detection</li><li>It solves the problems with FC layers in which 1) the spatial information is lost, 2) the image input size has to be fixed and 3) too many weights and prone to overfitting</li><li>It can be as simple as replacing the FC layers with Conv layers of same depth, eg. FC layer <strong>4096</strong> → Conv layer 16x16x<strong>4096</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*LIXwIS0ZBD03KpEwJsMAqA.png" /><figcaption>Illustration of FCN (<a href="https://arxiv.org/pdf/1411.4038.pdf">ref</a>)</figcaption></figure><ul><li>However, there is a problem at the last layer where there is a huge upsampling: <strong>a small feature map</strong> → <strong>the pixel-wise prediction of the same size as the input image</strong>.</li><li>Therefore, more sophisticated upsampling methods are needed</li></ul><h4>Upsampling method in FCN</h4><p><strong>1. Deconvolution / Transposed convolution (learnable)</strong></p><ul><li>Deconvolution has been mentioned in <a href="https://medium.com/dejunhuang/learning-day-40-dcgan-and-wgan-dde522ac383e">Day 40</a> regarding GAN</li><li>It can be treated as using <strong>thick</strong> paddings to increase the feature map size and perform convolution operation to obtain larger feature map</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/0*veZVCpqrSwDj4ozv.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/0*SpeB4k0hSuF8GVXR.gif" /><figcaption>Transposed convolution different padding strategies (<a href="https://datascience.stackexchange.com/questions/6107/what-are-deconvolutional-layers">ref</a>)</figcaption></figure><p><strong>2. Bilinear interpolation (non-learnable)</strong></p><ul><li>Simple and fast calculation</li><li>Bilinear just means it takes in 2 variables (x, y in below case) for interpolation as compare to linear interpolation</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*fSKlrT1xAqQ0Xy6X.png" /><figcaption>The red dots are data, green dot is the point to be interpolated (<a href="https://en.wikipedia.org/wiki/Bilinear_interpolation">ref</a>)</figcaption></figure><p><strong>3. Unpooling (non-learnable)</strong></p><ul><li>It was mentioned in <a href="https://medium.com/dejunhuang/learning-day-55-back-propagation-in-cnn-pooling-layer-c980e76780a7">Day 55</a> for back propagation of pooling layer</li><li>For max pooling, it records down the location of max values before pooling was performed. During unpooling, just need to place the values back to the max-value locations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*fOab4sJoLzbVFVr7yvUixA.png" /><figcaption>Max pooling and its corresponding unpooling operation. Switch variables are the location records that were kept for unpooling. (<a href="https://arxiv.org/abs/1505.04366">ref</a>)</figcaption></figure><h4>Another component working with upsampling: Skip-layer</h4><ul><li>In order not to loss too much information, we do upsampling not only from the last layer, but also from the shallower layers (where the feature maps are bigger) and combine them to form the final output.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/0*a6_n7dwdKaQ_nLka" /><figcaption>Illustration of skip-layer (<a href="https://leonardoaraujosantos.gitbook.io/artificial-inteligence/machine_learning/deep_learning/image_segmentation">ref</a>)</figcaption></figure><ul><li>Shallow layers (bigger feature maps, eg pool3 above) extract details</li><li>Deep layers (smaller feature maps, eg. conv7 above) extract semantics</li></ul><h4>Combining upsampling and skip-layer</h4><ul><li>For the final outputs from the last layer and all the branches formed by skip-layers, they use interpolation</li><li>For the intermediate stages in the skip layers, they use deconvolution with learnable weights whose initial values are obtained via interpolation</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N6NSPRC9z2HSzc6Z" /><figcaption>AlexNet-based FCN with the details in skip-layer FCN-8s (shallow feature map) (<a href="https://blog.csdn.net/attitude_yu/article/details/84697181">ref</a>)</figcaption></figure><ul><li>The FCN layers are colour-coded as follows:</li><li>— Blue: Conv layers</li><li>— Green: Pooling layer</li><li>— Yellow: Elementwise summation</li><li>— Orange: Deconvolution</li><li>— Grey: Crop (to unify the size)</li></ul><h4>Training guideline</h4><ul><li>SGD with momentum (0.9)</li><li>Learning rate: 1e-3 (AlexNet), 1e-4 (VGG16), 1e-5 (GoogLeNet)</li><li>Minibatch: 20</li><li>First 5 conv layers loaded with pre-trained weight from original CNN</li><li>The 6th and 7th conv layers are initialized with zero</li><li>Last layer of upsampling uses bilinear interpolation, no learning</li><li>The rest of upsampling are to be initialized with interpolation values, followed by deconvolution with learning</li></ul><h4>FCN performance</h4><ul><li>In terms of skip-layer, FCN-8s gives the best results</li><li>In terms of base network, FCN-VGG16 has the best performance, but also the slowest</li><li>The edge accuracy is still not fantastic due to thick padding and cropping operation</li></ul><h3>Reference</h3><p><a href="https://www.chinahadoop.cn/course/2038">link1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c377b46e3a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dejunhuang/learning-day-67-semantic-segmentation-1-fcn-8c377b46e3a8">Learning Day 67: Semantic segmentation 1 — FCN</a> was originally published in <a href="https://medium.com/dejunhuang">dejunhuang</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Day 66: Object detection 5 — YOLO v1, v2 and v3]]></title>
            <link>https://medium.com/dejunhuang/learning-day-66-object-detection-5-yolo-v1-v2-and-v3-780711e7c997?source=rss-fc04c8eeae27------2</link>
            <guid isPermaLink="false">https://medium.com/p/780711e7c997</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[daily-learning]]></category>
            <category><![CDATA[dejunhuang]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[De Jun Huang]]></dc:creator>
            <pubDate>Sun, 20 Jun 2021 18:17:53 GMT</pubDate>
            <atom:updated>2021-06-20T18:22:44.702Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Day 66: Object detection 5 — YOLO v1, v2 and v3</h3><h3>YOLO v1 (You Only Look Once)</h3><ul><li>In previous object detection algorithms (eg. Faster R-CNN, R-FCN), there are two problems to solve: classification and regression</li><li>In YOLO, changed to a pure regression problem</li><li>Use a neural network to directly predict 1) bounding box (bbox) location, 2) probability of including an object, and 3) probability of a certain class.</li><li>Use NMS (Non-Maximal Suppression) to filter down boxes</li></ul><h4>Network architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j_W1ZDjEQPZ5n6pcJ8rGoA.png" /><figcaption>YOLO v1 network architecture (<a href="https://arxiv.org/abs/1506.02640">ref</a>)</figcaption></figure><ul><li>The pretrained model with ImageNet has input size of 224x224, but is then resized to 448x448 for YOLOv1</li><li>It contains both conv and FC layers, unlike R-FCN</li><li>The final output is 7x7x<strong>30</strong> (for <a href="https://cv.gluon.ai/build/examples_datasets/pascal_voc.html#prepare-pascal-voc-datasets">Pascal VOC </a>dataset which determines the last dimension <strong>30</strong>. Will be explained later)</li></ul><h4>How to interpret output 7x7x30</h4><ul><li>The first two dimensions, 7x7, indicate that YOLOv1 has conceptually divided the input image into 7x7 grid cells.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/0*D8vY4reT40vlAh-l.jpeg" /><figcaption>Illustration of grid cell and bboxes (<a href="https://zhanghanduo.github.io/post/yolo1/">ref</a>)</figcaption></figure><ul><li>In each grid cell, its depth is 30 which is the third dimension of 7x7x<strong>30</strong></li><li>The depth of 30 includes three components: bbox1 (depth=5), bbox2 (depth=5) and number of classes (depth=20)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N1QP6KqwenFeV-RB.png" /><figcaption>Visualisation of output of YOLO v1 (<a href="https://www.harrysprojects.com/articles/yolov1.html">ref</a>)</figcaption></figure><p><strong>Bbox1 &amp; 2 (depth=5 each)</strong></p><ul><li>YOLO v1 uses two bboxes per grid cell.</li><li>First 4 layers contain the bbox1 locations, width and height: x, y, w, h.</li><li>The 5th layer contains a confidence score (probability of this grid cell containing any object and how well the bbox at this grid cell is positioned, from 0 to 1). If there is object inside, the score = IOU of predicted bbox and groundtruth.</li><li>The next 5 layers serve the same purpose for the 2nd bbox.</li><li>The last 20 layers is the probability of each object class at this grid cell as <a href="https://cv.gluon.ai/build/examples_datasets/pascal_voc.html#prepare-pascal-voc-datasets">Pascal VOC </a>dataset has 20 classes.</li><li>So the output dimension is generically written as: S*S*(B*5+C) where S is the grid size, B is the no. of bboxes and C is the no. of classes.</li></ul><h4>Loss function</h4><ul><li>It is a summation of 5 terms that have been well explained in another post <a href="https://towardsdatascience.com/yolov1-you-only-look-once-object-detection-e1f3ffec8a89">here</a>.</li><li>2 terms take care of the bbox location and size, 2 terms take care of the confidence score, 1 term takes care of the object classification.</li></ul><h4>YOLOv1 characteristics</h4><p><strong>Advantage</strong></p><ul><li>Fast.</li><li>Low false positive rate.</li><li>Able to learn abstract object features.</li></ul><p><strong>Disadvantages</strong></p><ul><li>Sacrifice accuracy for speed.</li><li>Prone to bbox locating error.</li><li>Not fantastic for small object detection (since the grid size 7x7 is pretty coarse).</li></ul><h3>YOLO v2</h3><p>YOLOv2’s backbone is Darknet-19.</p><h4>7 changes from YOLOv1</h4><ol><li><strong>Batch normalization</strong> included.</li><li><strong>Higher res classifier</strong>: Uses pretrained models that take in 448x448 image.</li><li><strong>Convolution with anchor boxes</strong> (instead of just 2 bboxes).</li><li><strong>Dimension cluster</strong>: use K-means to obtain the optimal no. of bboxes (resulted in B=5).</li><li><strong>Direct location prediction</strong>: As YOLO with anchor boxes itself is unstable for predicting x, y of bboxes, use sigmoid to bound value to 0–1 to determine if the bbox should even be at a certain location. So bboxes that are too far from the groundtruth will be set to 0.</li><li><strong>Fine-Grained features</strong>: object detection is done of a 13x13 feature map (instead of 7x7) to get finer details. At passthrough layer (similar to ResNet) to combine high res and low res features.</li><li><strong>Multi-scale training: </strong>YOLOv2 becomes a fully conv layers network, which make this possible: every 10 epochs, randomly choose different input sizes for training. (if FC layer is present, input size is fixed).</li></ol><h3>YOLO v3</h3><ul><li>Backbone is Darknet 53.</li><li>Different loss function: from softmax at v2 to logistic loss at v3.</li><li>Different no. of anchor boxes from 5 at v2 to 3x3 at v3.</li><li>Perform 3 detections. These detections are done at different feature maps of different sizes to detect features at different scales.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*MsH6fgNEw98QWpy1fsxOzA.png" /><figcaption>3 detections are performed at 3 different feature maps of different sizes to detect features at different scales (<a href="https://www.chinahadoop.cn/course/2038">ref</a>)</figcaption></figure><h3>Reference</h3><p><a href="https://www.chinahadoop.cn/course/2038">Link1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=780711e7c997" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dejunhuang/learning-day-66-object-detection-5-yolo-v1-v2-and-v3-780711e7c997">Learning Day 66: Object detection 5 — YOLO v1, v2 and v3</a> was originally published in <a href="https://medium.com/dejunhuang">dejunhuang</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Day 65: Object detection 4 — R-FCN]]></title>
            <link>https://medium.com/dejunhuang/learning-day-65-object-detection-4-r-fcn-db29d0320701?source=rss-fc04c8eeae27------2</link>
            <guid isPermaLink="false">https://medium.com/p/db29d0320701</guid>
            <category><![CDATA[daily-learning]]></category>
            <category><![CDATA[dejunhuang]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[De Jun Huang]]></dc:creator>
            <pubDate>Sat, 19 Jun 2021 20:46:03 GMT</pubDate>
            <atom:updated>2021-06-19T20:50:19.149Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Day 65: Object detection 4 — R-FCN</h3><h3>Past models</h3><h4>R-CNN</h4><ul><li>SS to get region proposals</li><li>Every region proposal will go through conv+fc layers</li><li>Separate classifier (object classification) and regressor (bounding box location)</li></ul><h4>SPP-Net</h4><ul><li>Feed entire image to conv layers followed by SPP pooing for feature extraction</li><li>No fine tuning</li><li>Separate classifier and regressor</li></ul><h4>Fast R-CNN</h4><ul><li>ROI pooling instead of SPP</li><li>Has fine tuning</li><li>End to end network with multi-task loss</li><li>Almost a fully network-based structure except for selective search step</li></ul><h4>Faster R-CNN</h4><ul><li>RPN for selective search. Faster R-CNN = RPN + Fast R-CNN</li><li>ROI pooling</li><li>Has fine tuning</li><li>End to end network with multi-task loss.</li><li>Fully network-based structure</li><li>Weight sharing for conv layers between RPN and Fast R-CNN</li></ul><h4>Drawbacks of past models</h4><ul><li>Based on traditional CNN structure: Conv + FC layers. Weight sharing exists in conv layers only.</li><li>RoI-wise sub networks: Each RoI will go through separate networks. No weight sharing between RoIs</li></ul><h3>R-FCN</h3><ul><li>Similar to the trend in CNN, trying to make R-CNN fully conv layer-based</li></ul><h4>Dilemma in detection and classification tasks</h4><ul><li>Detection needs to be sensitive to object translation (translation variance)</li><li>Classification needs to be insensitive to translation (translation invariance)</li><li>With deeper conv layers, translation invariance becomes more dominant</li></ul><h4>New techniques to boost translation variance</h4><ul><li>Position-sensitive score maps</li><li>Position-sensitive RoI pooling</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8mL6N1CnS9bYNfpsrtNIdg.png" /><figcaption>The centre coloured block the the score maps. The right most coloured block is the result of RoI pooling (<a href="https://arxiv.org/abs/1605.06409">ref</a>)</figcaption></figure><h4>Position-sensitive s<strong>core maps</strong></h4><ul><li>Each layer of score map (identified by different colours) is actually consists of C+1 layers with C being the no. of classes, +1 is the background class</li><li>Each layer of score map contain information of each class at a particular location. Eg. the first layer on the right (light blue), it represents the bottom-right grid of the pooling layer</li></ul><h4>Position-sensitive RoI pooling</h4><ul><li>It serves as a voting summary</li><li>It takes the average of each layer of score maps in one colour for every class and feed that score into the corresponding position</li><li>So 9 grids information is squeezed to 1 grid, but the thickness stays the same at C+1</li><li>In the above figure, the pooling layer is 3x3. Eg. If the bottom-right grid (light blue) has the highest score for a certain class Ci, that means there is a high probability that that region contains a class Ci object.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UvFs_gW1o39Ss61arspN6g.png" /><figcaption>In example 1 here, the RoI is bounding a person, and the voting map has generally high score (represented by light colours) (<a href="https://arxiv.org/abs/1605.06409">ref</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hAXIh-vzHq1Sp9XIHJqYzQ.png" /><figcaption>In example 2 here, the RoI is shifted, so the voting map only has a few high score and many low score areas (low score areas is represented by dark colours) (<a href="https://arxiv.org/abs/1605.06409">ref</a>)</figcaption></figure><h4>OHEM (Online Hard Example Mining)</h4><ul><li>Use RPN to get RoI and sort for both positive and negative samples</li><li>Pick the front N RoIs (harder examples) and negative examples in a way that positive:negative = 1:3</li><li>Hard examples to make it more robust</li></ul><h4>Training procedure</h4><ul><li>It is similar to faster R-CNN: train PRN and R-FCN alternatively</li></ul><h4>Performance improvement</h4><ul><li>Improve position sensitivity compared to Faster R-CNN</li><li>Improve training and test time by about 2.5x based on ResNet-101</li><li>OHEM does not affect time spent</li></ul><h3>Reference</h3><p><a href="https://www.chinahadoop.cn/course/2038">link1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db29d0320701" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dejunhuang/learning-day-65-object-detection-4-r-fcn-db29d0320701">Learning Day 65: Object detection 4 — R-FCN</a> was originally published in <a href="https://medium.com/dejunhuang">dejunhuang</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Day 64: Object detection 3 — Fast R-CNN and Faster R-CNN]]></title>
            <link>https://medium.com/dejunhuang/learning-day-64-object-detection-3-fast-r-cnn-and-faster-r-cnn-9c641b095547?source=rss-fc04c8eeae27------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c641b095547</guid>
            <category><![CDATA[dejunhuang]]></category>
            <category><![CDATA[daily-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[De Jun Huang]]></dc:creator>
            <pubDate>Fri, 18 Jun 2021 12:29:24 GMT</pubDate>
            <atom:updated>2021-06-18T12:30:14.005Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Day 64: Object detection 3 — Fast R-CNN and Faster R-CNN</h3><h3>Fast R-CNN</h3><h4>Advantages</h4><ul><li>Better performance</li><li>Faster than R-CNN and SPP-Net (training: 8.8x faster than R-CNN, single image test: 146x faster at 0.32s)</li><li>End to end object detection</li><li>All layers can be fine tuned</li></ul><h4>New techniques</h4><p><strong>1. RoI pooling for Selective Search</strong></p><ul><li>It is a special case of SPP pooling</li><li>At SPP pooling, there are various grid sizes for the same region</li><li>At ROI pooling, it only uses the finest grid sizes. (eg. 7x7 for VGG)</li><li>In each grid, perform max pooling</li><li>Finding a function f to establish bounding box as close to groundtruth box as possible. It does translation first, scaling second</li></ul><p><strong>2. Multi-task loss</strong></p><ul><li>Combine classification and regression losses to one</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*GK1SV9ZSt5dVqrrMKG8gyQ.png" /><figcaption>Multi-task loss (<a href="https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Girshick_Fast_R-CNN_ICCV_2015_paper.pdf">ref</a>)</figcaption></figure><h4>Training/Fine tuning procedure</h4><p><strong>Mini batch sampling</strong></p><ul><li>Batch size (128) = images in batch (2) x RoI in each image (64)</li><li>RoI grouping based on overlapping with groundtruth with following rules</li><li>(1) 25% objects with IoU ≥0.5</li><li>(2) 75% background with IoU=[0.1, 0.5)</li></ul><p><strong>Other details</strong></p><ul><li>Due to the large amount of RoIs, almost of of the time is used for FC layers calculations. Can be accelerated by using SVD</li></ul><h3>Faster R-CNN</h3><ul><li>Faster R-CNN = Fast R-CNN + RPN (Region Proposal Network)</li><li>Even faster (single image test: 0.198s)</li><li>Replace the last bit of non NN component, Selective Search, with NN structure RPN</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*1DizuwPUoPZGuhFNgkKE8w.png" /><figcaption>Faster R-CNN illustration (<a href="https://arxiv.org/pdf/1506.01497.pdf">ref</a>)</figcaption></figure><h4>Region Proposal Network (RPN)</h4><p><strong>Advantages</strong></p><ul><li>Enable weight sharing for conv layers</li><li>No more offline Selective Search</li><li>Less region proposals but higher quality</li></ul><p><strong>How it works</strong></p><ul><li>Taking conv 5 feature map from the earlier conv layers</li><li>In the sliding window, take <strong>k</strong> anchor boxes of various sizes</li><li>Use 3x3 conv layer to get 256-d layer</li><li>Use 1x1 conv layer to get 4<strong>k</strong>-d layer for regression</li><li>Use another 1x1 conv layer to get 2k-d layer for classification</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*kzcl3KpTYYDvxg0OJI8KqQ.png" /><figcaption>How RPN works (<a href="https://arxiv.org/pdf/1506.01497.pdf">ref</a>)</figcaption></figure><ul><li>For anchor box, eg. k=9 →3 scales (128, 256, 512) with 3 ratios (1:1, 1:2, 2:1)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*NTXEcup_vAEelzEIPLIH8g.png" /><figcaption>Example of anchor boxes at different scales and ratios (<a href="https://www.chinahadoop.cn/course/2038">ref</a>)</figcaption></figure><p><strong>RPN loss</strong></p><ul><li>Lcls for object or non object</li><li>Lreg uses smooth L1</li><li>mini-batch sampling:</li><li>— single image</li><li>— 128 positive samples. IoU &gt; 0.7 anchor boxes or the largest</li><li>— 128 negative samples. IoU&lt; 0.3 anchor boxes</li></ul><h4>Training procedure for Faster R-CNN</h4><p><strong>Step 1.</strong> Train RPN</p><ul><li>Initialise conv layers with pretrained weights from ImageNet</li><li>Generate region proposals and pass to R-CNN</li></ul><p><strong>Step 2.</strong> Train fast R-CNN</p><ul><li>Initialise conv layers with pretrained weights from ImageNet</li></ul><p><em>(notice the above two blocks of conv layers </em><strong><em>are different and not shared</em></strong><em>)</em></p><p><strong>Step 3.</strong> Fine tune RPN</p><ul><li>Initialise conv layers with Fast R-CNN weights</li><li>Fix conv layer, fine tune the rest</li><li>Generate better region proposals and pass to R-CNN</li></ul><p><strong>Step 4. </strong>Find tune Fast R-CNN</p><ul><li>Fix conv layers, fine tune the rest</li></ul><p><em>(Notice that the conv layers in step 3 and 4 </em><strong><em>are shared</em></strong><em>, those were the resultant layers trained in step 2)</em></p><h3>Reference</h3><p><a href="https://www.chinahadoop.cn/course/2038">link1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c641b095547" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dejunhuang/learning-day-64-object-detection-3-fast-r-cnn-and-faster-r-cnn-9c641b095547">Learning Day 64: Object detection 3 — Fast R-CNN and Faster R-CNN</a> was originally published in <a href="https://medium.com/dejunhuang">dejunhuang</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Day 63: Object detection 2— SPP-Net]]></title>
            <link>https://medium.com/dejunhuang/learning-day-63-object-detection-2-spp-net-3e04238f39d4?source=rss-fc04c8eeae27------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e04238f39d4</guid>
            <category><![CDATA[daily-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dejunhuang]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[De Jun Huang]]></dc:creator>
            <pubDate>Thu, 17 Jun 2021 19:49:04 GMT</pubDate>
            <atom:updated>2021-06-17T19:49:04.101Z</atom:updated>
            <content:encoded><![CDATA[<h3>SPP-Net</h3><ul><li>Improve on the drawbacks of R-CNN: slow, many calculations (all region proposals (~2000) in each image will go through CNN). Ie. <strong>Image → crop/warp →conv layers →fc layers →output</strong></li><li>Directly feed the entire image to CNN for once and extract region features at Conv5</li><li>Spatial Pyramid Pooling (SPP) to extract features at different size of regions</li><li>SPP-Net flow: <strong>Image →conv layers →spatial pyramid pooling →fc layers →output</strong></li><li>So it takes the advantage of parameters sharing at CNN layer, and can adapt to different input size (since input size is limited by the fc layers not conv layers, in R-CNN the crop has to be warped to fulfil certain size; SPP in SPP-Net elevates this size constraint)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9V6_9ouLivmkWxGqOMHbEw.png" /><figcaption>SPP illustration (<a href="http://kaiminghe.com/eccv14sppnet/index.html">ref</a>)</figcaption></figure><ul><li>How to select Region of Interest (ROI) here: Based on the feature maps at Conv5, select the strongest activations and project the bounding box backwards to the original image. That works because objects in feature maps and original images have the same relative position</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*J3epzEWAZff0iUL1sKKU7A.png" /><figcaption>How to choose ROI (<a href="https://www.chinahadoop.cn/course/2038">ref</a>)</figcaption></figure><h3>SPP-Net fine-tuning procedure</h3><ul><li>With the above SPP-Net flow as the basis</li><li>Load pre-trained model and calculate the SPP features in all ROI, <strong>F</strong></li><li>Use F to fine tune only the fc layers, fc6 →fc7 →fc8 (different from R-CNN which fine tunes the conv layers as well)</li><li>Calculate the new fc7 features and use them for SVM classifier</li><li>Use <strong>F</strong> for bounding box regression</li></ul><p>Remaining drawbacks inherited from R-CNN and new dragback</p><ul><li>Need to storage large amount of features</li><li>Multi-phase training</li><li>Faster than R-CNN but still quite slow</li><li>New drawback: cannot fine tune conv layers before SPP layer</li></ul><h3>Reference</h3><p><a href="https://www.chinahadoop.cn/course/2038">link1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e04238f39d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dejunhuang/learning-day-63-object-detection-2-spp-net-3e04238f39d4">Learning Day 63: Object detection 2— SPP-Net</a> was originally published in <a href="https://medium.com/dejunhuang">dejunhuang</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Day 62: Object detection — R-CNN]]></title>
            <link>https://medium.com/dejunhuang/learning-day-62-object-detection-r-cnn-3cf7aac3e594?source=rss-fc04c8eeae27------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cf7aac3e594</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[daily-learning]]></category>
            <category><![CDATA[dejunhuang]]></category>
            <dc:creator><![CDATA[De Jun Huang]]></dc:creator>
            <pubDate>Wed, 16 Jun 2021 17:22:25 GMT</pubDate>
            <atom:updated>2021-06-17T19:49:46.221Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Day 62: Object detection — R-CNN</h3><h3>Object detection</h3><ul><li>Detect an object + identify what it is</li><li>Detect an object — bounding box</li><li>Identify what the object is —classification</li><li>Models: R-CNN, SPP-NET, Fast, R-CNN, Faster R-CNN</li></ul><h3>Object detection with R-CNN</h3><ul><li>Region Based Convolutional Neural Networks (R-CNN)</li><li>Main idea: modify existing CNN and extract feature maps or fc layer features for object classification and bounding box regression</li></ul><h4>Region proposals — traditional ways to find bounding box</h4><ul><li>Selective Search (SS) — draw a set of bounding boxes, R, based on certain rules, and combine them based on similarity (in terms of colour, texture and size after combining)</li></ul><h4>R-CNN steps</h4><ol><li>Extract region proposal (~2,000) from input image</li><li>warp the regions to meet the CNN input size requirement by stretching or rescaling with black boarders</li><li>Using AlexNet as an example, extract the fc7 layer for SVM classification</li><li>Using the 5th conv layer for bounding box regression</li></ol><h4>R-CNN fine-tunning details</h4><ul><li>With the R-CNN steps above as the basis</li><li>Load pre-trained model and train on all region proposals</li><li>use log loss</li><li>change softmax to N+1 outputs instead of 1000 in eg. AlexNet</li><li>Positive label — if IoU with groundtruth ≥ 0.5</li><li>Negative label — if IoU&lt;0.5</li><li>IoU is the Intersection over Union</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*1xcifR0dEShYGp1O.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*CyPhU6EoLI0MU0eb.png" /><figcaption>IoU formula in graphical form (left) and examples of IoU values with visualization (right) <a href="https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/">ref</a></figcaption></figure><h4>Details for classifier:</h4><ul><li>Taking AlexNet as the example, train SVM classifier At fc7 layer</li><li>Each class (total N classes) has 1 SVM classifier (N SVMs)</li><li>Positive — groundtruth regions</li><li>Negative — IoU&lt;0.3 or wrong object</li></ul><h4>Details for regressor:</h4><ul><li>Taking AlexNet as the example, use pre-trained conv5 for bounding box regression</li><li>Each class 1 regressor</li></ul><h4>Test metrics:</h4><ul><li>True positive: IoU ≥0.5</li><li>False positive: IoU&lt;0.5</li><li>False negative: missed out groundtruth</li></ul><h4>Drawbacks of R-CNN</h4><ul><li>Long training time</li><li>Long inference time</li><li>Take up a lot of storage</li></ul><h3>Reference</h3><p><a href="https://www.chinahadoop.cn/course/2038">link1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cf7aac3e594" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dejunhuang/learning-day-62-object-detection-r-cnn-3cf7aac3e594">Learning Day 62: Object detection — R-CNN</a> was originally published in <a href="https://medium.com/dejunhuang">dejunhuang</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>