<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sahil - on Medium]]></title>
        <description><![CDATA[Stories by Sahil - on Medium]]></description>
        <link>https://medium.com/@sahiltinky94?source=rss-eccb7d1504eb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*L4lpWwCDZ843GbJ4xPKilQ.jpeg</url>
            <title>Stories by Sahil - on Medium</title>
            <link>https://medium.com/@sahiltinky94?source=rss-eccb7d1504eb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:48:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sahiltinky94/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Know about EfficientNet & Implementation from Scratch Using Pytorch]]></title>
            <link>https://sahiltinky94.medium.com/know-about-efficientnet-implementation-from-scratch-using-pytorch-b8ec0c0e6a33?source=rss-eccb7d1504eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8ec0c0e6a33</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Sahil -]]></dc:creator>
            <pubDate>Tue, 16 Apr 2024 08:38:53 GMT</pubDate>
            <atom:updated>2024-04-16T08:38:53.871Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi Guys! In this blogs, I will share my knowledge, after reading <a href="https://arxiv.org/pdf/1905.11946v5.pdf">this research paper</a>, what it is all about!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rcTRdt1ndYmpjG6Nj2qVnQ.png" /></figure><h3>Abstract</h3><ul><li>Study model scaling and identify that carefully balancing network depth, width and resolution can lead to better performance.</li><li>Propose — a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple called ‘<em>compound coefficient</em>’</li><li>Use — design a new baseline network and scale it up to obtain a family of models called ‘EfficientNets’ which achieve better accuracy and efficiency.</li><li>Result — EfficientNet-B7 achieve top-1 accuracy on ImageNet while being 8.4x smaller and 6.1 faster on inference than earliers ConvNets models.</li></ul><h3>Introduction</h3><ul><li>The process of scaling up ConvNets has never been well understood and there are many ways to do it.</li><li>Common way is to increase the depth or width.</li><li>Another less common way is to scale up models by image resolution.</li><li>Though it is possible to scale two or three dimensions arbitrarily, scaling requires tedious manual tuning and still often yields sub-optimal accuracy and efficiency.</li><li>Questions arises?</li></ul><p>Is there a principled method to scale up ConvNets that can achieve better accuracy and efficiency?</p><blockquote>Empirical study shows that it is critical to balance all dimensions of network width/depth/resolution and balance can be achieved by simply scaling each of them with constant ratio.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*dtv52uOvZlp9BGbOz6meKA.png" /><figcaption>Fixed Scaled Coefficient</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*EO-jcVLGUGHHxTeMcZ-hcQ.png" /><figcaption>If input image is bigger…</figcaption></figure><p>So, this paper was the first empirically quantify the relationship among all three dimensions of network width, depth and resolution.</p><h3>Compound Model Scaling</h3><blockquote><strong>Problem Formulation</strong></blockquote><p>In this section, this paper explained the mathematics how ConvNets is computed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*cPwIHSx5J4Uulj0AWSTMBA.png" /><figcaption>Output of each ConvNet Layer</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*ubdFj5qxXGiQVotP9jrQHw.png" /><figcaption>Output of ConvNet N</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*gWYt8VAl5_KjbPSkH9tdYA.png" /><figcaption>Definition of ConvNet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*rNzxKtUJAECpTHzGFbjR_Q.png" /><figcaption>For layer &lt;<em>i</em>&gt;</figcaption></figure><p>Now, as this paper proposed that the network should scaled based on width (w), depth (d) and resolution (r).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*tSR3rT-1tWBqNw8lI0w0Dw.png" /><figcaption>Objective</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*kJsJYL1YjQAXj8gHrxmosA.png" /></figure><blockquote>Scaling Dimensions</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O_DCrXsgyCJ9auZLa6V01w.png" /></figure><p>Setting baseline model of one multiplier, and others to keep constant. For example, increase width multiplier and keep other two constants value 1.</p><p>Based on these, the graph has shown that accuracy quickly saturated after certain value of the multiplier (width or depth or resolution).</p><blockquote>Compound Scaling</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*VuyUjDxC3ZDT0M-owlu4FA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*TewpaLwWOeoYj9bTflAtGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*y-9miZ4At2OCVGUcHtqLHQ.png" /><figcaption>Constraints on multipliers</figcaption></figure><h3>EfficientNet Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*iDYL8tSfe3wST0BaQojvaw.png" /><figcaption>Table for EfficientNet-B0 Baseline Architecture</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Z0r95Omo_ixiULzgu7ejA.png" /><figcaption>MBConv Block (Info from Google)</figcaption></figure><p>In this paper, the MBConv block information was provided too vague. I explored from Google and ChatGPT. This is where the idea came into this picture.</p><h3>Code</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7bd6a36277dcd4a51124e890ded6c06/href">https://medium.com/media/e7bd6a36277dcd4a51124e890ded6c06/href</a></iframe><p>In the above code, you have noticed that gamma was not used for resolution multiplier. During the training, we have to provide the resolution multiplier (r) and according to (r), we need to adjust the width multiplier (w) and depth multiplier (d). Make sure that these are under the constraint on these multipliers.</p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gul_uBweZrW1JjD8-XPr6w.png" /><figcaption>Comparison EfficientNet with earlier ConvNets models</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*w8N-HbaFewcEo78VBAUmwQ.png" /><figcaption>Inteference Latency</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OijXTVfMlJUWErwq_mc5lQ.png" /><figcaption>Comparison on different dataset for pre-Trained EfficientNet model with other ConvNet models</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xRS46-klfhscd0kMCN7Iwg.png" /></figure><p>I tried my best to build from scratch to get “close” with number of parameter mentioned in research paper. It is not much nearly but not specifically mentioned where the author has tweaked the parameter to match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*LTeXyefc9QrOETsB-0ls7w.png" /><figcaption>Fixed d,w,r and increasing pie value</figcaption></figure><p>Please do let us know if I missed anything or share your comments so we can also learn each other.</p><p>That’s all folks!</p><p>Thanks for reading it. Happy Learning! :D</p><p>Here my <a href="http://linkedin.com/in/sahil-40a621168">LinkedIn Profile</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8ec0c0e6a33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Know about MobileNet v2 & Implementation from Scratch Using Pytorch]]></title>
            <link>https://sahiltinky94.medium.com/know-about-mobilenet-v2-implementation-from-scratch-using-pytorch-8e589b55599?source=rss-eccb7d1504eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e589b55599</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sahil -]]></dc:creator>
            <pubDate>Wed, 24 Aug 2022 14:07:08 GMT</pubDate>
            <atom:updated>2022-08-24T14:07:08.681Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi Guys! In this blogs, I will share my knowledge, after reading this research paper, what it is all about!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*M4ZeUl8YS7ugwfQFNSsbtw.png" /></figure><h3>1. Abstract</h3><ul><li>This model is based on an <strong>inverted residual structure.</strong></li><li>Inverted Residual Structure is where the <strong>shortcut connections are between the thin bottleneck layers</strong>.</li><li>In a sense, the bottleneck layers, the <strong>intermediate expansion layer uses lightweight depth-wise convolutions </strong>to filter features as a source of non-linearity.</li></ul><h3>2. Introduction</h3><ul><li><strong>What did Neural Network brought</strong>? Neural network have revolutionised in many areas of machine intelligence, enabling human accuracy for challenging image recognition classification.</li><li><strong>However</strong>? The drive to improve accuracy often comes at a cost. For instance, modern state of the art networks require high computational resources beyond the capabilities of many mobile and embedded application.</li><li><strong>Objective</strong>: This network pushes the state of the art for the mobile application by significantly decreasing the number of operations and memory needed while retaining the same accuracy.</li><li><strong>Contribution</strong>: The Inverted Residual with linear bottleneck.</li></ul><blockquote>“The Inverted Residual with linear bottleneck” intuitionally as<br>a. The modules takes as an input a low dimensional compressed representation.<br>b. Further, it is first expanded to high dimension<br>c. Afterwards, it is filtered with a lightweight depthwise convolution.<br>d. Finally, projected back to a low-dimensional representation with a linear convolution.</blockquote><p><strong><em>Note</em></strong><em>: In this whole research paper, it explained in more detailed about their contribution. If you understood the intuition of their contribution, then you are done for this paper.</em></p><ul><li>This convolutional module is particularly <strong>suitable for the mobile</strong> designs because it allows to <strong>reduce the memory footprint needed during interference</strong> by never fully materialising large intermediate tensors (or layers).</li></ul><h3>3. Preliminaries</h3><h4>3.1 Depthwise Separable Convolution:</h4><ul><li>As I have already explained in <a href="https://sahiltinky94.medium.com/know-about-mobilenet-v1-implementation-from-scratch-using-pytorch-7dd594f4d99b">MobileNetV1</a> blog. Kindly visit if you are not familiar with.</li><li>Here, it uses kernel size = 3 (that will be reduced computationally by 8 or 9 times than standard convolution)</li></ul><h4>3.2 Linear BottleNeck</h4><ul><li>ReLU is used as non-linear activation.</li><li>This paper summarised two properties that are indicative of the requirement that the <a href="https://stats.stackexchange.com/questions/465208/meaning-of-manifold-of-interest#:~:text=Informally%2C%20for%20an%20input%20set,embedded%20in%20low%2Ddimensional%20subspaces.">manifold of interest</a> should be lie in a low-dimensional subspace of higher-dimensional activation space:</li></ul><ol><li>If manifold of interest remains non-zero volume after ReLU transformation, it corresponds to a linear transformation.</li><li>ReLU is capable of preserving complete information about the input manifold, but only if the input manifold lies in a low-dimensional subspace of the input space</li></ol><h4>3.3 Inverted Residual</h4><ul><li>The bottleneck blocks appear similar to residual block where each block contains an input followed by several bottlenecks then followed by expansion.</li><li>They used shortcut between the bottlenecks.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9PiJCFTdLOsQp79xA4zuDg.png" /><figcaption>Fig 1: Illustration of Table 1 and Fig 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*Xlv2_XYz5WOM7DCi3NFJhg.png" /><figcaption>Table 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*d3pTmmDz2q731VjJVL4Z5Q.png" /><figcaption>Fig 2: Diagram as per stride value in DW Convolution</figcaption></figure><h3>4. Model Architecture</h3><ul><li>Basic building block is a bottleneck depth-separable convolution with residual. (Detailed structure is shown in Table 1 and Fig 1)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*lmdSKIj-_xSSJ4cQDmgkOA.png" /><figcaption>Table 2: Architecture of MobileNetV2</figcaption></figure><ul><li>Initially, fully convolution layer with 32 filters, followed b6 19 residual bottleneck layers.</li><li>Used ReLU6 as non-linearity because of the robustness when used with low-precision computation.</li><li>It used 3x3 kernel size as is standard for modern networks and utilise dropout and batch normalisation during training.</li><li>expansion factor (t) = 6</li></ul><p>In Table 2,</p><ul><li>each line describes a sequence of 1 or more identical layers, repeated <strong>n</strong> times.</li><li>All layers in the same sequence have the same number <strong>c</strong> of output channels.</li><li><strong>The first layer of each sequence has a stride s and all others use stride 1.</strong></li><li>Expansion (t) is always applied to the input size. One point to be noted here, when expansion (t) is 1, then convolution (3x3) in first layer of bottleneck (in Fig 2) is not required.</li><li>they applied <strong>width multiplier</strong> to all layers except the very last convolutional layer. This improves performance for smaller models.</li></ul><h3>5. Coding</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/187e1da7021403ae3f61481cc0d1fafd/href">https://medium.com/media/187e1da7021403ae3f61481cc0d1fafd/href</a></iframe><h3>6. Experiment and Result</h3><blockquote>6.1 Performance on ImageNet and Comparison for Different Networks</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*mhzdXWDIbig-9aDBupllMw.png" /></figure><blockquote>6.2 Performance comparison of MobileNet v2 with SSDLite and other real time detector on the COCO dataset object detection</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*BKxV4txB-h5qRxhj5463Tg.png" /></figure><p>That’s all folks!</p><p>Thanks for reading it. Happy Learning! :D</p><p>Here my <a href="http://linkedin.com/in/sahil-40a621168">LinkedIn Profile</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e589b55599" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML-I: Linear Regression (Implementation Using Python from Scratch)]]></title>
            <link>https://sahiltinky94.medium.com/ml-i-linear-regression-implementation-using-python-from-scratch-b8b35435c791?source=rss-eccb7d1504eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8b35435c791</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Sahil -]]></dc:creator>
            <pubDate>Thu, 28 Jul 2022 06:51:22 GMT</pubDate>
            <atom:updated>2022-07-28T06:51:22.613Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi, Guys! In this blog, I will give you the idea about the Linear Regression in Machine Learning from Scratch.</p><p>Before reading, you should aware of Maths: Matrix and Linear Algebra.</p><h3>1. Theory</h3><p>As we have study from higher school about the equation of the line which is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*kv1FgnYf8U37o7oZ7uRbqw.png" /><figcaption>Eq. 1: Equation of the Line</figcaption></figure><p>Now, if we try to make it in general form of equation of the line using matrix and linear algebra, we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*k2c3LwR3-hL5aXvIpwO19A.png" /><figcaption>Eq. 2: General Equation of the Line using Matrix and Linear Algebra</figcaption></figure><p>Here, ‘T’ means transpose of the matrix.</p><p>Ideally, the input will be look like this ..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*_FXA_bIZLVMC6mxcHUKGRw.png" /><figcaption>Format Input of (X,Y)</figcaption></figure><h3>2. Coding</h3><p>Few assumption I’ve taken as:</p><p>Loss Tolerance value: 1e-20</p><p>Learning Rate: 1e-3</p><p>Error Measurement: MSE (Mean Square Error)</p><h4>2.1 My Custom Loss Function (MSE)</h4><p>I created MSE class from scratch which include compute loss value and gradient value.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95d5cd7a23728e94eb9bd7523389af6d/href">https://medium.com/media/95d5cd7a23728e94eb9bd7523389af6d/href</a></iframe><h4>2.2 My Custom Linear Regression</h4><ol><li>First, check the type of data. It should accept Numpy only</li><li>Check number of dimension. If only one, reshape it to (-1,1) otherwise ignore.</li><li>Set coefficient and intercept initial value as 1’s and 0&#39;s.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b513767bdbaa5f93382d986228cba49a/href">https://medium.com/media/b513767bdbaa5f93382d986228cba49a/href</a></iframe><h3>3. Testing and Compare sklearn and my custom model (Linear Regression)</h3><p>Test case 1: x and y both in 1D data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*skudPgKfjXXLc_OOLtBA2Q.png" /></figure><p>Test case 2: x and y both in 2D data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*oZeqiZ6TdFzddZaQdg2kzA.png" /></figure><p>Test case 3: x in 2D data and y in 1D data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*3a3gYpsK_RpLY9OMC8DHhQ.png" /></figure><p>Test case 4: x in 1D and y in 2D data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*jIVMGn9A-9gbN2FItnkpIQ.png" /></figure><h4>Conclusion:</h4><ol><li>Some of my custom model’s results are showing very close to the sklearn model.</li><li>There are few cases like in test case 2: my coefficient value: 0.666, 0.333 and sklearn coefficient value: 0.5, 0.5 which is difference of 0.1 approx. The reason may be due to loss tolerance I’ve set it.</li></ol><p>Here is the testing code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5caac1860478aee44c095eb0a69df3ba/href">https://medium.com/media/5caac1860478aee44c095eb0a69df3ba/href</a></iframe><p>That’s all, folks!</p><p>Thank you for reading it! Happing Learning :D</p><p>Here my <a href="http://linkedin.com/in/sahil-40a621168">LinkedIn Profile</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8b35435c791" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Know about MobileNet v1: Implementation From Scratch using Pytorch]]></title>
            <link>https://sahiltinky94.medium.com/know-about-mobilenet-v1-implementation-from-scratch-using-pytorch-7dd594f4d99b?source=rss-eccb7d1504eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/7dd594f4d99b</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sahil -]]></dc:creator>
            <pubDate>Thu, 31 Mar 2022 10:35:51 GMT</pubDate>
            <atom:updated>2022-03-31T10:35:51.725Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi Guys! In this blogs, I will share my knowledge, after reading <a href="https://arxiv.org/pdf/1704.04861.pdf">this research paper</a>, what it is all about!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t3vRMJfV3w5zvaJuzJEcKQ.png" /></figure><h3>1. Abstract</h3><ul><li>This network brought for mobile and embedded vision application</li><li>Its used <strong>depth-wise separable</strong> convolutions to build light weight deep neural networks.</li><li>In this paper, they introduced two hyper-parameters: <strong>trade-off between latency and accuracy</strong>. Its allow the model builder to chose right sized model for their application based on constraints of the problem.</li></ul><h3>2. Introduction</h3><ul><li>Although, it is true that “making deeper and more complicated networks” able to achieve higher accuracy.</li><li>However, if we observed from network point of view, it has not been more efficient w.r.t speed and size. In real world, such as robotics and self driving car, these need to be consider on computationally limited platform.</li></ul><h3>3. Prior Work</h3><ul><li>MobileNets primarily focus on <strong>optimising for latency</strong> yet yield small network. However, <strong>most paper only considered on “size” but never looked on “speed”</strong>.</li><li>As we stated in abstract, its used depth-wise separable convolution.</li></ul><h3>4. Comparison and Computational cost of standard convolution and depth-wise separable convolution:</h3><p>Let say,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*8Hj5ckBjiqr2qN3K6OItsg.png" /></figure><p><strong>In Standard Convolution,</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*Z75CxkEaa8kU7h_E1dXqWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*exq-xTQQd2CO-bfjpQqBVw.png" /><figcaption>Input * Kernel = Output</figcaption></figure><ul><li>No. of multiplication on 1 convolution operation:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*4OUV5PPP0mMXGHuEJoyaZg.png" /><figcaption>Same as Kernel Size</figcaption></figure><ul><li>Since there are “N” filters and each filter slides vertically and horizontally “Dp” times. Total number of multiplication:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y-3jD5g0N8cYlmNib7ElEA.png" /><figcaption>Total cost on Standard Convolution</figcaption></figure><p><strong>In Depth-wise separable convolution, </strong>its perform two operations: depth-wise convolution and point-wise convolution</p><p>In depth-wise convolution, convolution applied single channel at a time. (N=1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*Vt9NlnV1VfNi5iNZglPBKQ.png" /></figure><p>(.:. Rest of the above concept are same)</p><p>So, total cost of multiplication in depth-wise convolution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*1GkHYkLS2WcuHSC_hustnw.png" /><figcaption>Put N=1 in total cost on Standard convolution equation</figcaption></figure><p>In point-wise convolution, 1x1 convolution operation is applied on “M” channels).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*6PTSSPNe8Mv3fIthy7_5kQ.png" /></figure><p>Total cost of multiplication on point-wise convolution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*hbEu90CsQuSPeAz05yL-3g.png" /><figcaption>Put Dk=1 in total cost on Standard convolution equation</figcaption></figure><p>So, total cost of multiplication on depth-wise separable convolution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*RPVGB8LLQlt-5OK6Hzdoww.png" /><figcaption>Total cost of depth-wise separable convolution</figcaption></figure><p>The above explanation is same as mentioned in research paper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*6MTcw0BpJ2Q-86OloRYg9Q.png" /><figcaption>Definition of depth-wise separable convolution</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*OtDwirVNxASwIalMC8c06w.png" /><figcaption>Illustration between standard convolution and depth-wise separable convolution</figcaption></figure><p>You can also see the reduction cost as I proved it mathematically above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/1*iSjDMFXGFV-Rxvp4-Y-GXw.png" /></figure><h3>5. Network Architecture and Training</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*KmVC3xNSOXlsKj2qJzh9Xw.png" /><figcaption>Table 1: MobileNet Architecture</figcaption></figure><ul><li>All layers are followed by BatchNorm and ReLU activation function except final layer (FC).</li><li>Last layer used no activation function and feed into softmax layer for classification.</li><li>Since, we are using depth-wise convolution, we will follow as per right side architecture given below figure.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*w8Qp6IYzFUTGTa_Bsw6tnQ.png" /><figcaption>Left side: Standard Convolution, Right side: Depth-wise separable convolution</figcaption></figure><ul><li>Down-sampling is handled with strided convolution in the depthwise convolution as well as first layer also.</li><li>Average pooling reduces the spatial resolution to 1 before FC.</li></ul><p>Counting depth-wise and point-wise as separated layer, MobileNet has 28 layers.</p><p>If you observe the MobileNet architecture, the most of the computation will be on 1x1 Conv.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*4839X0u70DQafq6gLHmxtQ.png" /><figcaption>Resource Utilization Per Layer</figcaption></figure><p>During training…</p><ul><li>used less regularization and data augmentation techniques because small models have less trouble with overfitting.</li><li>No label smoothing.</li><li>Very little or no weight decay (l2 regularization) on depth-wise filters since their are so few parameters in them.</li></ul><h3>5.1 Code For MobileNet Architecture (as per Table 1)</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51efde6424cc8638e01cfe260317e874/href">https://medium.com/media/51efde6424cc8638e01cfe260317e874/href</a></iframe><h3>5.2 Thinner Model: Width Multiplier</h3><ul><li>To construct smaller and less computationally expensive model, they introduced a simple parameter <strong>α </strong>called width multiplier.</li><li>For given α, we simple multiply to input channel and output channel. The total computational cost for Depth-wise Separable Convolution with width multiplier:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*-7P_o8ExS69_VhGAIT5Vbg.png" /></figure><p>Note: Just replace M → <strong>α</strong>M and N → <strong>α</strong>N</p><ul><li>For width multiplier, α ∈ (0, 1]. So, α=1 is the baseline MobileNet model (Table 1). For α&lt;1 are reduced MobileNets. Typically, <strong>α</strong> set as 1, 0.75, 0.5 or 0.25.</li><li>When reduced α, then computational cost and number of parameter becomes α².</li></ul><h3>5.3 Code for MobileNets with width multiplier ‘α’</h3><ul><li>Just add width-multiplier in first convolution layer and self.nlayer_filter.</li><li>In layer_construct function, make it int() type because when multiplied with decimal return decimals but Conv module take int() type in input and output channel as argument.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4201c67c38cd17f3a84decce4f3c7d1d/href">https://medium.com/media/4201c67c38cd17f3a84decce4f3c7d1d/href</a></iframe><h3>5.4 Reduced Representation: Resolution Multiplier</h3><ul><li>Second hyper-parameter is resolution multiplier (<strong>ρ</strong>) to reduce the input image size.</li><li>When the input image size reduced, subsequently every layer will reduced by the same multiplier.</li><li>The total cost of Depth-wise Separable Convolution with width multiplier and resolution multiplier:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*Fgj0q33z1yct-a9CiJnnPQ.png" /></figure><p>(Note: ρ is multiplied to input image D_F in above equation)</p><ul><li>For resolution multiplier, ρ ∈ (0,1]. So, ρ=1 is baseline MobileNet model and for ρ&lt;1 are reduced computational models. Typically, ρ set as 224, 192, 160 or 128.</li></ul><p>(Note: When resolution multiplier applied, only the computational part (Millions of Multi-Add operations) is reduced not the number of parameters!)</p><p>In code, you have to resize your input image before feeding to the model.</p><h3>6. Experiment</h3><blockquote>6.1 Compare the parameters between Standard Conv and Depth-wise Separable Conv</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*DOMPKGlaioeJveEFkOxiEw.png" /></figure><p>From the observation above, with 7x times fewer parameters able to achieve nearly 70% with 1% difference.</p><blockquote>6.2 Comparison of Different width multiplier with same image resolution 224</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*aU-2c7NGlM89lthzdaFO2w.png" /></figure><blockquote>6.3 Comparison of Different resolution multiplier with same width multiplier=1</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*dVjNZmQ23A_fZGcpGxJrRw.png" /></figure><blockquote>6.4 Comparison with previous know models</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*wcvL1J4sV4OpMbHMs4-APg.png" /></figure><ul><li>MobileNet is nearly as accurate as <strong>VGG16 </strong>while being <strong>32 times smaller</strong> and <strong>27 times less compute intensive</strong>.</li><li>It is <strong>more accurate than GoogleNet</strong> while being <strong>smaller </strong>and <strong>more than 2.5 times less computation</strong>.</li></ul><blockquote>6.5 Comparison with smaller neural networks</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*ugqX-CsvxmnjumOuL_6oUw.png" /></figure><ul><li>Reduced MobileNet with width multiplier α = 0.5 and reduced resolution 160 × 160.</li><li>Reduced MobileNet is <strong>4% better than AlexNet</strong> while being <strong>45× smaller and 9.4× less compute than AlexNet</strong>.</li><li>It is also <strong>4% better than Squeezenet </strong>at about the same size and <strong>22× less computation</strong>.</li></ul><p>That’s all folks!</p><p>Thanks for reading it. Happy Learning! :D</p><p>Here my <a href="http://linkedin.com/in/sahil-40a621168">LinkedIn Profile</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7dd594f4d99b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Know about Inception v2 and v3; Implementation using Pytorch]]></title>
            <link>https://sahiltinky94.medium.com/know-about-inception-v2-and-v3-implementation-using-pytorch-b1d96b2c1aa5?source=rss-eccb7d1504eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1d96b2c1aa5</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Sahil -]]></dc:creator>
            <pubDate>Sat, 26 Jun 2021 11:00:59 GMT</pubDate>
            <atom:updated>2021-07-01T01:27:50.528Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi Guys! In this blogs, I will share my knowledge, after reading <a href="https://arxiv.org/pdf/1512.00567v3.pdf">this research paper</a>, what it is all about! Before I proceed it, I want you to know that I didn’t go and study very extensively. It was only means to understand that</p><ul><li>What this research paper is all about?</li><li>How was different than previous state-of-the-art model?</li><li>What was the result of this novel approach compared to old ones (previous ones)?</li></ul><p>So. all of these are written here as a key points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4t9Pns4oOT3xf4LeslD68g.png" /></figure><blockquote>Inception v2 is the extension of Inception using <strong>Factorizing Asymmetric Convolutions</strong> and <strong>Label Smoothin</strong>g.</blockquote><blockquote>Inception v3 (Inception v2 + BN-Auxiliary) is chosen as the best one experimental result from different Inception v2 models.</blockquote><h3>Abstract</h3><ul><li>Although increased model size and computational cost tend to translate to immediate quality gains for most tasks, <strong>computational efficiency and low parameter count are still enabling factors for various use cases.</strong></li><li>Here they exploring a ways to scale up the networks in the ways that aim at utilizing the added computation as efficiently as possible by suitably <strong>factorized convolutions</strong> and <strong>aggressive regularization</strong></li></ul><h3>Factorizing Convolutions with Large Filter Size</h3><ul><li>Since Inception networks are fully convolutional, each weight corresponds to one multiplication per activation. Therefore, <strong>any reduction in computational cost results in reduced number of parameters</strong>.</li><li>Convolutions with larger spatial filters (e.g. 5 × 5 or 7 × 7) tend to be disproportionally expensive in terms of computation.</li></ul><blockquote>For example; as per the research paper,</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*OzHsiEicnavvE-7MKaMmQg.png" /><figcaption>Mini-network replacing the 5x5 conv with 3x3 conv and 1x1 conv</figcaption></figure><blockquote>a 5 × 5 convolution with n filters over a grid with m filters is 25/9 = 2.78 times more computationally expensive than a 3 × 3 convolution with the same number of filters. Of course, a 5×5 filter can capture dependencies between signals between activations of units further away in the earlier layers, so <strong>a reduction of the geometric size of the filters comes at a large cost of expressiveness</strong>. However,<strong> we can ask whether a 5 × 5 convolution could be replaced by a multi-layer network with less parameters with the same input size and output depth</strong>. In left side figure given, If we zoom into the computation graph of the 5 × 5 convolution, we see that each output looks like a small fully-connected network sliding over 5×5 tiles over its input.</blockquote><ul><li>Having a two layer replacement for the 5 × 5 layer, it seems reasonable to reach this expansion in two steps: increasing the number of filters by √α in both steps.</li><li>It end up with a net (9+9)/25 × reduction of computation, resulting in a relative gain of 28% by this factorization.</li></ul><h4>But it raises two questions:</h4><p>Q1- Does this replacement result in any loss of expressiveness?</p><p>Q2- If our main goal is to factorize the linear part of the computation, would it not suggest to keep linear activations in the first layer?</p><p>Ans- We have ran several control experiments (for example see Figure 1) and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*XkUSuCZUMrTRSeVhit7llw.png" /><figcaption>Figure 1.</figcaption></figure><p>using <strong>linear activation was always inferior to using rectified linear units in all stages of the factorization</strong>.</p><p>They attribute this gain to the enhanced space of variations that the network can learn especially if we batch normalize the output activations. <strong>One can see similar effects when using linear activations for the dimension reduction components.</strong></p><h3>Spatial Factorization into Asymmetric Convolutions</h3><ul><li>We can ask the question whether one should factorize them into smaller, for example 2×2 convolutions. However, it turns out that one can do even better than 2 × 2 by <strong>using asymmetric convolutions, e.g. n × 1</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/1*GHz_5TkE9E7d_PF8PkxyIQ.png" /><figcaption>Figure 3: Mini-network replacing the 3 × 3 convolutions. The lower layer of this network consists of a 3 × 1 convolution with 3 output units</figcaption></figure><blockquote>For example (as per research paper) using a 3 × 1 convolution followed by a 1 × 3 convolution is equivalent to sliding a two layer network with the same receptive field as in a 3 × 3 convolution (see Figure 3)</blockquote><blockquote>Still <strong>the two-layer solution is 33% cheaper for the same number of output filters</strong>, if the number of input and output filters is equal. <strong>By comparison, factorizing a 3 × 3 convolution into a two 2 × 2 convolution represents only a 11% saving of computation.</strong></blockquote><ul><li>One can replace any n × n convolution by a 1 × n convolution followed by a n × 1 convolution and the computational cost saving increases dramatically as n grows (see Figure 6 below).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*EfwVW5zdQu8D5WXId2YDqw.png" /><figcaption>Figure 6: Inception modules after the factorization of the n × n convolutions.</figcaption></figure><h3>Utility of Auxiliary Classifiers</h3><p>They found that</p><ul><li><strong>Auxiliary classifiers did not result in improved convergence early in the training</strong>: the training progression of network with and without side head looks virtually identical before both models reach high accuracy.</li><li><strong>Near the end of training, the network with the auxiliary branches starts to overtake the accuracy of the network without any auxiliary branch and reaches a slightly higher plateau.</strong></li></ul><p>The auxiliary classifiers act as <strong>regularizer</strong>. This is supported by the fact that the main classifier of the network performs better if the side branch is batch-normalized or has a dropout layer.</p><h3>Efficient Grid Size Reduction</h3><ul><li>In order to avoid a representational bottleneck, <strong>before applying maximum or average pooling the activation dimension of the network filters is expanded.</strong></li><li>We can use two parallel stride 2 blocks: P and C. P is a pooling layer (either average or maximum pooling) the activation, both of them are stride 2 the filter banks of which are concatenated as in Figure 5.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*vDO5V6ytubDu9tbI6Tjx2g.png" /><figcaption>Figure 10</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db7f3afb23e8eafb9073680b2f4e17f2/href">https://medium.com/media/db7f3afb23e8eafb9073680b2f4e17f2/href</a></iframe><h3>Inception-v2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*sLR3Zxdd8mxyg51F3bm6xQ.png" /><figcaption>Table 1: Architecture of Inception-v2</figcaption></figure><ul><li>Factorized the traditional 7 × 7 convolution into three 3 × 3 convolutions.</li><li>For the Inception part of the network, we have 3 traditional inception modules at the 35×35 with 288 filters each. This is reduced to a 17 × 17 grid with 768 filters using the grid reduction technique.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*DmTA6yDr7lJrU7SGBxDGjA.png" /><figcaption>Figure 5: (3xInception) in Table 1</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9587668171aafa89946c9b0dd8a124ca/href">https://medium.com/media/9587668171aafa89946c9b0dd8a124ca/href</a></iframe><ul><li>This is is followed by 5 instances of the factorized inception modules as depicted in Figure 5.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51cb6de3b70fb1550b9dfbe157a15833/href">https://medium.com/media/51cb6de3b70fb1550b9dfbe157a15833/href</a></iframe><p>Figure 6: (5xInception) — Already there above. Under “Spatial Factorization into Asymmetric Convolutions” Section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*uR7iwysGaC9Q8jm_2PuJuQ.png" /><figcaption>Figure 7: (2xInception) in Table 1</figcaption></figure><ul><li>This is reduced to a 8 × 8 × 1280 grid with the grid reduction technique depicted in Figure 10 (Under “Efficient Grid Size Reduction” section).</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e337a3cbaaf3bcf37d99695b4b414b09/href">https://medium.com/media/e337a3cbaaf3bcf37d99695b4b414b09/href</a></iframe><ul><li>At the coarsest 8 × 8 level, we have two Inception modules as depicted in Figure 6, with a concatenated output filter bank size of 2048 for each tile.</li></ul><p><strong>This network is 42 layers deep, this computation cost is only about 2.5 higher than that of GoogLeNet and it is still much more efficient than VGGNet.</strong></p><h3>Model Regularization via Label Smoothing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*rJzK1g9saw9XlFyV5Vkl9Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*FJ7zQqSP41v5n2mMp8UYhA.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/da3257d571885716d62e88868734e687/href">https://medium.com/media/da3257d571885716d62e88868734e687/href</a></iframe><h3>Training Methodology</h3><ul><li>stochastic gradient utilizing the TensorFlow distributed machine learning system using 50 replicas running each on a NVidia Kepler GPU</li><li>Batch size=32</li><li>Epoch=100</li><li>RMSProp with 0.9 decay and 1.0 noise</li><li>learning rate of 0.045, decayed every two epoch using an exponential rate of 0.94. In addition, gradient clipping with threshold 2.0 was found to be useful to stabilize the training.</li></ul><h3>Experimental Results and Comparisons</h3><ul><li>Table 3 shows the experimental results about the recognition performance of our proposed architecture (Inceptionv2)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*muFkUnBlan96neeU4Ab3MQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*0BTimq0S0kbBNpXnq_QrAg.png" /></figure><p><strong>BN-auxiliary</strong> refers to the version in which the fully connected layer of the auxiliary classifier is also batch-normalized, not just the convolutions.</p><h3>Complete Code from Scratch</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a7c32d81d112dd3bb06b30d00a48b7f/href">https://medium.com/media/0a7c32d81d112dd3bb06b30d00a48b7f/href</a></iframe><p>That’s all the key-points for this research paper. Hope you got it.</p><p>Thank you for reading it and Have a nice day! :D</p><p>Here my <a href="http://linkedin.com/in/sahil-40a621168">LinkedIn Profile</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1d96b2c1aa5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Know about Inception and Implementation using Pytorch]]></title>
            <link>https://sahiltinky94.medium.com/know-about-inception-and-implementation-using-pytorch-21cc0fec0fee?source=rss-eccb7d1504eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/21cc0fec0fee</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Sahil -]]></dc:creator>
            <pubDate>Sat, 10 Apr 2021 13:28:28 GMT</pubDate>
            <atom:updated>2021-04-10T13:28:28.075Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi Guys! In this blogs, I will share my knowledge, after reading <a href="https://arxiv.org/pdf/1502.03167.pdf">this research</a> paper, what it is all about! Before I proceed it, I want you to know that I didn’t go and study very extensively. It was only means to understand that</p><ul><li>What this research paper is all about?</li><li>How was different than previous state-of-the-art model?</li><li>What was the result of this novel approach compared to old ones (previous ones)?</li></ul><p>So. all of these are written here as a key points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o9ILBJ_eBmOIAMWeUKOWaw.png" /></figure><blockquote>**It is the extension of GoogLeNet, which introduced Batch Normalization**</blockquote><h3>Abstract</h3><ol><li>What was a difficulty till now? Training DNN is complicated by the fact that the <strong>distribution in each layer’s input changes during training as the parameters of the previous layers change</strong>. This <strong>slows down the training</strong> by <strong>requiring lower learning rates</strong> and <strong>careful parameter initialization</strong> and <strong>makes it notoriously hard to train models with saturating non-linearities</strong>. This phenomenon as<em> internal covariate shift.</em></li><li>How to address? By normalizing layer inputs called <strong>Batch Normalization (BN)</strong>. BN <strong>allows us to use much higher learning rates</strong> and be <strong>less careful about initialization</strong>. It also <strong>act as a regularizer, in some cases eliminating the need for Dropout.</strong></li><li>What will achieve? BN achieves the <strong>same accuracy with 14 times fewer training steps</strong> and beats the original model by a significant margin.</li></ol><h3>Introduction</h3><ol><li>SGD is good but….? While stochastic gradient is simple and effective, it requires careful tuning of the model hyperparameters, specifically the learning rate used in optimization as well as the initial values for the model parameters.</li><li>How its affected? the inputs to each layer are affected by the parameters of all preceding layers. So that small changes to the network amplify as the network becomes deeper.</li><li>We already know why ReLU activation used? To avoid vanishing gradient (according to this paper). However, if we could ensure that the distribution of non-linearity inputs remains more stable as the network trains, then the optimizer would be less likely to get stuck in the saturated regime and the training would accelerate.</li><li>How<em> internal covariate shift</em> named? Change of distributions of internal nodes of a deep network, in the course of training, as Internal Covariate Shift.</li></ol><h3>Towards Reducing Internal Covariate Shift</h3><h4>Initially first cut approach was:</h4><ol><li>By fixing the distribution of the layer input x as the training progresses, we expect to improve the training speed.</li><li>It has long know by <a href="https://ieeexplore.ieee.org/document/6853582">this</a>, the network training converges faster if its inputs are whitened. (Whitened means — linearly transformed to have zero means and unit variances and decorrelated)</li><li>As each layer observes the input produced by the layers below, it would be advantageous to achieve the same whitening of the inputs of each layer.</li><li>By whitening the inputs to each layer, it would take a step towards achieving the fixed distributions of inputs that would be remove the ill (distortion) effects of the internal covariate shift.</li></ol><h4>Drawback about this first cut approach</h4><p>Whitening the layer inputs is expensive as</p><ul><li>it required computing covariance matrix</li><li>its inverse square root to produce the whitening activations.</li><li>as well as their derivatives of these transforms for backpropagation.</li></ul><p>What we were seeking for? An alternative that performs input normalization in a way, that is, differentiable and does not require the analysis of the entire training set after parameter update.</p><h3>Normalization via Mini-Batch Statistics</h3><ul><li>Because of the above drawback, they make two necessary simplifications.</li></ul><ol><li>First, instead of whitening the features in layer inputs and output jointly, they will normalize each scalar feature independently, by making the mean of zero and the variance of 1 (one).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*BNTeHiM2hWu33PTXSdKidA.png" /></figure><blockquote>Can change what layer can represent? Note that simply normalizing each input if a layer may change what the layer can represent. For instance, normalizing the input of a sigmoid would constrain them to the linear regime of the nonlinearity.</blockquote><p><strong>Objective</strong>: To make sure, the transformation inserted in the network can represent the identity transform.</p><p>To <strong>accomplish </strong>this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*kQlTjIq2Q_F3pgAyS0GqGw.png" /></figure><p>Now, this could only work when we use whole training networks but impractical when using stochastic optimization.</p><p>2. Second simplification, since we are using mini-batches in stochastic gradient training, <strong>each mini-batch produces estimates of the mean and variance of each activation</strong>.</p><ul><li>Consider a mini-batch <strong>B </strong>of size <strong><em>m</em></strong>:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/1*JgyHPP2nv4PlFSk2DadQSw.png" /></figure><ul><li>Normalize values:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*ue3BzwOKFMufU5t7o-h1uQ.png" /></figure><p>BN Transform Algorithm applied to activation over mini-batch</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*KkYzbIGt_WacaXv3frSbPQ.png" /><figcaption>Taken from research paper</figcaption></figure><p>Red arrow marked are need to be learned as training, and rest of the input values are fixed (obvious that input dataset remain unchanged).</p><p>The distribution of the values of any x(hat) has to be expected value of 0 and the variance of 1 as long as elements of each mini-batch are sampled from the same distribution.</p><h4>What about gradient during backpropagation?</h4><p>In whitening, it was not possible and expensive.</p><p>In this case: During training, we need to backpropagate the gradient of the loss l through this BN transformation, as well as compute the gradient with respect to the parameters of the BN transform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*WW10nTmWajQJNV1SWljdVA.png" /><figcaption>Derivatives of BN</figcaption></figure><p>Thus, BN transform is differentiable transformation that can introduces normalized activation into the network. This ensure that as model is training, layers can continue learning on input distribution that <strong>exhibit less internal covariate shift</strong>, <strong>thus accelerating the training</strong>.</p><h3>Training and Inference with Batch-Normalized Networks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*IgbwA0s4XkId7VS1AIC5yQ.png" /></figure><p>where</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*3WfJtNlkMg10ZvWDcoVeYA.png" /></figure><p>Since <strong>the means and variances are fixed during inference</strong>, the <strong>normalization is simply a linear transform applied to each activation</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*Dlzxi4PENWubhivUxi0l1A.png" /><figcaption>Taken from research paper</figcaption></figure><p>From 1–6, it is about training part. While training, weights are updated on the basis of loss l. And we have seen above the derivatives in BN transform, we can also update γ and β w.r.t loss l.</p><p>Line 7–12, it about inference part. At each mini-batch, it will calculate mean and variance with given sample size <em>m</em> and transform it with given trained γ and β values.</p><h3>Batch-Normalized Convolutional Networks</h3><p>Generally in traditional neural network which consist of an affine transformation followed by an element-wise nonlinearity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*FIa4-KxeZpPOm5CLvhh3kA.png" /></figure><p>We <strong>add BN transform immediately before the nonlinearity by normalizing</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*xKIXQuak-hKcRGToFthh5A.png" /></figure><p>For <strong>convolutional layers</strong>, we normalize all the activations in a mini-batch over all locations — i.e instead of just single value x, we take 2D feature maps of size p x q. Its become, considering batch size <em>m</em>, |B|=m.pq. We learn a pair of parameters γ and β per feature map, rather than per activation.</p><h3>Batch Normalization enables higher learning rates</h3><p>Considering traditional deep networks, which consists of many deep layers which lead to vanish or explode gradient as well as stuck in poor local minima.</p><p>Normally, large learning rate is set whenever increase the scale of layer parameters, which can amplify the gradient during backpropagation and lead to model explosion.</p><p>However, BN is unaffected by the scale of its parameters. Let say we increase by <em>‘a’.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*Rh1wbSdxygaM5pQIjXfrZA.png" /><figcaption>Taken from research paper</figcaption></figure><p>then,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*g3zRr1CHNctG-JZTP3AHbw.png" /><figcaption>Taken from research paper</figcaption></figure><p>Larger weights lead to smaller gradients and BN will stabilize the parameter growth.</p><h3>Batch Normalization regularizes the model</h3><p>They found this advantageous to the generalization of the network. Whereas Dropout is typically used to reduce overfitting, in a batch-normalized network they found that either removed or reduced in strength.</p><h3>Experiments</h3><h4>Accelerating BN networks</h4><ol><li>Increase learning rate: achieve a training speedup from higher learning rates with no ill effects.</li><li>Remove Dropout: removing dropout from Modified BN-Inception speeds up training, without increasing overfitting</li><li>Reduce l2 weights regularization: Modified BN-Inception the weights of this is reduced by a factor of 5.</li><li>Accelerate the learning rate decay: BN Model network train faster than Inception (Without BN), they lower the learning rate 6 times faster.</li><li>Remove Local Response Normalization: While Inception and other models benefits with this, but with BN, we don’t need it.</li><li>Shuffle training examples more thoroughly.</li><li>Reduce the photometric distortions.</li></ol><h4>Single-Network Classification</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*-GDwkuky-QNCF9eHNJ5zNA.png" /><figcaption>Taken from research paper</figcaption></figure><p>Model description as follow:</p><ul><li>Inception: Set initial learning rate of 15e-4</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ycAxF7b6xWn80Lg6KkLSaQ.png" /><figcaption>Inception Architecture</figcaption></figure><ul><li>BN-Baseline: Inception with BN.</li><li>BN-x5: Inception with BN but set the initial learning rate by factor of 5 i.e. 75e-4</li><li>BN-x30: Inception with BN but set the initial learning rate by factor of 30 i.e. 45e-3</li><li>BN-x5-Sigmoid: Like BN-x5 but with sigmoid activation function rather than ReLU.</li></ul><p><strong>Interestingly, increasing the learning rate further (BN-x30) causes the model to train somewhat slower initially, but allows it to reach higher accuracy.</strong></p><h3>Code Implementation Using Pytorch</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/971bde93c84b11149dffdccb63df3d17/href">https://medium.com/media/971bde93c84b11149dffdccb63df3d17/href</a></iframe><p>That’s all the key-points for this research paper. Hope you got it.</p><p>Thank you for reading it and Have a nice day! :D</p><p>Here my <a href="http://linkedin.com/in/sahil-40a621168">LinkedIn Profile</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21cc0fec0fee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Know about GoogLeNet and implementation using Pytorch]]></title>
            <link>https://sahiltinky94.medium.com/know-about-googlenet-and-implementation-using-pytorch-92f827d675db?source=rss-eccb7d1504eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/92f827d675db</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sahil -]]></dc:creator>
            <pubDate>Tue, 30 Mar 2021 15:24:06 GMT</pubDate>
            <atom:updated>2021-03-30T15:24:06.505Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi Guys! In this blogs, I will share my knowledge, after reading <a href="https://arxiv.org/pdf/1409.4842v1.pdf">this research paper</a>, what it is all about! Before I proceed it, I want you to know that I didn’t go and study very extensively. It was only means to understand that</p><ul><li>What this research paper is all about?</li><li>How was different than previous state-of-the-art model?</li><li>What was the result of this novel approach compared to old ones (previous ones)?</li></ul><p>So. all of these are written here as a key points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TtGqRk4slWDN9K5BDX0CmA.png" /></figure><h3>Abstract</h3><ol><li>What the purpose? for setting the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC2014)</li><li>What the main hallmark? the improved utilization of the computing resources inside the network.</li><li>What achievement? carefully crafted design that allows for increasing the depth and width of the network while keeping the computational budget constant.</li><li>How architecture novel idea? The architectural decisions were based on Hebbian principle and the intuition of multi-scale processing.</li></ol><h3>Introduction</h3><p>How can it be reduced to ‘fewer’ parameters than AlexNet? This GoogLeNet submission to ILSVRC 2014 actually used 12x fewer parameters than the winner architecture AlexNet, while being significantly more accurate.</p><h3>Motivation and High Level Considerations</h3><p>The most straightforward way to improving the performance is to increase their size — depth and width. However, it has two drawbacks.</p><p><strong>Drawback 1</strong>: Bigger size typically means larger number of parameters, which makes the enlarged network more prone to overfitting.</p><p><strong>Drawback 2</strong>: Uniformly increased network size is the dramatically increased use of computational resources.</p><p>The fundamental way <strong>to solve above issues</strong> would be moving from fully connected to sparsely connected architectures even inside the convolution.</p><p>Sparse Matrix → Fast Calculation (due to more 0’s present in it)</p><h3>Architecture Details</h3><ol><li>What the main idea? It is based on finding out how an optimal locals sparse structure in a convolutional vision network can be approximated and covered by readily available dense components.</li><li>How do take this? they are restricted to filter sizes 1x1, 3x3 and 5x5 (this decision was based more on convenience rather than necessity). It also means than suggested architecture is a combination of all those layers with their output filter banks concatenated into a single output vector forming the input of the next stage. Adding an alterative parallel pooling path in each such stage should have additional beneficial effect.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*hq-TELPV38u2p_xPK5QAbw.png" /></figure><p>3. There is one problem. What is it? It is that even a modest number of 5x5 convolutions can be prohibitively expensive on top of a convolutional layer with a large number of filters.</p><p>To resolve about drawback, just applying dimension reduction and projections whenever the computational requirements would increase too much otherwise. That is, 1x1 conv is apply before expensive 3x3 and 5x5 convolutions (also include rectified linear activation).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*IlMHbOpiOo_bAH2hJziD4A.png" /></figure><p>4. What is “Inception”? Inception is a network consisting of modules of the above type stacked upon each other with occasional max-pooling layers with stride 2 to half the resolution of the grid.</p><p>Advantages of this network:</p><p>Advantage 1: It allows for increasing the number of units at each stage significantly without an uncontrolled blow-up in computational complexity.</p><p>Advantage 2: It aligns with the intuition that visual information should be processed at various scales and then aggregated so that the next stage can abstract features from different scales simultaneously.</p><h3>GoogLeNet</h3><ol><li>All the convolutions, including those inside the Inception modules, used rectified linear activation.</li><li>The size of the receptive field in this network is 224x224 RGB with normalization.</li><li>“#3x3 reduce” and “#5x5 reduce” stands for the number of 1x1 filters in the reduction layer before then 3x3 and 5x5 convolution.</li><li>One can see the number of 1x1 in the projection layer after built-in max-pooling in the pool proj column.</li><li>All these reduction/projection layers used ReLU activation as well.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NRH0z_K9NlzLvadJfraW9A.png" /></figure><p>Why average pooling layer used instead of fully connected (FC)? It was found that a move from FC layers to average pooling improved the top-1 accuracy by about 0.6%, however use of dropout remain essential after removing FC layers.</p><p>What about gradients? Given the relatively large depth of the network, the ability to propagate gradients back through all the layers in an effective manner was a concern.</p><p>Solution: By adding auxiliary classifier connected to these intermediate layers, it would expect to encourage discrimination in the lower stages in the classifier, increase the gradient signal that get propagated back and provide additional regularization.</p><p>Where do these auxiliary classifier to be put it and what about inference time? These classifier take the form of smaller convolutional networks put on top of the output of the Inception (4a) and (4d) modules. During training, their loss gets added to the total loss of the network with the discount weight (the losses of the auxiliary classifier were weighted by 0.3). At inference time, these auxiliary classifiers are discarded.</p><p>Total Loss = Loss(Whole Network) + Discount*Loss(Auxiliary classifier)</p><p>The exact structure of the auxiliary classifier is as mentioned as per research paper:</p><ul><li>Average pooling with 5x5 filter size and stride 3, resulting in 4x4x512 in Inception (4a) and 4x4x528 for Inception (4d)</li><li>A 1x1 convolution with 128 filters for dimension reduction and ReLU</li><li>A FC layer with 1024 units and ReLU</li><li>A dropout with p=0.7</li><li>A linear layer with softmax loss as the classifier (predicting the same number of classes as the main classifier)</li></ul><h3>Training Methodology</h3><ol><li>SGD with momentum=0.9</li><li>Fixed Learning rate (decreasing the learning rate by 4% every 8 epochs)</li></ol><h3>Result</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*xR7GDAlw2nxTkU6Qj5dLHA.png" /></figure><h3>Code Implementation Using Pytorch</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0834a503ac9f220740b9ccd96b904966/href">https://medium.com/media/0834a503ac9f220740b9ccd96b904966/href</a></iframe><p>That’s all the key-points for this research paper. Hope you got it.</p><p>Thank you for reading it and Have a nice day! :D</p><p>Here my <a href="http://linkedin.com/in/sahil-40a621168">LinkedIn Profile</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92f827d675db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Know about VGG Model and Implementation Using Pytorch]]></title>
            <link>https://sahiltinky94.medium.com/know-about-vgg-model-and-implementation-using-pytorch-ca2c5ba4a1bf?source=rss-eccb7d1504eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca2c5ba4a1bf</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Sahil -]]></dc:creator>
            <pubDate>Tue, 30 Mar 2021 07:25:11 GMT</pubDate>
            <atom:updated>2021-03-30T07:25:11.061Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi Guys! In this blog, I will share my points after went through <a href="https://arxiv.org/pdf/1409.1556v6.pdf">VGG research.</a> Before i proceed it, I want you to know that I didn’t go and study very extensively. It was only means to understand that</p><ul><li>What this research paper is all about?</li><li>How was different than previous state-of-the-art model?</li><li>What was the result of this novel approach compared to old ones (previous ones)?</li></ul><p>So. all of these are written here as a key points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hybtDO9J3bV2RLs4fFr4lw.png" /></figure><h3>Abstract</h3><ol><li>What was the contribution in this paper? “Increase depth” using an architecture with very small (3x3) convolution filters.</li><li>What did its proven? Significant improvement on the prior-art configuration can be achieved by pushing the depth to 16–19 conv layers.</li></ol><h3>Introduction</h3><ol><li>What was the role for challenge? An important role in the advance of deep visual recognition architectures has been played by the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC).</li><li>What was history achievement?<br>The winner of ILSVRC 2011 which used high-dimensional shallow feature encodings. However, the winner of ILSVRC 2012 which was achieved by using deep convnets (AlexNet).</li><li>Attempting more…? With the CovNets becoming more of a commodity in the computer vision field, a<strong> number of attempts have been made to improve the original architecture of AlexNet in a bid to achieved better accuracy</strong>.</li><li>Convey…? They come up with significant more accurate CovNets architectures, which <strong>not only achieve the state-of-the-art accuracy on ILSVRC classification and localization task, but are also applicable to other image recognition datasets</strong>.</li></ol><h3>ConvNet Configuration</h3><h4>Architecture</h4><ol><li>Input (224x224) RGB: During training, the input to their ConvNet is a fixed-size 224x224 RGC image.</li><li>Preprocessing: The preprocessing they do is subtracting the mean RGB value, computed in the training set, from each pixel. (Like Normalization)</li><li>Stack of Conv layers: The image is passed through a stack of convolutional (conv.) layers, where they used filters with a very small receptive field: 3x3.<br>Also attempt to utilise 1x1 convolution filters to differentiale the result.<br>(See Table, C and D ConvNet Configuration)</li></ol><blockquote>Note: AlexNet used 11x11, 5x5 and 3x3 receptive field. But here, they used one receptive field throughout the whole network.</blockquote><p>4. Stride=1: The convolution stride is fixed to 1.</p><p>5. Padding=1: The padding is 1 pixel for 3x3 convolution layers.</p><p>6. Maxpooling: Spatial pooling is carried out by 5 max-pooling layers, which follow some of the conv layers. It is performed over a 2x2 pixel window, with stride 2.</p><p>7. ReLU: All the hidden layers are equipped with the rectification non-linearity.</p><blockquote>Note: Local Response Normalization (LRN) also tried but it does not improved the performance on the ILSVRC dataset and leads to increased memory consumption and computation time.</blockquote><h4>Configuration</h4><ol><li>Table Explain: The ConvNet configurations, evaluated in this paper, one per column. <br>Refer to the nets by their names (A-E) which differ only in the depth: from 11 weight layers in the network A (* conv and 3 FC layers) to 19 weight layers in the network E (16 conv and 3 FC layers)</li><li>Configuration of width: The width of conv layers (the number of channels) is rather small, starting from 64 in the first layer and then increasing by a factor of 2 after each max-pooling layer, until it reaches 512.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*RrI7CrWzRZIg_t9us0ECaw.png" /></figure><p>In Table 2, in spite of a large depth, the number of weights in this networks is not greater than the number of weights in a shallow net with increase widths and larger receptive fields. (144 millions weights in <a href="https://arxiv.org/pdf/1312.6229.pdf">Sermanet et al.</a>)</p><h3>Discussion</h3><p>What they have gained by using a stack of three 3x3 conv layers instead of a single 7x7 layer?</p><ul><li>Firstly, It makes the decision function more discriminative.</li><li>Secondly, Decrease the number of parameters.</li></ul><h3>Classification Framework</h3><h4>Training</h4><ul><li>Mini-batch gradient descent with momentum = 0.9</li><li>batch size = 256</li><li>Regularized by weight decay = 5x10^(-4)</li><li>First two FC used dropout with p=0.5\</li><li>Initial learning rate set to 10^(-2) and decrease by factor 10 when the validation set accuracy stopped improving. In total, learning rate was decreased 3 times and stopped after 370K iterations (74 epochs)</li><li>The initialization of weight was sampled from a normal distribution with zero mean and 10^(-2) variance. The biases were set to zero.</li><li>To obtain the fixed 224x224 ConvNet input images, they were randomly cropped from rescaled training images (one crop per image per SGD iteration). To further augment the training set, the crops underwent random horizontal flipping and random RGB color shift.</li></ul><h4>Implementation details</h4><p>Speed up 3.75 times on an off-the-shelf 4_GPU system as compared to using a single GPU. On a system quipped with four NVIDIA Titan Black GPUs, training a single net took 2–3 weeks depending on the architecture.</p><h3>Classification Experiments</h3><p>The dataset includes images of 1000 classes and is split into three sets: training (1.3M images), validation (50K images) and testing (100K images with held-out class labels).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*Jp9ZbdGVhtbj9wzVBXGW0A.png" /></figure><h3>Code Implementation using Pytorch</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3feabe7c02f3b278d6ae66c69fe07856/href">https://medium.com/media/3feabe7c02f3b278d6ae66c69fe07856/href</a></iframe><p>You can also cross-check the number of parameters of each VGG models</p><p>That’s all for the key points I have put it. Hope you got it! :D</p><p>Thank you for reading!</p><p>Here my <a href="http://linkedin.com/in/sahil-40a621168">LinkedIn Profile</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca2c5ba4a1bf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Word Embedding: GloVe]]></title>
            <link>https://sahiltinky94.medium.com/word-embedding-glove-dd27f630c663?source=rss-eccb7d1504eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd27f630c663</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sahil -]]></dc:creator>
            <pubDate>Fri, 19 Feb 2021 10:16:22 GMT</pubDate>
            <atom:updated>2021-02-19T10:16:22.037Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi Guys! In this blog, I would like to share points on this research paper ‘<a href="https://www.aclweb.org/anthology/D14-1162.pdf">GloVe: Global Vectors for Word Representation</a>’.</p><p>I went all through this research paper, and found that we really need to have a prior knowledge of Maths (Number theory) and Deep Learning concept.</p><p>You will really find confusing some of the equations, but these are comes from Number theory. If you have strong intuition with Number theory, you will feel ease. Otherwise I have also added the link whenever you encounter the part of number theory which you are not aware of it and just take the grasp of what its does and proceed it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gOovrQ2UK5waiRP79f8Deg.png" /></figure><p><strong>Requirement to have some grasp of knowledge:</strong></p><ul><li>Part 1: <a href="https://medium.com/datadriveninvestor/word-embedding-cbow-skip-gram-8262e22fa7c">Word Embedding — CBOW and Skip-gram</a></li></ul><h3>1. Introduction</h3><ul><li>Semantic vector space models of language represent each word with a real-valued vector.</li><li>These vectors can be used as features in a variety of applications (such as information retrieval, document classification, question answering, named entity recognition and parsing).</li><li>Most word vector methods <strong>rely on the distance or angle between pairs of word vectors as the primary method</strong> for evaluating the intrinsic quality of such a set of word representation.</li><li>Like usual, as I explained in Part 1 (Under Goal Section, 2nd point), that for measuring the quality of the resulting vector representations, with the expectation that not only similar words tends to have closer to each other but also have <em>multiple</em> <em>degree of similarity</em>[1]. For example, the analogy “king is to queen as man is to woman” should be encoded in the vector space by the vector equation as <em>king − queen</em> = <em>man − woman</em>.</li></ul><h4><strong>1.1 Drawback:</strong></h4><p>The two main model families for learning word vectors are <em>global matrix factorization method </em>(like <a href="https://medium.com/acing-ai/what-is-latent-semantic-analysis-lsa-4d3e2d18417a">LSA</a>) and <em>local context window methods</em> (like skip-gram model). Both families suffer significant drawbacks.</p><ul><li>Methods like LSA (Latent Semantic Analysis) efficiently provide statistical information, they do <strong>relatively poorly on the word analogy task</strong>, indicating a sub-optimal vector space structure.</li><li>Methods like skip-gram may do better on the analogy task, but they poorly utilize the statistics of the corpus since they <strong>train on separate local context windows </strong><em>instead of</em><strong><em> </em>on global co-occurrence counts</strong>.</li></ul><h4>1.2 Objective</h4><ul><li>Analyze the model properties necessary to produce linear directions of meaning and argue that global log-bilinear regression models are appropriate for doing so.</li><li>Propose a specific weighted least squares model that trains on global word-word co-occurrence counts and thus makes efficient use of statistics.</li><li>Demonstrate that this papers’ methods outperform other current methods on several word similarity tasks, and also on a common named entity recognition (NER) benchmark.</li></ul><h3>2. Related Work</h3><p>(Just don’t focus too much in this section. Just take a grasp about it or You can skip this section, if required. This blog only means to understand more focus on the state-of-the-art model.)</p><h4>2.1 Matrix Factorization Methods</h4><ul><li>We already know why matrix factorization was used? To decompose large matrices that capture statistical information about a corpus.</li><li>In LSA, the matrices are of “term-document” type, i.e., the rows correspond to words or terms, and the columns correspond to different documents in the corpus.</li><li>The Hyperspace Analogue to Language (<a href="https://researchinbox.wordpress.com/2010/03/22/introduction-to-hyperspace-analogue-to-language-hal/">HAL</a>) (Lund and Burgess, 1996), <a href="https://researchinbox.wordpress.com/2012/10/09/hal-example/">for example</a>, utilizes matrices of “term-term” type (known as co-occurrence matrix), i.e., the rows and columns correspond to words and the entries correspond to the number of times a given word occurs in the context of another given word.</li><li>A main problem with HAL and related methods is that the most frequent words contribute a disproportionate amount to the similarity measure: For example, the number of times two words co-occur with <strong><em>the</em></strong> or <strong><em>and</em></strong>, will have a large effect on their similarity even though there is relatively little about their semantic relatedness.</li><li>A number of techniques came for HAL shortcoming, such as COALS methods), in which the co-occurrence matrix is first transformed by an entropy- or correlation-based normalization. Then, later on, a variety of methods also proposed like positive pointwise mutual information (PPMI), Hellinger PCA (HPCA).</li></ul><h4>2.2 Shallow Window-Based Methods</h4><ul><li>To learn word representations that aid in making predictions within local context windows.</li><li>For example, Bengio et al. (2003) introduced a model that <em>learns word vector representations</em> as part of a <em>simple neural network architecture</em> for language modeling. (which I already summarized in Part 1, under Model Architecture, section 3.1)</li><li>The skip-gram and continuous bag-of-words (CBOW) models of Mikolov et al. (2013a)[2] propose a simple single-layer architecture based on the inner product between two word vectors.</li><li><a href="https://papers.nips.cc/paper/2013/file/db2b4182156b2f1f817860ac9f409ad7-Paper.pdf">Mnih and Kavukcuoglu (2013)</a> also proposed closely-related vector log-bilinear models, vLBL and ivLBL</li><li><a href="https://www.aclweb.org/anthology/W14-1618.pdf">Levy et al. (2014)</a> proposed explicit word embeddings based on a PPMI metric.</li><li>In the <strong>skip-gram and ivLBL models, the objective is to predict a word’s context given the word itself</strong>, whereas <strong>the objective in the CBOW and vLBL models is to predict a word given its context</strong>. Through evaluation on a word analogy task, these models demonstrated the capacity to learn linguistic patterns as linear relationships between the word vectors.</li><li>Unlike the matrix factorization methods, the shallow window-based methods <strong>suffer from the disadvantage that they do not operate directly on the co-occurrence statistics of the corpus</strong>. Instead, these models scan context windows across the entire corpus, which <strong>fails to take advantage of the vast amount of repetition in the data</strong>.</li></ul><h3>3. The GloVe Model</h3><p>Before proceeding, you need to know how co-occurrence matrix in NLP calculated. You can refer from <a href="https://medium.com/@apargarg99/co-occurrence-matrix-singular-value-decomposition-svd-31b3d3deb305">here</a>.</p><ul><li>How GloVe named as model? Their insights to construct a new model for word representation which they call GloVe, for Global Vectors, because <strong>the global corpus statistics are captured directly by the model</strong>.</li></ul><h4>3.1 Notation</h4><p>Let the matrix of word-word co-occurrence counts be denoted by X.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*2TXXfjpg3TAW7Ut1zVg9vQ.png" /><figcaption>Notation about X_ij</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*tfrUKokfravKTpesuPwVaw.png" /><figcaption>Notation about X_i</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*QRlxMR6JfYDuD2T3jeuX9A.png" /><figcaption>Notation about P_ij</figcaption></figure><h4>3.2 Example for showcase how co-occurrence probabilities can be extracted</h4><ul><li>Suppose we are interested in the concept of thermodynamic phase, for which we might take i = <em>ice </em>and j = <em>steam</em>.</li><li>The relationship of these words can be examined by studying the ratio of their co-occurrence probabilities with various probe words, <em>k</em>.</li><li>For words <em>k </em>related to <em>ice </em>but not steam, say <em>k </em>= <em>solid</em>, we expect the ratio P_<em>ik</em> /P_<em>jk</em> will be large. (Observe the Table 1 below)</li></ul><blockquote><strong>P_ik</strong> means probability that word <em>k</em>=<em>solid </em>appears in the context <em>i</em>=<em>ice </em>which it should be high as ice is (context as) the solid state. Also written as P(k=solid|i=ice)</blockquote><blockquote><strong>P_jk </strong>probability that word appears <em>k</em>=<em>solid</em> in the context <em>j</em>=<em>steam which </em>it should be low because steam is (context as) the solid state. Also written as P(k=solid|j=gas)</blockquote><blockquote>So overall, P_ik/P_jk = P(k=solid|i=ice)/P(k=solid|j=gas) should be high.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1eCxYHTHfZVoPrMuDeFdhg.png" /><figcaption>Table 1 shows these probabilities and their ratios for a large corpus</figcaption></figure><ul><li>It has been experimented and shows that <strong>compared to the raw probabilities, the ratio is better able to distinguish relevant words (solid and gas) from irrelevant words (water and fashion)</strong> as steam and ice are part of water.</li></ul><h4>3.3 Conclusion</h4><ul><li>The above argument suggests that the appropriate starting point <strong>for word vector learning should be with ratios of co-occurrence probabilities</strong> rather than the probabilities themselves.</li><li>Noting that the ratio P_<em>ik</em> /P_<em>jk</em> depends on three words <em>i</em>, <em>j</em>, and <em>k</em>, the most general model takes the form,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/1*27l_YNiTJfrL44OuBe06tA.png" /><figcaption>Eq. 1</figcaption></figure><p>where <strong><em>w </em></strong>∈ R^(d) are word vectors and <strong><em>˜w</em></strong> ∈ R^(d) are separate context word vectors. <strong><em>F</em></strong> may depend on some as-of-yet unspecified parameters (think of like its a function). The number of possibilities for <strong><em>F </em></strong>is vast, but by enforcing (or to make effective) a few desiderata (few something that is needed) we can select a unique choice.</p><h3>4. Math formulation</h3><ul><li>First, we would like <strong><em>F </em></strong>to encode the information present the ratio P_<em>ik</em> /P_<em>jk</em> in the word vector space.</li><li><a href="https://math.stackexchange.com/questions/3721681/vector-space-and-linear-structure#:~:text=The%20result%20is%20that%20the,both%20addition%20and%20scalar%20multiplication.&amp;text=Vectors%20of%20the%20form%20%CE%BB,under%20linear%20combinations%20of%20vectors."><strong>Since vector spaces are inherently (</strong>characteristic way<strong>) linear structures</strong></a>, the most natural way to do this is <strong>with vector differences</strong>. With this aim, we can restrict their consideration to those functions <strong><em>F</em></strong> that depend only on the difference of the two target words, modifying Eqn. (1) to,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*owFSThVIuQKh3xsYwqG6Og.png" /><figcaption>Eq. 2</figcaption></figure><ul><li>Next, it noted that the <strong>arguments of F</strong> in Eqn. (2) <strong>are vectors</strong> while the <strong>right-hand side is a scalar.</strong></li><li>While <strong><em>F </em></strong>could be <strong>taken to be a complicated function parameterized</strong> by, e.g., a neural network, <strong>doing so would obfuscate (</strong>uncertain<strong>) the linear structure</strong> we are trying to capture (that was the objective task we want). To avoid this issue, we can first take the <strong>dot product of the arguments</strong>, (as we already mentioned in above (2nd point) that vector spaces are inherently linear structure, so we can also perform dot product also)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*5gqrHEhuoJZvn5VdkOdnow.png" /><figcaption>Eq. 3</figcaption></figure><p>which prevents <strong><em>F </em></strong>from mixing the vector dimensions in undesirable ways.</p><ul><li>Next, note that for word-word co-occurrence matrices, the distinction between a word and a context word is arbitrary and that we are free to exchange the two roles. The symmetry can be restored in two steps. First, we require that <strong><em>F </em></strong>be a <a href="https://brilliant.org/wiki/homomorphism/"><strong>homomorphism </strong></a>between the groups (R,+) and (R&gt;0, ×), i.e.,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*6uFwaChuDNmmwOo3vMEelA.png" /><figcaption>Eq. 4</figcaption></figure><p>which, by Eqn. (3), is solved by,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*iHddtuAkg2Y7NDCsjSGBBg.png" /><figcaption>Eq. 5</figcaption></figure><ul><li>The solution to Eqn. (4) is <strong>F = exp</strong>, or,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*WqnIVoe3iXkKfqHQpgR_ag.png" /><figcaption>Eq. 6</figcaption></figure><ul><li>We note that Eqn. (6) would exhibit the exchange symmetry if not for the log(X_<em>i</em>) on the right-hand side. However, <strong>this term is independent of <em>k</em></strong> <strong>so it can be absorbed into a bias b_<em>i</em> for w_<em>i</em></strong> . Finally, <strong>adding an additional bias ˜b_<em>k</em> for ˜w_<em>k</em></strong> restores the symmetry,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*XAu_EsaoKYWbkAP6WLXSAw.png" /><figcaption>Eq<strong>. </strong>7</figcaption></figure><ul><li>Eqn. (7) is a drastic simplification over Eqn. (1), but it is actually ill-defined since <strong>the logarithm diverges whenever its argument is “zero” </strong>. (Here the <a href="https://tutorial.math.lamar.edu/classes/calcii/convergenceofseries.aspx">link</a> explained who we decide it’ll converge or diverge)</li><li>One resolution to this issue is <strong>to include an additive shift in the logarithm</strong> which maintains the sparsity of X while avoiding the divergences.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*f2vAO7C3U_nXvw68aOBLnA.png" /></figure><ul><li><strong>A main drawback to this model is that it “weighs all co-occurrences equally”, even those that happen rarely or never. </strong>Such <strong>rare co-occurrences are “<em>noisy</em>”</strong> and <strong>carry “<em>less information”</em> than the more frequent ones</strong> — yet even just the zero entries account for 75–95% of the data in X, depending on the vocabulary size and corpus.</li><li>They propose a <strong>new weighted least squares regression model </strong>that addresses these problems. Casting Eqn. (7) as a least squares problem and <strong>introducing a weighting function f(X_<em>ij</em>) into the cost function</strong> gives us the model</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*kZM0FAOOxTo9hjyB_VNuPA.png" /><figcaption>Eq. 8 Cost function of the model</figcaption></figure><p>where <strong>V </strong>is the size of the vocabulary.</p><ul><li>The weighting function f(X_<em>ij</em>) should obey the following properties:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*Lv3cSEu9JbQ5YnLNlUvATA.png" /></figure><ul><li>There is one class of functions, which satisfy above properties, that they found to work well can be parameterized as,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*0py-QRb64GFzjb6ZoXsdrQ.png" /><figcaption>Eq. 9</figcaption></figure><ul><li>The performance of the model depends weakly on the cutoff, which they fix to <strong>x_max = 100 </strong>for all our experiments. They found that <strong>α = 3/4</strong> gives a modest improvement over a linear version with α = 1.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*31KB4LgRVb1GOplgD3YAqQ.png" /><figcaption>Fig. 1 Weighting function f with α = 3/4.</figcaption></figure><h3>5. Relationship to Other Models</h3><ul><li>The starting point for the skip-gram or ivLBL methods is a model <strong>Q_<em>ij</em></strong> for the probability that word <em>j</em> appears in the context of word <em>i</em>. For concreteness, let us assume that <strong>Q_<em>ij</em> </strong>is a softmax,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*DYRCsX_66Sc_Ju_gqu0l1w.png" /><figcaption>Eq. 10</figcaption></figure><ul><li>They attempt <strong>to maximize the log probability</strong> as a context window scans over the corpus. T<strong>raining proceeds in</strong> an on-line, <strong>stochastic fashion</strong>, but the <strong>implied global objective function</strong> can be written as,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*doQ7EztBYXFSXse0_K5Nyw.png" /><figcaption>Eq. 11</figcaption></figure><ul><li>Evaluating the normalization factor of the softmax for each term in this sum is <strong>costly</strong>. However, the sum in Eqn. (11) can be evaluated much more efficiently if <strong>we first group together those terms that have the same values for <em>i</em> and <em>j</em>,</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*LJ2GEe-xdYM-VJdpKunz2w.png" /><figcaption>Eq. 12</figcaption></figure><p>where they have used the fact that the number of like terms is given by the co-occurrence matrix X.</p><ul><li>Recalling the notation for X_<em>i</em> and P_<em>ij.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*zHNb_5NXMhowJp1r2TilsQ.png" /><figcaption>Reformulate from P_<em>ij</em></figcaption></figure><ul><li>Replace X_ij into J, rewrite as,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*pYrb-niFoV9exQBrG65mNg.png" /><figcaption>Eq. 13</figcaption></figure><p>where <strong>H(P_<em>i</em> ,Q_<em>i</em>) is the cross entropy of the distributions P_<em>i</em> and Q_<em>i</em></strong> , which we define in analogy to X_<em>i</em> .</p><ul><li>As a weighted sum of cross-entropy error, <strong>this objective bears some formal resemblance to “the weighted least squares objective”</strong> of Eqn. (8). In fact, it is possible to optimize Eqn. (13) directly as opposed to the on-line training methods used in the skip-gram and ivLBL models.</li><li>Cross entropy error is just one among many possible <strong>distance measures between probability distributions</strong>, and it has the <strong>unfortunate property that distributions with long tails are often modeled “poorly” with too much weight given to the unlikely events</strong>.</li><li>This presents a <strong>computational bottleneck owing to the sum over the whole vocabulary</strong> in Eqn. (10) and it would be desirable <strong>to consider a “different distance measure” that did not require this property</strong> of Q.</li><li>A <strong>natural choice would be a “least squares objective”</strong> in which normalization factors in Q and P are discarded.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*gIPNIIgYoyk9GpSJoViyMA.png" /><figcaption>Eq. 14</figcaption></figure><p>where <strong>P_ <em>ij</em> = X_<em>ij</em> and Q_<em>ij</em> = exp((w_<em>i</em>)^T .˜w_<em>j</em>)</strong> are the unnormalized distribution.</p><ul><li>At this stage another problem emerges, namely that <strong>X_<em>ij</em> often takes very large values</strong>, <strong>which can complicate the optimization.</strong></li><li>An effective remedy is to <strong>minimize the squared error of the “logarithms of Pˆ and Qˆ” instead</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*jkRBXRBdpd4HFGwm8ZbEyg.png" /><figcaption>Eq. 15</figcaption></figure><ul><li>They observe that while the <strong>weighting factor X_<em>i</em> is preordained by the on-line training method inherent to the skip-gram and ivLBL models</strong>, it is by n<strong>o means guaranteed to be optimal</strong>. In fact, Mikolov et al. (2013a) observe that <strong>performance can be increased by filtering the data</strong> <strong>so as to reduce the “effective value of the weighting factor for frequent words”</strong>.</li><li>They introduce a <strong>more general weighting function</strong>, which we are free to take to depend on the context word as <strong><em>w</em></strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*wlHb51Sdkhxbss4ADo3dIA.png" /><figcaption>Eq. 16</figcaption></figure><p>which is similar as Eqn. (8)</p><h3>6. Complexity of the model</h3><ul><li>The computational complexity of the model depends on the <strong>number of nonzero elements in the matrix <em>X</em></strong>.</li><li>X. As this number is always less than the total number of entries of the matrix, the model scales no worse than <strong>O(|V|^2).</strong></li><li>At first glance this might seem like <strong>a substantial improvement over the shallow window based approaches (like skip-gram)</strong>, which <strong>scale with the corpus size, |C|</strong>. However,<strong> typical vocabularies have hundreds of thousands of words, so that (|V|^2) can be in the hundreds of billions, </strong>which is actually much larger than most corpora.</li><li>In order t<strong>o make any concrete statements about the number of nonzero elements in <em>X</em></strong>, it is necessary to <strong>make some assumptions</strong> about the <strong>distribution of word co-occurrences</strong>.</li><li>In particular, they assumed that the number of co-occurrences of word i with word j, <strong>X_<em>ij</em></strong>, <strong>can be modeled as a “power-law” function of the frequency rank of that “word pair”, r_<em>ij</em>.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/1*wfN0Xp27B53aAhO4d5YVWg.png" /><figcaption>Eq. 17 Power law function of frequency rank of that word pair</figcaption></figure><ul><li>The <strong>total number of words in the corpus</strong> is <strong>proportional to</strong> the <strong>sum over all elements of the cooccurrence matrix <em>X</em></strong>,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*ulc4MhoRY7944ZGZRWsQOQ.png" /><figcaption>Eq. 18</figcaption></figure><p>where they have rewritten the last sum in terms of the<strong> generalized harmonic number H_<em>n,m</em>. </strong>The upper limit of the sum<strong>, |X|, is the maximum frequency rank</strong>, <strong>which coincides with the number of nonzero elements in the matrix <em>X</em>. </strong>This number, <strong>|X|</strong>, is also equal to the <strong>maximum value of <em>r</em></strong> in Eqn. (17) such that <strong>X_<em>ij</em> ≥ 1</strong>, i.e., <strong>|X| = k^(1/α)</strong>. Rewritten from Eqn. (18) as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*DtSxpjCKsN0ieVK5Lxe0YA.png" /><figcaption>Eq. 19</figcaption></figure><ul><li>We are interested in <strong>how |X| is related to |C| when both numbers are large</strong>; therefore we are <strong>free to expand the right hand side of the equation for large |X|</strong>. We use the expansion of generalized harmonic numbers</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*-KOlhcPaqxKAM3AlMNiElQ.png" /><figcaption>Eq. 20 Apostol, 1976</figcaption></figure><p>giving,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*VjRXQjIk5Wenvx7JbQIzKg.png" /><figcaption>Eq. 21</figcaption></figure><p>where <strong>ζ (s) is the </strong><a href="https://www.britannica.com/science/Riemann-zeta-function"><strong>Riemann zeta function</strong></a><strong>.</strong></p><ul><li>In the limit that X is large, only one of the two terms on the right hand side of Eqn. (21) will be relevant, and which term that is depends on whether α &gt; 1,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*S4q9VzOibvSTj2HdG8KZHA.png" /><figcaption>Eq. 22</figcaption></figure><ul><li>For the corpora studied in this article, we observe that <strong>X_<em>ij</em> is well-modeled by Eqn. (17) with “α = 1.25”</strong>. In this case we have that <strong>|X| = O(|C|^0.8 )</strong>. Therefore we <strong>conclude that the complexity of the model is much better than the worst case O(V^2 )</strong>, and in fact <strong>it does somewhat “better than the on-line window-based methods” which scale like O(|C|).</strong></li></ul><h3>7. Corpora and training details</h3><h4>7.1 Trained our model on five corpora of varying sizes:</h4><ul><li>A 2010 Wikipedia dump with 1 billion tokens</li><li>A 2014 Wikipedia dump with 1.6 billion tokens</li><li>Gigaword 5 which has 4.3 billion tokens</li><li>The combination Gigaword5 + Wikipedia2014, which has 6 billion tokens</li><li>42 billion tokens of web data, from Common Crawl (For the model trained on Common Crawl data, we use a larger vocabulary of about 2 million words.)</li></ul><h4>7.2 Pre-step taken</h4><ul><li><strong>Tokenize</strong> and <strong>lowercase </strong>each corpus <strong>with the Stanford tokenizer</strong>.</li><li><strong>Build a vocabulary</strong> of the <strong>400,000 most frequent words</strong>.</li><li>Construct <strong>a matrix of cooccurrence counts X</strong>. In constructing X, we must <strong>choose how large the context window should be </strong>and<strong> whether to</strong> <strong>distinguish left context from right context</strong>.</li><li>In all cases we <strong>use a decreasing weighting function</strong>, so that<strong> word pairs that are “d” words apart contribute “1/d” to the total count. </strong>This is one way to account for<strong> the fact that very distant word pairs are expected to contain less relevant information about the words’ relationship to one another.</strong></li></ul><h4>7.3 Experiments</h4><ul><li>Set <strong>x_max </strong>= 100 and <strong>α </strong>= 3/4</li><li>Train the model using <strong>AdaGrad</strong></li><li><strong>Stochastically sampling </strong>nonzero elements from <strong><em>X </em></strong><em>(i.e. co-occurrence matrix)</em></li><li>Initial <strong>learning rate of 0.05</strong></li><li>We run <strong>50 iterations for vectors smaller than 300 dimensions</strong>, and <strong>100 iterations otherwise</strong></li><li>Use a <strong>context of ten words to the left</strong> and <strong>ten words to the right.</strong></li></ul><h4>7.4 What model gives?</h4><ul><li>The model generates <strong>two sets of word vectors</strong>, <strong>W </strong>and <strong>˜W</strong> .</li><li>When <strong><em>X</em> is symmetric</strong>, <strong>W</strong> and <strong>˜W</strong> <strong>are equivalent</strong> and <strong>differ only</strong> as a result of <strong>their random initializations; </strong>the <strong>two sets of vectors should perform equivalently.</strong></li><li>There is evidence that <a href="https://papers.nips.cc/paper/2012/file/459a4ddcb586f24efd9395aa7662bc7c-Paper.pdf"><strong>for certain types of neural networks</strong></a><strong>,</strong> <strong>training multiple instances of the network </strong>and <strong>then combining the results</strong> can <strong>help reduce overfitting and noise </strong>and generally <strong>improve results.</strong></li><li>With this in mind, we choose to use the sum <strong>W</strong> +<strong>˜W</strong> as our word vectors. Doing so typically <strong>gives a small boost in performance</strong>, with the <strong>biggest increase in the semantic analogy task</strong>.</li></ul><h3>8. Result</h3><ul><li>The GloVe model performs significantly better than the other baselines, often <strong>with smaller vector sizes and smaller corpora</strong>.</li><li>Our results <strong>using the word2vec</strong> tool are somewhat better than most of the previously published results. This is due to a number of factors, including <strong>our choice to use “negative sampling” </strong>(which typically works<strong> better than the hierarchical softmax</strong>)<strong>, the number of negative samples, and the choice of the corpus</strong>.</li><li>We note that <strong>increasing the corpus size does not guarantee improved results for other models</strong>, as can be seen by the decreased performance of the SVD-L model on this larger corpus.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*kbsIlPL9wjYf8ZQulRVGvQ.png" /></figure><blockquote>The singular vectors of this matrix constitute the baseline “SVD”. For the SVD baselines, we generate a truncated matrix <strong>X_trunc </strong>which retains the information of how frequently each word occurs with only the top 10,000 most frequent words. (matrix-factorization-based methods)</blockquote><blockquote>We also evaluate two related baselines: “<strong>SVD-S</strong>” in which we take the SVD of sqrt(X_trunc), and “<strong>SVD-L</strong>” in which we take the SVD of log(1+Xtrunc).</blockquote><blockquote>(<strong>SG†</strong>) denotes skip-gram and (<strong>CBOW†</strong>) denotes continuous-bag-of-words</blockquote><blockquote><strong>(CBOW∗</strong> ), last row in Table 3, denotes the vectors available on the word2vec website that are trained with word and phrase vectors on 100B words of news data.</blockquote><ul><li>Table 3 shows results on f<strong>ive different word similarity datasets</strong>.</li><li>A <strong>similarity score is obtained</strong> <strong>from the word vectors by</strong> <strong>first normalizing each feature across the vocabulary and then calculating the cosine similarity. </strong>We compute Spearman’s rank correlation coefficient between this score and the human judgments.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*z2b-JApy0ofab_KMGC4V6A.png" /></figure><ul><li>Table 4 shows results on <strong>the NER task with the CRF-based model.</strong></li><li>The GloVe model outperforms all other methods on all evaluation metrics, except for the CoNLL test set, on which the HPCA method does slightly better.</li><li>We conclude that <strong>the GloVe vectors are useful in downstream NLP tasks.</strong></li></ul><h3>References</h3><p>[1] <a href="https://www.aclweb.org/anthology/N13-1090.pdf">Linguistic Regularities in Continuous Space Word Representations</a></p><p>[2] <a href="https://arxiv.org/pdf/1301.3781v3.pdf">Efficient Estimation of Word Representations in Vector Space</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd27f630c663" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Word Embedding: CBOW & Skip-gram]]></title>
            <link>https://medium.datadriveninvestor.com/word-embedding-cbow-skip-gram-8262e22fa7c?source=rss-eccb7d1504eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/8262e22fa7c</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sahil -]]></dc:creator>
            <pubDate>Mon, 11 Jan 2021 09:36:29 GMT</pubDate>
            <atom:updated>2021-01-25T08:58:13.241Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi Guys! In this blog, I would like to share my knowledge on review this research paper ‘<a href="https://arxiv.org/pdf/1301.3781v3.pdf">Efficient Estimation of Word Representations in Vector Space</a>’ where it proposed CBOW and Skip-gram novel architecture using Deep Learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mAaLLj9PzezIfu4Yw_3kSQ.png" /></figure><h3>1. Introduction</h3><p>Word embedding means representing a word into continuous (or numerical) vector representation.</p><p>A simple models, <strong>statistical language modelling</strong> representation(like Bag-Of-Word (BoW), TF-IDF (Term Frequency-Inverse Document Frequency), N-gram model), has good choice with reason — simplicity, robustness and observation. However, its has certain limitation —</p><ul><li>Lack of semantics meaning. For example, <em>tasty </em>and <em>delicious </em>will consider different, however, meaning is same.</li><li>Amount of relevant in-domain data for automatic speech recognition is limited.</li><li>Scaling up words (like corpus which contain few billions of words) will not result in any signification progress.</li></ul><p>Therefore, we need advance techniques and that how Deep Learning comes into this stage where it also gives semantic meaning. For example, Glove, lda2vec, and many more. Also, we know that, nowadays, Deep Learning dominating in most of the research area (like object detection, language modelling, Text-to-speech, etc) with outperforming result than ML.</p><h3>2. Goal</h3><ul><li>Introduce techniques that can be used for learning <em>high-quality word vectors</em> from high data set of billions of words and with millions of words in the vocabulary.</li><li>For measuring the quality of the resulting vector representations, with the expectation that not only similar words tends to have closer to each other but also have <em>multiple</em> <em>degree of similarity</em>[1]. For example, the male/female relationship is automatically learned, and with the induced vector representations, “King - Man + Woman” results in a vector very close to “Queen.”</li></ul><p><strong>Objective: </strong>Maximize accuracy of these vector operation by developing new architecture that preserve linear regularities among words.</p><h3>3. Model Architecture</h3><ul><li>Focus on distributed representation of word learned by neural networks ( shown that they perform significantly better than <a href="https://medium.com/acing-ai/what-is-latent-semantic-analysis-lsa-4d3e2d18417a">Latent Semantic Analysis (LSA)</a> for preserving linear regularities among words)</li><li><a href="https://towardsdatascience.com/latent-dirichlet-allocation-lda-9d1cd064ffa2">Latent Dirichlet Allocation</a> (LDA) becomes computationally very expensive on large data sets.</li></ul><p>Below are related work that had been done previously…</p><h4>3.1 Feedforward Neural Net Language Model (NNLM) [2]</h4><p>(You can read from reference starting from <em>page number 1141</em>,if you have time, as there is many formulation and math involve. I will summarize here what actually it is.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCh-359oshF3B9LqpdGdsg.png" /><figcaption>From research paper [2]</figcaption></figure><ul><li>It consists of input, projection, hidden and output layers.</li><li>At input layer, <em>N</em> previous words are encoded using 1-of-<em>V</em> coding, where <em>V </em>is size of the vocabulary.</li><li>The input layer is then projected to a projection layer <em>C</em> that has dimensionality <em>N X D</em>, using shared projection matrix.</li></ul><blockquote>From research paper [2],</blockquote><blockquote>The parameters of the mapping <strong>C</strong> are simply the feature vectors themselves, represented by a <strong>|V| × m</strong> matrix; C whose <strong>row ‘i’</strong> is the feature vector <strong>C(i)</strong> for <strong>word ‘i’ </strong>and <strong>‘m’</strong> is the dimensionality.</blockquote><blockquote>Here I replace ‘m’ with ‘D’ to make it simple to understand as D correspond to D-dimension of each word vector.</blockquote><ul><li>NNLM architecture becomes <em>complex for computation</em> between the projection and the hidden layer, as values in the projection layer are dense.</li><li>Hidden layer is used to <strong><em>compute probability distribution over all the words in the vocabulary</em></strong>, resulting in an output layer with dimensionality <em>V</em>.</li><li>The function <strong><em>g</em></strong> may be implemented by a feed-forward or recurrent neural network or another parametrized function, with parameters <strong><em>ω</em></strong>. The overall parameter set is <strong>θ = (C,ω)</strong>. Notation written for above figure is given below, where <em>g </em>is the neural network and <em>C(i)</em> is the i-th word feature vector.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*_V6iZoMi8wCbnXjumkcuPw.png" /><figcaption>Notation of above figure</figcaption></figure><p><strong>Training —</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*donNOKlWN0USNNkLvLPxpw.png" /><figcaption>From research paper [2], page no. 1142</figcaption></figure><p>Common choice for this architecture:</p><ul><li><em>N </em>=10</li><li>projection layer, <em>C</em>, — might between 500 to 2000</li><li>hidden layer, <em>H</em>, — typically 500 to 1000 units</li></ul><p>Issues regarding this architecture —</p><ol><li>The computational complexity per training each training sample is:</li></ol><blockquote><em>Q = N X D + N X D X H + H X V</em></blockquote><blockquote>Please remember that each word w_(i) correspond to ‘V’ dimension as it is encode 1-of-V coding. So, for N words, it’ll be N X V dimensions in input layer. However, if we also see that the size of vocabulary will have a finite and fixed all over the network while training, right? All we left is number of words that can feed to input layer i.e. ’N’ which is vary and hyperparameter (to find the right value). That&#39;s why its neglect the ‘V’ term in training complexity. <br>I am following the review as per research paper accordingly.</blockquote><p>The first term is from input layer to projection layer — N X D</p><p>The third term is from hidden to output layer — H X V</p><p>The second term is from projection to hidden layer , which is D X H and also since we have N previous words, so need to perform ’N’ times more — N X D X H (this term is the most where complexity caused).</p><p>2. Since the output units have a size as same as vocabulary <em>V. </em>Generally, V is not small but high number of vocabulary which lead to slow down the performance.</p><p>Solution for (2nd) point—</p><ul><li>With binary tree representation of the vocabulary, the number of the output unit that need to be evaluated can go down to around log_base2(V), i.e., use <em>hierarchical softmax</em> where the vocabulary represented as <strong>Huffman binary tree</strong>.</li><li>The Huffman tree based hierarchical softmax requires about log_base2(<em>Unigram_perplexity</em>(<em>V</em>)). For example, when V=1 Million words, the result is 2x speedup in evaluation.</li></ul><p>Solution for (1st) point —</p><ul><li>the computational bottleneck <em>N X D X H</em>, they later propose architectures that <em>do not have hidden layers </em>and thus depend heavily on the efficiency of the softmax normalization (i.e on output layer).</li></ul><h4>3.2 Recurrent Neural Net Language Model (RNNLM)</h4><ul><li>Recurrent Neural Network based language model used to overcome limitations of previous ones, such as the need of specify the context length (<em>N</em>). Also this model does not have projection layer; only input, hidden and output layer.</li><li>We already know about RNN speciality is that connects hidden layer to itself using time-delayed connections, which allows to form Short-Term Memory (as i<em>nformation from the past</em> can be represented by the hidden layer state that get <em>updated</em> based on <em>current input</em> and state of <em>hidden layer</em> in previous time step)</li></ul><p>The complexity per training sample of RNN model is:</p><blockquote>Q = H X H + H X V</blockquote><p>word representation (dimensionality)— <em>D, </em>hidden layer (<em>H</em>).</p><p>Same issue as previous model —</p><ol><li>output layer <em>V</em> can be reduced to log(<em>V</em>) by using hierarchical softmax. So, the second term become <em>H X log(V)</em>.</li><li>The most computation come in first term, i.e., <em>H X H</em> — which is recursive of hidden layer in RNN.</li></ol><h3>4. Proposed — New Log-linear Models</h3><ul><li>The main observation from the previous works were the most of the complexity is caused by the non-linear hidden layer in the model.</li></ul><h4>4.1 Continuous Bag-of-Words Model</h4><ul><li>the first proposed architecture is similar to the feedforward NNLM (section 3.1), where <strong>non-linear hidden layer is removed</strong> and <strong>projection layer is shared for all words</strong>; all words get projected into same position (their vectors are averaged).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pNPRuA0jDmI6l_Xd2qt5rA.png" /><figcaption>From research paper [3], page no. 5</figcaption></figure><ul><li>Call this architecture a <em>bag-of-words</em> model as the <strong>order of words in the history</strong> doesn’t influence the projection.</li><li>Two things you need to know: <em>context</em> and <em>focus </em>word.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*C61vsnEnJH8kwf3OQGn4-A.png" /><figcaption>Figure 1. Word notation</figcaption></figure><p><em>Focus</em> word are those word (output) that this model need to predict from the given <em>context</em> word. So, from above example, w_3 was when timestep <em>t</em>=3; likewise with given timestep t, we can written as:</p><p>context word=w_(<em>t</em>-2) w_(<em>t</em>-1) w_(<em>t</em>+1) w_(<em>t</em>+2) →two history and two future words from (the focus word w_(t)) current time <em>t.</em></p><p>focus word=w(<em>t</em>)</p><ul><li>It found out that by <strong>building a log-linear classifier </strong>with f<strong>our future and four history words</strong> at the input got the best performance on the task.</li></ul><h4>4.1.1 More in-depth</h4><p>From the same notation given in Fig 1., Illustration of architecture look like</p><p>w_1, w_2, w_4, w_5 are context words which is fed to input layer; w_3 is focus word which need to be predicted in output layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*pqxgkGdUYxagW-u1QAOLRg.png" /><figcaption>Figure 2. Notation in CBOW architecture</figcaption></figure><p>Going to even more depth of the architecture, it looks like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*x6z7uv80d_jN46xFl0SmEQ.png" /><figcaption>Deep Learning Illustration of Figure 2.</figcaption></figure><blockquote>In above figure, I have written weight W_4<em>x</em>D since 4 inputs are fed. If N input are fed then weight become W_N<em>x</em>D.<br>Again, here also I said N; generally is W_N<em>x</em>V<em>x</em>D. Since V will be fixed length among the training network, we will have W_N<em>x</em>D. <br>** So, its upto you how you want to understands. either with N or N<em>x</em>V **</blockquote><p>Ideally, arrow represented as “Fully Connected”. However, as per research paper, its represent as Hierarchical softmax.</p><p>Let magnify more, between ‘D’ layer and Output Layer to see how hierarchical softmax looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*7nl8D2L8_05xbdSChD-KNQ.png" /><figcaption>Figure 3. Magnified between ‘D’ layer and output layer.</figcaption></figure><p>Now, come to the point while at training stage,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*kts5ap0TEC8CRJ3AIvrGGg.png" /></figure><p>For 5 words, it needs to train 5 times repeatedly with every different focus word as a output when network is training. So, if we have Millions of words, we need to train Millions times.</p><p><strong>Training Complexity—</strong></p><blockquote>Q = N X D + D X log_base2(V)</blockquote><p>First term comes from input layer (N) and Projection layer (N)— N X D</p><p>Second term comes from Projection layer (D) to Output layer (V) — D X log_base2(V); [NOT ‘V’ as it is NOT FULLY CONNECTED; Its is hierarchical structure i.e Huffman Coding] and If you know what the time complexity of Huffman Coding? It’s O(n*log(n)). This is very much similar with this second term, right?</p><p><strong>Note that the weight matrix between the input and the projection layer is shared for all word positions in the same way as in the NNLM.</strong></p><p><a href="https://www.datadriveninvestor.com/2020/11/27/deep-learning-amid-increased-physician-administrative-workload/">Deep Learning amid Increased Physician Administrative Workload | DataDrivenInvestor</a></p><h4>4.2 Continuous Skip-gram Model</h4><p>The second architecture is similar to CBOW, but instead of predicting the current word based on the context, it going to take focus word as an input to a log-linear classifier with continuous projection layer, and predict words within a certain range before (history) and after (future)the context word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*f8BWWj4vew1BD9sYT1fRuw.png" /><figcaption>From research paper [3], page no. 5</figcaption></figure><p>So, this time, we fed focus word (w_(t)) into architecture and get the predict within certain range history (i.e. w_(t-2), w_(t-1)) and future context (i.e. w_(t+1), w_(t+2)) word</p><h4>4.2.1 More In-depth</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*g5fFVgH01o8XMIabgzNQhg.png" /><figcaption>Figure 4.</figcaption></figure><p><strong>Training complexity —</strong></p><p>Most people will think that the training complexity for this architecture will be</p><blockquote>Q = D + D x log_base2(V)</blockquote><p>However, that wrong! Let me tell you why. See with the same example in Fig 4.;</p><p>Just like usual; network will go forward phase initially i.e. each focus word w_3 fed to input layer and transit to projection layer and finally get the output of all 4 context words (w_(1), w_(2), w_(4), w_(5)).</p><p>Now, in backward phase, we need to backpropagate to update the weights. Can its be happen to update weights of projection from coming all output layers? (See Fig below for illustration)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*TmdoQwVdIbGa1yM5T7RXxA.png" /></figure><p>No, it can’t be updated. So, we need to do perform 4 times repeated to update projection layer.</p><p>First from w_(3) →w_(1); and during backward stage, update projection layer flow from w_(1) →D →w_(3)</p><p>Second from w_(3) →w_(2); and during backward stage, update projection layer flow from w_(2) →D →w_(3)</p><p>Third from w_(3) →w_(4); and during backward stage, update projection layer flow from w_(4) →D →w_(3)</p><p>Fourth from w_(3) →w_(5); and during backward stage, update projection layer flow from w_(5) →D →w_(3)</p><p>So, you need to repeat it ‘C’ times where C is the number of context word (including history and future words). In above case, we taken 4 words so C=4 and repeat it ‘C’ times.</p><p>Add ‘C” times in Training complexity, we will get: <strong>Q = C × (D + D × log_base2(V ))</strong>, where C is the maximum distance of the words.</p><blockquote>From research paper [3], page no. 4 under section 4.2, last paragraph</blockquote><blockquote>Thus, if we choose C = 5, for each training word we will select randomly a number R in range between &lt;1 and C&gt; and then use R words from history and R words from the future of the current word as correct labels (i.e. This will require us to do R × 2 word classifications, with the current word as input, and each of the R + R words as output.)</blockquote><ul><li>In this experiments, it used C = 10.</li></ul><h3>5. Result</h3><ol><li>With different size of vocabulary ‘V’,</li></ol><p>The training data consists of several LDC corpora and is described in detail in (320M words, 82K vocabulary).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FE2tWUutnldwVfCOXwfzwA.png" /></figure><p>2. Selecting D=640, comparing with all other models,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D_P4xl2bUj8Ka_6nQx7o4g.png" /></figure><p>The CBOW architecture works better than the NNLM on the syntactic tasks, and about the same on the semantic one. Finally, the Skip-gram architecture works slightly worse on the syntactic task than the CBOW model (but still better than the NNLM), and much better on the semantic part of the test than all the other models.</p><p>3. Evaluated this models trained using one CPU only and compared the results against publicly available word vectors (like Google News),</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v567fWEYwjMnN87-3fBx0g.png" /></figure><p>4. Results of several models trained on the Google News 6B data set, with mini-batch asynchronous gradient descent and the adaptive learning rate procedure called Adagrad. They used 50 to 100 model replicas during the training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kosTNLkMhsyz3k_N39YvQw.png" /></figure><h3>References</h3><p>[1] <a href="https://www.aclweb.org/anthology/N13-1090.pdf">Linguistic Regularities in Continuous Space Word Representations</a></p><p>[2] <a href="https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf">A Neural Probabilistic Language Model</a></p><p>[3] <a href="https://arxiv.org/pdf/1301.3781v3.pdf">Efficient Estimation of Word Representations in Vector Space</a></p><p>That’s it. Hope you got the glimpse of how CBOW and Skip-gram architecture works.</p><p>My LinkedIn Profile <a href="https://www.linkedin.com/in/sahil-40a621168/">here</a>.</p><p>Thank you and Have a nice day :D</p><p><strong>Gain Access to Expert View — </strong><a href="https://datadriveninvestor.com/ddi-intel"><strong>Subscribe to DDI Intel</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8262e22fa7c" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/word-embedding-cbow-skip-gram-8262e22fa7c">Word Embedding: CBOW &amp; Skip-gram</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>