<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Apil Tamang on Medium]]></title>
        <description><![CDATA[Stories by Apil Tamang on Medium]]></description>
        <link>https://medium.com/@apiltamang?source=rss-64eccdd45f96------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*6RPIIbxO57eGLM5G.</url>
            <title>Stories by Apil Tamang on Medium</title>
            <link>https://medium.com/@apiltamang?source=rss-64eccdd45f96------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 21:16:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@apiltamang/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Routing Ports internally in Linux using iptables]]></title>
            <link>https://medium.com/@apiltamang/routing-ports-internally-in-linux-using-iptables-51c2b2890349?source=rss-64eccdd45f96------2</link>
            <guid isPermaLink="false">https://medium.com/p/51c2b2890349</guid>
            <dc:creator><![CDATA[Apil Tamang]]></dc:creator>
            <pubDate>Mon, 28 Oct 2019 17:20:23 GMT</pubDate>
            <atom:updated>2019-10-28T17:20:23.137Z</atom:updated>
            <content:encoded><![CDATA[<p>Simply what I have for port forwarding. A lot of web-resources mention *nginx* as a good alternative.</p><pre>(This is for if your local web-server runs on &quot;https&quot;, which operates on port 443 by default. Use 80 instead of 443 if using unsecured http communication.</pre><pre>sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 8443</pre><pre>sudo iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 443 -j REDIRECT --to-ports 8443</pre><pre>sudo iptables-save</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51c2b2890349" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to expose your MySQL Server to the Internet]]></title>
            <link>https://medium.com/@apiltamang/how-to-expose-your-mysql-server-to-the-internet-2d9c736587cd?source=rss-64eccdd45f96------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d9c736587cd</guid>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Apil Tamang]]></dc:creator>
            <pubDate>Fri, 25 Oct 2019 19:05:02 GMT</pubDate>
            <atom:updated>2019-10-25T19:05:02.823Z</atom:updated>
            <content:encoded><![CDATA[<p>More for my own records than to disseminate real knowledge. Very quick and dirty walkthrough. Meant for my Ubuntu server.</p><ol><li>SSH into the database server as: ssh <a href="/service/mailto:root@server1.cyberciti.biz">root@server</a>.mydb.uri</li><li>vi /etc/mysql/mysql.cnf.d/mysqld.cnf to edit the [mysqld] &gt; bind-address to read my server’s ip (local intranet ip issued by the router &lt;db_intranet_ip4&gt;. Use $ifconfig to view the ip address on the bash)</li><li>mysql&gt; CREATE DATABASE MyTable ;</li><li>mysql&gt; GRANT ALL ON MyTable.* TO user@&#39;%&#39; IDENTIFIED BY &#39;SOME_PASSWORD&#39;;</li></ol><p>The ‘%’ tells database to accept connection from any ip address</p><p>5. Open the iptables to port 3306</p><pre>$/sbin/iptables -A INPUT -i eth0 -p tcp --destination-port 3306 -j ACCEPT [enter]</pre><p>6. $iptables-save [enter]</p><p>Finally test connection using:</p><p>7. $mysql -u user -h &lt;db_intranet_ip4&gt; -p</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d9c736587cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing Apache Toree on OS X w/ Conda environment]]></title>
            <link>https://medium.com/@apiltamang/installing-apache-toree-on-os-x-w-conda-environment-9df158b5493b?source=rss-64eccdd45f96------2</link>
            <guid isPermaLink="false">https://medium.com/p/9df158b5493b</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[jupyter]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[conda]]></category>
            <dc:creator><![CDATA[Apil Tamang]]></dc:creator>
            <pubDate>Mon, 17 Dec 2018 18:58:38 GMT</pubDate>
            <atom:updated>2018-12-17T18:58:38.215Z</atom:updated>
            <content:encoded><![CDATA[<p>Looking for a good Spark + Jupyter integration solution. After giving spylon_kernel a try, and seeing that the project seemed to have been abandoned and not working for my case either, decided to give Apache Toree a shot.</p><p>After a couple of hits and misses, here’s the config that seemed to work.</p><ol><li>Create a new env. as follows: conda create --name apache_toree</li><li>Activate env: conda activate apache_toree</li><li>Export spark installation as follows: export SPARK_HOME=/path/to/spark-2.1-hadoop-2.7</li><li>Install toree as: pip install --upgrade toree</li><li>Then do the final step as follows: jupyter toree install --spark_home=$SPARK_HOME --sys-prefix</li><li>Should be it!</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9df158b5493b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Granting external users folder permissions in Ubuntu 16.04]]></title>
            <link>https://medium.com/@apiltamang/granting-external-users-folder-permissions-in-ubuntu-16-04-e0bdb8d5736c?source=rss-64eccdd45f96------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0bdb8d5736c</guid>
            <category><![CDATA[groups]]></category>
            <category><![CDATA[users]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[permission]]></category>
            <dc:creator><![CDATA[Apil Tamang]]></dc:creator>
            <pubDate>Wed, 07 Feb 2018 18:19:31 GMT</pubDate>
            <atom:updated>2018-02-13T01:47:02.613Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently me and a pal (let us call him Harry) are working on a project that requires sharing data quite frequently. In particular, this is a deep-learning project, and the need came about to sharing the weights so Harry can quickly load it in his own machine, and do things Harry does best… (still wonder what that is.. hmmm!)</p><p>Status: currently logged in using ssh to the remote PC running headless Ubuntu-16.04, and as user: <strong>apil</strong> (myself).</p><ul><li>First, add a user using: <strong>sudo adduser harry</strong>. Kicks off a short workflow (i.e. add name, password, telephone number) etc.</li><li>Second, add a new user-group using: <strong>sudo groupadd deeplearners.</strong></li><li>Now add the new user (and myself) to this group using:</li></ul><pre>$sudo usermod -a -G deeplearners harry<br>$sudo usermod -a -G deeplearners apil</pre><ul><li>And finally, to allow Harry to access files/folders inside a folder in my home account, do the following:</li></ul><pre>[Command to be executed by the owner of target folder]</pre><pre>$chown -R apil:deeplearners /home/apil/path/to/target/folder  </pre><pre>  - chown: change ownership<br>  - -R: recursive, so that all files/folders in target is affected</pre><pre>  - apil: the current owner of the targeted folder <br>  - deeplearners: the new group assuming ownership<br>  - /home/../path/to/folder: the target folder </pre><p>Hope Harry is nothing but good news :)</p><pre>&gt;&gt; what the &quot;ls -ld&quot; command shows in unix:<br>$ ls -ld some_dir<br>drwxrwxr-x 2 alex consult 4096 Feb 20 10:10 some_dir/<br> ^   ^  ^     ^     ^<br> |   |  |     |     |_____ directory&#39;s group<br> |   |  |     |___________ directory&#39;s owner<br> |   |  |_________________ permission for EVERYONE ELSE<br> |   |____________________ permission for GROUP<br> |________________________ permission for OWNER</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0bdb8d5736c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unicode, UTF-8, and ASCII encodings made easy]]></title>
            <link>https://medium.com/@apiltamang/unicode-utf-8-and-ascii-encodings-made-easy-5bfbe3a1c45a?source=rss-64eccdd45f96------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bfbe3a1c45a</guid>
            <category><![CDATA[encoding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[unicode]]></category>
            <dc:creator><![CDATA[Apil Tamang]]></dc:creator>
            <pubDate>Sat, 03 Feb 2018 17:35:47 GMT</pubDate>
            <atom:updated>2018-02-07T13:57:02.331Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>So I have an announcement to make: if you are a programmer working in 2017 and you don’t know the basics of characters, character sets, encodings, and Unicode, and I <em>catch</em> you, I’m going to punish you by making you peel onions for 6 months in a submarine. I swear I will.<strong><br>- </strong>(Joel Spolsky. Cofounder of stack-overflow. Taken from article <a href="https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/">here</a>)</blockquote><p>Truer words have not been spoken.</p><p>And let me add that after my doing my time in the submarine, peeling onions with red-shot eyes, you don’t want to be in the same boat (errr submarine) !</p><p>This post will really be a more condensed summary of what I’ve managed to gather from Joel’s <a href="https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/">blog</a>. Surprisingly after more than an hour of scouring the internet for more information on encodings, it was only by chance that I landed at this blog. It is by far the most concise and understandable read on the matter and deserves to be shared.</p><h3>ASCII</h3><p>First there was the C programming language, then there was ASCII. <strong>In ASCII, every letter, digits, and symbols that mattered (a-z, A-Z, 0–9, +, -, /, “, ! etc.) were represented as a number between 32 and 127</strong>. Most computers used 8-bits bytes then. This meant each byte could store 2⁸-1= 255 numbers. So each byte (or unit of storage) had more than enough space to store the basic set of english characters.</p><p>Life was good. Heck they even sent people to the moon! That is if you spoke english.</p><h3>Unicode</h3><p><strong>In order to accommodate the non-english characters, people started going a little crazy on how to use the numbers from 128 to 255 </strong>still available on a single byte<strong>.</strong> Different people would use different characters for the same numbers. Obviously, not only was it the wild wild west, but it quickly dawned that the extra available numbers could not even come close to represent the complete set of characters for some languages.</p><p><strong>The Unicode was a brave attempt to create a single character set that could represent every characters in every imaginable language systems.</strong> This required a paradigm shift in how to interpret characters. And in this new paradigm, each character was an idealized abstract entity. Also in this system, rather than use a number, each character was represented as a <strong>code-point. </strong>The code-points were written as: <strong>U+00639, </strong>where U stands for ‘Unicode’, and the numbers are hexadecimal.</p><h3>UTF-8</h3><p>The final piece we’re missing at this point is a system for storing and representing these code-points. <strong>This is what the encoding standards serve</strong>. After a couple of hits and misses, the UTF-8 encoding standard was born.</p><blockquote>In UTF-8, every code-point from 0–127 is stored in a single byte. Code points above 128 are stored using 2, 3, and in fact, up to 6 bytes.</blockquote><p>Remember that each byte consists of 8 bits, and the number of allowed information increases exponentially with the bits used for storage. Thus with 6 bytes (and not necessarily always 6), one could store as many as 2⁴⁸ characters. That would make for a very (very) large scrabble game!</p><p>Other benefits of UTF-8 meant that nothing changed from the ASCII so far as the basic english character-set was considered. Life continued to be good for english-speakers (a coincidence?). Everybody else had to hop some trains, but finally everybody agreed to use the same set of standards.</p><p><strong>Perhaps the single most important thing to remember from the discussions above is that there is NO string or text, without an accompanying encoding standard</strong>. If you are a web-developer like me, you will occasionally encounter this text: Content-Type: text/plain; charset=&quot;UTF-8&quot; in the code. This essentially tells the browsers (or any text parsers) the specific encoding being used.</p><h3>UTF-16</h3><p>Now that we know what UTF-8 is, extrapolating our understanding to UTF-16 should be fairly straight-forward. UTF-8 is named for how it uses a minimum of 8 bits (or 1 byte) to store the unicode code-points. Remember that it can still use more bits, but does so only if it needs to.</p><p><strong>UTF-16, in the other hand, uses a minimum of 16 bits (or 2 bytes) to encode the unicode characters</strong>. <strong>Java</strong>, whom I have a love-hate relationship with, natively uses this encoding. A <strong>Java</strong> char (Character) is thus a UTF-16 encoded unicode character and occupies a memory space of 2 bytes. The numerical value ranges from</p><ul><li>0x0000 (numerical value: 0) to</li><li>0xFFFF (numerical value: 65,536)</li></ul><p>This also means UTF-16 is NO longer backwards compatible with ASCII. Remember ASCII only used 1 byte or 8 bits.</p><h3>After thoughts</h3><p>This post is meant more as a quick post-it note to help me rehash my understanding of the concept rather than regurgitating all the technical details. It is also meant for an easy morning or bedtime read. And oh, did I tell you how much it stinks to peel onions in a submarine! That alone, and above all, should be your prime motivation for learning the material.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bfbe3a1c45a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unmasking a vanilla RNN: What lies beneath?]]></title>
            <link>https://medium.com/@apiltamang/unmasking-a-vanilla-rnn-what-lies-beneath-912120f7e56c?source=rss-64eccdd45f96------2</link>
            <guid isPermaLink="false">https://medium.com/p/912120f7e56c</guid>
            <category><![CDATA[rnn]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Apil Tamang]]></dc:creator>
            <pubDate>Sun, 10 Dec 2017 01:45:06 GMT</pubDate>
            <atom:updated>2017-12-10T02:42:29.239Z</atom:updated>
            <content:encoded><![CDATA[<p>RNNs have a reputation for being rather hard to understand. They are truly a bit mysterious, and can often seem inscrutable to the new learner. When one starts talking about LSTMs, or the rolled/unrolled versions of the RNN architectures, these discussions can definitely send some scurrying faster than the ghost that lived in Michelle Peifer’s <a href="http://www.imdb.com/title/tt0161081/">home</a>.</p><p>I think the reason for this is that there exists few materials that expose the internals of a RNN in a way that is easy to visualize. Without having a good mental image of what the building block is, I find myself times and often struggling to understand the same thing over and over again.</p><h3><strong>Core APIs</strong></h3><p>Now at the core of any RNN architecture is a simple RNN cell or its variant. This is what it takes to create a RNN Cell in PyTorch:</p><pre>rnn_pytorch = nn.RNN(input_size=10, hidden_size=20)</pre><p>and this is what it takes to create an RNN Cell in Tensorflow:</p><pre>rnn_tensorflow = tf.contrib.BasicRNNCell(num_units=20)</pre><p>Notice the usage of the word <strong>Cell</strong>. Somehow, the term <strong><em>cell</em></strong> seems very alien in neural networks, where we’re used to visualizing individual neuron and/or their arrangements in space with mutual connections. How does a RNN cell relate to the floating collections of individual neurons that we’re much used to in neural nets? This is the visual we’ll explore in the rest of this blog.</p><h3><strong>Traditional Views</strong></h3><p>A RNN <strong>cell</strong> should simply be thought of as a <strong><em>unit</em></strong> of the overall network that encapsulates certain parts of it in one homogeneous block. In most blogs and papers, an RNN cell (block) is shown as the follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NKhwsOYNUT5xU7Pyf6Znhg.png" /><figcaption>Typically how a simple RNN is shown (Left: Canonical form, Right: Unrolled form)</figcaption></figure><p>which is very different from how traditional fully-connected (FC) and Convolution networks (conv-nets) are depicted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mAlKuFfKfszuCTaItAP3_w.png" /><figcaption>Typical view of Neural Networks (Left: Fully-Connected, Right: Convolution Networks)</figcaption></figure><h3><strong>Under the hood</strong>:</h3><p>The best way to intimately understand the structure of a RNN cell maybe by creating one ourself! Here’s the basic RNN cell created from a scratch in PyTorch.</p><pre><strong>class</strong> <strong>CharLoopModel</strong>(nn.Module):<br>    <em># This is an RNN!</em></pre><pre>    <strong>def</strong> __init__(self, vocab_size, in, h):<br>        super().__init__()</pre><pre>        self.h = h<br>        self.e = nn.Embedding(vocab_size, in)<br>        self.l_in = nn.Linear(in, h)<br>        self.l_hidden = nn.Linear(h, h)<br>        self.l_out = nn.Linear(h, vocab_size)<br>        <br>    <strong>def</strong> forward(self, *cs):</pre><pre>        &#39;&#39;&#39;<br>        cs (list(list(int))): input signals to propagate forward.<br>        Example: ((23,32,34), (12,24,23), (12, 45,23))<br>        &#39;&#39;&#39;<br>        <br>        bs = cs[0].size(0)<br>        hidden_state = Variable(torch.zeros(bs, self.h)).cuda()<br>        <strong>for</strong> c <strong>in</strong> cs:<br>            inp = F.relu(self.l_in(self.e(c)))<br>            hidden_state = F.tanh(self.l_hidden(hidden_state+inp))<br>        <br>        <strong>return</strong> F.log_softmax(self.l_out(hidden_state), dim=-1)</pre><p>The code above gives us a very good idea of the internals of the RNN Cell. Note that the two chief constituents that participate to form a RNN are the <strong><em>self.l_in</em></strong> and <strong><em>self.l_hidden</em></strong> layers. Via the analysis of the <strong><em>__init__</em></strong> and the <strong><em>forward</em></strong> method, we can equivalently represent the composition as the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*lDNGu8wQiI7l0LNPgMdTKA.png" /></figure><p>From the figure above, it should be evident that an RNN Cell is really a set of two fully-connected linear layers. Both these layers have ‘h’ neurons, where ‘h’ is the user-specified dimension of the hidden state. The only special thing that happens internally is the addition of the “hidden state” to the output from the first layer. That’s it!!</p><p>The hidden-state forms a recurrent connection within the layer (and from where the RNN cell derives it names). In typical usage, the recurrent connection allows the network to remember what it learned in the previous step. RNNs have been used in many time or sequence dependent problems for precisely this reason.</p><h3><strong>Unrolled RNN Graph:</strong></h3><p>The unrolled RNN graph of the same can be imagined with very little additional effort. It shows how the recurrent connection flows from one-step to another allowing the RNN Cell to exhibit its memory-like characteristic.</p><p>Training the RNN is usually executed one step at a time. Thus, RNNs are also notorious for being difficult to train because they cannot utilize the fully parallel architectures of modern hardware. By the nature of the problem (and proposed solution), the computation can only proceed one step at a time forward. In the same way, back-propagation through a RNN cell involves evaluating the gradient at the very last time-step, and propagating it backwards one time-step back at a time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*kbL67eKOHPOa_a_F4n8QTw.png" /><figcaption><strong>An RNN Cell’s computational graph showed unrolled over time</strong></figcaption></figure><h3><strong>Finally,</strong></h3><p>RNNs have been used extensively for modeling problems that are time or sequence dependent. Their prowesses for problems, especially in the language-modeling domain such as machine translation, speech recognition, and text summarization, have become far superior to any traditional methods.</p><p>At its heart, an RNN cell (or any of its variant) is really a composition of linear dense layers that introduce recurrence via some moderated connections. Actually, modern RNN architectures rarely every use the basic RNN cell that we studied above. Rather, they most often use the LSTM cell, which is just a kind of RNN cell that introduce many more intra-recurrent connections. Ultimately, having a good understanding of the structure of the basic RNN cell can often help guide to more intuitive understanding of more complex cells and how they function.</p><h3><strong>Acknowledgement</strong></h3><p>Once again, I owe Jeremy’s <a href="http://fast/">fastai</a> course for many of the insights that were provided in this post, including the core code for the PyTorch implementation of the RNN from a scratch. The accompany source code on <a href="https://github.com/fastai/fastai/blob/master/courses/dl1/lesson6-rnn.ipynb">github</a> goes on to demonstrate the usage of this code for a sample text corpus, as well as explore additional variations of the RNN.</p><p>I had also previously implemented the RNN via pure tensorflow, but the <a href="https://github.com/apiltamang/tensorflow_rtp_materials/blob/master/week-7/Vanilla_RNN_Using_Embeddings.ipynb">end-to-end implementation</a>, including the demonstration, is much more involved herein. The key to this implementation was using tensorflow’s <strong><em>scan </em></strong>method, which essentially performed the dynamic unrolling of the computational graph, thus obfuscating the matter quite a bit!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=912120f7e56c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Entity Embeddings in one breath]]></title>
            <link>https://medium.com/@apiltamang/learning-entity-embeddings-in-one-breath-b35da807b596?source=rss-64eccdd45f96------2</link>
            <guid isPermaLink="false">https://medium.com/p/b35da807b596</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Apil Tamang]]></dc:creator>
            <pubDate>Sat, 02 Dec 2017 20:53:44 GMT</pubDate>
            <atom:updated>2017-12-10T02:43:21.874Z</atom:updated>
            <content:encoded><![CDATA[<p>Well maybe not, unless you’re routinely breaking the world-record in holding your breath! On the bright-side, entity embeddings is really as easy as it is fascinating to learn about.</p><p>The core idea of using entity embeddings is to use a different set of dimension to represent a categorical set of data.</p><p>A categorical set of inputs is a type of data where we have different categories (or types) that are unrelated amongst each other. For example: Volvo, BMW, Mercedes, etc. would be different categories of a car. In a naive numerical method, these types would typically be represented as one-hot encoded vectors. However, when we have thousands, and possibly millions of categories, storing these inputs as one-hot encoded vectors quickly become computationally and memory-wise very inefficient.</p><p>For e.g. google’s <a href="https://code.google.com/archive/p/word2vec/">word2vec</a> represents ~3 millions words and phrases as vectors of only 300 dimensions. Many state-of-the-art NLP models routinely use the word2vec model to power their algorithms.</p><p><strong>Problem</strong></p><p>To better understand entity embedding, we will create an embedding ourself. The corpus we use will be the IMDB movie ratings database. The data gives ratings for different movies by different users. For the ease of demonstration, we will talk through a database where ‘M’ users have reviewed ‘M’ movies on a scale from 1 to 5. This data can be viewed as a matrix of the following form (let us call this matrix <strong><em>W</em></strong>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1jwsXJ8fcTPSg7s_FAyP0A.png" /><figcaption>Visualizing the Matrix (W) of Movie Reviews from the dataset</figcaption></figure><p>The problem we’re fundamentally trying to solve is to come up with a model for collaborative filtering. In other words, given a user and a movie, how accurately can we predict that the user would rate the movie he/she has not already seen?</p><p>Needless to say, many classical methods do exist for attacking this problem. These class of methods fall under the domain of recommender systems, and often rely on specifically engineered features for analysis. The special thing about our approach is that not only do we come up with a good model, but we also end up with an entity embedding that produces some fascinating insight into the entities themselves (i.e movies and users).</p><h3><strong>Initialization</strong></h3><p>In our approach using entity embeddings, we first make the assumption that each user and movie in our dataset can be represented as a unique vector in an ‘N’ dimensional space, where N is much smaller than M. This process could be visualized as the follows, where each vector u and v are initialized via random sampling of real numbers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*qRI4FOau33Og6OFzmOosvw.jpeg" /><figcaption>Representing each ‘User’ and ‘Movie’ as vectors of length ‘N’</figcaption></figure><h3><strong>Hold your breath…</strong></h3><p>This is where you take a deep breath and dip a toe in the icy world of mathematics. Now that we have our random matrices <strong><em>U</em></strong> and <strong><em>V</em></strong> for the users and movies respectively, we want to take the dot product of each user and movie to produce a result matrix. This is akin to performing a forward-pass in the neural-network lingo. Let us call this matrix <strong><em>W`.</em></strong></p><p>Again, a single picture speaks a thousand words, so here is the action unpacked in one visualization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*tFp761cvzyomCq1OBJ8W-w.jpeg" /><figcaption>forward-propagation in our model (i.e. take dot product of each user with each movie)</figcaption></figure><p>After we have the result of the forward-pass (W`), we want to evaluate the error for each element on the matrix, and take its square. Finally we take the square root to get the root mean square error (MSE):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*K2Ug6KwMFfPgdPwSgIK0CA.gif" /><figcaption>Mean Squared Error</figcaption></figure><p>Once the mean-squared error is obtained, the process of gradient descent can be used to update the elements in matrices <strong>U</strong> and <strong>V, </strong>so that the error is made smaller. In most deep-learning frameworks such as <a href="https://www.tensorflow.org/">tensorflow</a> and <a href="https://www.pytorch.org/">pytorch</a>, gradient descent and update are provided by default and does not have to be implemented.</p><p>This process of forward-pass and gradient search/update is repeated until a the error is smaller than a specified tolerance. Very soon, the error should be quite small and prediction of each value in <strong>W`</strong> should be very close to the corresponding value in <strong>W.</strong></p><h3><strong>And let go…</strong></h3><p>This is where you let out your breath and realize that the math wasn’t so hard after all! In fact, if you ruminate on the solution, it is fascinating to realize that the random vectors we ascribed to each movie and user has now converged to some special results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*POXWMsLVcKsLBI1gx-uNpg.jpeg" /><figcaption>Visualizing the embedding vectors in N-dimensional space</figcaption></figure><p>What has happened is that each movie and user is now represented as optimized vectors in a different linear sub-space. In other words, each entity is now an embedding (vector) in new dimensions (hence the term <strong><em>entity embedding</em></strong>).</p><p>Think of these different dimensions as different features that we might talk about in movies. They might be things like how romantic a movie is, or if it’s in the sci-fi genre. It might even be something as if ‘Ryan Gosling’ was an actor in the movie(at which point we’d assume it is likely more romantic than anything else). The figure below demonstrates how the embeddings relate to the dimensions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*ejRZ1fABR-lTzbD_b9guwQ.jpeg" /><figcaption>Relationship of a movie and its embeddings.</figcaption></figure><p>Now, does it mean that this actually happened, i.e. are we really classifying movies and users on these dimensions (‘adventure’, ‘action’, ‘sci-fi’, ‘romantic’, or ‘Ryan Gosling’)? No… What we’ve found is a hidden (or latent) representation that works for our problem. We simply used the above list as a relatable example. Best of all, we did not have to use any feature engineering! i.e. no tagging of movies with any features, and no clustering of users or the reviews.</p><h3><strong>References:</strong></h3><p>Well, if this blew your mind away, then you’re not the only one. There are well-documented cases of other people’s mind being blown (<a href="https://www.youtube.com/watch?v=V2h3IOBDvrA&amp;feature=youtu.be&amp;list=PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM&amp;t=6147">youtube</a> see 1:50:00). The excel file for that should be available <a href="https://github.com/fastai/fastai/blob/master/courses/dl1/excel/collab_filter.xlsx">here</a>. Needless to say, I credit the <a href="http://www.fast.ai/">fast.ai</a> course for the exposition of this topic from this unique perspective!</p><p>Additionally, the implementation of the above from scratch in PyTorch can be referenced <a href="https://github.com/fastai/fastai/blob/master/courses/dl1/lesson5-movielens.ipynb">here</a>. This notebook also explores slightly more complex variation of the method, which involves adding dropout and hidden layers to create a non-linear model.</p><p>Happy embedding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b35da807b596" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A gentle dive into the anatomy of a Convolution layer.]]></title>
            <link>https://medium.com/@apiltamang/a-gentle-dive-into-the-anatomy-of-a-convolution-layer-6f1024339aca?source=rss-64eccdd45f96------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f1024339aca</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <dc:creator><![CDATA[Apil Tamang]]></dc:creator>
            <pubDate>Tue, 21 Nov 2017 02:30:54 GMT</pubDate>
            <atom:updated>2017-12-10T02:43:43.095Z</atom:updated>
            <content:encoded><![CDATA[<p>In the realm of deep learning, the convolution network is a workhorse for many awe-inspiring results. After 2012’s publication of the AlexNet, it is hard to cite a ground-breaking computer-vision architecture that does not use convolutional architecture at its very foundation.</p><p>With existing deep-learning frameworks, writing a convolution layer is often a one-line statement which abstracts away many structural details. And yet, sometimes it is a good idea to take a step back and peel open some abstractions. This blog is an attempt to elaborate one a specific anatomical feature of the convolution layer that is unfortunately overlooked in most blogs and discussions on this matter.</p><p>Many convolution architectures start off with an exterior convolution unit that map an input image of channel RGB into a series of interior filters. In most popular deep-learning frameworks, this code may look as the follow:</p><blockquote>out_1=Conv2d(input=image, filter=32, kernel_size=(3,3), strides=(1,1))<br>relu_out=relu(out_1)<br>pool_out=MaxPool(relu_out, kernel_size=(2,2), strides=2)</blockquote><p>To many, it is well understood that the result of the above is a series of filters 32 layers deep. What is not always understood is how an image with 3 channels is exactly mapped into these 32 layers! What isn’t also apparent is exactly how the max-pool operator is applied. For instance, is the max-pool applied over all the filter layers at once to effectively produce a single filter map at the end? Or, is it that the max-pool is applied to each filter independently to produce the same 32 layers of pooled filters?</p><h3><strong>The How?</strong></h3><p>A single picture speaks a thousand words, so here’s a diagram to show all the actions packed in the above code snippets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_Rz06Kz3JebUmTSv4bJDA.png" /><figcaption>Application of the convolution layer.</figcaption></figure><p>The most notable observation from the figure above is that each filter (i.e filter-1, filter-2, … et. al) in Step-1 actually comprises of a set of three convolution kernels (Wt-R, Wt-G, and Wt-B). Each one of this kernel is reserved respectively for the Red (R), Green (G), and Blue (B) channels in the input image.</p><p>During the forward propagation, the R,G, and B pixel values from the image are multiplied against the Wt-R, Wt-G, and Wt-B kernels respectively to produce an intermittent activation map (not shown in the figure). The outputs from the three kernels are then added to produce one activation map per filter.</p><p>Subsequently, each of these activations are subject to the ReLu function, and finally run through the max-pooling layer, the latter being primarily responsible for reducing the dimension of the output activation map. What we have at the end is a set of activation map, usually whose dimensions is now half of the input image, but now whose signals span a set of 32 (by our choice as the number of filters) two-dimensional tensors.</p><p>The output from a convolution layer often serves as the input to the subsequent convolution layer. Thus, if our second convolution unit started as the follows:</p><blockquote>conv_out_2 = Conv2d(input = relu_out, filters=64)</blockquote><p>then the framework needs to instantiate 64 filters, each filter using a set of 32 unique kernels.</p><h3>The Why?</h3><p>Another subtle, but an important point that escapes scrutiny is the discussion of why we used 32 filters for our first convolution layer. In many popular architectures, the number of filters used gets increasingly larger (e.g. 64 for the second, 128 for the third, and so on..) as we go deeper in the network.</p><p>Matt Zeiler, in this paper, employs a the deconvolution operator to visualize how the kernels at different layers and depth of a deep convolution architecture get tuned during the training process. The general consensus is that in an optimally trained convolution network, the filters at the very edge (close to the image) becomes sensitive to basic edges and patterns. The filters in the deeper layers become sensitized to gradually higher orders shapes and patterns. The phenomenon is very well summarized in these diagrams extracted from Matt’s paper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TopKS7puyHfrmPohXseLAQ.png" /><figcaption>Visualization of the activation of filters on the first and second (outermost) layers.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pfODaRvbWGoimmlHWjjBSw.png" /><figcaption>Visualization of the filters activation on the third layer.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IRPhCFHAEZNTcxmXttDoqg.png" /><figcaption>Visualizations of the filters activation on the 4th and 5th layers.</figcaption></figure><p>Another question that I wondered for a considerable time is why different filters, even in any given layer, get tuned to a specific shape or pattern. After all, there was nothing extraordinary about the weights in any kernel which would’ve guaranteed the observed outcome. Precisely to that point: the process of stochastic gradient descent (SGD) <strong>automagically</strong> corrects the weights so that the kernels acquire the specialized features above. It is only important that:</p><ul><li>the kernels (or weight matrices) be initialized randomly, so that we ensure each kernel is optimized to a unique solution space, and</li><li>we define enough filters to maximize the capture the various features in our dataset, while striking the balance against the incurred computational cost.</li></ul><p>And finally, many papers also suggest that the visualization of the filter activations often provide a window into the performance of a convolution architecture. A balanced and performant network often displays activations as discussed above, marked with the manifestation of well-defined edge and shape detectors. A network that over-fits, under-fits, and/or generalizes poorly often fail to show these observations. Hence, it is always a good idea to test the network using the process used in (2) to see if an experimental convolution network is yielding good results.</p><p>References:</p><ol><li>ImageNet Classification with Deep Convolutional Neural Networks, Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks">https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks</a></li><li>Visualizing and Understanding Convolutional Networks, Matthew D Zeiler, Rob Fergus <a href="https://arxiv.org/abs/1311.2901">https://arxiv.org/abs/1311.2901</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f1024339aca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Case Study: A world class image classifier for dogs and cats (err.., anything)]]></title>
            <link>https://medium.com/@apiltamang/case-study-a-world-class-image-classifier-for-dogs-and-cats-err-anything-9cf39ee4690e?source=rss-64eccdd45f96------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cf39ee4690e</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[cats-and-dogs]]></category>
            <category><![CDATA[fastai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Apil Tamang]]></dc:creator>
            <pubDate>Fri, 10 Nov 2017 22:04:08 GMT</pubDate>
            <atom:updated>2017-11-11T16:30:20.313Z</atom:updated>
            <content:encoded><![CDATA[<p>It is amazing how far computer vision has come in the last couple of years. Problems that are insanely intractable for classical machine learning methods are a piece of cake for the emerging field of deep learning (DL), aka deep neural networks (DNN). Even more outstanding is that the foundation of deep learning models is both mathematically and conceptually very easy to understand. Their implementations can arguably be quite complex, but the proliferation of DL frameworks mean that state of the art result for many traditionally hard computer vision problems is achievable in a few lines of code these days.</p><p>In this blog, we’ll attempt to use the <a href="https://github.com/fastai/fastai">fastai</a> library to build an image classifier that works amazingly well for a classification task. The library (thanks to Jeremy Howard and his team at <a href="https://www.usfca.edu/data-institute">USF</a>) makes deploying a model as easy as flipping a burger, so well, you can both make your burger and eat it!</p><p>The first step to take is to download the dogs and cats dataset from <a href="https://www.kaggle.com/c/dogs-vs-cats">kaggle</a>, and unarchive it at a certain path in your machine. Then you need to get a copy of the <a href="https://github.com/fastai/fastai">fastai</a> library from github, and install the proper development environment (i.e. Python 3.6, Cuda 8.0, CudaNN 6.0, Jupyter notebook etc.).</p><p><strong>Model Architecture</strong>: We’ll be using the <a href="https://arxiv.org/pdf/1611.05431.pdf">ResNext-50</a> architecture for this case study. This model was also the runner-up in the 2016 ILSVRC competition, and is arguably one of the state-of-the art in computer vision models.</p><pre>sz=299            # intended image size<br>arch=resnext50    # intended architecture</pre><p><strong>Data Sampler</strong>: We’ll be using a data-sampler that randomly provides horizontally flipped versions of sample images, with random zooming by a factor of 1.1</p><pre>tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)</pre><pre>data = ImageClassifierData.from_paths(PATH, tfms=tfms, bs=bs, num_workers=4)</pre><p><strong>Learning Model</strong>: The following line instantiates the model that we’ll use to sperform classifications with. The Resnext model we use will be a pre-trained model on the imagenet database.</p><pre>learn = ConvLearner.pretrained(arch, data, precompute=<strong>True</strong>, ps=0.5)</pre><p>In the following line, we actually go ahead and fit the model.</p><pre>learn.fit(1e-2, 1)</pre><pre>&gt;&gt; #(epoch, train_loss, val_loss, val_acc)<br>&gt;&gt; [ 0.       0.04081  0.01944  0.99388]</pre><p>And bam! We see that the model is already predicting dogs and cats with an accuracy of over 99% ! It is also worth mentioning that this prediction is based on the validation set, i.e. images of cats and dogs that the model never got to train on. Very impressive, if you ask me.</p><p>Now, let’s take a quick visual of how many cats and dogs the model got wrong. This is done by using a confusion matrix which is pretty self-explanatory:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*n9TxgFX0_7pSLwEEYgFIHQ.png" /><figcaption>Confusion Matrix for the Cats and Dogs (Epoch 1)</figcaption></figure><p>So, it seems like we incorrectly predicted 10 images of cats as dogs. Likewise, we incorrectly predicted 5 images of dogs as cats. Hmmm.. let’s see what is really going on!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7LI4f_rDoGgE1NehWhpjBA.png" /><figcaption>Images with truth labels of dogs, but that were labelled as cats by the network.</figcaption></figure><p>In the row of images above, the first two images are predictions that the network seemingly got wrong. One could argue that the first image has a dog that might resemble as a large cat, while the second image is quite blurry. The image on the middle is practically indistinguishable from a cat, while the two images on the right are neither that of a dog or a cat. No surprise then that the network would not get their class labels right.</p><p>Now let us look at the images that had the truth label of cats but got classified as dogs. Looking at the first two images in the first row, we see that the model thought they were dogs with a very strong confidence (i.e. the closer the number is to 1, the stronger the prediction that it is a dog). In fact, these images do have a picture of a dog each occupying a much larger volume of the image. No wonder then that the classifier (or for that matter, any human) would associate these images to the dog label.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*--k8EwdxCcuMZePVT8r-8A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dL_3M3N0j7wm0NEyb1A_Bg.png" /><figcaption>Images with truth labels as cats, but which the network thought were dogs.</figcaption></figure><p>There are two other images that is clearly not a dog or a cat, but mere posters. The fact that the network thinks these are also dogs with rather large confidences (0.9845 and 0.938929) suggests that the model could use some improvement. Additionally, there are a number of images of cats which the network has classified as dogs, but with much smaller confidence (i.e. the further the confidence is from 1.0, the more uncertainty that it is a dog). Remember that we only fitted the model once, so we have plenty of opportunities to do better.</p><p>In the next round of learning, we continue to fit our model using some very generalizable training tricks. We actually apply different learning rates for different sections of the Resnext architecture. The learning rate is also varied across iterations as shown below, and the learning rates reset to the initial magnitudes at the start of a new epoch. Additionally, the model weights are saved at the end of each epoch, and these weights used to make an averaged prediction at the end.</p><p>The code for the above is packed between these two lines:</p><pre>lr=np.array([1e-5,1e-4,1e-3])  # set a differential learning rate<br>learn.fit(lr, 2, cycle_len=1)  # fit the model with cyclic method</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*9o_ujDWvaDXTrg8xzxeZIQ.png" /></figure><p>The principal motivation for <strong>varying</strong> the learning rates, and taking <strong>averages </strong>of the prediction of the model from each epoch is based on the paper on <a href="https://arxiv.org/pdf/1704.00109.pdf">snapshot ensembles</a>. The core idea behind doing such is that varying learning rates, as well as taking an ensemble of predictions helps us find a minimum that is more stable and resistant to data perturbations. Thus such models should generalize better to unseen data, and help us actually use the model for useful future work.</p><p>The motivation for using <strong>different</strong> learning rates is something that was brought <strong><em>originally</em></strong><em> </em>for Ver. 2 of FastAI’s Part I class. The intuition driving this thought process has its roots in Matt Zyler’s <a href="https://cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf">paper</a> on deconvolution layers. It is shown that the shallow convolution layers (i.e those closest to the input images) are useful for learning basic geometric edges, thus we chose to barely, if at all, update the weights on these layers during training. In the same tangent of thought, the middle layers are shown to be detectors of higher order shapes (e.g. arcs, circles, patterns etc.), thus we choose to update the weights on these layers with a slightly larger learning rate. The deepest layers, i.e. the fully connected layers that we custom injected, need the most amount of updating and thus have the largest learning rate ascribed to them.</p><p>After running the training for a couple of epochs with the differential learning rates, followed by a varied learning rate we arrive at the following situation:</p><pre>#(epoch, train_loss, val_loss, val_acc)<br>[ 0.       0.0551   0.02114  0.99355]                         <br>[ 1.       0.03563  0.01902  0.99405]</pre><p>We see very slight improvements in both the validation loss and accuracy. It might not seem like a lot, but in fiercely competitive scenarios such as a kaggle competition, every bit of gain helps. Let us look at how the cats and dogs images are now classified using the confusion matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*FoudoRjqL4Gwg2fPFsOsaA.png" /></figure><p>Voila! Off of 2,000 images of cats and dogs, we’ve managed to fail on only 8 of them. And, we achieved to do that by running no more than 5 to 7 total iterations. I believe this is an incredible feat, given that we were able to accomplish this with pretty much off-the-shelves components and with very little, if any, customizations.</p><p>Looking ahead at the wrongly classified images, we see that the network is pretty much behaving like a human. As can be seen, three of these images are just posters, and one of these images have both a cat and dog in it. The image on the fourth column is a dog inside a cage, and for any model, we’d expect it to be a tough nut to crack. Thus with some meaningful intuition, we find ourselves almost forgiving of these misclassifications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*61-CJVhEToAN5wErxJPzjw.png" /><figcaption>Left 3: Images with truth label of cats, but classified as dogs. Right 2: Images with truth label of dogs, but classified as cats</figcaption></figure><p>The row of images below show images of dogs classified as cats by the network. Except for the first one, all the other images are indeed very poor representations of dogs. One is the underneath of a dog’s belly, the other a close snapshot of a dog’s teeth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rhhtjYrDgpTVhsKfu4agdQ.png" /><figcaption>Images (all) with truth labels of dogs, but were classified as cats.</figcaption></figure><p>I hope this was a gentle introduction to the kinds of datasets that deep-learning particularly excels at, i.e. images, texts, and signals that exist in massive quantity in the increasingly digital world. The page showing these processes are described with a lot more depth, discussion, and detail in this <a href="https://github.com/fastai/fastai/blob/master/courses/dl1/lesson1.ipynb">notebook</a>, although by default, it uses the Resnet-34 architecture. A more sparse version of this notebook using Resnext-50 should be found <a href="https://github.com/fastai/fastai/blob/master/courses/dl1/lesson1-rxt50.ipynb">here</a>, and it is possible that at the time of viewing, you might see slightly different results. Deep-learning, at its heart, is stochastic, so the results will slightly vary between consecutive executions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cf39ee4690e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yet Another ResNet Tutorial (or not)]]></title>
            <link>https://medium.com/@apiltamang/yet-another-resnet-tutorial-or-not-f6dd9515fcd7?source=rss-64eccdd45f96------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6dd9515fcd7</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Apil Tamang]]></dc:creator>
            <pubDate>Sat, 04 Nov 2017 19:07:07 GMT</pubDate>
            <atom:updated>2017-11-04T19:07:07.086Z</atom:updated>
            <content:encoded><![CDATA[<p>The purpose of this article is to expose the most fundamental concept driving the design and success of ResNet architectures. Many blogs and articles go on and on describing how this architecture is laid out, and show images of large networks as well as apprehensive accuracy tables and loss plots. I hope to excuse myself from reiterating the obvious, and hopefully really cutting to the kernel.</p><p>The ResNet architecture is often touted as the state-of-the-art in image classification tasks. The ability to construct arbitrarily deep layers, followed by the popular ‘we need to go deeper’ heuristics makes ResNet the prodigy of deep learning architectures. The constituent building unit of the ResNet architecture is the <strong>ResNet block</strong>, as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4H3-1yTTQddcZWg7aTO5FQ.png" /></figure><p>The key idea driving the design of ResNet is actually best described in the original paper itself rather than in any other blogs I’ve seen:</p><p><em>“So rather than expect stacked layers approximate H(x) (the underlying mapping function), we explicitly let these layers approximate a residual function F(x) := H(x) -x. The original function thus becomes F(x) + x. ”</em></p><p>If you’re like me, you’re trying to wrap your head around this and wondering what the hell’s going on here... What is the original mapping function, (F(x)), H(x), and why the residual? The original paper goes into a lot of detail, and between the residual function, the degradation diagnosis, and other technical jargon, the heart of the matter is sometimes lost in a casual read.</p><p>You see.. a deep neural net has a specific task, and that is: Given an x, the net’s one and only goal is to find an underlying useful mapping function H(x). The function H(x) is useful because it attempts to classify the given image into specified class labels, sub-segments, captions, or whatever the output it is you’re trying to produce. So let’s visualize it as this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0KHv7MgUlkOCbLXL6BIGwA.png" /><figcaption>Our Deep Net needs to find the mapping function H(x)</figcaption></figure><p>The residual way of thinking is to say: You know what.. let’s not try to map it to the mapping function H(x). Instead, let me try to map it to a different function F(x)!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*23PTFSipnkPHPKB67XTw2w.png" /></figure><p>Wait.. what is F(x)? This is where all the knowledge and intuition kicks in. The authors of ResNet do not want the net to try approximate just any random F(x). Instead, they want it (the net) to approximate a very well-defined, albeit simple, function F(x), such that F(x) := H(x) -x. This function is what the authors call by the name: “residual function”.</p><p>Now, <strong>here’s the twist</strong>: Our good-old pal H(x), the original mapping function, is still the only function that we really (truly) love and need! So, in order to fool our deep-learning net to find the function H(x), we actually modify the input so that instead of supplying the net with the original input: x, we actually feed it F(x)+x!</p><p>And there you have it: this is how the residual block and how the skip-connections work. Remember that the skip connection allows for an input channel which is F(x)+x, but what we’re really wanting to find is the (one-and-only) function H(x). At the heart of the matter, this is all there it is.</p><p>Original Reference Paper for ResNet:</p><p><a href="https://arxiv.org/abs/1512.03385">https://arxiv.org/abs/1512.03385</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6dd9515fcd7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>