<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by saahil on Medium]]></title>
        <description><![CDATA[Stories by saahil on Medium]]></description>
        <link>https://medium.com/@saahil?source=rss-9f0b277a519a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*9lNxSKHqqgbV_PEH.png</url>
            <title>Stories by saahil on Medium</title>
            <link>https://medium.com/@saahil?source=rss-9f0b277a519a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 21:57:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@saahil/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Trouble With Training RNNs]]></title>
            <link>https://medium.com/@saahil/trouble-with-training-rnns-2db69c26e8d7?source=rss-9f0b277a519a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2db69c26e8d7</guid>
            <dc:creator><![CDATA[saahil]]></dc:creator>
            <pubDate>Sat, 07 Jun 2014 23:39:33 GMT</pubDate>
            <atom:updated>2014-06-07T23:39:33.333Z</atom:updated>
            <content:encoded><![CDATA[<h4>Follow up post on Recurrent Neural Networks</h4><p>I introduced RNNs in an <a href="https://medium.com/@saahil/some-exciting-things-9c0ae9c58ce4">old post</a> in a very superficial manner. Recurrent networks are semantically applicable to tasks that are serial in nature or follow temporal consistency, e.g. predicting natural language, stock market trends, music progressions etc.</p><p>Let’s revisit these networks and try to see why they are difficult to train with conventional gradient descent methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*S5-_3urUYcnTK7dMU-AIbw.png" /><figcaption>Schematic of RNN. (<a href="http://arxiv.org/abs/1211.5063">On the difficulty of training Recurrent Neural Networks</a> by Pascanu et al.)</figcaption></figure><p>In the schematic of an RNN, the hidden states depend on the preceding state values by a recurrent weight matrix, <strong><em>W_rec.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*UqblKq6iiTLpv50PFz2N2Q.png" /></figure><p><em>(I’m going to use </em><strong>W </strong><em>in place of </em><strong>W_rec</strong><em> in the rest of the essay, because Medium doesn’t let me subscript. Input to hidden weights, </em><strong>W_in</strong><em>, plays the usual role as in an MLP)</em></p><p>Another popular variant of equation (2) is one where the non-linearity, <strong>σ</strong>, is applied on the linear combination of all weights and biases. But we will continue with this definition for the sake of convenience.</p><p>Recall the “unrolled” architecture of an RNN, where individual time steps’ MLPs are fully connected with the input and output nodes, and the hidden layers are fully connected to adjacent time units.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*9ZlaVKypawi0I7D3eyWV8w.png" /><figcaption>Taken from <a href="http://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf">Ilya Sutskever’s thesis</a></figcaption></figure><p>If we train such a network using <a href="http://www.nature.com/nature/journal/v323/n6088/abs/323533a0.html">backpropagation</a> algorithm, where the gradient of loss at a respective node output is used to descend the weight values, so as to reach a minima in the error function, the recurrent matrix, <strong><em>W</em></strong>, will be compounded per time step. This would be harmful as far as storing long term time dependencies goes, because by the time the error gradient at <em>t</em> would have been propagated to <em>t’&lt;&lt;t</em>, the norm compounding in <strong><em>W</em></strong> would have overpowered the time-dependency effect.</p><p>The compounding of the error gradient can happen in one of two opposite directions, both depending on the <em>largest eigenvalue</em> (also known as the spectral-radius) of the recurrent weights matrix. If the spectral radius is much less than 1, the gradient would vanish over time. On the other hand, if the spectral radius is bigger than 1, the gradient would explode over time.</p><p>Let’s demonstrate the delicate nature of training recurrent weight matrix, using an over-simplified architecture.</p><p>In (2), assume that there is no new input coming at every time step, so that the second term with <strong><em>u_t</em> </strong>becomes unnecessary. Furthermore, assume that <strong><em>x</em></strong> is a single dimension variable, which means that <strong><em>W</em></strong> and <strong><em>b</em></strong> have shapes [1, 1] and [1] respectively.</p><p>Our objective, then, is to start <strong><em>x</em></strong> with a zero value and reach a given target value, <strong><em>z </em></strong>in a set number of time steps. I chose <strong><em>z</em></strong> as 0.78 and 50 time steps, for no reason in particular. I present the training graph over 10000 different initialization sets of <strong><em>W</em></strong> and <strong><em>b</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*rmbbUcJAl-R7h4wI175lgQ.png" /></figure><p>The steep wall perpendicular to the parameter space represents an explosion in gradients of the loss function. Basically, when the largest eigenvalue vector in the parameter matrix is compounded, the curvature of the error surface compounds too and that’s what the wall illustrates.</p><p>But what is noteworthy is this — When the search routine is at a point on the top surface of the error curve, it makes it’s next step in a direction perpendicular to the face of the wall. Depending on the learning rate, it might then fall to ground “beyond” the valley where the error reaches it’s minimum. This is not such a big problem, because the search must come back to the valley region, given itself to explore the ground region.</p><p><em>Unless!</em> Unless of course the search direction bangs head-on into the wall again, at which point a small change in the norm of the update would take the search back to the top of the hill.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*x_XpQAMo3SjYUzF6WtL-Hw.png" /><figcaption>A more desirable training regime. Note the use of ‘J’ instead of ‘L’. This is because the raw error ‘L’ has to be regularized somehow to make the search space smoother.</figcaption></figure><p>Norm-based regularizers and gradient clipping methods might solve the problem of exploding or vanishing gradients to some extent, but they depend heavily on initialization of parameters in a right regime.</p><p>In a few weeks’ time, I’ll describe some analytical and structural methods to deal with these training issues, which have been known to work well.</p><blockquote>If you enjoyed this article, please recommend, tweet or share using the buttons below. That would be ace! ☺</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2db69c26e8d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some exciting things]]></title>
            <link>https://medium.com/@saahil/some-exciting-things-9c0ae9c58ce4?source=rss-9f0b277a519a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c0ae9c58ce4</guid>
            <dc:creator><![CDATA[saahil]]></dc:creator>
            <pubDate>Mon, 02 Jun 2014 22:18:58 GMT</pubDate>
            <atom:updated>2014-06-03T06:12:35.861Z</atom:updated>
            <content:encoded><![CDATA[<h4>I promised myself that I’ll write and publish more often</h4><h3>Up to now</h3><p>My <a href="https://medium.com/@saahil/whats-going-on-a2ae552015dc">last post</a> talks a bit about the interdisciplinary project that I was working on, at <a href="http://campar.in.tum.de/WebHome">uni</a> (for a lot of credits!) last semester. I can report that we were able to pull off some very decent results on the limited clinical data that we were able to gather in the span of two months. Unfortunately, we royally botched up the imaging setup by not taking enough care to register the channels of the camera together. Expectedly, our paper was rejected at <a href="http://miccai2014.org/">MICCAI 2014</a>. But a coffee loaded professor smiled really affectionately at us at 3am in the lab, which was very uplifting. The most notable observation from the segmentation results was that there was a clear advantage in using rudimentary classification techniques on extremely simple features like histogram values in a surrounding patch, <a href="http://en.wikipedia.org/wiki/Histogram_of_oriented_gradients">histogram of oriented gradients</a>, <a href="http://en.wikipedia.org/wiki/Local_binary_patterns">local binary patterns</a> or even raw intensities from all channels, than using computationally expensive techniques like <a href="http://deeplearning.net/tutorial/lenet.html">convolutional networks</a>. With a lot of improvements in the experimental setup, we have submitted a revised version to another machine vision conference and I’ll post here if and when it is accepted/rejected.</p><p>My guided research project at the <a href="http://www4.informatik.tu-muenchen.de/index_en.shtml">software and systems engineering chair</a> went pretty smoothly, and I am rather proud of the <a href="http://mediatum.ub.tum.de/node?id=1198433">technical report</a> we produced at the end of it. <a href="http://www4.in.tum.de/~mendezfe/">Daniel</a> and <a href="http://www.iste.uni-stuttgart.de/se/menschen/wagner.html">Stefan</a> are fantastic supervisors with very precise and constructive critiques. We aim to write and publish a full study soon.</p><h3>Recurrent Neural Networks (RNNs)</h3><p>Time series networks are <a href="http://en.wikipedia.org/wiki/Multilayer_perceptron">MLPs</a> which are not fully connected, but recur in time steps. What this basically means is this — an “instance” of a fully connected MLP is copied <em>n </em>times over in time, where <em>n </em>is the number of time steps. A fully connected instance at time <em>t </em>is connected to instances <em>t-1 </em>and <em>t+1</em> through shared weights that go from every hidden node at <em>t</em> to every hidden node at <em>t-1</em> and <em>t+1</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*9ZlaVKypawi0I7D3eyWV8w.png" /><figcaption>An example of RNN unrolled in time (<a href="http://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf">Training Recurrent Neural Networks by Ilya Sutskever</a>)</figcaption></figure><p>When such a time series network is trained by traditional <a href="http://www.nature.com/nature/journal/v323/n6088/abs/323533a0.html">back-propagation </a>on error gradients, it suffers from one of two peculiar analytical problems — <a href="http://arxiv.org/abs/1211.5063">exploding gradients or vanishing gradients</a>. When the error gradients are back-propagated through what is essentially a set of identical weight vectors, the gradients grow smaller and smaller (or larger, depending on the eigenvalues of the weight matrix), until they become insignificant for training purpose.</p><p>At <a href="http://brml.org/">Biomimetic Robotics and Machine Learning lab</a>, where I’m writing my thesis, we are looking at how the exploding/vanishing gradient problem in RNN training can be solved without resorting to complex analytical methods that involve working with <a href="http://machinelearning.wustl.edu/mlpapers/paper_files/icml2010_Martens10.pdf">ugly Hessians</a> (pdf) and optimization of sub-loss functions which have a massive computational appetite or unkempt <a href="http://dl.acm.org/citation.cfm?id=1246450">LSTM architectures</a> with forget gates. Maybe dropout or simple norm-based regularizers? I’ll write a longer post on RNNs with some preliminary results soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c0ae9c58ce4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What’s going on?]]></title>
            <link>https://medium.com/@saahil/whats-going-on-a2ae552015dc?source=rss-9f0b277a519a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2ae552015dc</guid>
            <dc:creator><![CDATA[saahil]]></dc:creator>
            <pubDate>Fri, 04 Oct 2013 17:23:10 GMT</pubDate>
            <atom:updated>2013-10-04T17:23:10.422Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*LIoW-KCJNHWxZd7b.jpeg" /><figcaption>Risserkogel trek, 24th aug. 2013</figcaption></figure><h4>…Long due updates</h4><p>In all my time of using Posterous and Wordpress before that, I had no idea just how big Tumblr had gotten. But since <a href="http://www.theverge.com/2013/4/30/4281780/posterous-is-shutting-down-tomorrow-here-are-the-best-alternatives">Posterous shut down</a> and I was looking for a new place to write stuff at, I thought I’d give Tumblr a shot. But everybody seems to be on it. As a result, I didn’t get my first 8 or 9 choices of usernames(I usually go: first name, full name, some classic album names, names of characters from favourite books, give up!). Turns out Medium is a very cool alternative and the text editor, especially, is pretty darn amazing! Being here, this might be a good incentive to keep publishing more. But for now, here are some things I’ve been doing since October last year.</p><h3>Academics</h3><p>After taking some introductory courses in machine learning in the first semester, I did a lab called <a href="http://en.wikipedia.org/wiki/Deep_learning">Deep Learning</a> in Real World. Deep learning, basically, is a multi-layered neural network with each layer trying to imitate the layered informational structure of the actual data being looked at. For eg.: in trying to classify boozecats from grumpy cats, an NN layer may first be able to distinguish a horse from a feline creature, followed by a layer to learn a cheetah from a cat, followed by features on the cat’s face, and so on. In our task,we figured a supervised image segmentation method, using <a href="http://deeplearning.net/tutorial/lenet.html">Convolutional NNs</a>, to separate bone pixels from cartilage pixels in MRI scans(3d or 2d) of human knees. For tear and injury related diagnoses, this network reduces the manual annotation time by a factor of almost 40. Still room for a LOT of improvement. Nevertheless, felt like a boss!</p><p>But a more significant result of the above experiment was that I was able to convince somebody in <a href="http://campar.in.tum.de/WebHome">computer aided medicine</a> to let me in to some of their research related to medical imaging and segmentation. The cool people of the chair have constructed a multi-spectral camera in-house and I’m really excited about being able to use that! Obviously, there are some problems related to calibration and the fact that I’d have to use unsupervised methods and <a href="http://en.wikipedia.org/wiki/Principal_component_analysis">PCAs</a> to do a lot of reconstruction. But what’s life if not a succession of punches to the belly?</p><p>I’ve also been working with the <a href="http://www4.informatik.tu-muenchen.de/index_en.shtml">SE chair</a> for almost an year now and have learnt a lot about the importance of empirical methods and philosophy in science, even something as high-nosed as requirements engineering or engineering process improvement. I’m doing a guided research with my supervisor, <a href="http://www4.in.tum.de/~mendezfe/">Daniel</a>, who I have the absolute best discussions with almost every week. The study seems very promising and if I manage to publish something relevant and mildly significant, maybe they will let me continue here after the degree. There’s still time for that, though, and we’ll see.</p><h3>Others</h3><p>One of the top things about living in a place like Munich is that it is delightfully close to the lower Alps region and a great trekking spot is never more than an hour long train journey away. It started with a winter trek to Lenggries last year, and I’ve gone to about 4 more hikes after that. A friend who left for US some months ago said to me that he regretted not having gone to the mountains more often(he went once every month, on average). I plan to not have the same regret when I leave, so I’ll do hopefully do some more treks this year.</p><p>My current save of FM13 is not going well. I’m managing Southampton and at the midway mark of the season, they’ve never managed to come out of the bottom 5. The media predicted relegation in the beginning of the season, and it might end up being so. I’m just glad I didn’t start with Swansea, whose media predictions and squads were only slightly better but with way worse youth and training facilities. If I take them to the Champions League in 5 or 6 seasons, maybe United or Milan will offer me a job, but who knows.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2ae552015dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>