<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by RAKESH on Medium]]></title>
        <description><![CDATA[Stories by RAKESH on Medium]]></description>
        <link>https://medium.com/@jrakesh2727?source=rss-83c196f2dfdf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zR0IPfAU3-61g-XRBA153g.jpeg</url>
            <title>Stories by RAKESH on Medium</title>
            <link>https://medium.com/@jrakesh2727?source=rss-83c196f2dfdf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:19:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jrakesh2727/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DEMYSTIFYING OPTIMIZATION ALGORITHMS IN DEEP LEARNING]]></title>
            <link>https://medium.com/data-science-community-srm/demystifying-optimization-algorithms-in-deep-learning-a107350e13c2?source=rss-83c196f2dfdf------2</link>
            <guid isPermaLink="false">https://medium.com/p/a107350e13c2</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[optimization-algorithms]]></category>
            <dc:creator><![CDATA[RAKESH]]></dc:creator>
            <pubDate>Mon, 27 Jul 2020 08:07:04 GMT</pubDate>
            <atom:updated>2020-07-27T08:07:04.469Z</atom:updated>
            <content:encoded><![CDATA[<p>Deep learning algorithms involve optimization in many contexts. For example, performing inference in models such as PCA involves solving an optimization problem. We often use analytical optimization to write proofs or design algorithms. Of all the many optimization problems involved in deep learning, the most diﬃcult is neural network training.</p><p>Optimization algorithms used for training of deep models diﬀer from traditional optimization algorithms in several ways. Machine learning usually acts indirectly. In most machine learning scenarios, we care about some performance measure W, which is deﬁned with respect to the test set and may also be intractable. We, therefore, optimize W only indirectly. We reduce a diﬀerent cost function L(θ) in the hope that doing so will improve W. This is in contrast to pure optimization, where minimizing L is a goal in and of itself. Optimization algorithms for training deep models also typically include some specialization on the speciﬁc structure of machine learning objective functions.</p><p>The below mentioned are few of the widely used optimizing algorithms which I will be covering in this post -</p><p>• <strong><em>Vanilla Gradient Descent<br>• Momentum Based Gradient Descent<br>• Nesterov Accelerated Gradient Descent<br>• AdaGrad<br>• RMSProp<br>• Adam</em></strong></p><p><strong>Vanilla Gradient Descent:</strong> It is the first update rule we used to follow by directly updating the weights and biases as follows:</p><p><strong><em>ω t+1 = ωt + η ∇ω<br>ω = ω − η * ∂L(ω) / ∂ω<br>η- Learning Rate</em></strong></p><p><strong>Momentum Based Gradient Descent:</strong> Now instead of updating the weights directly let’s accelerate a bit in the direction, it’s moving so that it takes lesser steps in moving in the same direction.</p><p><strong><em>υt = γ * υt−1 + η ∇ωt<br> ωt+1 = ωt − υt<br>ωt+1 = ωt − γ * υt−1 − η ∇ω</em></strong></p><p>The intuition behind this is as we progress further and further down a series or direction, we can place lesser and lesser importance to the later gradients as we move along the same direction because the ‘gamma’ is getting powered up as we go ahead means it starts decaying and more importance to the last step we have taken.<br><strong><em>υt−1</em></strong> -History of the movement and<strong><em> γ</em></strong>-<strong><em> </em></strong>Ranges from 0–1<br>Here, we take an exponentially decaying weighted sum, whereby as we move further and further into the series, the weight decays more.</p><figure><img alt="Countour plot of MBGD(29-epochs) &amp; VGD(40-epochs)" src="https://cdn-images-1.medium.com/max/553/1*h4WYo7gSpArxsh4w5QhU6g.png" /><figcaption>MBGD(29-epochs) &amp; VGD(40-epochs)</figcaption></figure><p>We can see from the above graph that in a for just 29 epochs even in the regions having gentle slopes MBGD can take large steps because the momentum carries along. But whereas in b which took 40 epochs then too couldn’t be able to cross the plateau.</p><blockquote><strong>IS MOVING FAST ALWAYS GOOD?</strong><br><strong>Would there be a situation where momentum would cause us to run pass our goal?</strong><em><br>No, moving fast always isn’t a good idea because there comes a problem of overshooting and oscillating about the minima. So this is the common issue for both the MBGD and VGD of overshooting with a high learning rate. So to come up with the problem of overshooting they introduced what is called Nesterov accelerated gradient descent.</em></blockquote><p><strong>Nesterov Accelerated Gradient Descent:</strong> So the problem associated with the momentum and vanilla gradient descent of oscillation can be overcome by the Nesterov method. What we do here is first we go with the history and then before moving ahead we then calculate the derivative at that point for the second step whereas in momentum we used to calculate the derivative before taking a step and then move by its history and current which would lead to overshoot problems.<br><strong><em>ωt+1 = ωt — γ * (υt−1 — η ∇ ωt)</em></strong></p><p>What we can see from the above equation is that the movement occurs in two steps -<br>i. First is with the history term — <strong><em>γ * υt−1</em></strong><br>ii. Next is with the weight term — <strong><em>η∇ωt</em></strong></p><p>So when moving both steps each time it is possible to overshoot the minima between the two steps. So we can consider first moving with the history term, then calculate the second step from where we were located after the first step (ωtemp ).</p><p>Using the above intuition, the Nesterov Accelerated Gradient Descent solves the problem of overshooting and multiple oscillations.</p><p><strong><em>ωtemp = ωt — γ * υt-1</em></strong></p><p>[ Compute based on movement with the history]<br><strong><em>ωt+1 = ωtemp − η * ∇ ωtemp</em></strong></p><p>[Move further in the direction of the derivative of ωtemp.]<br><strong><em>υt = γ * υt-1 + η * ∇ ωtemp</em></strong></p><p>[Update history with movement due to derivative of ∇ ωtemp]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*cdO993_XSo7CSVltido05A.png" /><figcaption>NAG’S Movement</figcaption></figure><p>Looking ahead helps NAG in correcting its course quicker than MBGD. Hence the oscillations are smaller and the chances of escaping the minima valley are also smaller.</p><p><strong>AdaGrad: </strong>The motivation for adagrad comes from the sparse and dense features from the data set. When in particular feature if most of the points are ‘0’ then there is no point in updating weights since it means as it is because of the ‘0’ value. So what we can do is for the cases when there is less number of 1’s then 0’s we can boost the learning rate for such features and decay the learning rate for sparse features.</p><p>The basic intuition for the Adagrad is to decay the learning rate for parameters in proportion to their update rule.</p><p><strong><em>υt = υt−1 + (∇ ωt)²</em></strong></p><p>In the case of dense features, it increments for most iterations, resulting in a larger υt value.<br>For sparse features it does not increment much as the gradient value is often 0, leading to a lower υt value.</p><p><strong><em>ωt+1 = ωt − (∇ ωt) * η /√((υt) + ε)</em></strong></p><p>From the above equation, we can say that the denominator term ‘υt’ serves to regulate the learning rate η that is why it is called the adaptive learning rate.<br>i. For dense features, υt is larger, √(υt ) becomes larger thereby lowering η.<br>ii. For sparse features, υt is smaller, √(υt ) becomes much smaller thereby lowering η to a smaller extent. The ‘ε’ term is added to the denominator √(υt ) + ε to prevent a divide-by-zero error from occurring in the case of very sparse features i.e. where all the data points yield zero up till the measured instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*NXLKjDgHJkvw0muhvFx1SA.png" /><figcaption><strong>Black</strong> — Vanilla gradient descent, <strong>Red</strong> — Momentum based gradient descent,<br> <strong>Blue</strong> — Nesterov accelerated gradient descent, <strong>Green</strong> — AdaGrad</figcaption></figure><p>So from the figure, we can observe that for the curves(black, red, and blue) my ‘w’ was not changing initially only ‘b’ was changing. That’s the problem with these algorithms the issue was that my ‘w’ was a very sparse feature that’s why it’s derivative was zero in most cases and was not updating and ‘b’ was mostly dense feature it was updating.<br>From the green curve, we can see that parameters corresponding to sparse features get better updates. But still misses the convergence the RMS Prop comes into the picture to solve this issue of converging.</p><p>The LR decays very aggressively as the denominator grows(not good for parameters corresponding to dense features)</p><p><strong>RMS Prop:</strong> The learning rate decays aggressively as the denominator grows which is not suitable for the parameters corresponding to dense features. So the motivation or the intuition for RM prop is that why not decay the denominator and prevent its decay? Right!.<br>Here we are taking the exponential decaying growth to make the curve to converge to the minima.</p><p><strong><em>υt = beta * υt−1 + (1 — beta) * (∇ ωt)² <br>ωt+1 = ωt − (∇ ωt) * η /√((υt) + ε)</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*ys6Xqxp7oLpAl_O0clyzzw.png" /><figcaption><strong>Brown</strong>-RMS Prop</figcaption></figure><p>From the above contour plot, we can say that Adagrad was stuck when it was close to convergence(it was no longer able to move in the vertical (b) direction because of the decayed learning rate). RMS prop overcomes this problem by being less aggressive on the decay.</p><p><strong>Adam:</strong> We know that both momentum and RMS prop were using history. The momentum was using a history of gradient and RMS was using the history of the square of the gradient, right!.<br>Now in case of momentum, this cumulative history helps in moving fast out of the flat surface and used to update the current weight. Whereas in rms prop, history was used to adjust the learning rate suitably used to shrink or grow.</p><blockquote><strong>Can we be to combine it both ?</strong><br>Yes, we can taking the advantage of momentum as well as rms for faster movement and in the mean time using rms to prevent the learning rate from getting blown up or null.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*Bv5TrC_fn8czJIblPuyN-g.png" /><figcaption><strong>Cyan</strong>-Adam</figcaption></figure><p>From the above graph, we can see that it overshoot the minima since it uses momentum based gd.<br><strong><em>mt = beta1 * vt-1 + (1 — beta1)(wt) <br>vt = beta2 * vt — 1 + (1 — beta2)(wt)2 <br>wt + 1 = wt — mt * η /√((vt) + ε) </em></strong><br>The equation above takes advantage of the momentum and below takes the advantage of rms prop. And the updates are performed as shown in the last equation. The adam also provides what is called the <strong>Bias Correction</strong> which ensures that the training is smoother or says the inputs are not like weird kind of. The mt and vt are updated using the below-mentioned equations.<br><strong><em>mt = mt / (1 — beta1^t) <br>vt = vt / (1-beta2^t)</em></strong></p><p>The concepts were from:<br>- <a href="https://padhai.onefourthlabs.in/">https://padhai.onefourthlabs.in/</a><br>- <a href="https://www.deeplearningbook.org/contents/optimization.html">https://www.deeplearningbook.org/contents/optimization.html</a></p><p>#deeplearning #neuralnets #onefourthlabs #optimization_algorithm</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a107350e13c2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-community-srm/demystifying-optimization-algorithms-in-deep-learning-a107350e13c2">DEMYSTIFYING OPTIMIZATION ALGORITHMS IN DEEP LEARNING</a> was originally published in <a href="https://medium.com/data-science-community-srm">DataX Journal</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>