<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ramsane on Medium]]></title>
        <description><![CDATA[Stories by ramsane on Medium]]></description>
        <link>https://medium.com/@ramsane?source=rss-86fcaa602ad6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mShYQXeP0lCJ0db-aFqUqA.jpeg</url>
            <title>Stories by ramsane on Medium</title>
            <link>https://medium.com/@ramsane?source=rss-86fcaa602ad6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:14:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ramsane/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Learning Rate in Gradient Descent —  what could possibly go wrong]]></title>
            <link>https://medium.com/@ramsane/learning-rate-in-gradient-descent-what-could-possibly-go-wrong-6e58614941a0?source=rss-86fcaa602ad6------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e58614941a0</guid>
            <category><![CDATA[learning-rate]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[ramsane]]></dc:creator>
            <pubDate>Thu, 03 Oct 2019 11:48:29 GMT</pubDate>
            <atom:updated>2019-10-11T04:42:14.214Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Rate in Gradient Descent — what could possibly go wrong</h3><h4>Explained with Linear Regression</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*_ChtfJzlHqpndjNUb5LCxg.png" /></figure><blockquote>Learning rate is one of The most important thing to consider in whole of machine learning. But choosing the correct learning rate is pretty much impossible all the time.</blockquote><blockquote>But…, having the knowledge of what it can do is useful in some cases where we can at least guess what could be the appropriate learning rate.</blockquote><blockquote>We will see how even the slightest change in learning_rate can improve the speed drastically or could break your model. I mean it can overshoot the minima and it might never return .</blockquote><blockquote>The data we will use in this blog is created from <a href="http://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_regression.html">sklearn make_regression</a>.</blockquote><pre><strong>X, y = make_regression(n_samples=75, n_features=1, random_state=0, noise=20, bias=100)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*0vN34Kpllu0tV7bwnSdP-g.png" /><figcaption>plot</figcaption></figure><ul><li>We will draw the regression line(in red) with learned Linear Regression and we put that as a benchmark.</li><li>We choose Linear Regression over any other because it is easy to understand ( and easy to code too).</li><li>we will focus mainly on how learning rate effects on optimizing least squared error of Linear Regression.</li></ul><blockquote><strong>Note:</strong> Here we won’t be explaining linear regression in much detail.</blockquote><p>Since we have 1-D data, we can easily fit a line to it. (We have to predict y given X).</p><ul><li>This line is what we call <strong><em>‘ weight vector</em></strong>’ or ‘ <strong><em>hypothesis</em></strong>’.</li><li>By using this line (the plane in higher dimensional data), we will fit the line with minimum error possible. We have only one x and output for that x. We will try to fit the line/plane that is best for all the data points in this dataset.</li><li>After we find the best line for this data, we will<strong> predict y</strong> given any <strong>new x</strong>.</li></ul><p>Let’s say our equation of our line is :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*qMdIm38dktakbUj7VzhVeg.png" /></figure><ul><li>After we find the best line for this data, given any new <strong>x</strong>, we will just substitute our data ( x_1 in this case), in the equation which is optimized with better w_0 and w_1 values, to predict <strong>y</strong>.</li></ul><p>To make our calculation simpler ( calculating the Gradient Descent with vector multiplication rather than traditional for loops.. ), we will rewrite the equation as :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*jmiBaqkUZ8gq6u0LRFFSsw.png" /></figure><h4>Initial Weights :</h4><p>The initial weights should be random. But we chose some specific numbers just to make sure that is not anywhere near the Optimal Weights. Reshaping is done to make it easier while computing cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*JbijcGFt61Ib5DjqcFqmVQ.png" /><figcaption>Initial weights ( W_0, W_1 )</figcaption></figure><h4>Cost Function :</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*KND0bBqPSOrMniCfyMirXw.png" /><figcaption>Mean Squared Error of Initial Weights</figcaption></figure><p>For Linear Regression, cost function computes the MSE ( Mean Squared Error).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/1*9lJBpFeD63oZjk5s1Uko0A.png" /><figcaption>Mean Squared Error in vector Notation</figcaption></figure><blockquote><strong><em>Note :</em></strong>The fraction (1/2) is just to make the optimization simpler<em>.</em></blockquote><ul><li>This is how our regression with initial weights looks like. This is why we chose the weights manually instead of generating them randomly.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*pJ4ePFFUVAbTYAGgyC-1HQ.png" /></figure><h4>Optimisation Function :</h4><ul><li>Our optimisation function is purely <strong><em>convex function</em></strong>. So there will be only <strong>one minimum</strong> and it is <strong><em>GLOBAL.</em></strong></li><li>So we can apply <strong>Gradient Descent</strong> to find out where the minimum value is.</li><li>The <strong>minimum</strong> lies in the <strong>darkest place in the 3-D plot</strong>( below fig.). We will approximate this using Gradient Descent.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*CH604kfGjnCgCwphi60RFw.png" /><figcaption>Cost Function (least Squares) which is convex</figcaption></figure><h3><strong>3. Gradient Vs Stochastic Gradient Descent:</strong></h3><p>There are many interpretations of what is what regarding this gradient descent method. I will try to explain to the best of my knowledge.</p><ul><li><strong>Gradient Descent: </strong><em>Its cost function will take all the training samples to do a single update. We can call this as a single epoch. But if we have a large data set, I mean literally huge(like 100 Million data points), applying GD is a nightmare. It would take days to reach the minima. With high dimensional data and with large data set, it is even worse.</em></li></ul><p><em>But the advantage of using GD is </em><strong><em>correctness</em></strong><em>, and it is best suitable when you have small data.</em></p><ul><li><strong>Stochastic Gradient Descent: </strong>Its <em>cost function will take one training sample to update the weight vector( in one epoch). Using SGD will be faster because you use only one training sample and it will start converging from the first epoch itself.</em></li></ul><p>The advantage of using SGD is, it often <strong><em>converges faster than GD</em></strong>. But the cost function is not well minimized when compared to GD. In the case of GD, when it reaches minima, it will keep oscillating there itself. Whereas in GD, it will try to reach minima with each epoch.</p><ul><li><strong>Batch SGD / Batch GD: </strong>In this variation of GD, we will make a batch of training samples which are taken at random from the training sample(per epoch) to update the weight vector. Some times people call it mini-batch SGD too. It doesn’t matter what you call it unless you are referring to the same thing.</li></ul><p>With Batch SGD, we are trying to bridge the gap between GD(correctness) and SGD(fastness). We will converge faster as we are not using whole data while updating the weight vector. And it will be the close approximation to the minima as we are using a batch of training sample instead of a single point.</p><p>As batch size increases, the time it will take to reach minima(closer approximation) will increase, and the accurate it will get. To put it in simple words, SGD becomes GD.</p><h3>4. The behaviour of Gradient Descent with different Learning Rates</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*g49Wb0L7bDu0TSzB0xoxig.png" /></figure><ul><li>As we discussed in the previous section, there is a subtle difference between Gradient and Stochastic Gradient Descent. This function is basically a Stochastic Gradient descent with some batch size. People call it Batch_SGD some times.</li><li>We can perform Gradient Descent with it by just passing training size as batch_size.</li><li>The gradient descent<strong><em>( ) </em></strong>function returns two lists.., <em>weights </em>and <em>costs. </em>Each of which contains what is its value after each epoch. This is useful to plot the graphs.</li><li>With [Stochastic]Gradient Descent in our hands.., let’s try to see how the Gradient Descent optimizes for different learning rates.</li><li>I wrote some code to get some kind of viz using matplotlib. The code is in my GitHub. Feel free to use it and you can play with your own data set. But make sure that it is small.</li><li>I have tested Gradient Descent with so many learning rates. Here we will show 4 cases where three of them are in such a way that gradient descent is slow, reasonably fast, fastest of all.</li><li>The last one is a very interesting case where the gradient Descent will never take you to Minimum. It will overshoot minima in the first epoch itself. And it will never go in the direction of minima.</li></ul><h4><strong>GD with Learning Rate=0.01 (</strong>2100 iterations)<strong>:</strong></h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FaDehv2rbTOY%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DaDehv2rbTOY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FaDehv2rbTOY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c15352e5faab2fb91f76fc3fdcdcb014/href">https://medium.com/media/c15352e5faab2fb91f76fc3fdcdcb014/href</a></iframe><h4>GD with Learning Rate=0.1 (300 iterations):</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKcXy97XfV_4%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKcXy97XfV_4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKcXy97XfV_4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/27bc1dc735c6405de612ff800de9bdd4/href">https://medium.com/media/27bc1dc735c6405de612ff800de9bdd4/href</a></iframe><h4>GD with Learning Rate=1.0 (50 iterations):</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F0b17Ggvhkbk%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D0b17Ggvhkbk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F0b17Ggvhkbk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed3f8bf5f4eff03d30e27a30b4799a73/href">https://medium.com/media/ed3f8bf5f4eff03d30e27a30b4799a73/href</a></iframe><p>From all these three videos, we can observe that it reached minima faster with increasing learning rate. But If you keep on increasing it, it would overshoot after some epochs. This can be observed with following videos.</p><h4>GD with Learning Rate=1.50 (100 iterations):</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FfGnHMD-c4x8%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DfGnHMD-c4x8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FfGnHMD-c4x8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/362c2ead7617b529c7b8c996d6bf8f1c/href">https://medium.com/media/362c2ead7617b529c7b8c996d6bf8f1c/href</a></iframe><h4>GD with Learning Rate=1.75 (150 iterations):</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZRIRtLqA0bU%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZRIRtLqA0bU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZRIRtLqA0bU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8118606e04d32add8bd9d042041a51ec/href">https://medium.com/media/8118606e04d32add8bd9d042041a51ec/href</a></iframe><h4>GD with Learning Rate=1.80 (250 iterations):</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FK7u1_V0bplY%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DK7u1_V0bplY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FK7u1_V0bplY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d3c9b304c4bd03f655c5bf878ee9abdd/href">https://medium.com/media/d3c9b304c4bd03f655c5bf878ee9abdd/href</a></iframe><h4>GD with Learning Rate=1.85 (500 iterations):</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FEcK04wkFlps%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DEcK04wkFlps&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FEcK04wkFlps%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e4d4c5e407820e321a0d16b4ddccae9/href">https://medium.com/media/6e4d4c5e407820e321a0d16b4ddccae9/href</a></iframe><p>You can get the idea on how it can overshoot with the increase in learning rate. Now, we will see how subtle change in the learning rate will effect on obtaining the minima.</p><h4>GD with Learning Rate=1.8898(500 iterations):</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FnXYdzZ5FHAQ%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DnXYdzZ5FHAQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FnXYdzZ5FHAQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4490eacae9c8a4d125211804ce9a001/href">https://medium.com/media/e4490eacae9c8a4d125211804ce9a001/href</a></iframe><h4>GD with Learning Rate=1.8899 (500 iterations):</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F16peJemxOCM%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D16peJemxOCM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F16peJemxOCM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/35ad8080314554252b1b2966d529dd95/href">https://medium.com/media/35ad8080314554252b1b2966d529dd95/href</a></iframe><p>These are created with few learning rates. Jupyter notebook is available in the github repository. Feel free to change the dataset and see how the decision boundary is changing.</p><p>you can get some non-linear data, add few additional features and see how the decision boundary is changing. As creating videos is done via matplotlib. It might take a while. Try smaller data sets. And this is not scalable as it based on matplotlib.</p><p><a href="https://github.com/ramsane/ML_Inside_Out">ramsane/ML_Inside_Out</a></p><p><strong>PS: As mentioned earlier, this function is generic and works for batch sizes from 1 to length(data set). Observe how each update step will be with different batch sizes in the cost function (3D)</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e58614941a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>