<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jatin Prajapati on Medium]]></title>
        <description><![CDATA[Stories by Jatin Prajapati on Medium]]></description>
        <link>https://medium.com/@jatinskytek1?source=rss-41fdddcea6cc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*QfgKMcvQ0muMMG6n</url>
            <title>Stories by Jatin Prajapati on Medium</title>
            <link>https://medium.com/@jatinskytek1?source=rss-41fdddcea6cc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:08:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jatinskytek1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Optimizers made easy: A beginner-friendly explanation with real-world examples.]]></title>
            <link>https://medium.com/@jatinskytek1/optimizers-made-easy-a-beginner-friendly-explanation-with-real-world-examples-9bde43cfc8df?source=rss-41fdddcea6cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bde43cfc8df</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[adam-optimizer]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <category><![CDATA[optimizer]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Jatin Prajapati]]></dc:creator>
            <pubDate>Sat, 22 Feb 2025 14:46:24 GMT</pubDate>
            <atom:updated>2025-02-24T17:13:23.596Z</atom:updated>
            <content:encoded><![CDATA[<h3>Optimizers in Machine Learning: A Beginner’s Guide with Real-World Examples</h3><p>Hey everyone! I’m Jatin, and if optimizers in Machine Learning sound intimidating to you, don’t worry — you’re not alone. In this article, I’ll break down optimizers in the simplest way possible, using real-world examples to make learning fun and easy. Let’s dive in!</p><p>The origin of optimizers lies in calculus, specifically in the concept of derivatives. A derivative helps us understand how a function changes and identifies points where it reaches its minimum or maximum — just like an optimizer finds the best parameters for a machine learning model.</p><p>Example —Let’s take a simple convex function.</p><p>f(x) = X²+2x<br>df/dx = 2x+2 = 0</p><p>At x = -1, this function will give minimum value.<br>f(-1) = (-1)² + 2*(-1) = -1.</p><p>But what if this can be solved by gradient descent. Gradient descent checks for points to reach minima.</p><p>let initialization value x = 4</p><figure><img alt="This graph shows the how gradient descent finds the minima value of a function." src="https://cdn-images-1.medium.com/max/600/1*0iFdBUSboOMQUDDS5st-cA.gif" /><figcaption>This graph shows the how gradient descent finds the minima value of a function.</figcaption></figure><p>Gradient descent work —</p><p><em>x_new = x — αlpha ⋅ gradient(x)</em></p><p>let, <em>αlpha </em>(learning rate) = 0.01</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*P_dNvqZiZUOj0OfFIIqjqg.png" /><figcaption>With iterations gradient descent finds minima and reach minima.</figcaption></figure><p>Through these iterations, <strong>gradient descent slowly converges to x=−1</strong>, just like our analytical method!</p><p><strong>Intution</strong> — The idea behind <strong>gradient descent</strong> is that while we can analytically compute minima for functions using derivatives, real-world problems involve highly complex functions with multiple variables. In such cases, solving for the minimum directly becomes impractical or even impossible due to computational complexity.</p><p>Gradient descent provides an efficient way to iteratively move towards the best possible solution by following the direction of steepest descent, making it useful for optimizing large-scale machine learning models.</p><p><strong>Problem with Gradient Descent (Simple Explanation)<br></strong>Gradient descent is a powerful method to find the lowest point (minimum) of a function, like finding the deepest point in a valley. However, it has some challenges:</p><p>1️⃣ <strong>Getting Stuck in Local Minima</strong> 🏔️<br>✅ Imagine you’re hiking in the mountains and trying to find the lowest point. But if you stop at a small dip instead of the deepest valley, you’ve found a <strong>local</strong> minimum, not the <strong>global</strong> minimum.</p><p>2️⃣ <strong>Slow Convergence ⏳<br></strong>✅If the learning rate (step size) is too small, it takes a long time to reach the minimum. If it’s too big, you might keep overshooting and never settle.</p><p>3️⃣ <strong>Oscillations 🔄<br></strong>✅On steep paths, gradient descent may bounce back and forth, making it inefficient.</p><h3>Why Do We Need Optimizers?</h3><p>Optimizers are designed to <strong>overcome the problems</strong> of basic gradient descent. If an optimizer can efficiently navigate through a function with local minima, slow convergence, or oscillations, it proves that it can also handle <strong>complex real-world datasets</strong>.</p><h3>How Optimizers Help?</h3><p>✅ <strong>Escape Local Minima</strong> — Advanced optimizers adjust learning to avoid getting stuck.<br>✅ <strong>Faster Convergence</strong> — They speed up learning by adapting step sizes.<br>✅ <strong>Stable Updates</strong> — They reduce unnecessary jumps and oscillations.</p><p>Since real-world machine learning problems involve <strong>millions of parameters</strong> and <strong>complex loss surfaces</strong>, using better optimizers makes training efficient and improves model performance.</p><p><strong>Types of optimizers</strong> — <br>Basic Optimizers</p><ul><li><strong>Gradient Descent (GD)</strong> — Uses the entire dataset to update weights (slow but stable).</li><li><strong>Stochastic Gradient Descent (SGD)</strong> — Updates weights after each data point (fast but noisy).</li><li><strong>Mini-Batch Gradient Descent</strong> — Uses small batches for a balance between GD &amp; SGD.</li></ul><h4>First-Order Optimizers (Focus Here)</h4><ul><li>Momentum</li><li>Nesterov Accelerated Gradient (NAG)</li><li>Adagrad</li><li>RMSprop</li><li>Adam</li><li>Varients of Adam (Nadam, Radam, Adamax, Adabelief)</li></ul><h4>Second-Order Optimizers (Based on Second Derivative)</h4><ul><li>Newton’s Method</li><li>BFGS (Broyden–Fletcher–Goldfarb–Shanno)</li><li>L-BFGS (Limited-memory BFGS)</li><li>Hessian-Free Optimization</li></ul><h4><strong>1. Momentum</strong></h4><p>Momentum in physics means an object in motion keeps moving unless something stops it.<br>In optimization, <strong>Momentum</strong> helps gradient descent move faster by remembering the past direction and reducing sudden changes. Think of it like rolling a ball down a hill — it won’t stop immediately at small bumps but keeps going in the general downward direction. This helps the optimizer avoid getting stuck in small local minima and reach the best solution faster.</p><p><em>The original </em><strong><em>Polyak Momentum </em></strong><em>formulation is →<br>vt = beta * v(t-1) + alpha* gradient(x)<br>x_new = x — vt</em></p><p><em>Adjusted Momemtum formula →<br>vt = beta * v(t-1) + (1 — beta) * gradient(x)<br>x_new = x — αlpha ⋅ vt</em></p><p>I’m using the <strong>adjusted momentum formulation</strong> because it provides a clearer visualization of how momentum builds up and smooths updates. The original Polyak formula can cause excessive overshooting, making it harder to see the gradual accumulation of momentum. The adjusted version better demonstrates how momentum accelerates optimization while maintaining stability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*MYnj2op_IkZsXgBs_3XZ6w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*KsQ9iMhhERMmAgdIHx6yJw.gif" /><figcaption>Image given above are showing gradient descent path on Rastrigin function without &amp; with momentum.</figcaption></figure><p><em>vt</em> = current momentum<br><em>v(t-1) = previous momentum<br>beta = momentum factor<br>αlpha = learning rate</em></p><p>Simple gradient descent can get stuck in small fluctuations (local minima), slowing down progress. Momentum helps by adding extra velocity, allowing the optimizer to move past these small dips and reach a better solution faster. This helps gradient descent find the optimal solution more efficiently.</p><p><strong>Simple Gradient Descent:</strong> Stops at local minima.<br><strong>Momentum Gradient Descent:</strong> Moves smoothly, avoiding small traps.</p><p>Limitations — <br><strong>A).</strong> <strong>Sensitive to Learning Rate &amp; Initialization</strong> — Momentum needs a properly tuned learning rate and a good starting point to gain velocity and avoid getting stuck in local minima.<br><strong>B</strong>)<strong>. Overshooting</strong> — Due to accumulated momentum, the optimizer may overshoot the optimal solution and oscillate or fall into another local minimum. (We can see in the above image)</p><h4><strong>2. NAG (Nestrov Accelerated Gradient)</strong></h4><p>NAG improves the momentum optimizer by adjusting the update direction before applying the gradient, helping to reduce overshooting and improve convergence.</p><p><strong><em>Real</em></strong><em> →<br>look_ahead = x — beta * vt<br>vt = beta * v(t-1) + αlpha* gradient(look_ahead)<br>x_new = x — vt</em></p><p><strong><em>Adjusted</em></strong><em> →<br>look_ahead = x — beta * vt<br>vt = beta * v(t-1) + (1 — beta) * gradient(look_ahead)<br>x_new = x — αlpha ⋅ vt</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*-dZurAvicwRbVZdmkLcxQg.gif" /><figcaption>Gradient descent with NAG optimizer</figcaption></figure><p>NAG penalize the momentum to reduce overshooting and reach optimal solution.<br>Still for optimal solution, paramters tuning is necessary.</p><h4><strong>3. Adagrad</strong></h4><p>Adaptive Gradient Algorithm is a smart optimization technique that automatically adjusts how fast a machine learning model learns. Unlike traditional gradient descent, which uses a fixed learning rate, Adagrad makes learning faster for rarely changing parameters and slower for frequently changing ones.</p><p>Example — Let we have two paramters, a and b. Initialize a= b =10.<br>&gt;Loss function = 5a²+ 0.1b²<br>Gradient(a) = 10a<br>Gradient(b) = 0.2b</p><p>Using Simple Gradient Descent -<br>Formula →x_new = x-alpha*gradient(x)<br>a) Take High learning rate like alpha = 0.5<br>a_new = 10–0.5*10*10 = -40<br>b_new = 10–0.5*0.2*10 = 9</p><p>b) Take Low learning rate like alpha = 0.01<br>a_new = 10–0.01*10*10 = 9<br>b_new = 10–0.01*0.2*10 =9.98</p><p>As from above calculations, it can be been that for higher learning rate parameter a will not converege and for lower learning rate b will take a huge number of iterations to converge. Therefore fixed learning rate should not be used.</p><p><em>G(t) =G(t-1)+g(t)²<br>new_alpha = αlpha/(sqrt(G(t))+epsilon)<br>x_new = x —new_alpha⋅ g(t)</em></p><p>G(t) = sum of past gradients of all parameters (g(t)²+g(t-1)²+g(t-2)²………)<br>g(t) = current gradient <br>epsilon is added to prevent division by zero.</p><p>Now lets check<br>Using Adaptive gradient — <br>a) High learning rate — 0.1<br><em>On paramerter a,<br>G(t) = 0 + (10*10)² = 100<br>new_learning_rate of a = 0.1/sqrt(G(t)) = 0.1/sqrt(100) = 0.01</em><br>learning rate becomes low from 0.1 to 0.01</p><p>a_new = 10-0.01*(10*10) =9</p><p><em>On paramater b,</em><br><em>G(t) = 0 + (0.2*10)² = 4<br>alpha = 0.1/sqrt(G(t)) = 0.1/sqrt((10*10)) = 0.5</em><br>learning rate becomes high form 0.1 to 0.5</p><p>a_new = 10–0.5*(0.2*10) =9</p><p>As we can see, <strong>Adagrad automatically adjusts the learning rate</strong> for different parameters, helping them converge at an optimal pace. This adaptability is a major advantage over standard gradient descent.</p><p>Now, let’s compare <strong>Adagrad vs. Momentum-based Gradient Descent</strong> to see how Adagrad improves convergence. We’ll use a <strong>smoothed step function</strong> to highlight the differences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*M45glkClH_sNZ0tHY118pA.gif" /><figcaption>Gradient descent on Smoothed step function.</figcaption></figure><p>Adagrad automatically increase the learning rate to converege towards minima while sgd with/without momentum struggles.</p><h4>4. RMSprop</h4><p>RMSprop is an improved version of Adagrad. While Adagrad <strong>automatically adjusts the learning rate</strong> for each parameter, it <strong>slows down over time</strong> because it keeps accumulating all past squared gradients.</p><p>RMSprop fixes this by using an <strong>exponentially decaying average</strong> of past gradients instead of summing them all up, preventing the learning rate from becoming too small. 🚀</p><p><em>G(t) = </em>γ*<em>G(t-1)+(1-</em>γ<em>)*g(t)²<br>new_alpha = αlpha/(sqrt(G(t))+epsilon)<br>x_new = x — new_alpha⋅ g(t)</em></p><p>This formula when explored-<br>G(t) = (1-γ)*g(t)² +γ*(1-γ)*g(t-1)²+γ²*(1-γ)*g(t-2)²+γ³*(1-γ)*g(t-3)²……………….</p><p>By adding γ decay rate, RMSprop gives <strong>exponentially decreasing importance</strong> to past gradients. This means recent gradients influence the update more, while older gradients gradually fade away.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*t7IjVd52OP6n4o-T1nD6Aw.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*7dKndjgQAY2QWt1iJKgCwQ.gif" /><figcaption>RMSprop optimizer gradient descent</figcaption></figure><p>RMSprop accelerates convergence in flat regions by exponentially decaying past gradients. This adaptive learning rate also helps stabilize updates in non-convex functions.</p><h4>5. Adam</h4><p>Adam (Adaptive Moment Estimation) is an optimization algorithm that combines <strong>momentum</strong> (to smooth updates) and <strong>RMSprop</strong> (to adapt the learning rate). It effectively handles both flat and steep regions, making it efficient for deep learning tasks.</p><p><em>Momentum →vt = beta * v(t-1) + (1 — beta) * gradient(x)<br>Adaptive learning rate →G(t) = </em>γ*<em>G(t-1)+(1-</em>γ<em>)*g(t)²</em></p><p><em>new_alpha = αlpha/(sqrt(G(t))+epsilon)</em></p><p><em>x_new = x — new_alpha⋅ v(t)</em></p><p>In the formula it combines both momentum and rmsprop. Adam also adds up bias correction. As at starting point v(t) and G(t) are zero which leads slower updates at initial stage and underestimate momemtum and rmsprop.</p><p><strong>At start</strong>, vt = (1-beta)* gradient(x)<br>because v(t-1) is zero, therefore (1-beta) leads to slower update in gradient. Same for second order rmsprop.</p><p>Bias correction reason →</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*eIlng_mkmb7L4B85nWfexw.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*zigG5oJPgCG-tiVWAzLi7w.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*fLlvQjlFkbhHt3Rxw0FLZw.png" /><figcaption>Effect of bias and after bias correction are shown</figcaption></figure><p>Beta = 0.9 &amp; gamma = 0.999 are taken,<br><strong>In 2nd image x²+sin(5x) (momentum)</strong> →</p><ul><li><strong>Without Bias Correction:</strong> Initial updates are smaller due to the underestimated moving average of gradients.</li><li><strong>With Bias Correction:</strong> Corrected momentum increases early updates, leading to faster acceleration.</li></ul><p><strong>In 3rd image x²+sin(3x) (RMSprop)</strong>→</p><ul><li><strong>Without correction</strong>, Gt​ starts small, leading to <strong>larger updates</strong> in the early iterations and make training unstable.</li><li><strong>With correction</strong>, Gt​ is divided by (1−β**t​), making early updates <strong>more conservative</strong> (smaller).</li></ul><p>Adam incorporates bias correction and combines momentum with adaptive gradient learning, allowing it to handle <strong>non-convex landscapes (steep valleys) and accelerate convergence in flat regions</strong>. This enables Adam to efficiently navigate complex optimization problems and reach a good solution, though <strong>tuning parameters is still essential for finding the global optimum</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*xOAb-LgiBKBhheNk0XtAyg.gif" /><figcaption>Adam optimizer gradient descent</figcaption></figure><h4>6. Variants of Adam (Nadam, Radam, Adamax, Adabelief)</h4><p>There are multiple variants of Adam which improve the performance of adam in some way.</p><p>Nadam — Added nestrov momentum which makes converegence faster.<br>Radam — Rectifies Adam’s variance issue for stable learning in starting.<br>Adamax — Uses an infinity norm for more stability.<br>Adabelief — Adjust step sizes based on thebelief in the gradient.</p><p>Among them, <strong>AdaBelief</strong> improves convergence in non-convex problems more effectively using momentum.</p><h3><strong>Conclusion</strong> →</h3><p>Adam is a powerful adaptive optimizer that combines <strong>momentum and an adaptive learning rate</strong>, leading to fast convergence. However, for <strong>non-convex optimization</strong>, I believe <strong>AdaBelief</strong> performs even better by dynamically adjusting step sizes based on the gradient’s trustworthiness.</p><p>If you’re working with complex deep learning models, <strong>AdaBelief might be the key to achieving more stable and efficient training.</strong></p><p>However, as demonstrated in this article, first-order gradient-based optimizers perform well on 1D Rastrigin &amp; other functions but struggle on highly non-convex 2D functions like Rastrigin, Ackley, Rosenbrock etc. The presence of multiple local minima and oscillations makes convergence challenging.</p><p>To tackle this, hybrid approaches such as <strong>Simulated Annealing, Evolutionary Strategies, or second-order methods</strong> can help escape local minima and improve optimization. Choosing the right optimizer depends on the function landscape and the need for exploration versus exploitation.</p><p>We will explore 2D highly non-convex functions in our next article. Stay tuned!</p><p><strong>Thanks for reading! 🚀</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bde43cfc8df" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>