<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by CARE - Center for Applied Research &amp; Engineering on Medium]]></title>
        <description><![CDATA[Stories by CARE - Center for Applied Research &amp; Engineering on Medium]]></description>
        <link>https://medium.com/@care-ocw?source=rss-3bc19b781ef4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*T3wKF3UoUDUyK94eTHm7-g.png</url>
            <title>Stories by CARE - Center for Applied Research &amp;amp; Engineering on Medium</title>
            <link>https://medium.com/@care-ocw?source=rss-3bc19b781ef4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:25:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@care-ocw/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Image Augmentation in Keras]]></title>
            <link>https://care-ocw.medium.com/image-augmentation-in-keras-c2ae76a68a80?source=rss-3bc19b781ef4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2ae76a68a80</guid>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[image-augmentation]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[CARE - Center for Applied Research & Engineering]]></dc:creator>
            <pubDate>Thu, 11 Jan 2024 08:47:02 GMT</pubDate>
            <atom:updated>2024-01-11T08:47:02.375Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*fTpNdJTaBdsdXpppZIQkhg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*fTpNdJTaBdsdXpppZIQkhg.jpeg" /></figure><h3>Introduction</h3><p>In the dynamic field of computer vision and deep learning, the significance of training data quality and diversity cannot be overstated. Image augmentation is a key technique that aids in enhancing dataset richness and, consequently, improving model generalization. In this blog post, we’ll explore the world of image augmentation using Keras, a popular deep learning library, using a practical example of augmenting a simple image dataset in a folder.</p><h4>Understanding Image Augmentation:</h4><p>Image augmentation involves applying various transformations to existing images within your dataset, creating diverse variations without requiring additional labeled data. These transformations may include rotations, flips, zooms, shifts, and more, effectively simulating real-world scenarios. By augmenting your dataset, you empower your model to learn from a more comprehensive set of examples, making it more resilient and capable of handling a wider range of input variations.</p><h4>Getting Started with Keras</h4><p>Keras offers a straightforward way to implement image augmentation, using the `ImageDataGenerator` class. Let’s explore a basic example by augmenting a simple image dataset stored in a folder.</p><pre>from keras.preprocessing.image import ImageDataGenerator<br>import matplotlib.pyplot as plt<br>import os<br>from keras.preprocessing import image<br>import numpy as np<br><br># Define the path to the folder containing your images<br>image_folder_path = &#39;path_to_your_image_folder&#39;<br><br># Create an ImageDataGenerator instance<br>datagen = ImageDataGenerator(<br>    rotation_range=20,<br>    width_shift_range=0.2,<br>    height_shift_range=0.2,<br>    shear_range=0.2,<br>    zoom_range=0.2,<br>    horizontal_flip=True,<br>    fill_mode=&#39;nearest&#39;<br>)<br><br># Load and augment a sample image from the folder<br>img_path = os.path.join(image_folder_path, &#39;sample_image.jpg&#39;)<br>img = image.load_img(img_path)<br>x = image.img_to_array(img)<br>x = np.expand_dims(x, axis=0)<br><br># Generate augmented images<br>augmented_images = datagen.flow(x, batch_size=1)<br><br># Display the original and augmented images<br>plt.figure(figsize=(8, 4))<br>for i in range(9):<br>    plt.subplot(3, 3, i + 1)<br>    augmented_image = augmented_images.next()[0].astype(&#39;uint8&#39;)<br>    plt.imshow(augmented_image)<br>    plt.axis(&#39;off&#39;)<br>plt.show()</pre><h3>Common Augmentation Techniques:</h3><p><strong>1. Rotation: </strong>Rotating images by a specified angle, introducing variations in orientation.</p><p><strong>2. Shift:</strong> Shifting the image horizontally and vertically, simulating changes in position.</p><p><strong>3. Zoom:</strong> Zooming into or out of an image, replicating the effect of different camera perspectives.</p><p><strong>4. Flip:</strong> Flipping images horizontally or vertically, capturing reflections or different viewpoints.</p><p><strong>5. Shear:</strong> Shearing an image, distorting its shape along a specified axis.</p><p><strong>6. Brightness and Contrast Adjustments: </strong>Modifying brightness and contrast levels to handle varying lighting conditions.</p><p><strong>Best Practices:</strong></p><p><strong>1. Balance:</strong><br> Maintain a balance between augmentation and the original dataset, ensuring the augmented dataset remains representative.</p><p><strong>2. Evaluate Performance:</strong><br> Monitor the model’s performance on both the training and validation sets to prevent overfitting.</p><p><strong>3. Experiment:</strong><br> Experiment with augmentation parameters to find the optimal configuration for your specific use case.</p><h4>Conclusion</h4><p>Image augmentation stands as a powerful technique in your deep learning toolbox, enabling you to create a more diverse and robust dataset for training your models. With Keras, implementing image augmentation is seamless, facilitating the enhancement of your model’s performance and its adaptability to real-world scenarios. Experiment with different augmentation techniques and parameters to discover the ideal configuration for your specific application. Happy augmenting!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2ae76a68a80" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Deeper Look into Gradient Based Learning for Neural Networks]]></title>
            <link>https://medium.com/data-science/a-deeper-look-into-gradient-based-learning-for-neural-networks-ad7a35b17b93?source=rss-3bc19b781ef4------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad7a35b17b93</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[CARE - Center for Applied Research & Engineering]]></dc:creator>
            <pubDate>Fri, 20 Sep 2019 02:32:22 GMT</pubDate>
            <atom:updated>2019-09-26T03:03:06.313Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*SjtKOauOXFVjWRR7iCtHiA.gif" /><figcaption><a href="http://ruder.io/content/images/2016/09/saddle_point_evaluation_optimizers.gif">http://ruder.io/content/images/2016/09/saddle_point_evaluation_optimizers.gif</a></figcaption></figure><p>In Deep Learning, the way we try to approximate a function between input and output, is by randomly initialising parameters of the network first and then gradually updating them to find the right configuration of these parameters by minimising a loss function which in most of the cases is non convex in nature (many local minima instead of a single global minima) and thus training a neural network is a nondeterministic combinatorial optimisation problem because we can not guarantee that the point of convergence when achieved is the global minima or a saddle point on the error surface.</p><p>A whole bunch of algorithms are proposed to handle this situation out of which popular ones are SGD, momentum based, NAG, RMSprop and ADAM. They all are some variants of classical Gradient Descent Algorithm and ADAM which is a combination of RMSprop and momentum is considered to be current state of the art. <br>This article is trying to explain all of them and how they all are trying to overcome limitations of their ancestors.</p><h3>1. Gradient Descent Update Rule</h3><p>Consider that all the weights and biases of a network are unrolled and stacked into a single vector θ in some high diminutional vector space and the loss associated with it be ϑ(θ). Any change in the direction or magnitude of θ will give us a new value of ϑ(θ).</p><p>While training the network, at each epoch our goal is to find another vector ∆θ such that the new loss ϑ(θ + ∆θ) is less than the previous loss ϑ(θ), and thus at each epoch the goal is to find ∆θ such that following inequality holds,</p><p>ϑ(θ + ∆θ) — ϑ(θ) &lt; 0</p><p>Before we move forward in our quest to find ∆θ, let’s quickly recap <strong>Taylor Series</strong> which states that any non polynomial function can be exactly approximated as an infinite sum of polynomial terms,</p><p>ƒ(x) = c0 + c1 x + c2 x² + c3 x³ + . . . .</p><p>The only condition to satisfy is that the first, second and all higher order derivative of LHS and RHS should exactly match and ƒ(x) should be an infinitely differentiable function.</p><blockquote>cos(θ) at θ = 0, can we written as cos(θ) = 1- 1/2 θ² as the second order Taylor approximation. Because we are only using first three terms (c1= 1, c2 = 0, c3 = 1/2) instead of all infinite polynomial terms, the approximation works fine only in a close neighbourhood of θ around 0.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*eEzj4aj6N1a3ihF9WukM1w.gif" /><figcaption><a href="https://bespokeblog.files.wordpress.com/2011/07/sintay.png">https://bespokeblog.files.wordpress.com/2011/07/sintay.png</a></figcaption></figure><p>In general, Taylor Series of ƒ at x is the following power series</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*0vP6ejLjrDOn59bNzcm-Ng.png" /></figure><p>For a small ∆x, ƒ(x) ≈ ƒ(x +∆x).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*0-aBrb0afdO_7XnC_twpKQ.png" /></figure><p>For the multivariate function ƒ(x), just like the loss ϑ(θ) and ϑ(θ + ∆θ) which depends of the vector θ and θ + ∆θ, following is the first order Taylor approximation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/1*ErD-y0KPVyUScMaiUAq9WA.png" /></figure><p>And this is the second order Taylor approximation of ƒ(x + ∆x)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*Zws257BaX_YavkgaBZigAQ.png" /></figure><p>Taking the first order Taylor approximation of loss function</p><p>ϑ(θ + ∆θ) = ϑ(θ) + ∆θ ∇ϑ(θ)<br>ϑ(θ + ∆θ) — ϑ(θ) =∆θ ∇ϑ(θ)</p><p>We know that to minimise the loss function, loss at each step should be less than the previous one and this inequality (ϑ(θ + ∆θ) — ϑ(θ) &lt; 0) should hold, and thus from the first order Taylor approximation implies that the dot product ∆θ ∇ϑ(θ) &lt; 0. <br>Because A dot B (A B cos(ϕ)) can only be negative when the angle between them is greater then 90 degree, maximum reduction in loss at any epoch is possible when ∆θ is opposite to the direction of gradient of loss ∇(θ) w.r.t θ</p><p>This gives us the well known <strong>Gradient Descent</strong> update rule</p><p>θ := θ — ∇ϑ(θ)</p><blockquote>Because we are using first order Taylor approximation of the loss function which is good only in the near neighbourhood of θ, it is a necessary condition that ∆θ must be small. So we multiply η &lt;&lt; 0 with ∆θ in our update rule.</blockquote><blockquote>θ := θ — η ∇ƒ(θ)</blockquote><blockquote>In practice, larger η also causes overshooting in machine learning and is termed as the learning rate and is a hyper parameter.</blockquote><h4>Limitations</h4><p>One of the limitations of Vanilla Gradient Descent is that in the region of gentle slopes, computed gradients are very small resulting in a very slow convergence. One may simply increase the value of η but it may violate the assumption of first order Taylor approximation of loss function, also larger learning rate may cause overshooting.</p><p>Another point to consider is that near the minima loss functions are slightly ellipsoidal and gradients in these regions from the first order Taylor approximation of loss function are almost orthogonal to the true direction of convergence, causing a zig-zag path and a slow convergence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*hytYy8vTsTggPFqolLt3Og.png" /></figure><blockquote>Although it is not very clear, but second order Taylor approximation may solve this problem. but calculation of ∇ ²ƒ(θ) (Hessian) is computationally very expensive. There are some proposed implicit ways of using hessian by setting the learning rate η as 1/ β, where β is the dominant eigenvalue of the hessian. <a href="https://www.cs.princeton.edu/courses/archive/fall18/cos597G/lecnotes/lecture3.pdf">Click here</a> to read more about this proposed solution.</blockquote><h3>2. Using Momentum for Learning</h3><p>To resolve the slow convergence problem of gradient descent algorithm, one of the way may be to update θ in the direction of exponentially weighted average of gradients computed in pervious steps.</p><p>momentum_t = γ * momentum_t -1 + η ∇ϑ(θ_t) <br>θ_t+1 := θ_t - momentum_t</p><p>where ∇ϑ(θ_t) represents gradients calculated at step t.</p><p>momentum_0 = 0<br>momentum_1 = momentum_0 + η∇ϑ(θ_1) = η∇ϑ(θ_1)<br>momentum_2 = momentum_1 + η ∇ϑ(θ_2) = γ η∇ϑ(θ_1) + η∇ϑ(θ_2)<br>momentum_3 = momentum_2 + η ∇ ϑ(θ_3) <br>=&gt; γ²η∇ϑ(θ_1) + γη∇ϑ(θ_2) +η∇ϑ(θ_3)</p><p>Setting γ &lt;&lt; 1 insures that the higher power of γ causes decay in the gradients computed in previous steps and maximum weight is given to the last step. <br>Momentum causes a fast convergence even in the regions with a gentle slope where gradient descent takes a lot of time to converge.</p><h4>Limitations</h4><p>Although momentum causes relatively fast convergence than Gradient Descent but it usually overshoots the target and causes oscillations around the minima.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*K3i65aLdM6wJEba0Fi1msA.gif" /><figcaption>A pendulum oscillating around minima because of momentum</figcaption></figure><h3>3. Nesterov Accelerated Gradient Descent (NAG)</h3><p>NAG is a simple improvement of Momentum Based Gradient Descent algorithm. As we already discussed, in momentum we are also considering the historic momentum vector in update rule with a gradient of error surface.</p><ul><li>Initialising momentum = 0</li><li>Update step where momentum term is also included<br>θ_t+1 = θ_t - {γ * momentum +η * ∇ϑ(θ_t)}</li><li>Add the current gradient into momentum<br>momentum = momentum + η * ∇ϑ(θ_t)</li></ul><p>Imagine at step = t, we are at point A on error surface. Gradients calculated at point A alone are taking us to point B and momentum alone is taking us to point C and both are taking us to point D.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*GTkJcpA1aOnocedMDmAQ3Q.png" /></figure><p>In this scenario, momentum alone is overshooting and the gradients calculated at point A is even adding into it causing even a larger overshoot.</p><p>NAG says that instead of calculating gradients at point A and adding them into momentum to land from point A to D, one should calculate gradients at point C (look ahead point) and add that into momentum to land from A to D’, and thus reducing the oscillations around minima as compared to momentum.</p><ul><li>Initialising momentum = 0</li><li>Calculating look ahead point<br>look_ahead = θ_t + momentum</li><li>NAG update with gradient calculated at look_ahead point<br>θ_t+1 = θ_t — {γ * momentum + η*∇ϑ(θ_look_ahead)}</li><li>Adding the gradient at look ahead into momentum<br>momentum = momentum + η*∇ϑ(θ_look_ahead)</li></ul><h3>4. Adagrad and RMSprop for Adaptive Learning Rate</h3><p>So far we have been looking for better algorithms to navigate faster through small gradients error surfaces. Another issue while training large neural networks is uneven sparsity in many features. Imagine a weight w1 associated with a feature x1 generating an activation h(w.x + b) and L2 loss is applied to train the network. The gradient of loss with respect to w1 will be very small if x1 is zero in most of the cases.</p><p>∂ϑ / ∂w1 = (h(w.x + b) — y) * h’(w.x + b) * x1 . (multiplied with x1)</p><p>This imbalance in sparsity of different features in deep neural networks causes a significant variation in the values of gradients calculated resulting in larger updates of weights of non sparse features as compared to that of sparse features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*UBpYckScxWW86EQ3NpmPYw.png" /><figcaption>NPTEL NOC-IITM, Deep Learning CS7015</figcaption></figure><p>In the above image, the feature associated with w is highly sparse, that resulted in the path of convergence shown in above image.</p><p>The key idea of Adagrad is to assign adaptive learning rate to different weights according to the sparsity of the feature in a way that the weights with larger gradients are assigned smaller learning rate as compared to weights with smaller gradients.</p><p>This can simply be achieved by accumulating the previous gradients and dividing the learning rate from it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*8vV5jVfsPTvw7tHYqpiNww.png" /></figure><p>where ϵ is a hyper parameter and usually set as 1e — 6<br>This is <strong>Adaptive Gradients Update Rule (Adagrad) </strong>and for some reasons, it will not work without the square root in denominator and there is no clear answer to that.<br>v is the update vector which is accumulating the gradients and this way we are adjusting the learning rate according to the sparsity of the feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*m05L8s62XqzSq799Sw0aqQ.png" /><figcaption>NPTEL NOC-IITM, Deep Learning CS7015</figcaption></figure><p>In the above image, the green curve represents the convergence path taken by Adagrad. It takes a shorter path than momentum or NAG and is faster in sparsity cases but may not converge. <br>The reason is if the value of v_t is significantly large for features which are not sparse, we are simply killing the gradients over many iterations.</p><h4>RMSprop</h4><p>We want to divide η with square root of accumulated gradients for each weight for adaptive learning rate but we do not want “over accumulation” of gradients for less sparse features to prevent killing of step size.</p><p>A possible solution can be to take an exponential weighted average of square of gradients (like in momentum) instead of simply adding that.</p><p>v_t = β * v_t + (∇ w_t)² and setting β between 0 to 1 (Default settings for β is 0.9 or 0.95)</p><p>It seems that the accumulation of a ‘component’ of weighted average of square of gradients instead of entire gradients can generate better results.</p><p>v_t = β * v_t + (1 — β) * (∇ w_t)² where by multiplying 1 — β, we are only accumulating a component of square of gradient as weighted average in v_t.</p><p>If we replace the accumulation step of Adagrad with above equation, that becomes RMSprop (Adaptation of learning rate through prorogation of root of weighted mean of square of gradients).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*Ju5EuLuHqpU7zl3bEvtVeQ.png" /></figure><p>Following image shows the convergence path taken by RMSprop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*-8Ic-2tHV5ZmYhuHOIE8KA.png" /><figcaption>NPTEL NOC-IITM, Deep Learning CS7015</figcaption></figure><h3>ADAM Optimiser — Combining everything to get the current state of the art</h3><ul><li>Venilla Gradient Descent<br>Repeat till convergence:<br> . . . θ = θ — η * ∇θ . . . . . (i)</li><li>Momentum based Gradient Descent<br>Initialise m = 0<br>Repeat till convergence:<br> . . . m = γ * m + η * ∇θ . . . . . (ii)<br> . . . θ = θ — m . . . . . (iv)</li><li>RMSprop Update Rule with adaptive learning<br>Initialise v = 0<br>Repeat till convergence:<br> . . . v = β * v + (1 — β) * (∇θ)² . . . . . (v)<br> . . . θ = θ — {η / √(v + ϵ)} * ∇θ . . . . . (vi)</li></ul><p>In ADAM, the key idea is to combine momentum and RMSprop, that is calculate m in eq (ii) but without η and replace m with ∇θ in eq (vi).<br>That makes it Adaptive Moments</p><p>Initialise m = 0, v = 0<br>Repeat till convergence:<br> . . . m = β1 * m + (1 — β1) * ∇θ . . . . . (vii)<br> . . . v = β2 * v + (1 — β2) * (∇θ)² . . . . . (viii)<br>. . . θ = θ — {η / √(v + ϵ)} * m . . . . . (ix)</p><p>This is the update rule for ADAM. One should notice that eq (vii) is exactly same as eq (ii). γ is renamed as β1 and we are only accumulating a fraction of gradient (1 — β1) by applying the same logic of RMSprop for adaptive learning rate.<br>General Settings for ADAM<strong> </strong>are β1 = 0.9, β2 = 0.999, η = 0.01, ϵ = 1e — 6</p><h4>Bias Correction in ADAM</h4><p>Above update rule of Adam works well with the Batch Gradient Descent. But in mini batch gradient and stochastic gradient descent, the above update rule is not correct. The reason behind that is, the gradients which are calculated in mini batch gradient descent and SGD are not the true gradients as we are not using the entire training data to compute them, because of which there is a random variations in gradients calculated in each mini batch and can point towards conflicting directions.</p><p>To solve this problem one should consider the gradients computed in mini batch or SGD as a random variable with some probability distribution and thus the moments (m and v) should be equal to the expectation of this probability distribution.</p><p>Condition to hold is 𝔼[m] = 𝔼[∇θ]<br>But,<br>m0 = 0<br>m1 = β1 * m0 + (1 — β1)* ∇θ_1= (1 — β1)∇θ_1</p><p>m2 = β1 * m1 + (1 — β1)* ∇θ_2 = <br>=&gt;β1{β1 * m0 + (1 — β1) * ∇θ_1}+ (1 — β1) * ∇θ_2<br>=&gt;β1(1 — β1) * ∇θ_1 + (1 — β1) * ∇θ_2</p><p>m3 = β1²(1 — β1) * ∇θ_1 + β1(1 — β1) * ∇θ_2 + (1 — β1) * ∇θ_3</p><p>In general,<br>mt = ∑ β1^(t-i) * {( 1 — β1) * ∇θ_i} . . . . . i starting from 1 to t<br>=&gt; mt = (1 — β1) * ∑ β1^(t-i) * ∇θ_i</p><p>=&gt;𝔼[mt] = 𝔼[ (1 — β1) * ∑ β1^(t-i) * ∇θ_i ]<br>=&gt; 𝔼[mt]= 𝔼[∇θ] {(1 — β1) * ∑ β1^(t-i)}<br>=&gt; 𝔼[mt]= 𝔼[∇θ] * (1 — β1^t)</p><p>But because we want 𝔼[m] to be equal to the to the expectation of the probability distribution of gradients (𝔼[∇θ]), we will have to correct m by dividing it by (1 — β1^t).</p><p>m’ = m / (1 — β1^t)<br>v’ = v / (1 — β2^t) . . . . (same logic apply to v)</p><p>Following is the ADAM update rules with bias correction</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*Xbhk-UPfW4yYgeTyK-d9Yg.png" /></figure><p>ADAM is the current state of the art for training neural network and is considered the first preference to train a Deep Neural Network for the last four years. Although other algorithms specially RMSprop with some weight decay strategy works as good as ADAM. One can consider the analogy of RMSprop as manual transmission in a car while ADAM is automatic transmission. <br>ADAM is widely used in all the computer vision work and has generated very good results on almost all the popular convolutional neural network architectures including very deep networks like Resent 50.</p><h3>New Findings</h3><p>There are some recent research papers which show that ADAM does not generate good results in some conditions. <br>Also, there have been some interesting new researches in this area.</p><ul><li>cyclic learning rates: <a href="https://arxiv.org/abs/1506.01186">https://arxiv.org/abs/1506.01186</a></li><li>ADAM W (decoupled weight decay) : <a href="https://arxiv.org/abs/1711.05101">https://arxiv.org/abs/1711.05101</a></li></ul><p>Recently in 2019, Google Brain Researchers have proposed layer wise adaptive moments optimiser for batch training (LAMB) algorithm to train their popular language model BERT. <br>Earlier it took 3 days to train BERT with ADAM W but LAMB did that in 76 minutes. Here is the paper <a href="https://arxiv.org/pdf/1904.00962.pdf">https://arxiv.org/pdf/1904.00962.pdf</a> for further reference. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad7a35b17b93" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-deeper-look-into-gradient-based-learning-for-neural-networks-ad7a35b17b93">A Deeper Look into Gradient Based Learning for Neural Networks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>