<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ashwath Salimath on Medium]]></title>
        <description><![CDATA[Stories by Ashwath Salimath on Medium]]></description>
        <link>https://medium.com/@ashwaths?source=rss-51153ad007e0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zgbYgdT1RiIxt2budAFk6w.jpeg</url>
            <title>Stories by Ashwath Salimath on Medium</title>
            <link>https://medium.com/@ashwaths?source=rss-51153ad007e0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:51:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashwaths/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to use the Learning Rate Finder in TensorFlow]]></title>
            <link>https://medium.com/octavian-ai/how-to-use-the-learning-rate-finder-in-tensorflow-126210de9489?source=rss-51153ad007e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/126210de9489</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[beginner]]></category>
            <dc:creator><![CDATA[Ashwath Salimath]]></dc:creator>
            <pubDate>Tue, 09 Apr 2019 23:58:03 GMT</pubDate>
            <atom:updated>2019-04-09T23:58:03.563Z</atom:updated>
            <content:encoded><![CDATA[<p>When working with neural networks, every data scientist must make an important choice: the learning rate. A bad learning rate can stop your network from successfully training. In this article, I’m going to teach you a simple, robust way to find a good learning rate for your neural networks.</p><p><strong>Why does the learning rate matter so much?</strong> Let’s look at a few different scenarios. First, consider a minimal learning rate, much smaller than the ideal learning rate. Each iteration of training, the network updates its weights using the below formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*fNx0JnCEHE87Gm9T.png" /></figure><p>Where θ is the set of the weights that a network learns which successfully fit the data. The ⍺ refers to the learning rate which controls the update of the network weights. J(θ) is called the loss function. A loss function helps to measure the correctness of the model in terms of its ability to estimate the relationship between X (input data) and y (output label).</p><p>As you can see in the above formula, if alpha is really small, then the network won’t update its weights by much at all. So in that scenario, the network takes a longer time to find the set of weights that successfully fit the data.</p><p>On the other hand, if alpha is really large, then the network updates its weights in large increments. So in that scenario, we overshoot the ideal weights, and worse, may continue to miss the optimum weights forever as our optimizer ricochets around.</p><p>Therefore, a very large learning rate results in loss divergence, i.e. loss increases very fast, and we never reach the minimum, whereas a very low learning rate results in loss plateau at first and takes a long time to reach the minimum. Between these two extremes, there exists a good learning rate. Thus, the learning rate is a crucial hyper-parameter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K0ltbXIgtNLEXsXN.png" /><figcaption>Impact of learning rate on training (<a href="https://www.jeremyjordan.me/nn-learning-rate/">Visualization</a> by Jeremy Jordon)</figcaption></figure><p>We can use the learning rate finder (LRFinder) to find a good learning rate.</p><h3>What is LRFinder</h3><p>The idea of the learning rate finder (LRFinder) comes from a <a href="https://arxiv.org/pdf/1506.01186.pdf">paper</a> called “Cyclical Learning Rates for Training Neural Networks” by Leslie Smith. The learning rate finder is a method to discover a good learning rate for most gradient based optimizers.</p><p>The LRFinder method can be applied on top of every variant of the <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent">stochastic gradient descent</a>¹, and most types of networks. However, some complex network setups, e.g. GANs, may not work with the LRFinder (at least not without some thought and research).</p><h3>How to use the LRFinder</h3><p>Given an untrained neural network, a loss function and training data, take the following steps:</p><ol><li>Start with a very small learning rate (e.g. 1e-10) and exponentially increase the learning rate with each training step. Here’s how to do this in the model function of your network:</li></ol><pre># decay_learning_rate = learning_rate *<br>                        decay_rate ^ (global_step / decay_steps)</pre><pre>learning_rate = tf.train.exponential_decay(1e-10, global_step=global_step, decay_steps=your_value, decay_rate=your_value)</pre><pre># So it can be seen in TensorBoard later<br>tf.summary.scalar(&#39;learning_rate&#39;, learning_rate)</pre><p>You need to manually set the <em>decay_steps</em> and <em>decay_rate</em> based on your network. To see this code in situ, have a look at our <a href="https://github.com/Octavian-ai/learning-rate-finder-tensorflow">example</a>.</p><p>2. Train your network as normal.</p><p>3. Record the training loss and continue until you see the training loss grow rapidly.</p><p>4. Use <a href="https://www.tensorflow.org/guide/summaries_and_tensorboard">TensorBoard</a> to visualize your TensorFlow training session. Analyse the loss to determine a good learning rate (I’ll explain how in the next section).</p><h4>How to determine a good learning rate</h4><p>You can identify a learning rate by looking at the TensorBoard graph of loss against training step<strong>. You want find the section where loss is decreasing fastest, and use the learning rate that was being used at that training step.</strong></p><p>Here are the TensorBoard graphs from a real world <a href="https://github.com/Octavian-ai/learning-rate-finder-tensorflow">example</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IpkirJUNqKK95a4LN0P3Wg.png" /><figcaption>Loss graph</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MnkareUmU3vNDHcmOgbIoQ.png" /></figure><p>We are able to identify the regions of loss plateau, reducing loss and loss explosion in the above loss graph as shown below. Loss plateau refers to the region where the learning rate is too small for the network to make any progress at minimizing its loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mo1cw6dPTMx5hw8v0ekJZA.png" /></figure><p>A good estimate of a learning rate is the one being used during the fastest rate of loss decrease. In the below graph, the steepest part lies roughly at training step 4.4k.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NalshL5B83BsZu4PQDvTRA.png" /><figcaption>The loss at training step 4.401k</figcaption></figure><p>Using the graph of learning rate against training step, we can find out what learning rate was being used at step 4.4k. It was 1.03e-5 . This is the learning rate we’ll use for our network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B_MRxV2f-1H8Wutjl0yM-Q.png" /></figure><p>This method is an approximation. It works fairly reliably in practice. Here are some of the assumptions that it relies upon:</p><ol><li>A good learning rate generally performs well across all training steps (e.g. if we encounter a learning rate that performs well at a particular training step, then we can use it for all of training).</li><li>We increased the learning rate fast enough that a good one was tried before the network found its parameters converged on a loss minima (if it got that far before the learning rate increased enough to cause divergence)</li></ol><h3>LRFinder vs. Grid search</h3><p>Grid search involves full training with different values of the learning rate. Then, we choose the learning rate which had the lowest final loss as a good learning rate. Training the network many times, to try each different learning rate takes a lot of time and resources. In the case of LRFinder, we only train the network once. Therefore LRFinder requires much less time and resources than grid search. However, grid search is robust and exhaustive in a way LRFinder cannot be. For more complex networks, or hard to optimize hyper-parameters, grid search may be a necessary tool.</p><p>To better understand the affect of optimizer and learning rate choice, check this <a href="https://medium.com/octavian-ai/which-optimizer-and-learning-rate-should-i-use-for-deep-learning-5acb418f9b2">article</a> by David Mack. The article shows that the right hyper-parameters are crucial to training success, yet can be hard to find.</p><h3>Additional resources</h3><ul><li><a href="https://pypi.org/project/keras-lr-finder/">keras</a> implementation</li><li><a href="https://youtu.be/JNxcznsrRb8?t=4m45s">fast.ai</a> library implementation</li></ul><h4>About Octavian</h4><p>Octavian’s mission is to develop systems with human-level reasoning capabilities. We believe that graph data and deep learning are key ingredients in making this possible. If you are interested in learning more about our research or contributing, <a href="https://octavian.ai/">check out our website</a> or <a href="/service/mailto:hello@octavian.ai">get in touch</a>.</p><h4>Acknowledgements</h4><p>I would like to thank David Mack for coming up with the idea of writing an article about LRFinder, also helping in reviewing and editing the article. I would like to thank Andrew Jefferson for his input on the learning rate finder method implementation in TensorFlow.</p><p>1. <a href="http://ruder.io/optimizing-gradient-descent/">An overview of gradient descent optimization algorithms</a> by Sebastian Ruder.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=126210de9489" width="1" height="1" alt=""><hr><p><a href="https://medium.com/octavian-ai/how-to-use-the-learning-rate-finder-in-tensorflow-126210de9489">How to use the Learning Rate Finder in TensorFlow</a> was originally published in <a href="https://medium.com/octavian-ai">Octavian</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attention is not quite all you need]]></title>
            <link>https://medium.com/octavian-ai/attention-is-not-quite-all-you-need-cb605ef3aff6?source=rss-51153ad007e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb605ef3aff6</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[attention]]></category>
            <dc:creator><![CDATA[Ashwath Salimath]]></dc:creator>
            <pubDate>Thu, 08 Nov 2018 20:52:06 GMT</pubDate>
            <atom:updated>2018-11-12T07:53:03.465Z</atom:updated>
            <content:encoded><![CDATA[<p>Whilst working on <a href="https://github.com/Octavian-ai/mac-graph">MacGraph</a>, a neural network that answers questions using knowledge graphs, I came across a problem: <strong>How to tell if something’s not present in a list.</strong></p><p>In this article, we’ll show you our solution, <strong>the focus signal</strong>, and show how it performs reliably across a range of different datasets and architectures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JSKBB7SK3eJuhnEc2SjV9w.png" /></figure><h4>Background</h4><p>In traditional programming, it’s easy to tell if something is not in a list: run a for loop over the items, and if you get to the end without finding it, then it’s not there. However, neural networks are not quite so simple.</p><p>Neural networks are composed of differentiable functions, so that they can be trained using <a href="https://en.wikipedia.org/wiki/Gradient_descent">gradient descent</a>. Equality operators, for loops and if conditions, the standard pieces of traditional programming used to solve this task, do not work that well in neural networks:</p><ul><li>An equality operator is essentially a <a href="https://en.wikipedia.org/wiki/Heaviside_step_function">step function</a>, which has <a href="https://stats.stackexchange.com/questions/271701/why-is-step-function-not-used-in-activation-functions-in-machine-learning">zero gradient almost everywhere</a> (and therefore breaks gradient descent back-propagation)</li><li><a href="https://www.tensorflow.org/api_docs/python/tf/cond">If conditions</a> generally use a boolean signal to switch branches, which again is often the output of a problematic step function</li><li><a href="https://www.tensorflow.org/api_docs/python/tf/while_loop">While loops</a> can be inefficient on GPUs, and sometimes not even useful as often neural network libraries require all data to have the same dimensions (e.g. TensorFlow executes on a <a href="https://www.tensorflow.org/guide/low_level_intro">statically defined tensor graph</a>).</li></ul><p>A popular neural-network technique for working with lists of items (e.g. translating sentences treating them as lists of words) is to apply “<a href="https://distill.pub/2016/augmented-rnns/">attention</a>”. This is a function where a learnt “query” of what the network is looking for is compared to each item in the list, and a weighted sum of the items similar to the query is output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*dPAnpcnxo9mqNq1yJhnNkQ.png" /></figure><p>In the above example,</p><ol><li>The query is dot-producted with each item in the list to compute a “score”. This is done in parallel for all items</li><li>The scores are then passed through <a href="https://en.wikipedia.org/wiki/Softmax_function">softmax</a> to transform them into a list that sums to 1.0. The scores can then be used as a probability distribution.</li><li>Finally, a weighted sum of the items is calculated, weighting each item by its score</li></ol><p>Attention has been very successful and forms the basis of <a href="https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html">current best-in-class translation models</a>. The mechanism has worked particularly well because:</p><ul><li>It’s fast and simple to implement</li><li>Compared to a <a href="https://en.wikipedia.org/wiki/Recurrent_neural_network">Recurring Neural Network </a>(e.g. <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">LSTM</a>) it is much more able to refer to past values in the input sequence. An LSTM has to learn to sequentially retain past values together in a single internal state across multiple RNN iterations, whereas attention can recall past sequence values at any point in a single forward pass.</li><li>Many tasks can be solved by rearranging and combining list elements to form a new list (e.g. attention models have been important components in many current best-in-class <a href="https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html">translation</a>, <a href="https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html">question-answering</a> and <a href="https://cs.stanford.edu/people/dorarad/mac/blog.html">reasoning</a> models)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kOM2OSpwn6VrsQVnP9Tqtw.png" /></figure><h4>Our problem</h4><p>Despite attention’s versatility and success, it has a deficiency that plagued our work on <a href="https://github.com/Octavian-ai/mac-graph">graph question answering</a>: <strong>attention does not tell us if an item is present in a list.</strong></p><p>This first happened when we attempted to answer questions like “Is there a station called London Bridge?” and “Is Trafalgar Square station adjacent to Waterloo station?”. Our tables of graph nodes and edges have all this information for attention to extract, but attention itself was failing to successfully determine item existence.</p><p>This happens because attention returns a weighted sum of the list. If the query matches (e.g. scores highly) against one item in the list, the output will be almost exactly that value. If the query did not match any items, then a sum of all the items in the list is returned. Based on attention’s output, the rest of the network cannot easily differentiate between those two situations.</p><h4>Our solution</h4><p>The simple solution we propose is output a scalar aggregate of the raw item-query scores (e.g. before using softmax). This signal will be low if no items are similar to the query, and high if many items are.</p><p>In practice this has been very effective (indeed, the only robust solution of the many we’ve tested) at solving existence questions. From now on we will refer to this signal as “focus”. Here is an illustration of the attention network we showed earlier, with the focus signal added:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BTbEFzt4Zh14RFWMK8LTkw.png" /><figcaption>Attention with focus signal, using a summation for aggregation of raw scores</figcaption></figure><h3>Experimental setup</h3><p>Finding that the focus signal was essential for <a href="https://github.com/Octavian-ai/mac-graph">MacGraph</a> succeeding on some tasks, we tested this concept on a range of datasets and model architectures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wLdx4J5VjGgz9VcgPuciiQ.png" /><figcaption>The experiment’s neural network</figcaption></figure><p>We constructed a network that takes a list of items and a desired item (the “query”), and outputs whether that item was in the list. The network takes the inputs, performs attention (optionally with our focus signal), transforms the outputs through a couple of residual layers, then outputs a binary distribution¹ of whether the item was found.</p><p>The network loss is calculated using <a href="https://www.tensorflow.org/api_docs/python/tf/nn/sparse_softmax_cross_entropy_with_logits">softmax cross entropy</a> and trained using the <a href="https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer">Adam optimizer</a>. Each network variation has its ideal learning rate determined prior to training by using the <a href="https://towardsdatascience.com/estimating-optimal-learning-rate-for-a-deep-neural-network-ce32f2556ce0">learning rate finder</a>.</p><p>In our experiments, we vary the network by:</p><ul><li>Including / removing the focus signal from the attention step (“Use focus” below)</li><li>Varying the attention score function</li><li>Varying the focus signal aggregation function</li><li>Varying the first residual transformation layer’s activation function (“Output activation” below)</li></ul><p>We then apply these network variations to a few different datasets, listed in the next section.</p><p>Here are all of the network configurations we tested:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*HiVu2-Xwn3BUbgRbTFVgxQ.png" /></figure><p>The focus signal was found to be essential to the network reliably achieving &gt;99% accuracy across the range of datasets and network configurations.</p><p>Our experiments’ code is <a href="https://github.com/Octavian-ai/attention-focus">open-source in our GitHub</a>.</p><h4>Datasets</h4><p>Each dataset is a set of examples, each example contains the input features <em>List of items</em>,<em> Desired item</em> and the ground truth output, <em>Item is in list. </em>An item is an N dimensional vector of floating point numbers.</p><p>Each dataset was constructed so 100% accuracy is possible.</p><p>We tested on three different datasets, each with a different source of items:</p><ul><li>Orthogonal one-hot vectors of length 12</li><li>Many-hot vectors (e.g. random strings of 1.0s and 0.0s) of length 12</li><li><a href="https://en.wikipedia.org/wiki/Word2vec">Word2vec</a> vectors of length 300</li></ul><p>Each dataset has balanced answer classes (i.e. an equal number of True and False answers)</p><p>The one-hot and many-hot vectors were randomly generated. Each training example has a list of two items. Each dataset has 5,000 training items. Here’s one of the training examples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F9b1I1FhMBaeqjciOBRiZA.png" /><figcaption>A training example in the orthogonal one-hot dataset. (Vectors shown as length 3 for visual simplicity)</figcaption></figure><p>For our word2vec dataset, we used pre-calculated word2vec embeddings trained on Google News, which can be downloaded <a href="https://code.google.com/archive/p/word2vec/">here</a>.</p><p>The word2vec dataset has item lists of up to length 77, each representing a shuffled, randomly chosen sentence from a 300-line <a href="https://en.wikipedia.org/wiki/Croatian_War_of_Independence">wikipedia article</a>. Each desired-item is a randomly chosen word2vec vector from the words within the article.</p><h3>Results</h3><p>The focus signal was found to be essential for reliably determining item existence:</p><ul><li>Without the focus signal, no network configuration achieved &gt;56% accuracy for all datasets</li><li>For orthogonal dataset vectors, no network achieved &gt;80% accuracy without the focus signal</li><li>Networks with the focus signal achieved &gt;98% accuracy in all datasets and configurations, networks without the focus signal only achieved &gt;98% in 4 out of 9 configurations</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Py-qOdY5tCZhMRM5C0YayQ.png" /></figure><p>It must be noted that the network without focus signal did achieve &gt; 98% accuracy in some configurations and datasets.</p><h3>Conclusion</h3><p>In this article we’ve introduced a new concept for attention networks, the “focus signal”. We’ve shown that it’s a robust mechanism for detecting the existence of items in a list, which is an important operation for machine reasoning and question answering. The focus signal successfully detected item existence in all datasets and network configurations tested.</p><p>We hope that this work helps other teams in tackling the challenge of item-existence and adds more clarity to considering what architecture to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ecOXP5BYbX8H8AcSnDFmw.png" /></figure><h4>Octavian’s research</h4><p>Octavian’s mission is to develop systems with human-level reasoning capabilities. We believe that graph data and deep learning are key ingredients to making this possible. If you interested in learning more about our research or contributing, <a href="https://octavian.ai/">check out our website</a> or <a href="/service/mailto:hello@octavian.ai">get in touch</a>.</p><h4>Acknowledgements</h4><p>I would like to thank David Mack for the initial idea and helping with much of the editing of this article. I would also like to thank Andrew Jefferson for writing much of the code and guiding me through this write-up. Finally, I would like to thank Octavian for giving me an opportunity to work on this project.</p><h3>Footnotes</h3><ol><li>The output transformation layers were copied from MacGraph and could possibly be simplified. The sigmoid (<strong>σ</strong>) layer is definitely unnecessary in this model as it limits the range of the inputs to softmax and therefore prevents the model from being able to train to zero loss (although it can still achieve 100% accuracy since the output is argmax’ed to a one-hot-vector and compared with the label). We don’t believe this complexity invalidates any of the results, but felt compelled to call it out.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkt8IqIrY-9pY3WFpTma7A.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb605ef3aff6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/octavian-ai/attention-is-not-quite-all-you-need-cb605ef3aff6">Attention is not quite all you need</a> was originally published in <a href="https://medium.com/octavian-ai">Octavian</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>