<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pooja Aggarwal on Medium]]></title>
        <description><![CDATA[Stories by Pooja Aggarwal on Medium]]></description>
        <link>https://medium.com/@poojaaggarwal?source=rss-a57a661d1fc2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*aqYzO6Sp4FYTKxxj.</url>
            <title>Stories by Pooja Aggarwal on Medium</title>
            <link>https://medium.com/@poojaaggarwal?source=rss-a57a661d1fc2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:13:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@poojaaggarwal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Adaptive Gradient Clipping]]></title>
            <link>https://medium.com/analytics-vidhya/adaptive-gradient-clipping-f8fe04613009?source=rss-a57a661d1fc2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8fe04613009</guid>
            <category><![CDATA[gradient-clipping]]></category>
            <category><![CDATA[recurrent-neural-network]]></category>
            <category><![CDATA[exploding-gradient]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[adaptive-learning]]></category>
            <dc:creator><![CDATA[Pooja Aggarwal]]></dc:creator>
            <pubDate>Mon, 13 May 2024 23:05:30 GMT</pubDate>
            <atom:updated>2024-05-13T23:20:47.313Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Dynamically tackle Exploding Gradients!</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DBIJtbQtBHRVptavzxG9Ng.jpeg" /></figure><p>Adaptive gradient clipping(AGC) is a technique used in deep learning to prevent the gradients from becoming too large during training. It dynamically adjusts the scaling of gradients based on their magnitude relative to the model’s parameters, leading to a more stable optimization process and improved convergence of models.</p><h4><strong>How it works ?</strong></h4><ol><li><strong>Gradient Calculation:</strong> During training, the model computes gradients for each parameter with respect to the loss function. These gradients indicate the direction and magnitude of the change needed to minimize the loss.</li><li><strong>Norm Calculation:</strong> AGC calculates the norm (magnitude) of both the gradients and the parameters themselves. The norm of a vector is essentially its length in a mathematical sense. For the gradients, this norm represents the overall magnitude of the changes needed for the parameters to reduce the loss. Similarly, the parameter norm represents the overall magnitude of the parameters.</li><li><strong>Clipping Condition:</strong> AGC compares the ratio of the gradient norm to the parameter norm with a predefined threshold. This ratio indicates how much a single gradient step will change the parameter relative to its current value. If this ratio exceeds the threshold, it suggests that the gradients are too large relative to the parameter values, which could lead to instability during optimization.</li><li><strong>Gradient Scaling:</strong> If the ratio exceeds the threshold, AGC scales down the gradients to prevent them from becoming too large. This scaling factor is adaptive, meaning it changes based on the current gradient and parameter values. By scaling down the gradients, AGC ensures that the updates to the parameters remain within a reasonable range, preventing them from growing too large and causing instability in the optimization process.</li></ol><h4><strong>How is it different from Traditional Gradient Clipping ?</strong></h4><ul><li><strong>Dynamic Threshold:</strong> AGC does not rely on a fixed threshold. Instead, it computes a dynamic threshold based on the ratio of the gradient norm to the parameter norm.</li><li><strong>Adaptive Scaling:</strong> The scaling factor for each gradient is determined individually based on this ratio. Gradients with larger ratios (relative to their parameter norms) are scaled down more aggressively than those with smaller ratios.</li></ul><h4>Conclusion</h4><p>In essence, the key insight behind AGC is that <strong>a small change in a small number can be more significant than the same change in a large number</strong>.</p><p>Therefore, adapting the clipping threshold relative to the magnitude of both gradients and parameters allows for more effective control over the optimization process, ensuring stability while allowing for efficient learning across different scales within the model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8fe04613009" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/adaptive-gradient-clipping-f8fe04613009">Adaptive Gradient Clipping</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing the right parameters for pre-training BERT using TPU]]></title>
            <link>https://medium.com/analytics-vidhya/choosing-the-right-parameters-for-pre-training-bert-using-tpu-4584a598ca50?source=rss-a57a661d1fc2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4584a598ca50</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Pooja Aggarwal]]></dc:creator>
            <pubDate>Thu, 14 Jan 2021 06:30:59 GMT</pubDate>
            <atom:updated>2021-01-14T15:50:20.814Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0HbTDvL0CrOkJRacTu-a0Q.png" /><figcaption>source: <a href="https://346034.smushcdn.com/236617/wp-content/uploads/2017/03/Are-You-Giving-Your-Customers-Too-Many-Options-e1490278059717.jpg?lossy=0&amp;strip=1&amp;webp=1">Google Search</a></figcaption></figure><p>Pre-training a BERT model is not easy and many articles out there give a great high-level overview on what BERT is and the amazing things it can do, or go into depth about a really small implementation detail. This leaves aspiring Data Scientists, like me a while ago, often looking at Notebooks out there, thinking “It looks great and works, but why did the author choose this number of batch size or this sequence length instead of another?</p><p>In this article, I want to give some intuition on how to make some of the decisions to find the right configuration to pre-train the BERT model.</p><p>The full article with code and outputs can be found on <a href="https://github.com/google-research/bert/blob/master/README.md#pre-training-with-bert">Github </a>as a Notebook.</p><p>Let us begin looking each input parameter.</p><h4><strong>1. Do lower Case</strong></h4><p>You can pre-train “uncased” or “cased” version of the BERT model by setting <em>do_lower_case</em> to true or false.</p><blockquote>Uncased means that the text has been lowercased before tokenization, e.g., John Smith becomes john smith. The Uncased model also strips out any accent markers. Cased means that the true case and accent markers are preserved.</blockquote><p>You can decide its value based on the type of task and the language of the input data.</p><p><strong><em>Type of task</em></strong></p><p>For tasks like,</p><ol><li>Named Entity Recognition</li><li>Part-of-Speech tagging</li><li>Sentiment Detection</li></ol><p>Case information is an important signal. Imagine the word “us”. It could be pronoun representing “we” on the sentence or the country “USA” .</p><p>For the other tasks, we should use “uncased” model only. It does not have unnecessary duplicates e.g., both dhoni and Dhoni will be present in “cased” vocab but not in it and so, able to represent language better with same size vocab.</p><p><strong><em>Input Language</em></strong></p><p>Stripping accent from Devanagari and other non-latin scripts changes the meaning of word and hence, the meaning of sentence.</p><p>Example: कुरान (kuran) becomes करान (kran, ु is removed)</p><p>It is therefore recommended to use “cased” model for non-Latin languages.</p><h4><strong>2. Maximum Sequence length</strong></h4><p>The <em>max_seq_length </em>specifies the maximum number of tokens of the input. The input tokens are truncated or padded based on its value. Its is set in power of two e.g., 64, 128, 512.</p><p>You can decide its value based on the end task. Are you looking to do prediction on articles, sentences or phrases ?</p><p>One should keep larger value for the articles and smaller value for the phrases.</p><h4>3. Train Batch Size</h4><p>The train batch size is a number of samples processed before the model is updated. Larger batch size are preferred to get stable enough estimate of what the gradient of the full dataset would be. The batch size is always set in power of two e.g., 512, 1024, 2048.</p><p>Final input shape looks like (batch_size, max_seq_length, embedding_size). The embedding size is generally 768 for BERT based language models and sequence length is decided based on the end task as discussed above.</p><p>Our motive is to utilize our resource fully. So, you should set train batch size to maximum value based on the available ram.</p><h4>4. Maximum Prediction Per Sequence</h4><p>The max_predictions_per_seq is the maximum number of masked LM predictions per sequence. This is computed as follow.</p><pre>max_predictions_per_seq= (max_seq_length* masked_lm_prob)</pre><pre>For instance<br>max_seq_length= 512 and max_seq_length = 0.15<br>max_predictions_per_seq = 77</pre><p>where masked_lm_prob is the percentage of words that are replaced with a [MASK] token in each sequence.</p><h4><strong>5. Number of Training Steps</strong></h4><p>Training steps are number of times we pass the batch to train the model. It is computed as follow</p><pre>steps = (epoch * examples)/batch size<br>For instance<br>epoch = 100, examples = 1000 and batch_size = 1000<br>steps = 100</pre><p>where epoch is the number of times you want to pass the complete dataset. You should keep its value more than 1.</p><p>In real world, we usually keep a very high value and stop manually as soon as the loss bottoms-out.</p><h4><strong>6. Learning Rate</strong></h4><blockquote><em>learning rate, a positive scalar determining the size of the step.</em></blockquote><p>we should not use a learning rate that is too large or too small. When the learning rate is too large, gradient descent can inadvertently increase rather than decrease the training error. When the learning rate is too small, training is not only slower, but may become permanently stuck with a high training error.</p><h4>7. Number of Warm-up Steps</h4><p>If your data set is highly differentiated, you can suffer from a sort of “early over-fitting”. If your shuffled data happens to include a cluster of related, strongly-featured observations, your model’s initial training can skew badly toward those features — or worse, toward incidental features that aren’t truly related to the topic at all.</p><p>Warm-up is a way to reduce the primacy effect of the early training examples. Without it, you may need to run a few extra epochs to get the convergence desired, as the model un-trains those early superstitions.</p><p>The learning rate is increased linearly over the warm-up period. If the target learning rate is p and the warm-up period is n, then the first batch iteration uses 1*p/n for its learning rate; the second uses 2*p/n, and so on: iteration i uses i*p/n, until we hit the nominal rate at iteration n.</p><p>Warm-up period is usually set to 1% of total training period.</p><h4>Conclusion</h4><p>In this articles, I discussed input parameters to pre-train the BERT model and explained the factors that help to decide their configuration.</p><p><em>Cheers!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4584a598ca50" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/choosing-the-right-parameters-for-pre-training-bert-using-tpu-4584a598ca50">Choosing the right parameters for pre-training BERT using TPU</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is BERT?]]></title>
            <link>https://medium.com/analytics-vidhya/what-is-bert-e758ee2b2ab5?source=rss-a57a661d1fc2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e758ee2b2ab5</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Pooja Aggarwal]]></dc:creator>
            <pubDate>Tue, 12 Jan 2021 05:13:20 GMT</pubDate>
            <atom:updated>2021-01-12T16:08:00.771Z</atom:updated>
            <content:encoded><![CDATA[<p><em>BERT stands for Bidirectional Encoder Representations from Transformers. Each word here has a meaning to it and we will understand by the end of this article</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/0*8lXh39jAEbhhIguf.png" /></figure><h3>Introduction</h3><p><em>BERT is a general purpose framework to solve varieties of NLP tasks, including Question Answering (SQuAD v1.1), Classification (MNLI), and others. It learns language from large pure plain text by understanding text cohesion and is then educated further by fine-tuning on smaller labelled dataset, specific to natural language tasks.</em></p><h3>Attention Mechanism: self attention</h3><p><em>Same word can have different meanings in different contexts. For example:</em></p><p><em>I arrived at the bank after crossing the _____</em></p><p><em>we cannot know whether this “bank” refers to a river or a financial institute without having knowledge of the end word.</em></p><p><em>BERT learns word context based on all its surrounding words, just like humans! This is achieved using self attention. Let us understand it</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/50/0*PTjpXz5C_1OQx1ul" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/0*A1s9tG47gW-nx5LQ" /></figure><p><a href="https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&amp;utm_medium=demystifying-bert-groundbreaking-nlp-framework">Source</a></p><p><em>What does “it” in this sentence refer to? Is it referring to the street or to the animal? It’s a simple question to a human, but not as simple to an algorithm.</em></p><p><em>Self-attention solves this problem. It allows the model to look at the other words in the sentence to get a better understanding of “it”.</em></p><p><em>Associating each word in a sentence with every other word is called self-attention. Let’s see how we can calculate self-attention by an example.</em></p><p><em>Consider this phrase — “Action gets results”.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/60/0*Mt7GYdDRfZ6SoXiT" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*mmp-Gq5fiRHMwM3r" /></figure><p><a href="https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&amp;utm_medium=demystifying-bert-groundbreaking-nlp-framework">Source</a></p><ol><li><em>We created three vectors: Query Vector, Key Vector, Value Vector for the given phrase. These vectors are trained and updated during the training process. We’ll know more about their roles once we are done with this section</em></li><li><em>The score for the first word “Action” is calculated by taking the dot product of the Query vector (q1) with the keys vectors (k1, k2, k3) of all the words.</em></li><li><em>Then, these scores are divided by 8 which is the square root of the dimension of the key vector</em></li><li><em>Next, these scores are normalized using the softmax activation function.</em></li><li><em>These normalized scores are then multiplied by the value vectors (v1, v2, v3) and sum up the resultant vectors to arrive at the final vector (z1). It is then passed on to the feed-forward network as input:</em></li></ol><p><em>So, z1 is the self-attention vector for the first word of the input sequence “Action gets results”.</em></p><p><strong><em>That’s it — attention is just a fancy name for weighted average!</em></strong></p><h3>Transformer Encoder</h3><p><em>Transformer encoder is a type of neural network that accepts a sequence as input (e.g. a sequence of words) and produces some output (e.g. sentiment prediction). Unlike traditional recurrent networks such as LSTMs, which process each sequence element in turn, the encoder processes all elements simultaneously by forming direct connections between individual elements through a process known as Self Attention. Not only does this enable greater parallelization, but it also results in higher accuracy across a range of tasks.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/60/0*NpXz39ASt4CjHNM1" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/0*4Ven0JVa7WnOSmLJ.png" /></figure><p><a href="http://jalammar.github.io/illustrated-transformer/">Source</a></p><p><em>Self attention does not take ordering of words into account. To address this, the transformer encoder adds a vector to each input embedding that determines the position of each word, or the distance between different words in the sequence.</em></p><h3>BERT Architecture</h3><p><em>The BERT architecture is built on top of the Transformer Encoder.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/60/0*vnHNbf8JonH0XkCV" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/0*NC6gUdw-4pykKWbg" /></figure><p><a href="https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270">Source</a></p><p><em>Before feeding word sequences into BERT, 15% of the words in each sequence are replaced with a [MASK] token. The model then attempts to predict the original value of the masked words, based on the context provided by the other, non-masked, words in the sequence. In technical terms, the prediction of the output words requires:</em></p><ol><li><em>Adding a classification layer on top of the encoder output.</em></li><li><em>Multiplying the output vectors by the embedding matrix, transforming them into the vocabulary dimension.</em></li><li><em>Calculating the probability of each word in the vocabulary with softmax.</em></li></ol><h3>Conclusion</h3><p><em>BERT generates embeddings by taking information from both the left and the right side of a token’s context using the transformer model and hence called Bidirectional Encoder Representations from Transformers.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e758ee2b2ab5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/what-is-bert-e758ee2b2ab5">What is BERT?</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>