<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shadman Sobhan on Medium]]></title>
        <description><![CDATA[Stories by Shadman Sobhan on Medium]]></description>
        <link>https://medium.com/@shadmansobhan114?source=rss-33889b8e0d1a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*7TEGMEzSLGgGSlyc</url>
            <title>Stories by Shadman Sobhan on Medium</title>
            <link>https://medium.com/@shadmansobhan114?source=rss-33889b8e0d1a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 19:07:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shadmansobhan114/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Attention- please- Part 3 (Transformer Architecture)]]></title>
            <link>https://medium.com/@shadmansobhan114/attention-please-part-3-transformer-architecture-225e802cedac?source=rss-33889b8e0d1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/225e802cedac</guid>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[encoder-decoder]]></category>
            <category><![CDATA[transformer-architecture]]></category>
            <dc:creator><![CDATA[Shadman Sobhan]]></dc:creator>
            <pubDate>Thu, 01 May 2025 17:56:46 GMT</pubDate>
            <atom:updated>2025-05-01T17:56:46.797Z</atom:updated>
            <content:encoded><![CDATA[<h4>In-depth Discussion on “Transformer” Architecture</h4><p>Transformer Architecture has revolutionized modern AI &amp; Deep Learning. At present, we use large language models (LLMs) almost every day, and this is possible only because of the Transformer Architecture. Transformer architecture employs the Attention Mechanism. In the previous two parts of “Attention- please” we discussed the Attention mechanism in detail. In the final part, we will talk about Transformer Architecture, the most important thing to know in modern AI.</p><h4><strong>Background:</strong></h4><p>The Attention Mechanism was first proposed for machine translation. Typical Encoder-Decoder based RNN networks were used for machine translation. The encoder part generated a vector based on the given input, and the decoder generated the translated output. This structure had severe limitations, like poor quality translation, difficulty in translating long sentences, and difficulty in identifying keywords.</p><p>These problems could be solved if the decoder could put different importance/ attention on different words rather than processing the whole sentence at the same time. This is done by the “<a href="https://medium.com/@shadmansobhan114/attention-please-part-1-9ec0aa5759fa">Attention Mechanism</a>” where each RNN unit’s context vector is generated by putting different importance on different outputs of the Encoder.</p><p>RNN processes sequentially. So, if a RNN has, say, 20 units/time steps, then the final unit will have to wait for 19 time steps. Self-Attention Mechanism, taking motivation from CNN Architecture solved this problem. In the “<a href="https://medium.com/@shadmansobhan114/attention-please-part-2-self-attention-multi-head-attention-8a530978ab1c">Self-Attention</a>” mechanism, a token(here, a word) looks at itself and other words to understand the context.</p><p>When we use only one attention head, the model looks at the entire sentence, but it can only learn one way of focusing. Instead of using just one attention, “<a href="https://medium.com/@shadmansobhan114/attention-please-part-2-self-attention-multi-head-attention-8a530978ab1c">Multi-Head Attention</a>” uses several, and it can capture diverse relations between tokens/words. So, multi-head attention can capture much more diverse contextual information than single-headed self-attention.</p><h4>Why Transformer Architecture is Used Everywhere:</h4><ol><li><strong>Parallelization: </strong>Unlike RNNs, which process data sequentially, Transformers process entire sequences in parallel using self-attention. This makes training faster and more hardware-efficient, especially on GPUs/TPUs.</li><li><strong>Scalability: </strong>Transformers scale well with data and model size. Increasing model parameters (e.g., GPT-3, GPT-4) leads to better performance.</li><li><strong>Long-Range Dependencies: </strong>Self-attention lets Transformers directly relate any two positions in the input sequence, regardless of their distance. RNNs struggle with this due to vanishing gradients.</li><li><strong>Unified Framework for Multiple Modalities: </strong>The<strong> </strong>Transformer architecture works well not only for Texts (BERT, GPT), but also for Vision (e.g., ViT), Audio (e.g., Whisper), and Multimodal tasks (e.g., CLIP, Flamingo).</li><li><strong>Transfer Learning and Pretraining: </strong>Transformers are well-suited for pretraining on large corpora and fine-tuning for specific tasks. This is the foundation of modern NLP systems (BERT, T5, GPT, etc.).</li><li><strong>State-of-the-Art Performance: </strong>Across a wide range of benchmarks — translation, question answering, summarization, image classification — Transformers consistently outperform previous methods.</li></ol><h4>Transformer Architecture:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1pasy9RzKaPSmo6y5s2RWg.png" /><figcaption>Transformer Architecture</figcaption></figure><p>Each input token/words are converted to vector embeddings. Transformers operate on the entire sequence in parallel, treating inputs as a set of vectors rather than a sequence. So there is <strong>no built-in mechanism to understand token position or sequence</strong>. Thus, we inject positional information explicitly by adding position-specific vectors to each token’s embedding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*6-W0ZMu8jpQ3A5wAT88eMA.png" /></figure><p>Positional embeddings are added to input embeddings rather than being concatenated.<br>- Addition preserves the shape<br>- It acts like a bias, injecting positional structure into each embedding dimension<br>- Simpler and more efficient than concatenation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*rTIId30AwtJjGVAKRJi5Kw.png" /><figcaption>Input Processing before Encoder</figcaption></figure><h4>Encoder:</h4><p>Now, the converted embeddings go through the “Multi-Head Attention” mechanism. The output MultiHead(Q, K, V) and vector embeddings X are added and normalized. You can read about Multi-Head Attention from <a href="https://medium.com/@shadmansobhan114/attention-please-part-2-self-attention-multi-head-attention-8a530978ab1c">here</a>.<br> <strong>Z1 = LayerNorm ( X + MultiHead (Q, K, V) )</strong></p><p><strong>LayerNorm/ Normalization Layer: </strong>This layer stabilizes and accelerates training by normalizing the input. <br>In deep networks, the distribution of inputs to each layer can shift during training — this is known as <strong>internal covariate shift</strong>. Normalization helps mitigate this by keeping inputs to layers more stable.</p><p>Unlike BatchNorm (which normalizes across the batch), LayerNorm normalizes across the features (dimensions) of each example, making it ideal for variable-length sequences and attention mechanisms. Why residual connection is used here is beyond the scope of this article. If you are interested, you can read from <a href="https://medium.com/@shadmansobhan114/understanding-residual-network-resnet-skip-connection-c444a1accfe9">here.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*V5m9kwyQZyH8pUmqgKLw9w.png" /></figure><p><strong>Feed-Forward Network:</strong> After passing the normalization layer, the vectors enter the Feed-Forward Network. The Feed Forward Network (<strong>FFN</strong>) in a Transformer is a fully connected layer applied independently to each position in the sequence. It provides the model with non-linearity and depth beyond attention mechanisms. The equations of the layer are</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*9x-iLyPdlw9z7LYJ0NVSrA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*joMA7cc24pIHryCB7gPRJQ.png" /><figcaption>Feed-Forward Network</figcaption></figure><p>First hidden layer of FFN performs a linear transformation. Then, applying ReLU activation, another linear transformation is done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gP6wmbaXAA_jPSCMcJjIjw.png" /><figcaption>Transformer Encoder</figcaption></figure><p>This is the overall <strong>Transformer Encoder structure.</strong></p><p><strong>This Encoder is repeated N times. Generally N=6.</strong></p><h4>Decoder:</h4><p>Suppose the generated sentence is “<strong>I will go</strong>”. So, let’s see how this sentence is generated.</p><p>The decoder part takes help from previously generated words and the encoder output to generate a new word/token. Similar to input word processing, the output words also generate input and positional embeddings, which are passed to the decoder.</p><p>At time step <strong>t = 1</strong>, the model has no previously generated words. The decoder always starts with a special token, often called: &lt;S&gt; or &lt;SOS&gt; ( Start or Start of Sequence). Then it passes through <strong>Masked Multi-Head Attention </strong>and Normalization layer as before.</p><p><strong>Masked Multi-Head Attention</strong> is a special type of <strong>Multi-Head Attention</strong>. In standard Multi-Head Attention, each word in a sequence can attend to all other words — past, present, and future. <br>On the other hand, Masked Multi-Head Attention restricts this by allowing each word to attend only to previous and current positions, not future ones. Suppose there are 8 words. For calculating Multi-Head Attention for the 4th word, we will use all 8 words. But to find out Masked Multi-Head Attention for the 4th word, we will use the 1st to the 3rd word. That’s the difference between Masked Multi-Head Attention and Multi-Head Attention.</p><p>Queries come from <strong>the Normalization layer of the decoder</strong> &amp; Keys, Values come from <strong>the output of the Encoder</strong>. This is called <strong>Cross-Attention</strong>. It allows the decoder to generate a sentence according to the extracted information of the encoder and the already generated sequence. It is called <strong>“cross”</strong> because it operates <strong>across two different sequences.</strong></p><p>The remainder is mostly the same as the encoder. The final output passes through a linear layer as in an FFN. Finally, we use<strong> softmax</strong> to get a probability distribution over the next token. The token with the highest probability is the next word. In this case, we are assuming “I” is the word with the highest probability. So “I” gets generated. Now, “I” will go to the generated outputs.</p><p>The next word will be generated using both &lt;S&gt; and &lt;I&gt;. In this way, the decoder will continue generating words. After the sequence is generated, the decoder will generate &lt;EOS&gt; or End of Sequence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hWqA4yOC05gyvkFN-nB8VA.png" /><figcaption>How decoder generates</figcaption></figure><p>This is the overall working mechanism of the Transformer Architecture. I hope you had fun reading it and found it interesting. <br>Thank youuuu. :D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=225e802cedac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attention- please- Part 2 (Self-Attention & Multi-Head Attention)]]></title>
            <link>https://medium.com/@shadmansobhan114/attention-please-part-2-self-attention-multi-head-attention-8a530978ab1c?source=rss-33889b8e0d1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a530978ab1c</guid>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[multi-head-attention]]></category>
            <category><![CDATA[self-attention]]></category>
            <dc:creator><![CDATA[Shadman Sobhan]]></dc:creator>
            <pubDate>Wed, 30 Apr 2025 13:19:37 GMT</pubDate>
            <atom:updated>2025-04-30T13:19:57.695Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Neural Networks Focus &amp; On Depth Discussion on “Attention” Mechanism</h4><p>Various State-of-the-Art models, LLMs, and Transformers use a mechanism called “attention” through which they understand where to focus/ which part is important. In the previous <a href="https://medium.com/@shadmansobhan114/attention-please-part-1-9ec0aa5759fa">part</a>, we discussed basic Seq2Seq Attention. In this article, we will explore two different types of Attention mechanisms: Self-Attention and Multi-Head Attention.</p><blockquote>Let’s first review the basic ideas we talked about earlier.<br>The Attention Mechanism was first proposed for machine translation. Typical Encoder-Decoder based RNN networks were used for machine translation. The encoder part generated a vector based on the given input, and the decoder generated the translated output. This structure had severe limitations, like poor quality translation, difficulty in translating long sentences, and difficulty in identifying keywords. <br>These problems could be solved if the decoder could put different importance/ attention on different words rather than processing the whole sentence at the same time. This is done by the “Attention Mechanism,” where each RNN unit’s context vector is generated by putting different importance on different outputs of the Encoder.</blockquote><p>If we talk about RNN, it processes sequentially. That means if it has 3 blocks, then block 1 will generate an output that will work as the input of block 2, and so on. So, if a RNN has, say, 20 units/time steps, then the final unit will have to wait for 19 time steps or in other words, wait until the previous 19 units generate an output. This leads to slow training. For GRU/LSTM-based RNNs, which solve vanishing gradient, as they are complex, they require more time.</p><p>On the other hand, CNN processes input in parallel using kernels/filters. “Self-Attention Mechanism” takes motivation from CNN.</p><h4>Self-Attention:</h4><p>In the “Self-Attention” mechanism, a token(here, a word) looks at itself and other words <strong>to understand the context</strong>. That’s why it is called Self-Attention.</p><p>“The bank is closed”- In this sentence, the meaning of “bank” is a financial institution. However, bank could have also meant riverbank. For translation/ any other task, it is very important to understand contextual meaning. For contextual understanding, the other words of a sentence also play an important role. And that’s why the self-attention mechanism is very important.</p><p>The related formulas of the mechanism are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*XCS19hb58OnJAqp6hsjT2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*p2gtFf5jQCahhZFLGrYfjw.png" /></figure><p>Let’s understand how self-attention is calculated:<br><strong>(1)</strong> At first, the input words/tokens are converted to their vector representation. If there are 5 words/tokens (a token is the basic unit of text), we will get 5 vectors: x1,x2,…., x5. The vectors will be contained in the matrix <strong><em>X</em></strong>.</p><p><strong>(2)</strong> Now, Query(Q), Key(K), Value(V) are calculated using equation 1. Here, the three weights are learnable parameters.<br>In self-attention, each token in the sequence, <br>- creates a <strong>query vector</strong> → What it’s looking for (context)<br>- compares it with <strong>key vectors</strong> of all tokens → Which tokens might be relevant?<br>- uses the scores to compute a weighted sum of <strong>value vectors</strong> → What information to gather from each token.<br>Suppose the model is processing the word “<strong>it</strong>” in: <em>“The dog chased the cat because </em><strong><em>it</em></strong><em> was fast.”<br>- </em>The <strong>query</strong> for “<strong>it</strong>” asks: <em>“What does ‘it’ refer to?”<br>- </em>It looks at <strong>keys</strong> from other tokens: “dog”, “cat”, etc.<br>- It assigns higher attention to “cat” (maybe).<br>- It gathers the <strong>value</strong> of “cat” and others proportionally to form a context-rich output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SMHaub-v6myCyVz31S7g0A.png" /><figcaption>Self-Attention</figcaption></figure><p><strong>(3)<em> </em></strong>The raw attention score is calculated between a <strong>Query</strong> and all the <strong>Key </strong>vectors using the dot product, like in equation 2, and this measures <strong>how much attention token i should pay to token j</strong>. Then they are scaled as in equation 3. Generally, the magnitude of a dot product grows as the dimension increases. Thus, they are scaled down to prevent large values from making the softmax too sharp. This keeps gradients stable and training efficient.</p><p><strong>(4) </strong>Equation 4 computes softmax for each token <strong>i</strong> over all tokens <strong>j</strong>. The result α_ij​ is a <strong>normalized weight</strong> showing how important token j is to token i.</p><p><strong>(5)</strong> The output z is a weighted sum of all value vectors, where the weights come from attention scores α_ij​.</p><p><strong>(6)</strong> The final self-attention output matrix Z is formed by stacking all z’s. Z= [z1 z2 ….z_n]. The shape is n x d_v. <br><strong>n</strong> is the sequence length or number of tokens. For the given diagram, n = 4.<br><strong>d_v</strong> is the dimension of the value vector, which is a hyperparameter. The shape of z1, z2, z3 is 1 x d_v.</p><p>The importance of Z will be clear after understanding the <strong>Transformer </strong>architecture.</p><h4>Multi-Head Attention:</h4><p>One separate self-attention is also called “Head”. When we use <strong>only one attention head</strong>, the model looks at the entire sentence, but it can only learn one way of focusing. It is like having just one camera to take a picture. That one camera angle might miss important details. Instead of using just one attention “camera,” multi-head attention uses several, and it can capture diverse relations between tokens/words like: <em>“The dog chased the cat because </em><strong><em>it</em></strong><em> was fast.” </em>Earlier, the word “it” could capture only one relation, like “it refers to cat”. But using multi-head attention, it can capture: “it refers to cat”, “it (the cat) was fast”, “it was chased by the dog”, etc.<br>So, multi-head attention can capture much more diverse contextual information than single-headed self-attention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*gZB_Uoz7E6hwr47ykwSo1w.png" /><figcaption>Multi-Head Attention</figcaption></figure><p>Now we will dive deeper to understand Multi-Head Attention.<br><strong>(1)</strong> Just like in self-attention, the input consists of token vectors:<br> If there are 5 tokens, we get 5 vectors: x_1​,x_2​,…,x_5​, which are stored in the matrix X.</p><p><strong>(2)</strong> Instead of computing one set of Q, K, V matrices, we compute them separately for each head using different learned weight matrices as in equation 3.</p><p><strong>(3) </strong>For every head, we follow the same self-attention process: <br>-Each token uses its query vector to compare with all key vectors (from all tokens)<br>- This gives a similarity score (attention score)<br>- Apply softmax to get normalized attention weights<br>- Use these weights to take a weighted sum of value vectors and produce output Z_i​ for head i.<br>Each head learns to focus on different types of relationships — some may capture short-term dependencies, others long-range dependencies.</p><p><strong>(4)</strong> Each head gives an output matrix Z_i. We concatenate them along the feature dimension to form a matrix according to equation 5. This stacking step gathers all the different perspectives from each head.</p><p><strong>(5)</strong> The concatenated matrix Z is projected back to the original dimension using a learnable weight matrix W^O as in equation 6. This mixes information from all heads and returns the final multi-head attention output.</p><p>All the Z_i values can be calculated in parallel, as no part has a value that is dependent on others.</p><p>I hope the concept and calculation of self-attention &amp; multi-head attention are clear. These two mechanisms are the foundation of the Transformer architecture, which has changed the Deep Learning Field. Thank you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a530978ab1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attention- please- Part 1 (Seq2Seq Attention)]]></title>
            <link>https://medium.com/@shadmansobhan114/attention-please-part-1-9ec0aa5759fa?source=rss-33889b8e0d1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ec0aa5759fa</guid>
            <category><![CDATA[seq2seq]]></category>
            <category><![CDATA[transformer-model]]></category>
            <category><![CDATA[transformer-architecture]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <dc:creator><![CDATA[Shadman Sobhan]]></dc:creator>
            <pubDate>Tue, 29 Apr 2025 06:28:55 GMT</pubDate>
            <atom:updated>2025-04-30T16:16:27.148Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Neural Networks Focus &amp; On Depth Discussion on “Attention” Mechanism</h4><p>Paying attention is mandatory for humans. But what about machines? They cannot do it like humans, right? Through the advancements of Machine Learning, Deep Learning, and Neural Networks( We will mention it as “NN” onwards), machines are becoming more humanlike. They can identify images, predict outcomes, and do many more things. But in any task, it is very important to understand “what to process” or “exactly what to do”, and so focusing/paying attention is required.</p><p>Various State-of-the-Art models, LLMs, and Transformers use a mechanism called “attention” through which they understand where to focus/ which part is important. In this article, we will discuss in detail the “Attention Mechanism”.</p><h4>Why Attention is Important:</h4><p>The Attention Mechanism was first proposed for machine translation. So let’s understand machine translation in brief.</p><p>Suppose we want to translate “Ami ar konodin somoy noshto korbo na”- a Bengali sentence into English.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Awr07dPbFnjXTO6blAYMXg.png" /><figcaption>Neural Network for Machine Translation</figcaption></figure><p>The network has 2 parts. One is an <strong>encoder</strong> that receives the Bengali text/ <strong>input</strong>. It reads the input word by word and summarizes the input into a form that the model can work with, like a set of vectors (hidden states) capturing the <em>meaning</em> and <em>context</em> of the input, and expresses it as just one vector.</p><blockquote>For someone who didn’t read about the machine translation RNN architecture:<br>The h_0 is the first hidden state. From each neuron of the encoder, a hidden state is generated (h_1, h_2, etc.), and they are sent to the next state. The last neuron generates the last hidden state, which is a vector that represents the overall meaning and context of the input.</blockquote><p>The second part is <strong>the decoder</strong>. The decoder takes the encoder’s output (the memory) and generates the target output (e.g., English translated sentence here), step by step. It predicts the next output (word or token) based on the encoder’s information and the previously generated outputs (what it has translated so far).</p><p>This process is very different than human translation because humans would never memorize the whole sentence. Humans would read part of the sentence, then translate it, and so on. This encoder-decoder-based machine translation has some limitations:<br>1. When the sentence is very long, it would be difficult to represent everything by a fixed-sized vector.<br>2. For longer sentences, often important works/info would get lost.<br>3. A generated longer sentence would often be poor in quality.<br>4. The model could not focus back on specific parts when needed.</p><p>Now let’s try to implement this by another method:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nKrt1rJZ5G08SG5-S5qYgA.png" /><figcaption>Machine Translation with Attention Mechanism</figcaption></figure><p>At every decoding step, the model decides where to “focus” among all the encoder’s outputs, rather than treating all input words equally. Suppose, while outputting “I”, the model looks at every output of the encoding step, but puts most “attention” to the output whose input is “Ami” (Which means I in Bengali). In this way, every output is determined by putting attention on one/two specific outputs of the Encoder. And so this mechanism is called “<strong>Attention Mechanism</strong>”.</p><p>Now, how much attention <em>each decoder step</em> will give to the <em>output of each encoder step</em> depends upon the value of α associated with that encoder. Attention computes weights (α) for each input word at each decoding step &amp; Higher weight = the model is paying more attention to that encoder word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*7GxKt9j92VxzORsNcUyvNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*e3U3LNaO4Zkvgf2vW_Z1Cg.png" /></figure><p>Let’s now understand the whole process step by step:</p><p><strong>(1) </strong>The encoder processes the input sentence and returns hidden states..<strong> </strong>The final encoder hidden state at time step t is a concatenation of forward and backward hidden states at time t, as seen in <strong>equation 1</strong>. This effectively captures context from previous and next words.</p><p><strong>(2)</strong> For each output word to be generated at time step i, the decoder looks at its own previous state s&lt;i-1&gt; &amp; calculates attention scores e&lt;i,j&gt; for all encoder states using <strong>equation 2</strong>.</p><p><strong>(3)</strong> The attention weight is generated by using a score function. The score function evaluates how well the input word at position j aligns with the current decoding step i.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*uZaAO5VGT0EPfOInoalz3w.png" /><figcaption>Computing Scores</figcaption></figure><p>At first, let’s discuss about <strong>dot product</strong>. This computes the <strong>cosine-like similarity</strong> between the decoder and encoder states. It measures how aligned their directions are in the hidden space. Very fast and efficient &amp; Works well if s&lt;i−1&gt; and h&lt;j&gt; are of the same dimensionality. But the limitation is that there are no learnable parameters, so it can’t adapt or transform the representation. Besides, it is assumed that the encoder and decoder are in the same representation space, which may not always be ideal.<br><strong>General Scoring</strong> is a <strong>learnable dot product</strong>. The encoder hidden state h&lt;j&gt; is first transformed via a matrix W_a​ to match the decoder’s representation space. More flexible &amp; can learn to put more weight/importance on certain dimensions of h&lt;j&gt;. The limitation is slightly more computational cost than the dot product, and this is still a linear transformation.</p><p><strong>Additive scoring</strong> is a tiny Neural Network. It can model complex relationships and works well even when the encoder and decoder dimensions differ. The parameters learnt here are shared across time steps</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*qcAeHCKn-KUicPhbz-d4wQ.png" /><figcaption>Blue colors indicate learnable parameters</figcaption></figure><p><strong>(4) </strong>Using equation 3, the scores are converted into a probability distribution using the <strong>softmax</strong> activation function, and the sum of all attention weights is 1. This process ensures context vector C&lt;i&gt; is calculated as weighted average that we use in equation 5. Without normalization, this weighted sum would be unstable, and the decoder might over- or under-emphasize inputs arbitrarily.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*v7kz69vUBU0JCV4td-MIYw.png" /><figcaption>Softmax activation</figcaption></figure><p>Softmax creates a sharp or smooth focus. If one e&lt;i,j&gt; is much larger than others, softmax will give high attention weight to that token. If scores are similar, attention gets spread out across multiple tokens. This gives the model flexibility to focus on one key word (like translating a noun) or spread attention over a phrase (like idiomatic expressions). Also as softmax is fully differentiable, the attention weights can be learned through backpropagation during training. That’s why <strong>softmax</strong> is used.</p><p>Equation 4 just makes sure that attention weights are a probability distribution.</p><p><strong>(5) </strong>The context vector is a weighted average of encoder hidden states. Each h&lt;j&gt; is <strong>scaled by how relevant</strong> it is (attention weight α&lt;i,j&gt;) to the decoder at step i.</p><p>(6) The decoder’s new hidden state is calculated using equation 6. The function <em>f</em> is a GRU/LSTM unit.</p><p>(7) <strong>Final output prediction: </strong>Equation 7 predicts the probability of the next word in the output vocabulary. Suppose the third generated word is “go”. It predicts what might be the possible word like “go” or “visit” or “eat”, etc.</p><p>This equation computes the probability distribution over the target vocabulary for the next word y&lt;i&gt;, given: <br>All previous predicted tokens y&lt;1&gt;,…,y&lt;i−1&gt; &amp; the entire input sequence x.</p><p>The decoder’s updated state (s&lt;i&gt;) and context vector (C&lt;i&gt;) go through a linear transformation g(.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*G2-snQkipQGyd1VMNU0hVw.png" /></figure><p>The transformation g(.) is given in <strong>equation a</strong>. But the output of equation a is unnormalized. So we use softmax, like in equation b, to convert them into a probability distribution. Now it becomes easier to understand what might be the possible next word. <br>Suppose “go”-0.5, “visit”-0.4, “eat”-0.1. So the next word will be “go”.</p><p>I hope now it is clear how the “Attention Mechanism” works. The specific attention described in this article is called <strong>“Seq2Seq Attention” </strong>because <strong>it takes a sequence as input</strong> and <strong>generates another sequence as output</strong>.</p><p>Four key types of attention mechanisms are used in or beyond Seq2Seq.</p><h4><strong>Bahdanau Attention</strong> (Additive Attention):</h4><p>This uses Bahdanau Scoring to calculate the score.</p><h4><strong>Luong Attention</strong> (Multiplicative Attention):</h4><p>This uses dot product/ linear scoring to calculate the score.</p><p>The other two are <strong>Self-Attention &amp; Multi-Head Attention</strong>. We will talk about them in <a href="https://medium.com/@shadmansobhan114/attention-please-part-2-self-attention-multi-head-attention-8a530978ab1c">the next article</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ec0aa5759fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bidirectional RNN & Deep RNN]]></title>
            <link>https://medium.com/@shadmansobhan114/bidirectional-rnn-deep-rnn-8039ce939d4a?source=rss-33889b8e0d1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8039ce939d4a</guid>
            <category><![CDATA[bidirectional-rnn]]></category>
            <category><![CDATA[bidirectional-lstm]]></category>
            <category><![CDATA[deep-rnn]]></category>
            <dc:creator><![CDATA[Shadman Sobhan]]></dc:creator>
            <pubDate>Sun, 27 Apr 2025 18:09:02 GMT</pubDate>
            <atom:updated>2025-04-29T06:33:32.020Z</atom:updated>
            <content:encoded><![CDATA[<p>Neural Networks face limitations when the data is image or sequence data. For images, we use Convolutional Neural Networks &amp; for sequence data, we use RNNs. Even though RNNs are great at working with sequence data, it had two main limitations: Vanishing Gradient &amp; Exploding Gradient; Vanishing Gradient being the most severe. To solve this problem, the GRU or Gated Recurrent Unit was proposed. LSTM, a more complex and powerful architecture than GRU, handles long-term dependencies even more effectively.</p><p>These are the three blocks/units for Sequence Data Analysis: Recurrent Neural Network blocks, GRU(Gated Recurrent Unit), and LSTM(Long Short-Term Memory). Typical Neural Networks built by these three blocks are unidirectional. But for better context understanding, it is required to be bidirectional. Besides, even if we have built a Neural Network using these blocks, the network will have only a temporal dimension. For more processing and learning, the network needs to be deep.</p><p>So we will analyze bidirectional RNNs and then look into Deep RNNs in this article.</p><h4>Bidirectional RNN:</h4><p>Suppose we have a sentence: “The bank was closed”. Typical RNNs ( we are including GRU and LSTMs here too as Recurrent units) read a sentence from left to right. So it will not understand what “bank” actually means, like it’s a riverbank or a financial bank.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MlFWNU5PZKTUhU0yO9OA7A.png" /><figcaption>Unidirectional RNN architecture, which reads in only one direction</figcaption></figure><p>But what if it could also read backwards?<br>It would have understood that, as “closed” is related to “bank,” then it must be the financial bank because a riverbank cannot be closed. So, even if we use GRU, LSTM, etc., it must have the capability to read from both directions. <br>And that’s why BRNN (Bidirectional Recurrent Neural Network) is preferred over a Unidirectional one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M6gsn9TA8lJ5pmpiKf6HUw.png" /><figcaption>Bidirectional RNN</figcaption></figure><p>The first layer/ row reads from left to right, and the second row reads from right to left. The same word/input goes to two blocks at the same time, and the final output is generated using both the blocks’ decisions.</p><p>Overall, BRNN works much better as it reads from both directions and has better context. <br>But there is a limitation, <strong>it needs the whole sentence</strong> because the second row in the architecture, which reads from backwards, cannot start until it gets the last word of a sentence. Like in a speech recognition system, it will wait until the speaker stops speaking.</p><h4>Deep RNN:</h4><p>What we see in RNNs is just a single layer, as the two figures above. If we just stack them, we will get the architecture of Deep RNNs like the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DBK_6B1nbnhtsOFikd87Mg.png" /><figcaption>Deep RNN architecture</figcaption></figure><p>A typical RNN has only one layer across many time steps, while a deep RNN has multiple stacked layers, each unfolded across the same sequence of time steps.</p><p>However, as RNN has temporal dimension (time step-based dimension), it will be cumbersome if a deep RNN has many layers. So, generally, RNN does not have many layers like a traditional Deep Neural Network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*YafEjTExe1CsKVOby4RPHg.png" /><figcaption>Formula for the hidden layer in deep RNN</figcaption></figure><p>That’s it :D.</p><p>So to summarize:<br>1. NNs do not work well when the data is sequential.<br>2. Recurrent units solve this problem.<br>3. Recurrent units introduce the vanishing gradient problem, and it does not have long-term dependency.<br>4. These problems are solved by GRU and LSTM.<br>5. RNN can be built using Recurrent units/GRUs/LSTMs.<br>6. But RNNs built by this are unidirectional.<br>7. For clear context understanding, RNN must be bidirectional.<br>8. So, BRNN is utilized.<br>9. For better processing, deep RNNs are also used. They are made by stacking many layers of RNNs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8039ce939d4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Long Short-Term Memory (LSTM)]]></title>
            <link>https://medium.com/@shadmansobhan114/long-short-term-memory-lstm-3d1b5f18c049?source=rss-33889b8e0d1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d1b5f18c049</guid>
            <category><![CDATA[sequence-data]]></category>
            <category><![CDATA[lstm-time-series]]></category>
            <category><![CDATA[lstm]]></category>
            <dc:creator><![CDATA[Shadman Sobhan]]></dc:creator>
            <pubDate>Sat, 26 Apr 2025 09:33:40 GMT</pubDate>
            <atom:updated>2025-04-29T06:34:10.060Z</atom:updated>
            <content:encoded><![CDATA[<p>Neural Networks face limitations when the data is image or sequence data. For images, we use Convolutional Neural Networks &amp; for sequence data, we use RNNs. Even though RNNs are great at working with sequence data, it had two main limitations: Vanishing Gradient &amp; Exploding Gradient; Vanishing Gradient being the most severe. To solve this problem, the GRU or Gated Recurrent Unit was proposed. In this article, we will explore LSTM, a more complex and powerful architecture than GRU, which is designed to handle long-term dependencies even more effectively.</p><p>Let’s first compare GRU and LSTM, which will later help us understand the LSTM unit’s architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*uShjjEnrn3M-f_GQ2GNV_g.png" /></figure><p>The equations for an LSTM are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*sMorkBpleRJUi-0qqT5zWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kfG0fZrcKtW6Is2EYDgU_g.png" /><figcaption>LSTM unit</figcaption></figure><p>The 6 components of LSTM are:</p><ol><li><strong>Forget Gate:</strong><br>When f_t=0 or very low, the previous cell state c_t−1​ has little to no influence on the current cell state c_t​, basically “forgetting” past information.<br> Since the final hidden state h_t​ is computed based on the updated cell state c_t and the output gate, a low f_t means that the memory ( c_t-1​) will not influence the output.<br>Therefore, <strong>the forget gate controls how much past information is removed, which is why it’s named the forget gate.</strong></li><li><strong>Input Gate:</strong><br>The input gate controls how much of the candidate cell state value(or new input) is added to the cell state. The updated cell state then contributes to the computation of the final hidden state (h_t​), along with the output gate. <br>Therefore, the input gate indirectly influences the final hidden state by controlling what new information enters the memory. <br><strong>It is named the “input gate” because it controls how much of the new input (candidate cell state) is allowed into the cell state.</strong></li><li><strong>Output Gate:</strong><br>The output gate controls how much of the cell state’s information(c_t) will be present in the final hidden state h_t.<br> It applies a filter to the current cell state (c_t)​, deciding what part of the memory should be exposed to the next layer or time step.<br> Even if the cell holds important information, the output gate can choose to hide or reveal it based on context.<br><strong> That’s why it’s called the output gate — as it governs what information exits the cell as output.</strong></li></ol><p>Before continuing, let’s summarize the 3 gates-<br><strong>Forget Gate</strong> ⇒ Controls how much previous cell state info will be forgotten.<br><strong>Input Gate</strong> ⇒ Controls how much new input (candidate cell state) will be present in the cell state and final hidden state.<br><strong>Output Gate</strong> ⇒ Controls how much cell state information will be available in the output.</p><p><strong>4. Candidate Cell State: </strong><br>The candidate cell state​ represents the new potential content that could be added to the cell state.<br> It is generated based on the current input x_t​ and the previous hidden state h_t−1​, passed through a tanh⁡ activation to keep the values between -1 and 1.<br> However, this candidate value is not directly written into the memory; the input gate decides how much of it should be added.<br> <strong>It is called a candidate because it’s a candidate for the new cell state.</strong></p><p><strong>5. Cell State Update / Long-Term Memory:</strong><br>The cell state update is a combination of forgetting old information and adding new information.<br> This updated cell state acts as the long-term memory and heavily influences future computations. <br>This is considered “Long-Term Memory” because it contains information from previous cell states, which in turn depend on previous cell states. It is calculated based on what to forget (forget gate) and what to add/memorize (input gate).</p><p><strong>6. Hidden State / Short-Term Memory:</strong><br>The hidden state is computed by applying the output gate to the updated cell state. It represents the current output of the LSTM and is passed to the next time step or used for final predictions.<br> This is considered “Short-Term Memory” because it focuses on the important part of the current cell state relevant right now, after filtering through the output gate.</p><h4>Why LSTM is better than RNN:</h4><p>1. RNN suffers from the vanishing and exploding gradient problem due to repeated matrix multiplication. LSTM solves this by applying addition and element-wise multiplication.</p><p>2. RNN cannot capture long-term dependency. In LSTM, if Forget Gate (f_t) value is near 1, then previous state information can flow freely without losing any info, and so LSTM can capture long-term dependency.</p><h4>Why LSTM is better than GRU:</h4><ol><li>Stronger long-term memory management:<br>The forget gate in LSTM allows it to selectively erase or keep information, helping it capture long-term dependencies better than GRU, which combines memory operations in a simpler way.</li><li>More powerful handling of complex sequences:<br>Thanks to the separate cell state and hidden state, LSTM can store and update memory more effectively in tasks with complicated, long sequences compared to GRU’s single hidden state.</li></ol><p>For sequence data analysis, we can use RNN, GRU, and LSTM, especially GRU and LSTM.</p><p>For speed and simplicity → GRU<br>For maximum memory power and handling of long-term dependencies → LSTM.</p><p>For sequence data analysis, RNN, GRU, and LSTM are 3 units. Which one we will use depends greatly on the type of work we are doing. There is no universal choice. Thank you for reading this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d1b5f18c049" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gated Recurrent Unit (GRU)]]></title>
            <link>https://medium.com/@shadmansobhan114/gated-recurrent-unit-gru-8c82eddae668?source=rss-33889b8e0d1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c82eddae668</guid>
            <category><![CDATA[rnn-architecture]]></category>
            <category><![CDATA[gated-recurrent-unit]]></category>
            <dc:creator><![CDATA[Shadman Sobhan]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 16:21:39 GMT</pubDate>
            <atom:updated>2025-04-29T06:34:59.766Z</atom:updated>
            <content:encoded><![CDATA[<p>Neural Networks face limitations when the data is image or sequence data. For images, we use Convolutional Neural Networks &amp; for sequence data, we use RNNs. Even though RNNs are great at working with sequence data, it had two main limitations: Vanishing Gradient &amp; Exploding Gradient; Vanishing Gradient being the most severe. To solve this problem, the GRU or Gated Recurrent Unit was proposed, which introduced <strong>the Gating Mechanism</strong>. This mechanism dynamically controls the flow of information, retains relevant information, and discards unnecessary information.</p><p><strong>This article is structured in the following way: first, the architectures and equations of RNN and GRU will be given for easier understanding of GRU. Then, the vanishing gradient problem of RNN will be discussed. After that, individual gates and components of GRU will be described, followed by how GRU solves vanishing gradient problem that is present in RNN.</strong></p><p>Suppose a Recurrent Neural Network is working with a sentence: “The <strong>cat</strong>, which already ate its food, <strong>was</strong> full”. Now, whether it will be “was” or “were” depends on the subject. The subject is singular, so it will be “was”.<br>But we see, the subject “cat” is very far from “was”. In this case, the neural network should have <strong>long-term dependency, </strong>which means the decision of a step is dependent on a step that is very far away. So even in an RNN, using backpropagation, we might face Vanishing Gradient &amp; Exploding Gradient. GRU can be a possible solution, as will be discussed shortly.</p><p>At first, we will see both unit architectures &amp; formulas of RNN with GRU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*jJjDHU7JZHGhLgXYVVsMnw.png" /><figcaption>RNN unit ( Activation functions might change)</figcaption></figure><p>Following is a GRU unit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3u9TIPn9OBXCrlylNRxoMg.png" /><figcaption>GRU Unit</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*_Gy2gVSlm0E3Yw9I6U9uHw.png" /><figcaption>RNN Formulas</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*EVV2vc0WMfgqlsSJvTdTQw.png" /><figcaption>GRU Formulas</figcaption></figure><h4>Vanishing Gradient Problem in RNN:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*uPHvXhJlaLSCcJZ2vx0MzQ.png" /></figure><p>During backpropagation, gradients get multiplied many times by W_hh​ and the derivative of ϕ. If those values are &lt; 1, the gradients shrink exponentially. This makes it hard to learn long-term dependencies.</p><h4>Gates in GRU:</h4><p>A GRU consists of two main gates:</p><p><strong>1. Reset Gate (r_t)/ Short Term Memory</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*Kons6dDzA8zGz5M98PcXKg.png" /></figure><p>We can consider the Reset gate like a memory cell. When the value is 0, it holds no memory &amp; independent of the past (hidden state). When the value is 1, it holds memory &amp; depends on the past (hidden state).</p><p><strong>When r=0, it works like a reset button, or the previous value is forgotten. So, r is called <em>the Reset Gate</em>.</strong></p><p><strong>Reset Gate decides how much of the immediate past to forget when generating the next state, so it’s handling the <em>most recent memory</em>, not the long-term trend. Thus, it is called short-term memory.</strong></p><p>2.<strong> Update Gate (z_t)/ Long Term Memory</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*zFBLXUNxrzWTfMBrrwLrIg.png" /></figure><p><strong>When z=0, it means don’t update or keep the previous value. And when z=1, it means update or consider a new value. So z is called <em>Update Gate</em>.</strong></p><p>Suppose we want to remember the subject of a sentence: <em>“The cat, which had already eaten its food, was full.”<br></em>When the GRU reads “The cat”, it stores this in h_t. As more words come (like “which had already eaten…”), the update gate z_t​ can choose to keep the memory of “cat” by setting z_t ≈ 0.<br> So h_t=h_t−1​, memory doesn’t change.<br> When GRU finally reaches the word “was”, it still “remembers” “The cat” — thanks to the update gate holding on to that info. So, the update gate allows important info to persist unchanged across many time steps.</p><p><strong>The update gate decides whether to keep past knowledge. When set to low, it holds onto older information. This enables long-term memory , something vanilla RNNs struggle with. So, Update Gate is also called Long Term Memory.</strong></p><h4>Candidate Hidden State:</h4><p>The candidate hidden state is generated using the input &amp; previous hidden state.</p><p><strong>It is called Candidate Hidden State because it is a candidate to be selected as the new hidden state.</strong></p><p>If reset gate = 1 or high value, that means the candidate hidden state heavily depends on the past hidden state, or is more reliant on past &amp; if reset gate = 0 or low value, that means the candidate hidden state heavily depends on input or forgetful of past.<br>Now when update gate = 1 or high value, present hidden state = candidate value &amp; if update gate = 0 or low value, present hidden state = previous hidden state.</p><p>I know all of this is becoming confusing, so let’s summarize everything:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*z6aPNHeEh5YHPaJgSlPJBg.png" /></figure><p>I hope all this is clear. Now let’s go to the most important part.</p><p>We see that the present hidden state value can be either input-dependent or past-dependent. So we can control the information flow.<br>Now for the aforementioned sentence: “The <strong>cat</strong>, which already ate its food, <strong>was</strong> full”. If each stage of GRU is made past dependent, then “was” will be dependent on “cat” without making the gradient too high or too low.</p><p>In RNN, the values of each hidden state are multiplied by a weight before they reach the next stage. But in GRU, if z=0 or a very low value, then directly present hidden state = past hidden state. So, no problem of weight dependency, which in turn can make the present hidden state value much higher or lower than the very early stage hidden state value, which occurs in RNN.</p><p><strong>It can be compared to a direct connection between two related words, even though there are many words present between them. This is called Long-Term Dependency.</strong></p><p>Thus, GRU solves the most difficult problem present in vanilla RNN. That is vanishing gradient.</p><p>Overall, the benefits of a GRU compared to an RNN are:</p><ol><li>Solves the Vanishing Gradient Problem</li><li>Maintains Long-Term Dependencies</li><li>Adaptable Information Flow: GRU controls how much past information should be carried.</li><li>Efficient Training: GRUs converge faster than RNNs in practice due to better control over memory and gradient flow.</li></ol><p>In summary, GRUs offer a practical balance between performance and simplicity. By intelligently controlling memory flow, they overcome RNN limitations and remain an efficient alternative to more complex architectures like <a href="https://medium.com/@shadmansobhan114/long-short-term-memory-lstm-3d1b5f18c049">LSTMs</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c82eddae668" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[List of My Articles]]></title>
            <link>https://medium.com/@shadmansobhan114/my-articles-930db1680d66?source=rss-33889b8e0d1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/930db1680d66</guid>
            <dc:creator><![CDATA[Shadman Sobhan]]></dc:creator>
            <pubDate>Thu, 24 Apr 2025 17:28:14 GMT</pubDate>
            <atom:updated>2025-05-01T17:58:19.942Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello everyone. In my Medium account, I try to write articles on Machine Learning, Deep Learning &amp; Neural Networks and their variations.</p><p>Thanks to <a href="https://www.deeplearning.ai/"><em>deeplearning.ai</em></a> &amp; various <em>Medium Articles</em> for providing me with resources.</p><p>Following is a list of my articles &amp; what they are about, including mathematical expressions and visualizations:</p><h4>Machine Learning:</h4><ol><li><a href="https://medium.com/@shadmansobhan114/machine-learning-basics-cfc59aee32f5">Machine Learning Foundations &amp; Gradient Descent</a></li><li><a href="https://medium.com/@shadmansobhan114/underfitting-overfitting-how-to-solve-them-acd5ead7891c">Underfitting &amp; Overfitting- How to solve with Regularization</a></li><li><a href="https://medium.com/@shadmansobhan114/machine-learning-regression-algorithms-part-1-55f7bea9dc09">Machine Learning Regression Algorithms — Part 1</a></li></ol><h4>Convolutional Neural Networks:</h4><ol><li><a href="https://medium.com/@shadmansobhan114/convolutional-neural-networks-af9649848507">Convolutional Neural Networks</a></li><li><a href="https://medium.com/@shadmansobhan114/understanding-classical-cnn-architectures-4f7f206cd973">Understanding Classical CNN Architectures</a></li><li><a href="https://medium.com/@shadmansobhan114/understanding-residual-network-resnet-skip-connection-c444a1accfe9">Residual Network (ResNet) &amp; Skip Connection</a></li><li><a href="https://medium.com/@shadmansobhan114/1-x-1-convolution-filter-bottleneck-layer-a8d5e527e2ed">Inception Network &amp; Bottleneck Layer</a></li></ol><h4>Sequence Data Analysis:</h4><ol><li><a href="https://medium.com/@shadmansobhan114/recurrent-neural-network-rnn-for-e611aecf277b">Recurrent Neural Network (RNN) &amp; Sequence Data Analysis</a></li><li><a href="https://medium.com/@shadmansobhan114/gated-recurrent-unit-gru-8c82eddae668">Gated Recurrent Unit (GRU)</a></li><li><a href="https://medium.com/@shadmansobhan114/long-short-term-memory-lstm-3d1b5f18c049">Long Short-Term Memory (LSTM)</a></li><li><a href="https://medium.com/@shadmansobhan114/bidirectional-rnn-deep-rnn-8039ce939d4a">Bidirectional RNN &amp; Deep RNN</a></li></ol><h4><strong>Attention Mechanism &amp; Transformer Architecture:</strong></h4><ol><li><a href="https://medium.com/@shadmansobhan114/attention-please-part-1-9ec0aa5759fa">Attention- please- Part 1 (Seq2Seq Attention)</a></li><li><a href="https://medium.com/@shadmansobhan114/attention-please-part-2-self-attention-multi-head-attention-8a530978ab1c">Attention- please- Part 2 (Self-Attention &amp; Multi-Head Attention)</a></li><li><a href="https://medium.com/@shadmansobhan114/attention-please-part-3-transformer-architecture-225e802cedac">Attention- please- Part 3 (Transformer Architecture)</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=930db1680d66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recurrent Neural Network(RNN) & Sequence Data Analysis]]></title>
            <link>https://medium.com/@shadmansobhan114/recurrent-neural-network-rnn-for-e611aecf277b?source=rss-33889b8e0d1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e611aecf277b</guid>
            <category><![CDATA[recurrent-neural-network]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[natural-language-process]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Shadman Sobhan]]></dc:creator>
            <pubDate>Thu, 24 Apr 2025 16:15:12 GMT</pubDate>
            <atom:updated>2025-04-29T06:35:56.376Z</atom:updated>
            <content:encoded><![CDATA[<h3>Recurrent Neural Network (RNN) &amp; Sequence Data Analysis</h3><p>Typical Neural Networks might work very well, but their limitations become visible when the data is image or sequence data. For images, we use Convolutional Neural Networks. In this article, our main topic is RNN, which mainly works for sequence data. We will talk about their architecture, strengths, and limitations.</p><h4>Sequence Data:</h4><p>Sequence data is a special kind of data where the <em>order/position matters</em>. Like “The boy eats rice” is a sentence, but if we change the order, “Rice eats the boy” it will be a meaningless sentence 😂. Some examples of sequence data are:<br>📄 Text: “The cat sat on the mat” (order of words matters).<br>🎵 Audio: Speech or music waveforms (order of sound samples matters).<br>📈 Time Series: Stock prices, sensor readings (values change over time).<br>🎬 Video Frames: A series of images (frames) forming motion.<br>🧬 DNA/Protein Seq: ACGT… (biological sequences where order affects function).</p><p>Some examples of using sequence data in machine learning are:<br>Speech Recognition, Music Generation, DNA Sequence Analysis, Machine Translation, etc.</p><p>Let’s try to use a Neural Network to <strong>identify names</strong> from a sentence. Suppose a sentence is <em>“John went for a walk, but he returned with his friend Patrick ”</em>.</p><h4>Pre-processing steps:</h4><p>We have to pre-process the dataset to reduce complexity and ease the representation of words. The steps are:</p><ol><li><strong>Converting to Lower-Case:</strong><em> john went for a walk</em>, <em>but he returned with his friend Patrick</em></li></ol><p>2. <strong>Remove Punctuation:</strong><em> john went for a walk but he returned with his friend Patrick</em></p><p><em>3. </em><strong>Tokenization: </strong>Tokenization is the process of breaking down text into smaller pieces, called tokens. A word might be a token.<br><strong>“</strong><em>john” “went” “for” “a” “walk” “but” “he” “returned” “with” “his” “friend” “patrick” (“” this means individual words)</em></p><p><em>4. </em><strong>Removing Stop-Words: </strong>Stop words are the common words in a language that are filtered out before or during text processing because they are generally not useful for understanding the meaning of a sentence in most tasks. Some common stop words are: “a”, “an”, “the”, “in”, “on”, “and”, “is”, “are”, “was”, “be”, “with”, “to”, etc. <br>After removing the stop-words we get:<br><strong>“</strong><em>john” “went” “walk” “he” “returned” “his” “friend” “patrick”</em></p><p>5. <strong>Stemming/Lemmatization: </strong>Stemming chops off the end of words to find the “root” form , but it doesn’t care about grammar or if the result is a real word. <br>studies → studi<br>better → better<br>Lemmatization reduces a word to a valid dictionary base form.<br>studies → study<br>better → good<br>After applying lemmatization, we get:<br><strong>“</strong><em>john” “go” “walk” “he” “return” “his” “friend” “patrick”</em></p><p>6. <strong>Vectorization:</strong> Computers can only understand “numbers”. They can’t understand words. So we have to express words as numbers. This process is called vectorization. There are mainly 3 methods. Bag of Words, TF-IDF, Word Embeddings. <br>Here, for simplicity, we will use <strong>the dictionary representation of words.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hNU4T5zImuQHesW3yBRHeA.png" /><figcaption>Bag of Words Representation using Vocabulary for Name Identification</figcaption></figure><p>We will create a dictionary containing many words as shown in the image. The dictionary contains 10000 words. From the dictionary, we will select the words we have.<br>Like “patrick” which is the 6005 th word of the dictionary. Now, how do we express it as a number?<br>For that, we will use “<strong>One-hot encoding</strong>”. As “patrick” is the 6005th word, its one-hot encoding representation will be a vector of length 10000. All positions will be 0 except the 6005th position, which will be 1.<br>In this way, we will express all the words, and thus we can represent words as vectors as we wanted.</p><blockquote>This part is optional. Here we will just know why pre-processing is required:<br>If we don’t use pre-processing, words like “go”, “gone”, “went”, etc., will all need to be placed in the dictionary. Besides, we just need keywords.<br>“John went for a walk, but he returned with his friend Patrick.” &amp; “john go walk he return friend patrick ”- we can at least understand the basic idea. So to reduce complexity we have to apply pre-processing.</blockquote><h4>Why Neural Networks Are Not Enough:</h4><p>Now, there are two reasons<strong> we cannot use a simple Neural Network </strong>architecture here.</p><ol><li><strong>Different Input Size</strong>:<br>For our given sentence, the input size was 8. But if we consider another sentence like “Zami &amp; Mahedi went to the restaurant”, whose processed form is “zami mahedi go restaurant”. So the input size is now 4. Thus, as we change the sentence, the size will be different. <br>This variability in length means the input size changes from sentence to sentence. However, neural networks like standard NNs or CNNs require a fixed input size, because: <br>-The number of input neurons is fixed<br>-The number of weights is based on the input size<br>This makes it difficult for normal NNs or CNNs to directly handle variable-length sequences.<br>A possible solution is padding, where Shorter sequences are padded with zeros (or a special token) and Longer ones are truncated.</li><li><strong>No memory:</strong><br>For sequential data, memorizing word position and order is crucial. Consider the sentences:<br> “I am happy” vs “I am not happy.”<br> Although they contain similar words, the presence of “not” before “happy” completely changes the meaning.<br>In typical neural networks (NNs), inputs are processed independently — there’s no memory of previous words. This means NNs don’t understand the sequence or structure of the input.<br> Even though Convolutional Neural Networks (CNNs) can capture local patterns using filters (e.g., looking at 2–3 words at a time), they still lack an understanding of the global sequence or relative position of words in the sentence.<br>Therefore, traditional NNs and CNNs are not ideal for tasks where context and word order matter.</li><li><strong>Not Sharing Features: </strong>Typically, Neural Networks do not share features. Each position has different weights.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x5haxs_LeKSaPhf_7PMUcg.png" /><figcaption>Changing the position of an input yields a different output.</figcaption></figure><p>If we use an NN in Name recognition, it won’t understand when a name appears in the middle or at the end, as an NN does not share features. Different positions of a “Name” will result in different outputs. So, for sequence data, this is another great limitation of an NN.</p><p>To solve these problems <strong>Recurrent Neural Network(RNN)</strong> was proposed.</p><h4>Unidirectional RNN:</h4><p>The architecture of an unidirectional RNN is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FiNZC6tJzEJUw5me_wXVoQ.png" /><figcaption>RNN architecture. h_0 is the initial hidden state, generally taken as 0.</figcaption></figure><p>The related eqn. are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*H5dZmKWNbWkoNlnU1zuWIA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*jJjDHU7JZHGhLgXYVVsMnw.png" /><figcaption>RNN Unit</figcaption></figure><p>As the weights are all equal, and so varying input size is super easy.<br>Secondly, each stage takes input from the previous stage. Thus, RNN has memory.<br>All the weights are the same, and so, even after changing the position of a word, it is multiplied by the same weight. So, feature sharing is used.</p><p>However, there are two main problems with this architecture.<br>1. It only considers the previous stage input. It does not consider the later input like x4 considers x3, but not x5.<br>2. The input and Output sizes are equal, which might not be true always, like for machine translation.</p><h4>Different Types of RNN:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-_ipsyPvgqqEpbUvv7YSLg.png" /><figcaption>Many-to-one RNN for Sentiment Analysis. There will be only one output: positive or negative.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v_iRsU-qqtYistujiMfZCg.png" /><figcaption>One-to-many RNN for image captioning.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JWnGIfZz9Aw9dq71QDCkYQ.png" /><figcaption>RNN for <strong>Machine Translation</strong>. It has two parts. The encoder takes the input sentence (in the source language) and processes it word by word using an RNN. It compresses the entire sequence into a fixed-length context vector (also called the hidden state or feature vector), which captures the overall meaning of the sentence. The decoder is another RNN that takes the context vector from the encoder and generates the translated sentence one word at a time in the target language.</figcaption></figure><p>Even though RNNs are powerful for sequential data, they face several limitations, especially with long sentences. Let’s consider the sentence: “The curious cat jumped over the fence and explored the garden full of flowers.”</p><p>In this case, the word <strong>“curious”</strong> influences the interpretation of <strong>“flowers.”</strong> However, because these words are <strong>far apart</strong>, a standard RNN <strong>struggles to maintain</strong> this long-range dependency. The earlier word <strong>loses its impact</strong> as we move further in the sequence.</p><p>When training RNNs using backpropagation through time (BPTT), the model updates its weights based on the error gradients flowing backward through the sequence. But:</p><p><strong>Vanishing Gradient Problem</strong>:<br> The gradients shrink exponentially as they pass through each time step. This means <strong>earlier words have little to no impact</strong> on the learning. So, long-term dependencies are lost.</p><p><strong>Exploding Gradient Problem</strong>:<br> On the flip side, gradients can sometimes grow uncontrollably large, leading to <strong>unstable training</strong> and <strong>numerical overflow</strong>.</p><p>The <strong>vanishing gradient problem</strong> is more severe because it causes the model to <strong>completely forget long-term dependencies</strong>, making it nearly impossible for RNNs to learn relationships between distant words in a sequence. Unlike exploding gradients, which <strong>can be clipped </strong>and <strong>easily detected by NaN</strong>, vanishing gradients silently <strong>halt learning</strong> in early layers or time steps.</p><p>To solve the Vanishing Gradient problem, we generally use GRUs &amp; LSTMs instead of typical vanilla RNNs, which will be discussed in <a href="https://medium.com/@shadmansobhan114/recurrent-neural-network-rnn-for-e611aecf277b">the next article</a>.</p><p>Thank you. I hope the basic uni architecture, importance, and limitations of RNNs are clear.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e611aecf277b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Regression Algorithms — Part 1]]></title>
            <link>https://medium.com/@shadmansobhan114/machine-learning-regression-algorithms-part-1-55f7bea9dc09?source=rss-33889b8e0d1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/55f7bea9dc09</guid>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[polynomial-regression]]></category>
            <dc:creator><![CDATA[Shadman Sobhan]]></dc:creator>
            <pubDate>Thu, 17 Apr 2025 09:31:02 GMT</pubDate>
            <atom:updated>2025-04-17T09:31:02.016Z</atom:updated>
            <content:encoded><![CDATA[<h3>Machine Learning Regression Algorithms — Part 1</h3><p>Regression algorithms are a type of supervised machine learning model used to predict a continuous numerical value based on input features. These algorithms analyze the relationship between independent variables (features) and a dependent variable (target) and use this relationship to make predictions on new, unseen data.</p><p>There are many regression algorithms. We will discuss only the popular and important ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*zIuArcdvv1cAOzwQt-pXSg.png" /></figure><h4>Linear Regression:</h4><p>The Linear Regression algorithm creates a linear relationship between features (inputs) and targets (outputs).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*VSUt2BBSbLwelhrIdQldhQ.png" /></figure><p>Imagine you have a set of target values represented by red dots on a graph. These dots are scattered, and the goal is to model a straight line that best fits this data using linear regression.</p><p>In the graph, there are two lines:<br><strong>The Best Fit Line</strong>: This line is drawn in such a way that it passes near most of the red dots, representing the line that provides the most accurate predictions. <br><strong>The Poor Fit Line</strong>: This line does not align well with the target values. It fails to capture the pattern of the data and results in less accurate predictions.<br>Linear regression has to model the best-fit line. The equation that represents linear regression is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*_t950UHDrguEkI_U" /><figcaption>Equation 1</figcaption></figure><p>For a straight line, <em>m represents the slope, and c represents the y-axis intercept point.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*GoAwxtY1fEkcl46a1Isw9g.png" /></figure><p>That’s why in linear regression equation 1, we used m &amp; c. <br>Again, m is sometimes replaced by <em>w(weight)</em> and c by <em>b(bias)</em>. This will be clear from our next example.</p><p>The price of a house depends on many factors like size, age, location, no of rooms, etc. For this example, we are assuming that housing price depends only on size and no. of rooms. So a representative equation might be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*zlgswSav_Y0o0H2n" /></figure><p>Here, <em>5 &amp; 4 act as weight and 20 as bias.</em></p><p>Weights determine how important each feature is. If the size increases by 1 unit, the price increases by <strong>5</strong> units &amp; if the room number increases by 1 unit, the price increases by <strong>4</strong> units.<br>So, a bigger weight means <strong>more influence</strong> on the final price.<br>The feature that holds more value has a higher weight. Like a student&#39;s overall result mainly depends on the final exam, and so the final exam will have more weight.</p><p>Even if all the feature has a value of 0, there might still be a value of the ‘target variable’, and it is represented by bias. Bias allows the model to <strong>shift</strong> the prediction up or down. For example, if the size and number of rooms are 0, the house price will still be 20, which might represent the land value.</p><p>If there is only one feature (independent variable) and one target(dependent variable), it is called simple linear regression.<br>If there is more than one feature (independent variable) and one target(dependent variable), it is called multiple linear regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*nYA_KIPpPokmgFqsy_yIAw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*j9a9AqyskJUesoiCBkpsWg.png" /><figcaption>Source: When only one feature is present, the relation is expressed as a <strong>straight line</strong>. And when more than one feature is present, the relation is expressed as a <strong>hyperplane</strong>. <a href="https://thaddeus-segura.com/multiple-linear/">Thaddeus</a></figcaption></figure><p>Other related topics, such as <a href="https://medium.com/@shadmansobhan114/machine-learning-basics-cfc59aee32f5">error calculation, gradient descent</a>, <a href="https://medium.com/@shadmansobhan114/underfitting-overfitting-how-to-solve-them-acd5ead7891c">overfitting, and underfitting</a>, were discussed in detail. Therefore, I will not elaborate on them here.</p><h4>Polynomial Regression:</h4><p>Linear Regression captures a linear relationship between the input features and the target variable, where the power (degree) of each feature is 1. The model assumes the relationship can be represented as a straight line (or a hyperplane in higher dimensions).<br>Polynomial Regression captures a non-linear relationship between the input features and the target variable by including higher-degree terms (i.e., powers &gt; 1) of one or more features. Despite this, it is still linear in the parameters (weights).</p><p>Now, a question might arise: Why do we need polynomial regression?<br>Suppose we plotted some features like in the following image. Now we see that a straight line can never estimate the feature values. So linear regression is not a good choice.<br>But we can realize that a curve can somehow express the feature values. So, a curve is a much better option compared to a straight line in this case. <br>And <strong>Polynomial Regression is used to model a curve line.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QGMAQbChDCM-BLtLFGaaoA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*mLY5r3RL_kJk1d9w51DvVA.png" /></figure><p>The highest power of x is the degree of the polynomial. Let’s plot random polynomials:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9VBaDeO9WTGU_0Qho8nzvA.png" /></figure><p>We see, as the degree of the polynomial increases, the plot becomes more complex, and so by increasing the polynomial degree, we can capture more complex relations. <br>Higher degrees might lead to overfitting, and so it is important to find the correct degree of the polynomial. We can tend to two methods for correct degree selection:</p><ol><li><strong>Forward Selection:</strong> Start with a lower-degree polynomial (e.g., degree 1). Gradually increase the degree. At each step, evaluate the model’s performance. Stop when the improvement becomes insignificant or the model starts to overfit.</li><li><strong>Backward Selection: </strong>Start with a higher-degree polynomial (e.g., degree 5 or 10). Gradually decrease the degree by removing higher-order terms. Keep evaluating the model until it maintains good performance with fewer terms.</li></ol><p>The difference between Linear and Polynomial Regression is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*sbFrowCXKizIfJzwKWtrzQ.png" /></figure><h4>Lasso, Ridge, Elastic Net Regression:</h4><p>All of them are variations of standard Linear Regression. <strong>Everything is similar except the loss function</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*C05JZElKd3LGgTWPx-RcUA.png" /></figure><p>Their details can be read from this article, “<a href="https://medium.com/@shadmansobhan114/underfitting-overfitting-how-to-solve-them-acd5ead7891c">Underfitting &amp; Overfitting- How to solve with Regularization.</a>”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55f7bea9dc09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Underfitting & Overfitting- How to solve them]]></title>
            <link>https://medium.com/@shadmansobhan114/underfitting-overfitting-how-to-solve-them-acd5ead7891c?source=rss-33889b8e0d1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/acd5ead7891c</guid>
            <category><![CDATA[underfitting]]></category>
            <category><![CDATA[overfitting]]></category>
            <category><![CDATA[regularization-techniques]]></category>
            <category><![CDATA[l2-regularization]]></category>
            <category><![CDATA[l1-regularization]]></category>
            <dc:creator><![CDATA[Shadman Sobhan]]></dc:creator>
            <pubDate>Wed, 16 Apr 2025 09:41:28 GMT</pubDate>
            <atom:updated>2025-04-29T06:37:01.938Z</atom:updated>
            <content:encoded><![CDATA[<h3>Underfitting &amp; Overfitting- How to solve with Regularization</h3><p>While working with machine learning models, the two most common problems we face are underfitting and overfitting. When the model works poorly on both the training and the testing set, it is called underfitting, and when the model works too well on the training set but poorly on the testing set, it is called overfitting.</p><p>The cost function, gradient descent, and training were explained in <a href="https://medium.com/@shadmansobhan114/machine-learning-basics-cfc59aee32f5">the previous article</a>. Here, we will dive a bit deeper to understand <em>Overfitting and Underfitting. </em>After that, we will discuss the steps we can take to solve them. Finally, we will explore three regularization techniques and how they can help address overfitting.</p><h4>Underfitting :</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*UJbKeXHSz6C2BMvpo6TaoQ.png" /><figcaption>Underfitting Model</figcaption></figure><p>The red and blue dots represent the training and testing data points, respectively. In this case, the model fails to accurately predict both sets — it performs poorly on the training data <strong>and</strong> the testing data. This situation is known as <strong>underfitting</strong>. <br>An underfitting model is unable to capture the underlying patterns in the data, resulting in high errors regardless of whether it’s seeing familiar or new examples.<br>You can think of it like a student who hasn’t understood the material well — he struggles to answer questions from both his textbook (training set) and his exam (testing set).</p><p>The main reasons for underfitting are: <br>1. Model Too Simple: A Simple model cannot capture complex relations between features.<br>2. Not Enough Training: Too few training iterations or epochs and very low learning rate.<br>3. High Bias: The Model assumes too much and can’t adapt to the data’s actual complexity.<br>4. Missing Features: Important features (like “Location” in house pricing) are missing.</p><h4>Overfitting :</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*faM-BeimVHYtiYjCJ1zudA.png" /><figcaption>Overfitting Model</figcaption></figure><p>The red and blue dots represent the training and testing data points, respectively. In this case, the model fits the training data almost perfectly (red dots) but fails to generalize well to the testing data (blue dots). This situation is known as <strong>overfitting</strong>.<br>An overfitting model captures not only the underlying patterns but also the noise and minor fluctuations in the training data. As a result, it performs exceptionally well on the training set but poorly on new, unseen data. The left graph shows that a very <em>complex model</em> is learnt that works very well on the training set but does not work well on the testing set.<br>You can think of it like a student who has memorized all the answers from his textbook without truly understanding the concepts — he excels in practice questions (training set) but struggles when the exam questions (testing set) are phrased differently.</p><p>The main reasons for overfitting are:<br>1. Model Too Complex: Using a very deep neural network or high-degree polynomial for a simple dataset, i.e., too many parameters for too little data.<br>2. Not Enough Training Data: The model memorizes the limited examples, and it doesn’t see enough variation to learn general patterns.<br>3. Too Many Features: High dimensionality (too many input variables). So the model can “cheat” by finding irrelevant correlations.<br>4. Too Many Training Epochs: The model keeps training even after it has learned the patterns. Eventually, it starts memorizing noise and outliers.<br>5. Lack of Cross-Validation: Without checking performance on validation data during training, overfitting may go unnoticed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8IK3_YdksxfN3Ifq-iuWxw.png" /><figcaption><em>Underfitted vs. Fit vs. Overfitted model (Source: </em><a href="https://www.v7labs.com/blog/overfitting"><em>v7labs</em></a><em>)</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*BVxHa9VyG6M0Gn0mqc8YPQ.png" /></figure><h4>Bias-Variance Tradeoff:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*z-awIv8taVsi7OE5asjTDw.png" /></figure><p>Bias is the difference between the average prediction of the model and the correct value that the model is trying to predict. A model that has high bias means the model is too simple to capture the true complexity of the data, and it over-assumes everything. And so there is always a high error between target value and predicted value, be it training set or testing set. So high bias refers to underfitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*KdkukDYeZlkztgFTULZp6w.png" /></figure><p>Variance refers to the degree of variability or sensitivity in a model’s predictions when trained on different subsets of the data. When the training subset changes, a model that tries to capture overly complex relationships between features and targets — relationships that might not truly exist — is more likely to overfit. This leads to high variance, where the model becomes overly responsive to small fluctuations or noise in the training data, resulting in poor generalization to new, unseen data.</p><p>Expected error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*2w4e4qVSV4yNigyzhb01mA.png" /></figure><p>So, Total error is the sum of Bias², variance, and the irreducible error.</p><p>Irreducible error is the error that can’t be reduced no matter what we do. It is a measure of the amount of noise in our data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sEx-BfzRScXJNiHTD0vgyw.png" /><figcaption>Bias-Variance Bulls eye diagram</figcaption></figure><p>The red dot represents the ideal prediction (the bullseye), and the blue dots represent the model’s predictions. <br>When all the arrows (predictions) are consistently far from the bullseye, no matter how you shoot, it indicates high bias. When the arrows are scattered all over the place, this is high variance.</p><p>Underfitting ⇉ Both training and testing set points will be grouped together but consistently further away from the ideal value. So, high bias and low variance occur.<br>Overfitting ⇉ Training set points will be clustered close to the ideal value, but testing set points will be scattered all over the place. So, low bias and high variance occur.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*sFDV3JHrmGR3wxEV85P7ug.png" /><figcaption>Source: <a href="https://medium.com/data-science/understanding-the-bias-variance-tradeoff-165e6942b229">Seema@Medium</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RjaalVZqtMYjD90vAqZanw.png" /><figcaption>Bias-Variance Tradeoff (Source: <a href="https://serokell.io/blog/bias-variance-tradeoff">Serokell</a>)</figcaption></figure><p>If our model is simple with few parameters, then it may have high bias and low variance. On the other hand, if our model has a large number of parameters, then it’s going to have high variance and low bias. So we need to find the balance without overfitting and underfitting the data.</p><p>As we try to reduce bias, variance will increase and vice versa. So, to keep total error low, we need to find an optimal solution that balances both bias and variance. This is called the bias-variance tradeoff.</p><h4>How to solve underfitting:</h4><p>We can try the following ways to solve underfitting:</p><ol><li>Use a More Complex Model: Upgrade from a linear model to a polynomial model or a neural network. Also, increasing the number of layers/neurons in deep learning can help.</li><li>Train for More Epochs: Try increasing the number of iterations/epochs, as the model might need more time to learn.</li><li>Reduce Regularization: Too much L1/L2 regularization can restrict the model. We will discuss regularization shortly.</li><li>Add More Relevant Features: This provides more information to make the model complex.</li></ol><h4>How to solve overfitting:</h4><p>We can try the following ways to solve overfitting:</p><ol><li>Simplify the Model: Use a less complex algorithm or fewer layers/parameters.</li><li><strong>Regularization:</strong> We will discuss this later in detail.</li><li>Add More Training Data: More data helps the model see more variation and reduces the chance of memorization.</li><li><strong>Data Augmentation:</strong> For images/text, slightly altering data (rotate, crop, rephrase) feels like new data without actually collecting more. Training the model on variations of the same dataset will increase the size of the training dataset.</li><li><strong>Early Stopping:</strong> Stopping training once the validation loss starts increasing. It prevents the model from continuing to memorize the training data. This will be discussed in detail in Neural Networks. Early Stopping is like stopping a student from overstudying till confusion hits.</li><li><strong>Dropout (for Neural Networks):</strong> We will discuss this in Neural Networks in detail.</li><li><strong>Cross-Validation:</strong> A training technique we will discuss in detail later.</li></ol><h4>Regularization:</h4><p>Regularization is a technique used in machine learning to prevent overfitting by adding a penalty term to the loss function. This penalty discourages the model from becoming too complex and helps ensure the model doesn’t rely too heavily on any one feature or noise in the data.</p><p>For example, suppose we train a model and it develops the relation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*rYDi6fo7LMQjcZmDL2T3dg.png" /></figure><p>It’s reasonable to expect that eye color should have little to no relation to weight. Through regularization, the coefficient w3​ associated with eye color will be penalized and driven toward zero (or a very small value), effectively reducing the impact of this feature. This process helps the model focus on more meaningful features like height and calorie intake.</p><p>Regularization limits the model’s flexibility, making it less likely to fit noise or irrelevant features, which helps it generalize better to unseen data.</p><h4>L1 Regularization:</h4><p>L1 regularization(LASSO regression: <strong>L</strong>east <strong>A</strong>bsolute <strong>S</strong>hrinkage and <strong>S</strong>election <strong>O</strong>perator) adds the absolute value of each coefficient as a penalty term to the loss function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*9eu48-vEiExJmOoo9nKmcw.png" /></figure><p><strong><em>λ</em></strong> represents the strength of the regularization and is a hyperparameter chosen by the user. Equally, if set to 0, it simply transforms into an Ordinary Least Squares (OLS) problem. <br><strong>We try to find the optimal value of w &amp; λ, which will minimize the loss.</strong></p><p>Increasing <strong><em>λ </em></strong>(or implementing strong regularization) would require the coefficient <strong>β</strong> to decrease proportionally, as we are still trying to minimize the function. As a result, at <strong><em>λ</em> = </strong>∞, the coefficients <strong>β</strong> would be shrunk to 0.</p><p>λ Very high ⇉ Underfitting<br>λ Very low ⇉ Overfitting</p><p><strong>Selecting λ:</strong> The regularization parameter λ is not learned from the data but is selected manually or automatically by using techniques like <strong>cross-validation</strong>. The general idea is to test various values of λ and choose the one that results in the best model performance (e.g., lowest test error). We will discuss k-fold validation later on. <br>Using k-fold cross-validation, you would:</p><ol><li>Train the model on the training data with various values of λ.</li><li>Calculate the error on the validation fold for each value of λ.</li><li><strong>The λ that minimizes the validation error would be selected as the best regularization parameter.</strong></li></ol><p><strong>Selecting w: </strong>The value of w is selected through gradient descent.</p><p><strong>Advantage of L1 Regularization:</strong></p><ol><li><strong>Featur e Selection (Sparsity): </strong>L1 regularization can shrink some weights to exactly zero, effectively removing irrelevant features from the model. The reason for why this happens is because if a feature is not relevant or correlated to the variable being predicted, the residual sum of squares (RSS) for that feature will be unusually higher than those of the other features. <br>Loss=RSS+Regularization term. If a feature does not help predict, then any value of w​ for that feature won’t help reduce the overall RSS much. So, when minimizing the entire cost function (RSS + regularization term), the optimizer decides: “This feature isn’t helping reduce error, but it’s increasing the penalty — might as well make its weight zero to reduce the cost.”<br>Therefore, to minimize the function, <em>w</em> = 0 would be required. This leads to sparse models, which are simpler and often more interpretable. Sparsity is helpful when there are many features, but only a few are truly important.</li><li><strong>Collinearity:</strong> If there are features that are also correlated to other features, L1 regularization would choose only one of these features to include in the model and reduce the other’s coefficient to 0.</li><li><strong>Simpler model</strong>: By penalizing large weights, L1 regularization prevents the model from fitting noise in the training data. Helps improve generalization on unseen/test data and makes the model simpler.</li></ol><h4>L2 Regularization:</h4><p>L2 regularization (Ridge Regression), adds the square of each coefficient as a penalty term to the loss function. This discourages the model from assigning too much importance (large weights) to any single feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*WDn2UAzBlXxMlJiCmtxfDA.png" /></figure><p>The effect of changing λ is almost similar to L1; however, the coefficients rarely shrink to exactly 0, whereas in L1, the coefficients would shrink to exactly 0.</p><p>Choosing λ and w is similar to the L1 regularization.</p><p><strong>Reason of squaring:<br>1. </strong>The squared term is differentiable everywhere, including at zero. This makes the gradient descent optimization smoother and more efficient, as we can compute gradients easily.<br><strong>2.</strong> Another reason is heavier Penalty on Larger Weights. Suppose there are two values, 0.5 &amp; 2. Their ratio is 4. After squaring them, the ratio would be 16. So the model is discouraged from giving too much importance to any one feature unless it’s truly needed.<br><strong>3. </strong>L2 regularization shrinks weights uniformly, encouraging the model to distribute the influence across features. This is useful when many features contribute a little, rather than just a few contributing a lot.</p><p><strong>Advantage of L2 Regularization:</strong></p><ol><li><strong>Maintains All Features: </strong>Unlike L1 (which can zero out weights), L2 tends to shrink weights but keeps all features in the model. Useful when all features carry some information, even if it’s small.</li><li><strong>Smooth Optimization: </strong>The squared penalty is differentiable and convex, making optimization with gradient descent efficient and stable.</li><li><strong>Collinearity:</strong> If there are features that are also correlated to other features, L2 regularization would choose all features but spread weights among them.</li></ol><p>L1 vs L2: The difference between the two types of regularization techniques are :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*Q53IWQIthn3kcSuurZ4iAQ.png" /></figure><h4>Elastic Net Regularization:</h4><p>Elastic Net Regularization prevents overfitting by combining the strengths of L1 (Lasso) and L2 (Ridge) regularizations.<br>Elastic Net adds both L1 and L2 penalties to the loss function</p><p><strong>Lasso (L1)</strong> tends to select only one feature from a group of highly correlated features and ignore the rest.<br><strong>Ridge (L2)</strong> distributes the coefficients among correlated features.<br><strong>Elastic Net</strong> combines both: it encourages sparsity (like Lasso) and handles multicollinearity (like Ridge).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*0yAgIM5vluztls-J4dY6Mg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*_CV8neQldqT3alZLTBJQ4Q.png" /></figure><p>Other optimization techniques will be discussed in other articles, especially in Neural Networks.</p><p>In most cases, overfitting is a more serious problem than underfitting, and so it is more important to learn how to solve it. Here we discussed the steps we can take to solve these two problems. Hope everything is clear. Thank you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acd5ead7891c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>