<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ashutosh on Medium]]></title>
        <description><![CDATA[Stories by ashutosh on Medium]]></description>
        <link>https://medium.com/@ashutoshs81127?source=rss-6a002b0d0b5d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*-eO2NNtYRmgD9DZi</url>
            <title>Stories by ashutosh on Medium</title>
            <link>https://medium.com/@ashutoshs81127?source=rss-6a002b0d0b5d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 16:35:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashutoshs81127/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why Modern Transformers Use RMSNorm Instead of LayerNorm]]></title>
            <link>https://medium.com/@ashutoshs81127/why-modern-transformers-use-rmsnorm-instead-of-layernorm-5f386be7156c?source=rss-6a002b0d0b5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f386be7156c</guid>
            <dc:creator><![CDATA[ashutosh]]></dc:creator>
            <pubDate>Tue, 18 Nov 2025 09:29:24 GMT</pubDate>
            <atom:updated>2025-11-18T09:29:24.905Z</atom:updated>
            <content:encoded><![CDATA[<p><em>And Why Bias Terms Are Disappearing</em></p><p>if you’ve looked at any modern Transformer architecture — LLaMA, PaLM, Chinchilla, T5 — you might’ve noticed something surprising:</p><p>➡️ <strong>LayerNorm is gone.</strong><br> ➡️ <strong>RMSNorm has taken its place.</strong><br> ➡️ <strong>Bias terms are disappearing too.</strong></p><p>On paper, the change looks tiny. In practice, it’s a major architectural shift that impacts model speed, memory efficiency, and training stability.</p><p>Let’s break down what’s going on.</p><h4>🔹 <strong>LayerNorm — What It Used to Do</strong></h4><p>The original <strong>Transformer</strong> used <strong>LayerNorm</strong>, defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*PtaGx38EmSwlx5enUT4fwA.jpeg" /></figure><p>Step-by-step, this means:</p><ol><li>Compute the <strong>mean</strong> of the activations xxx.</li><li>Subtract that mean: center around zero.</li><li>Compute the <strong>variance</strong> and normalize by the standard deviation.</li><li>Multiply by <strong>γ (gamma)</strong> to scale.</li><li>Add <strong>β (beta)</strong> to shift.</li></ol><p>This helped keep activations stable — not too large, not too small — and enabled deeper networks to train.</p><p>And for years, it worked perfectly well.</p><h4>🔹 <strong>RMSNorm — The Simpler Alternative</strong></h4><p>RMSNorm, introduced later, asks a simple question:</p><p><em>Do we actually need the mean subtraction?</em></p><p>Turns out… <strong>no</strong>.</p><p>RMSNorm drops two things:</p><p>❌ No mean subtraction<br> ❌ No bias term (β)</p><p>It normalizes only by the <strong>root mean square</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*aHf35SYFIzYfOMxYacMo6w.jpeg" /></figure><p>That’s it.</p><p>What’s surprising is that RMSNorm performs <strong>just as well</strong> as LayerNorm on most tasks.</p><p>So why bother with LayerNorm’s extra computation if the simpler version works fine?</p><h4>🔥 <strong>Why the Switch Happened</strong></h4><h4>1. It’s Faster</h4><p>Removing the mean subtraction and bias term results in:</p><ul><li>fewer arithmetic operations</li><li>fewer parameters to read/write</li><li>less memory movement</li></ul><p>Even though these operations seem tiny compared to big matrix multiplications, they matter a lot.</p><h4>A key insight from Ivanov et al. (2023):</h4><p>➡️ Matrix multiplies = 99.8% of FLOPs<br> ➡️ But LayerNorm &amp; similar ops = <strong>25% of runtime</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*KH6CX1dm7bxxq3AsAy0cBg.jpeg" /></figure><p>Why?<br> Because memory access costs more than FLOPs.</p><p>RMSNorm reduces this overhead</p><h4>2. It’s More Memory-Efficient</h4><p>Bias parameters seem small, but each one still requires:</p><ul><li>storage</li><li>loading from GPU memory</li><li>movement through hardware pipelines</li></ul><p>Multiply this across hundreds of layers and billions of parameters — and the cost adds up.</p><p>Modern training pipelines care deeply about memory bandwidth.<br> RMSNorm = fewer parameters → fewer memory transfers → faster models.</p><h4>3. It Even Helps Stability</h4><p>Surprising but true:<br> Models often train <strong>more stably</strong> without bias terms.</p><p>The largest LLMs (LLaMA, PaLM, Chinchilla) report better stability by:</p><ul><li>Removing bias in Linear layers</li><li>Using RMSNorm</li><li>Using Pre-Norm (LayerNorm before residual)</li></ul><p>These design patterns have become the “rules” for stable scaling.</p><h3>🔹 Empirical Evidence: Faster &amp; Sometimes Better</h3><p>Ablations from Narang et al. (2020):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*dJungUCTTLzGshZNBNTIuw.jpeg" /></figure><p>RMSNorm trains <em>slightly faster</em> and even reaches a <em>better final loss</em>.</p><p>Not huge — but it’s free performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f386be7156c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Pre-Norm Became the Default in Transformers]]></title>
            <link>https://medium.com/@ashutoshs81127/why-pre-norm-became-the-default-in-transformers-4229047e2620?source=rss-6a002b0d0b5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4229047e2620</guid>
            <dc:creator><![CDATA[ashutosh]]></dc:creator>
            <pubDate>Mon, 10 Nov 2025 08:46:34 GMT</pubDate>
            <atom:updated>2025-11-10T08:46:34.234Z</atom:updated>
            <content:encoded><![CDATA[<p>Understanding Pre-Norm vs Post-Norm</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*gGoc03mET85_hDaR4cV5Xg.jpeg" /></figure><h4>🧩 Why Pre-Norm Became the Default in Transformers (and What’s Next)</h4><p>In the world of large language models (LLMs), architectures like GPT, LLaMA, and Gemma share a common DNA — the Transformer. <br>But even within this shared structure, researchers experiment with small architectural tweaks that can have massive effects on training stability and performance.</p><p>One of the most interesting design debates revolves around a seemingly small detail:</p><p><strong>👉 Where should we place Layer Normalization in a Transformer block?</strong></p><p>This choice — known as <strong>Pre-Norm vs Post-Norm </strong>has shaped the evolution of every major Transformer model since 2017. <br>Let’s unpack what it means, why it matters, and what new innovations are emerging.</p><h4>🧠 Setting the Stage: Variations in Modern Transformers</h4><p>Over the past few years, many Transformer architectures have converged toward a LLaMA-like design — a stack of attention and feed-forward layers connected by residuals.</p><p>Still, there’s no perfect consensus on all the details. <br>Different models mix things up:<br><em>- Some use </em><strong><em>LayerNorm</em></strong><em>, while others prefer </em><strong><em>RMSNorm</em></strong><em> (a simpler normalization variant).<br>- Some use </em><strong><em>serial</em></strong><em> versus </em><strong><em>parallel</em></strong><em> layer layouts.<br>- Some tweak activation functions or hidden sizes.</em></p><p>But one choice stands out as <strong>nearly universal</strong> among modern models:<br>- The switch from <strong>Post-Norm</strong>(used in the original Transformer) to <strong>Pre-Norm</strong>.</p><h3>⚙️ Understanding Pre-Norm vs Post-Norm</h3><p>Let’s recall what a Transformer block does. Each block typically includes:<br><em>1. Multi-Head Self-Attention <br>2. Feed-Forward Network (MLP) <br>3. Residual Connections (skip connections) <br>4. Layer Normalization (for stability)</em></p><p>The question is: <strong>where do we apply LayerNorm?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*Unnf9CVL2HD_T4-IiibCwQ.jpeg" /></figure><h3>💡 A Visual Analogy</h3><p><strong>Post-Norm (original Transformer):</strong></p><pre>y = LayerNorm(x + Attention(x))</pre><p><strong>Pre-Norm (modern Transformer):</strong></p><pre>y = x + Attention(LayerNorm(x))</pre><h3>🚀 Why the Shift to Pre-Norm?</h3><p>When the <strong>“Attention Is All You Need” </strong>paper came out in 2017, it used Post-Norm. <br><em>But soon after, researchers noticed problems:<br>- Training deeper models became </em><strong><em>unstable</em></strong><em>. <br>- Gradients would </em><strong><em>explode</em></strong><em> or </em><strong><em>vanish</em></strong><em>. <br>- Training required </em><strong><em>careful learning rate warm-up</em></strong><em> schedules to avoid collapse.</em></p><p>Then came <strong>Pre-Norm</strong>, which flipped the order — and suddenly, training became much smoother:<br><em>- Models were </em><strong><em>more stable</em></strong><em>. <br>- They didn’t need learning rate warm-up tricks. <br>- Gradients flowed easily through deep networks.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ntT6NdL5u3COwL3wjAXJvQ.jpeg" /><figcaption>Pre-vs-post-norm, the data</figcaption></figure><p>Almost every major modern LLM — GPT-2, GPT-3, LLaMA, Falcon, Mistral — now uses <strong>Pre-Norm</strong>.</p><h3>📊 The Stability Argument</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gt0jePEY6VsGsJv_aligYA.jpeg" /></figure><h4>The key reason Pre-Norm helps comes down to <strong>gradient flow</strong>.</h4><p>In a Transformer, the <strong>residual connection</strong> acts like a clean “identity highway” through the network.<br>It helps<strong> gradients</strong> move smoothly from output back to input, preventing the vanishing gradient problem.</p><p>When you use<strong> Post-Norm</strong>, you normalize after adding that residual, which disrupts the identity connection.<br>When you use <strong>Pre-Norm</strong>, you normalize before the heavy computation — leaving the residual path untouched<em>.</em></p><p>As a result:</p><ul><li><strong>Pre-Norm</strong> → Gradients stay consistent and stable.</li><li><strong>Post-Norm</strong> → Gradients often blow up or fade away.</li></ul><h3>🧪 What Research Found</h3><p>Papers like <strong>Xiong et al. (2020)</strong> and <strong>Salazar &amp; Nguyen (2019) </strong>showed clear empirical results:</p><ul><li><strong>Pre-Norm</strong> training curves are smooth and stable.</li><li><strong>Post-Norm</strong> often shows <em>loss spikes</em> and <em>unstable gradient norms</em>.</li><li>With <strong>Pre-Norm,</strong> warmup schedules can be reduced or removed entirely.</li></ul><p>In short:<br><strong>Pre-Norm</strong> = <strong>stability without extra tricks.</strong></p><h3>🔄 The Latest Innovation: “Double Norm”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*ar8x-tRzW9npXquJfoXABg.jpeg" /></figure><p>Even though Pre-Norm dominates today, researchers haven’t stopped experimenting.<br>A recent trend introduces a twist: adding an <strong>extra LayerNorm</strong> at the end of each block.</p><p>This design — sometimes called <strong>Double Norm</strong> — looks like this:</p><ol><li>Apply LayerNorm before attention (like Pre-Norm).</li><li>Apply another LayerNorm <em>after</em> the block.</li></ol><p>Recent models adopting this:</p><ul><li><strong>Grok and Gemma 2 </strong>→ LayerNorm before <em>and</em> after each block.</li><li><strong>Olmo 2</strong> → LayerNorm only after each block.</li></ul><p>Early results suggest this approach might make <strong>very large models</strong> even more stable during training.</p><h3>⚠️ Why Not Put LayerNorm Inside the Residual?</h3><h4><em>“Why is LayerNorm inside the residual stream bad?”</em></h4><p>Here’s the intuition:</p><p>The <strong>residual stream</strong> carries an identity signal directly through the model — from the input to the top layer.<br>This path allows information (and gradients) to flow freely through dozens or even hundreds of layers.</p><p>If you insert LayerNorm <em>inside</em> that residual, you distort this clean identity connection.<br>That breaks the smooth gradient flow and leads to the very problems we’re trying to avoid — instability, exploding gradients, and loss spikes.</p><p><strong>Pre-Norm (modern):</strong></p><pre>def transformer_block(x):<br>    x = x + multihead_attention(layernorm(x))<br>    x = x + feed_forward(layernorm(x))<br>    return x</pre><p><strong>Double Norm (new idea):</strong></p><pre>def transformer_block(x):<br>    x = x + multihead_attention(layernorm(x))<br>    x = x + feed_forward(layernorm(x))<br>    x = layernorm(x)  # additional normalization at the end<br>    return x</pre><ul><li><a href="https://arxiv.org/abs/1910.05895">Transformers without Tears: Improving the Normalization of Self-Attention</a></li><li><a href="https://arxiv.org/abs/2002.04745">On Layer Normalization in the Transformer Architecture</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4229047e2620" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Choose the Right Embedding Model for Your LLM Application]]></title>
            <link>https://medium.com/@ashutoshs81127/how-to-choose-the-right-embedding-model-for-your-llm-application-7f5b257fd389?source=rss-6a002b0d0b5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f5b257fd389</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[cohere]]></category>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[sentence-transformers]]></category>
            <dc:creator><![CDATA[ashutosh]]></dc:creator>
            <pubDate>Sun, 14 Jul 2024 10:58:06 GMT</pubDate>
            <atom:updated>2024-07-14T10:58:35.566Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>What are embeddings and embedding models?</strong></h4><p>An embedding is an array of numbers (a vector) representing a piece of information, such as text, images, audio, video, etc. Together, these numbers capture semantics and other important features of the data. The immediate consequence of doing this is that semantically similar entities map close to each other while dissimilar entities map farther apart in the vector space</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*regQ0iSaBuRYsQgvEIn0FA.png" /></figure><p>In today’s AI applications, embeddings are typically created using large language models (LLMs) that are trained on a massive corpus of data and use cutting-edge algorithms to learn complex semantic relationships in the data.</p><h4>Open Source Embedding Model</h4><ul><li>Easier to work with Locally.</li><li>Free (if processing locally) and may be cheaper for paid inference.</li><li>May not be supported by any provider for cloud inference.</li><li>It is useful when you have a lot of local data you want to process.</li><li>You don’t want to share your data with an embedding API.</li><li>You want more control of your search pipeline.</li></ul><p>e.g [GTE-Base,GTE-Large,GTE-Small,E5-Small,MultiLingual,RoBERTa (2022),MPNet V2,E5-Base]</p><h4>Closed Source Embedding Model</h4><ul><li>The closed Source model has very fast inference (for every token cost is added)</li><li>May perform better in certain use cases(Multilingual, instruct embeddings).</li><li>May have free trials that fit your use case (Voyage AI).</li></ul><p>Useful When</p><ul><li>You don’t have much data (&gt;50M Tokens) or are willing to pay to embed your data.</li><li>Don’t want to host your infrastructure.</li></ul><p>e.g [OpenAI, Google, Anthropic, Microsoft]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*tqr_uZioakx5BZl9A9cPJw.png" /></figure><h4>The Massive Text Embedding Benchmark</h4><p>(MTEB) Leaderboard is a good starting point for getting an overview of the current landscape of the wide range of proprietary and open-source text embedding models. For each embedding model, the MTEB lists various metrics, such as the model size, memory usage, embedding dimensions, maximum number of tokens, and its score for tasks such as retrieval, summarization, clustering, Reranking, classification, etc.</p><ul><li>Evaluated on 56 datasets across 8 tasks.</li><li>More Than 200 models on the leaderboard.</li><li>Contain both closed source and open source.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*osRJ4_93ildk-OZvjffHVg.png" /><figcaption><a href="https://huggingface.co/spaces/mteb/leaderboard">https://huggingface.co/spaces/mteb/leaderboard</a></figcaption></figure><p><strong>Here are a few considerations for choosing the best model for your application:</strong></p><p><strong>Task: </strong>On the top of the MTEB Leaderboard, you will see various tabs. You first have to decide whether you want an all-rounder or if you can find a task (classification, clustering, retrieval, summarization, etc.) that matches your specific use case. For example, if you are building an RAG application, you might want to look closer at the “Retrieval” task. You might also want to refine your selection if you have more specific needs, such as a specific language (e.g., English, Chinese, French, Polish) or domain (e.g., law).</p><p><strong>Max tokens:</strong> This is the maximum number of tokens that can be converted into a single embedding. For common RAG applications, a good chunk size for an embedding is typically about a single paragraph of text or less. In this case, models with max tokens of 512 should be sufficient. However, there will be special cases where you need to embed longer source texts that require models with a larger context window.</p><p><strong>Embedding Dimensionality: </strong>The number of dimensionality output by the model.</p><h4>Smaller Dimensions Embedding Model</h4><ul><li>Faster Computation and searches.</li><li>Reduced memory footprint.</li><li>Larger compression could mean less accurate Semantic representation.</li></ul><h4>Larger Dimensions Embedding Model</h4><ul><li>Higher expressiveness better captures relationships, patterns, and features.</li><li>Slow searches and larger memory footprint.</li></ul><h4>Model Size</h4><p>These give you an idea of the computational resources required to run the model. While retrieval performance scales with model size, it is important to note that model size also directly impacts latency. Additionally, it is worth mentioning that larger models can also overfit and thus underperform in production. Thus, you want to aim for a good latency-performance trade-off for a production setup. Ideally, start with a small, lightweight model to be able to build a baseline upon which you can iterate quickly. Once your pipeline works, you can switch out the model with a bigger and more performant one later.</p><h4>General Vs Domain Specific</h4><ul><li>Most Embedding models are General, trained on a wide variety of data.</li><li>Benchmark covers a wide variety of data as well take them with a grain of salt when considering domain-specific use cases.</li><li>Think of Medical, Scientific, or legal domain data.</li><li>A general embedding model could be at the middle /high school level of understanding of these topics.</li><li>Fine-tune a Model could be at a PhD level of understanding on the given topic.</li><li>It needs training data to fine-tune an embedding model.</li></ul><h4>Most Common Embedding Models</h4><h4><strong>Cohere Multilingual Embedding Model</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*K8gsOnCVkQbeKI_Xt5AxeA.png" /><figcaption><a href="https://cohere.com/embed">https://cohere.com/embed</a></figcaption></figure><ul><li>100+ Languages are Supported.</li><li>Best in Class Performance for Multilingual Search.</li><li>Closed Source.</li></ul><p><strong>Use Cases</strong></p><ul><li>Multi-lingual Semantic search: Enhance search results across languages.</li><li>Aggregate customer Feedback: Manage and organize feedback from various languages in international operations.</li></ul><h4><strong>Open AI Embeddings</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*f59Wbl0AerrwTHaQB8W4cg.png" /><figcaption><a href="https://openai.com/api/pricing/">https://openai.com/api/pricing/</a></figcaption></figure><ul><li>Simple API, with great documentation.</li><li>Excellent performance.</li><li>Relatively cheap, until a certain scale is reached.</li><li>Closed Source.</li><li>Dimension can be reduced without the loss to much performance.</li><li>Open AI docs recommended cosine similarity for most use cases.</li></ul><h4><strong>Voyage AI Embedding</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*Ydqb7zz8CSafZ77Ak1Wlow.png" /><figcaption><a href="https://docs.voyageai.com/docs/embeddings">https://docs.voyageai.com/docs/embeddings</a></figcaption></figure><ul><li>Top of MTEB leaderboard(May 2024).</li><li>Instruction-tuned embedding offers(may result in better performance for certain tasks).</li><li>1024 embeddings dimensions(much lower than any similarity performing models).</li><li>50M tokens embedded free with a credit card.</li><li>Can start without a credit card.</li><li>Fast Inference.</li></ul><h4><strong>Sentence Transformer</strong></h4><ul><li>Python framework for sentence, text, and image embedding.</li><li>Works with the most embedding models on HuggingFace(a platform that hosts opensource ML models)</li><li>Typically pretty slow, but robust(support a wide variety of models)</li></ul><h4>Fast Embed</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*gON6piGdJ42tKRx5gCPI3w.png" /><figcaption><a href="https://qdrant.github.io/fastembed/examples/Supported_Models/#supported-text-embedding-models">https://qdrant.github.io/fastembed/examples/Supported_Models/#supported-text-embedding-models</a></figcaption></figure><ul><li>Python library for very fast embedding generation.</li><li>Very lightweight and faster to install than a sentence transformer.</li><li>Limited in the no of models it supports.</li><li>Supports GPU acceleration.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f5b257fd389" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>