<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Navin Aananthan on Medium]]></title>
        <description><![CDATA[Stories by Navin Aananthan on Medium]]></description>
        <link>https://medium.com/@navinaananthan?source=rss-b9399f5c1ce2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*934ZBBXK9_CeGoEjWkiOcw.jpeg</url>
            <title>Stories by Navin Aananthan on Medium</title>
            <link>https://medium.com/@navinaananthan?source=rss-b9399f5c1ce2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:45:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@navinaananthan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Complete Guide to Optimizers in Deep Learning]]></title>
            <link>https://medium.com/@navinaananthan/a-complete-guide-to-optimizers-in-deep-learning-46b8b34f1ca1?source=rss-b9399f5c1ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/46b8b34f1ca1</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[adagrad-optimizer]]></category>
            <category><![CDATA[deep-learning-optimizers]]></category>
            <category><![CDATA[rmsprop]]></category>
            <category><![CDATA[nesterov-momentum]]></category>
            <dc:creator><![CDATA[Navin Aananthan]]></dc:creator>
            <pubDate>Sat, 07 Feb 2026 04:18:21 GMT</pubDate>
            <atom:updated>2026-02-07T04:18:21.037Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Optimisers are not just formulas — they are intelligent strategies that guide neural networks through complex loss landscapes</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZkGVnePUNeWwL6jo.png" /></figure><h3>1. Why Do We Need Optimizers?</h3><p>Deep learning is about <strong>minimizing a loss function</strong> by tuning millions (sometimes billions) of parameters.</p><p>Basic gradient descent updates weights as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*s0Papw9BUf7i_kz0tQcDLA.png" /></figure><p><strong>But basic gradient descent has major problems:</strong></p><ul><li>❌ Very slow convergence</li><li>❌ Zig-zag motion in narrow valleys</li><li>❌ Sensitive to learning rate</li><li>❌ Gets stuck near saddle points</li><li>❌ Poor scaling for deep networks</li></ul><p><strong>Optimizers solve this by:</strong></p><ul><li>Speeding up convergence</li><li>Reducing oscillations</li><li>Automatically adapting learning rates</li><li>Improving numerical stability</li></ul><h3>2. Visualizing Optimization—Loss Surfaces &amp; Contour Maps</h3><p>Optimization is easier to understand if we <strong>visualize loss as terrain</strong>.</p><ul><li>High regions → high error</li><li>Low regions → minimum error</li></ul><p><strong>Contour Maps</strong></p><ul><li>Closely spaced contours → steep slopes</li><li>Widely spaced contours → flat regions</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W2vE_HSl_97CI7Pd.png" /></figure><p><strong>Problem</strong></p><ul><li>Gradient descent oscillates in steep directions</li><li>Moves very slowly in flat directions</li></ul><p>Optimizers reshape this motion.</p><h3>3. Exponentially Weighted Moving Average (EWMA)</h3><p>This is the <strong>core mathematical idea behind Momentum, RMSProp, and Adam</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*nbUCHJTMxv4xIikggDxqaw.png" /></figure><p><strong>Intuition:</strong></p><ul><li>Gives <strong>more importance to recent values</strong></li><li>Smooths noisy signals</li><li>Retains memory of past trends</li></ul><p>📌 Used to smooth:</p><ul><li>Gradients</li><li>Squared gradients</li><li>Velocity terms</li></ul><h3>4. Momentum-Based Gradient Descent</h3><p><strong>Problem with Standard GD:</strong></p><ul><li>Zig-zag motion in narrow valleys</li><li>Slow traversal of flat regions</li></ul><p><strong>Solution: Add Momentum</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*x1cpdkybFT6l5tFimnQVGg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*OPBqki30nzKRso3yWA6Daw.jpeg" /></figure><p><strong>Intuition:</strong></p><p>Think of a <strong>ball rolling downhill</strong>:</p><ul><li>Keeps gaining speed in consistent direction</li><li>Smooths oscillations</li><li>Moves faster through flat regions</li></ul><p><strong>Benefits:</strong></p><ul><li>Faster convergence</li><li>Reduced oscillations</li><li>Better saddle point escape</li></ul><h3>5. Nesterov Accelerated Gradient (NAG)</h3><p>Momentum improves speed but sometimes <strong>overshoots minima</strong>.</p><p>NAG introduces <strong>look-ahead correction</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*PzhQijJ7he4gkJLl0HQMzg.png" /></figure><p><strong>Key Idea:</strong></p><blockquote>“Look before you leap.”</blockquote><p>Instead of computing the gradient at the current point, NAG computes the gradient <strong>after taking a momentum step</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ppF-_Rhh53VuyZUy.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/0*7YqmWhAXjFSYeYQJ" /></figure><h3>Adaptive Learning Rate Optimisers</h3><p>Instead of using <strong>one learning rate</strong>, adaptive methods give <strong>each parameter its own learning rate</strong>.</p><p>This solves:</p><ul><li>Feature imbalance</li><li>Sparse data learning</li><li>Gradient scaling issues</li></ul><h3>7. AdaGrad—Adaptive Gradient Algorithm</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*rV0aH6nAG1RlsTWHsJ5aDQ.png" /></figure><p><strong>Key Behavior:</strong></p><ul><li>Infrequent features → <strong>larger updates</strong></li><li>Frequent features → <strong>smaller updates</strong></li></ul><p><strong>Intuition:</strong></p><blockquote><em>Learn fast from rare events, slow down for common ones.</em></blockquote><p><strong>Major Problem:</strong></p><ul><li>Learning rate decays continuously</li><li>Eventually learning <strong>stops</strong></li></ul><h3>9. RMSProp—Root Mean Square Propagation</h3><p>RMSProp solves <strong>AdaGrad’s decaying learning rate problem</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*aJ3NkxLVzs8BFwz2-j4IzA.png" /></figure><p><strong>Key Idea:</strong></p><blockquote><em>Forget old gradients gradually.</em></blockquote><p><strong>Benefits:</strong></p><ul><li>Stable learning rate</li><li>Faster convergence</li><li>Works well for RNNs</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*GbVox3vRB_iAWp-v.gif" /></figure><h3>10. Adam — Adaptive Moment Estimation</h3><p>Adam uses <strong>two running averages</strong>:</p><p><strong>10.1 First Moment (Momentum)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*XNcrcePWFyHAA1Y7B9cNfg.png" /></figure><p><strong>10.2 Second Moment (RMSProp)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*IeGUHp3rUejn6ETz61l2Ug.png" /></figure><p><strong>10.3 Bias Correction</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*i1WGRpCfndpr4c1EpMk3Jg.png" /></figure><p><strong>10.4 Final Adam Update</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*SgNxX6cj5A-cHX-23S0ywA.png" /></figure><p>Adam combines:</p><ul><li>Momentum → first moment</li><li>RMSProp → second moment</li><li>Bias correction</li></ul><p><strong>Why Adam is powerful:</strong></p><ul><li>Smooth updates</li><li>Fast convergence</li><li>Works well for sparse data</li><li>Minimal tuning needed</li></ul><p>Optimizers are the backbone of training deep neural networks. While gradient descent gives us a direction to minimize loss, modern optimizers make this process <strong>faster, more stable, and more efficient</strong>. From basic <strong>SGD</strong> to advanced methods like <strong>Momentum, RMSProp, AdaGrad, and Adam</strong>, each optimizer addresses specific challenges such as <strong>slow convergence, vanishing gradients, exploding gradients, noisy updates, and sparse data learning</strong>.</p><p>In conclusion, <strong>optimizers transform theoretical learning into practical, scalable deep learning systems</strong>, enabling us to train models with millions or even billions of parameters efficiently. A solid grasp of optimizers is therefore essential for anyone aiming to build robust and high-performing neural networks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46b8b34f1ca1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Performance in Spring Boot: How Caching Reduces Network Calls and Database Operations]]></title>
            <link>https://medium.com/@navinaananthan/optimizing-performance-in-spring-boot-how-caching-reduces-network-calls-and-database-operations-cc95b79b446c?source=rss-b9399f5c1ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc95b79b446c</guid>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[cache]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[cacheable]]></category>
            <category><![CDATA[caching]]></category>
            <dc:creator><![CDATA[Navin Aananthan]]></dc:creator>
            <pubDate>Sat, 22 Nov 2025 07:33:37 GMT</pubDate>
            <atom:updated>2025-11-22T07:34:22.791Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Caching isn’t just an optimization—sometimes, it’s a necessity to make development efficient and systems usable.</blockquote><h3><strong>What Is Caching?</strong></h3><p>Caching is a mechanism where frequently accessed data is temporarily stored in a fast-access storage layer, typically in memory. Instead of fetching the same data repeatedly from a database or external service, the application can return the cached version almost instantly.</p><blockquote>If the data doesn’t change often, why fetch it again and again? Store it once, reuse it many times.</blockquote><p>Caching helps reduce:</p><ul><li>Repeated database queries and Database load</li><li>Network latency</li><li>CPU and I/O overhead</li><li>Response time for frequently accessed data</li></ul><p>This makes applications faster, more efficient, and capable of handling higher loads—especially when the database or service is remote.</p><h3>Types of Caching</h3><ol><li><strong>ConcurrentMapCache or in-memory caching:</strong> Data is stored directly in the system’s main memory, making access to this data extremely fast. Spring Boot can manage in-memory cache using its default cache providers like ConcurrentMapCache or integrate with stronger solutions like EhCache or Hazelcast.</li><li><strong>Distributed caching: </strong>Useful in microservices architectures where instances need to share a common cache state. Solutions like Redis or Memcached are typically used in these scenarios, allowing instances to maintain cache coherence across a distributed system.</li><li><strong>Disk-based caching: </strong>This involves storing cache data on a disk. This is slower than in-memory caching but can be useful for caching large datasets that do not fit into memory.</li></ol><h3>How Spring Boot Facilitates Caching</h3><p>Spring Boot provides a seamless caching mechanism that can be integrated with just a few annotations, making it extremely developer-friendly. Here are the key components:</p><ul><li><strong>@EnableCaching:</strong> This annotation is used to enable caching capability in a Spring Boot application.</li><li><strong>@Cacheable:</strong> This is used on methods to indicate that the result of the method should be stored in the cache. The next time the method is called with the same arguments, the result is retrieved from the cache instead of executing the method.</li><li><strong>@CacheEvict:</strong> This annotation is used to remove one or more entries from a cache. This is useful when cached data is no longer valid and needs to be updated.</li><li><strong>@CachePut:</strong> This annotation updates the cache with the result of the method execution, without interfering with the method execution itself.</li></ul><h3>How Is Caching Implemented in Java?</h3><p>Before we look at Spring Boot’s built-in caching support, it helps to understand how caching is traditionally implemented in plain Java.</p><p>At its core, caching is just storing data in memory using a structure like a dictionary, where the key represents the input and the value represents the computed or fetched result.</p><pre>import java.util.HashMap;<br>import java.util.Map;<br><br>public class SimpleCacheExample {<br><br>    private Map&lt;Long, String&gt; cache = new HashMap&lt;&gt;();<br><br>    public String getUser(Long id) {<br>        // Check if data already exists in cache<br>        if (cache.containsKey(id)) {<br>            System.out.println(&quot;Returning data from cache...&quot;);<br>            return cache.get(id);<br>        }<br><br>        // Simulate a database call<br>        System.out.println(&quot;Fetching from database...&quot;);<br>        String user = &quot;User_&quot; + id;<br><br>        // Store result in cache for future requests<br>        cache.put(id, user);<br>        return user;<br>    }<br><br>    public static void main(String[] args) {<br>        SimpleCacheExample example = new SimpleCacheExample();<br><br>        example.getUser(1L); // DB call<br>        example.getUser(1L); // Cache hit<br>    }<br>}</pre><p>In this example:</p><ul><li>The <strong>first call</strong> fetches data (simulating a DB call).</li><li>The <strong>second call</strong> returns the same data instantly from memory.</li></ul><p>This illustrates the core idea behind caching:<br> <strong>don’t repeat work if the result is already available and still valid.</strong></p><h3>Spring Boot Caching with @Cacheable</h3><p>Spring Boot makes caching incredibly simple thanks to its built-in caching abstraction. Instead of manually managing maps or writing boilerplate logic, you can enable caching with just a few annotations.</p><p>The first step is to enable caching in your Spring Boot application:</p><pre>@EnableCaching<br>@SpringBootApplication<br>public class Application {<br>    public static void main(String[] args) {<br>        SpringApplication.run(Application.class, args);<br>    }<br>}</pre><p>Once caching is enabled, you can apply the @Cacheable annotation to any method whose result should be cached.</p><p>Here’s an example:</p><pre>@Service<br>public class UserService {<br><br>    @Cacheable(value = &quot;users&quot;)<br>    public User getUserById(Long id) {<br>        System.out.println(&quot;Fetching from DB...&quot;);<br>        return repository.findById(id).orElse(null);<br>    }<br>}</pre><h4>What happens here?</h4><ul><li>The <strong>first time</strong> the method is called with a particular ID, Spring executes the method and stores the result in the cache under the key &quot;users::id&quot;.</li><li>The <strong>next time</strong> the same method is called with the same ID, Spring skips the method execution and returns the cached result instantly.</li></ul><p>So instead of repeatedly hitting the database—especially in scenarios where data doesn’t change often—caching helps serve responses much faster.</p><h4>@CachePut — Update Cache Without Skipping Method Execution</h4><p>@CachePut forces the method to run and updates the cache with the new value.<br> It’s useful when data changes—for example, updating a user profile.</p><pre>@Service<br>public class UserService {<br><br>    @CachePut(value = &quot;users&quot;, key = &quot;#user.id&quot;)<br>    public User updateUser(User user) {<br>        System.out.println(&quot;Updating DB and cache...&quot;);<br>        return repository.save(user);<br>    }<br>}</pre><ul><li>The method always executes</li><li>Cache entry is updated with the new result</li></ul><h4>@CacheEvict — Remove Entries From Cache</h4><p>@CacheEvict is used when data is deleted or becomes outdated.</p><pre>@Service<br>public class UserService {<br><br>    @CacheEvict(value = &quot;users&quot;, key = &quot;#id&quot;)<br>    public void deleteUser(Long id) {<br>        System.out.println(&quot;Deleting user and evicting cache...&quot;);<br>        repository.deleteById(id);<br>    }<br>}</pre><p>You can also clear the entire cache:</p><pre>@CacheEvict(value = &quot;users&quot;, allEntries = true)<br>public void clearUserCache() {<br>    System.out.println(&quot;All user cache entries cleared!&quot;);<br>}</pre><h3>Available Cache Providers in Spring Boot</h3><p>Spring Boot supports multiple caching providers — from simple in-memory caches to distributed ones suitable for microservices.</p><pre>| Cache Provider    | Type                  | Best For                            |<br>| ------------------| --------------------- | ----------------------------------- |<br>| Simple (default)  | In-memory             | Small projects, Dev/Testing         |<br>| Caffeine          | In-memory (advanced)  | High performance, eviction policies |<br>| Ehcache           | File + memory         | Local persistence, enterprise use   |<br>| Hazelcast         | Distributed           | Clusters &amp; microservices            |<br>| Redis             | Distributed in-memory | Large scale, cloud + multi-instance |</pre><p>Caching may seem like a small feature, but it can dramatically improve application performance—especially when working with remote databases or frequently accessed data. In my case, what started as a simple need to speed up local development led me to explore Spring Boot’s caching capabilities, and the improvement was immediate.</p><p>With just a few annotations like @Cacheable, &quot;and,&quot; Spring provides an elegant and powerful way to reduce repetitive database calls, minimize latency, and boost responsiveness. Adding the right cache provider—whether an in-memory option like Caffeine or a distributed store like Redis—can scale this approach even further for production environments.</p><p>In the end, caching isn’t just an optimization—it’s a smart strategy for building efficient, scalable, and user-friendly systems. And in scenarios where every millisecond counts, caching can make all the difference.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc95b79b446c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Model Context Protocol (MCP): The Missing Link Between AI Models and the Real World]]></title>
            <link>https://medium.com/@navinaananthan/model-context-protocol-mcp-the-missing-link-between-ai-models-and-the-real-world-911cb4380d57?source=rss-b9399f5c1ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/911cb4380d57</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[mcp-client]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[model-context-protocol]]></category>
            <category><![CDATA[mcp-protocol]]></category>
            <dc:creator><![CDATA[Navin Aananthan]]></dc:creator>
            <pubDate>Sat, 20 Sep 2025 05:34:51 GMT</pubDate>
            <atom:updated>2025-09-20T05:34:51.400Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Even the most sophisticated models are constrained by their isolation from data — trapped behind information silos and legacy systems.</em></p><p>AI models are powerful — but they don’t know everything. Out of the box, a language model can generate text, answer questions, and summarize documents, but it struggles when it needs real-time data, secure access to APIs, or integration with enterprise tools.</p><p>This is where the <strong>Model Context Protocol (MCP)</strong> comes in.</p><h3>What Is MCP, Really?</h3><p>The <strong>Model Context Protocol</strong>, introduced by Anthropic, is an <strong>open standard</strong> that allows AI models to connect with external data sources, APIs, and tools in a safe, structured way.</p><p>Think of MCP as the <strong>HTTP for AI models</strong> — just like the web needed a common protocol to let browsers talk to servers, AI needs a standard way to talk to the outside world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*M931JnCyBfzoPWYp" /></figure><h3>What makes MCP essential?</h3><p>Large Language Models (LLMs) are amazing at understanding and generating text — but they have a blind spot: they can only work with what they already know. When you need them to handle up-to-date information from your documents, databases, or apps, they hit a wall. To be truly useful, AI needs not just understanding, but access to the right context — and the ability to act on it, like updating a spreadsheet, sending an email, or creating a task.</p><p>Until now, connecting AI to external tools was messy. Developers had to write custom code or create separate integrations for every service, making solutions fragile and hard to maintain.</p><p>This is where <strong>Model Context Protocol (MCP)</strong> changes the game. Introduced by Anthropic in November 2024, MCP is an open standard that makes it easy for AI to connect with platforms like GitHub, Google Drive, or Asana. Once connected, AI can read documents, create tasks, or update records across multiple systems — all using simple language commands.</p><p>In short, MCP turns AI from a static knowledge engine into a dynamic, context-aware assistant, capable of working seamlessly with the tools and data you use every day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*R9yAJ1GnWIBiEjef93yx9w.png" /></figure><p>You Might Be Wondering: Isn’t MCP Just Another Form of RAG?</p><p><strong>Retrieval-Augmented Generation (RAG)</strong></p><ul><li><strong>What it does:</strong> Enhances an LLM by retrieving relevant information (from a vector database, knowledge base, or documents) at query time.</li><li><strong>Scope:</strong> Focused mainly on <strong>textual knowledge retrieval</strong>.</li><li><strong>Goal:</strong> Reduce hallucinations and give the LLM access to <em>up-to-date</em> or <em>domain-specific</em> data.</li><li><strong>Typical use case:</strong> A chatbot that pulls answers from your company’s knowledge base.</li></ul><p><strong>Model Context Protocol (MCP)</strong></p><ul><li><strong>What it does:</strong> Provides a <strong>standardized interface</strong> for LLMs/AI agents to access <em>all kinds of context and tools</em> — not just knowledge, but also actions (e.g., fetch files, query APIs, update a document, send an email).</li><li><strong>Scope:</strong> Much broader than RAG. It’s about <strong>plugging the model into the ecosystem</strong> of data and applications.</li><li><strong>Goal:</strong> Make it easy, scalable, and open-standard to “wire up” models to <em>any external source or tool</em>.</li><li><strong>Typical use case:</strong> An AI assistant that not only answers questions but also edits your Notion docs, pulls data from Salesforce, and sends you a summary email — all via MCP integrations.</li></ul><h3>Why does MCP matter?</h3><p>Depending on where you sit in the ecosystem, MCP can have a range of benefits.</p><ul><li>Developers: MCP reduces development time and complexity when building, or integrating with, an AI application or agent.</li><li>AI applications or agents: MCP provides access to an ecosystem of data sources, tools and apps which will enhance capabilities and improve the end-user experience.</li><li>End-users: MCP results in more capable AI applications or agents which can access your data and take actions on your behalf when necessary.</li></ul><h3>MCP vs AI Agents</h3><p>You might be wondering: aren’t <strong>AI agents</strong> already doing all this? Well, yes and no. AI agents are like digital helpers — they can perform tasks, make decisions, and interact with data. The catch is, connecting them to new tools or data sources usually means writing custom code or building one-off integrations. It’s doable, but messy and inconsistent.</p><p>This is where <strong>MCP</strong> comes in. Think of it as a universal adapter for AI. MCP doesn’t act on its own — it’s the bridge that lets AI agents seamlessly access documents, databases, APIs, or other tools. With MCP, your AI agent can focus on solving problems, while the protocol ensures it has all the context it needs to do so, without the headache of custom integrations.</p><p><strong>AI agents are the doers, MCP is the enabler.</strong> Together, they turn static AI into a dynamic, context-aware assistant.</p><p>In the coming blogs, we’ll dive into how MCP actually works in practice — showing step by step how to connect AI agents to real-world tools and data sources. By the end, you’ll see how simple it can be to turn your AI from a static assistant into a fully context-aware, action-ready powerhouse.</p><h3>References</h3><p>[1]Anthropic <a href="https://www.anthropic.com/news/model-context-protocol">https://www.anthropic.com/news/model-context-protocol</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=911cb4380d57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Types of RAG: Part 2]]></title>
            <link>https://medium.com/@navinaananthan/types-of-rag-part-2-91a07db7f8b8?source=rss-b9399f5c1ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/91a07db7f8b8</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[information-retrieval]]></category>
            <category><![CDATA[speculative-rag]]></category>
            <dc:creator><![CDATA[Navin Aananthan]]></dc:creator>
            <pubDate>Sat, 30 Aug 2025 07:09:33 GMT</pubDate>
            <atom:updated>2026-07-07T15:41:14.089Z</atom:updated>
            <content:encoded><![CDATA[<p>Having covered the first three types of Retrieval-Augmented Generation (RAG) in Part One, we’ll now move on to explore the next three approaches. These methods build on the same core principles but introduce new ways to retrieve and integrate information, offering fresh perspectives on how RAG can be applied to real-world tasks.</p><h3>1. Speculative RAG</h3><p>Speculative retrieval-augmented generation (RAG) is an innovative framework designed to improve the efficiency and accuracy of RAG systems by dividing the generation process into two distinct steps:</p><ol><li><strong>Drafting:</strong> The first step involves a smaller, specialized language model known as the RAG Drafter. This model generates multiple answer drafts from different subsets of retrieved documents. Each subset is carefully selected to represent diverse perspectives, minimizing redundancy and enhancing the quality of drafts.</li><li><strong>Verification:</strong> The drafts generated by the RAG Drafter are then passed to a larger, generalist LM, known as the RAG Verifier. This model evaluates the drafts, focusing on selecting the most accurate and reliable one based on the rationale provided. The verification process uses the language modeling capabilities of the generalist LM to ensure that the final output is both accurate and contextually relevant.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tE3vCQWjUGYTigjU.png" /></figure><ol><li><strong>Specialist RAG Drafter:</strong> Instead of overwhelming the big LLM, a smaller “drafter” model takes the lead. It looks at the question (Q) and the documents (1, 2…) and creates multiple draft answers (α1, α2, α3…). Each draft is based on a different subset of the documents.</li><li><strong>Rationales:</strong> The drafter also provides explanations (β1, β2, β3…) for why it chose those specific documents for each draft. Think of it like showing its work.</li><li><strong>Evaluation:</strong> The main LLM (the “Generalist LM”) now steps in. It looks at each draft answer <em>and</em> its explanation. It then picks the best answer (A) based on this evaluation.</li></ol><h3>2. Fusion RAG</h3><p>If standard RAG is like typing a single question into a search engine and hoping you used the right keywords, <strong>Fusion RAG (or RAG-Fusion)</strong> is like having a research assistant who rewrites your question five different ways, searches all of them, and synthesizes the best results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o7EcS3JzGNmhWGLq.png" /></figure><ul><li><strong>Multi-Query Generation:</strong> The system sends your original question to an LLM and asks it to generate 3 to 5 variations. For example, “How many vacation days?” becomes “What is the PTO policy?”, “How is employee leave calculated?”, and “Annual time off limits.”</li><li><strong>Parallel Retrieval:</strong> The system takes all of those new queries and runs a separate vector search for <em>every single one</em> at the same time. This results in multiple lists of retrieved documents.</li><li><strong>Reciprocal Rank Fusion (RRF):</strong> Now the system has several lists of documents, and it needs to merge them. It uses an algorithm called <strong>Reciprocal Rank Fusion</strong>. Instead of looking at raw search scores, RRF looks at the <em>rank</em> of the documents. If a specific HR document shows up as the #1 or #2 result across three different query variations, RRF gives it a massive mathematical boost.</li><li><strong>Final Generation:</strong> The noise is filtered out, and only the highest-consensus documents from the RRF merger are sent to the final LLM to generate your answer.</li></ul><h3>3. Agentic RAG</h3><p><strong>Agentic RAG </strong>is all about injecting intelligence and autonomy into the RAG framework. It’s like giving a regular RAG system a major upgrade, transforming it into an autonomous agent capable of making its own decisions and taking actions to achieve specific goals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3pBAqxPdTpb0CwHl.jpeg" /></figure><ul><li><strong>Query Input</strong>: The process starts with a clear objective or query from the user, such as explaining a concept, offering tailored advice, or solving a complex problem. This query serves as the basis for the model’s actions.</li><li><strong>Search and Retrieve</strong>: The model accesses a knowledge base or database, which goes through a preprocessing phase to structure the information efficiently for later use. The preprocessed data is stored in a <strong>Vector Store</strong> or <strong>Knowledge Graph</strong> for easy retrieval.</li><li><strong>Question Answering Check</strong>: After generating the initial response, the model checks if the question has been sufficiently answered. This is a key decision point in the process. If <strong>Yes</strong>, the process continues with the final response being provided to the user. If <strong>No</strong>, the process moves forward to the next phase.</li><li><strong>Agent Intervention and Actions</strong>: If the initial response doesn’t answer the query adequately, the model takes action through an <strong>Agent</strong>. The Agent may perform additional tasks, such as using external <strong>Tools</strong> or invoking further <strong>Actions</strong> to gather more relevant information. LLM determines autonomously which information to retrieve and which actions to take to achieve the goal. This can be done by defining a Chain of Thought prompt for the LLM in that step.</li><li><strong>Iterative Process and Dynamic Adjustment</strong>: The process is dynamic and iterative. The agent continuously assesses the model’s progress and adjusts actions in real-time, refining its strategy for better results. This step involves evaluating the intermediate results and taking corrective measures, such as re-querying or altering the retrieval approach.</li><li><strong>Completion of the Goal</strong>: Once the model confirms that it has met the objective or solved the problem, it produces a final output or response, which is delivered to the user. If further adjustments are needed, the process may loop again until the query is fully resolved.</li></ul><p>So far, we’ve explored the different types of RAG at a high level. In the upcoming blog series, we’ll dive deeper into each type in detail. If you have any questions in the meantime, feel free to reach out to me at <a href="https://topmate.io/navin_aananthan">https://topmate.io/navin_aananthan</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91a07db7f8b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Fundamental Analysis in Stock Market]]></title>
            <link>https://medium.com/@navinaananthan/introduction-to-fundamental-analysis-in-stock-market-5f326d1a3888?source=rss-b9399f5c1ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f326d1a3888</guid>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[stock-analysis]]></category>
            <category><![CDATA[fundamental-analysis]]></category>
            <category><![CDATA[long-term-investment]]></category>
            <dc:creator><![CDATA[Navin Aananthan]]></dc:creator>
            <pubDate>Sat, 30 Aug 2025 06:52:35 GMT</pubDate>
            <atom:updated>2025-08-30T06:52:35.331Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Fundamental Analysis (FA) is a holistic approach to study a business. When an investor wishes to invest in a business for the long term (say 3–5 years), it becomes essential to understand the business from various perspectives. It is critical for an investor to separate the daily short term noise in the stock prices and concentrate on the underlying business performance.</em></blockquote><p>Several Indian companies — such as <strong><em>Infosys, TCS, Page Industries, Eicher Motors, Bosch India, Nestlé India, and TTK Prestige</em></strong> — have consistently delivered over 20% compounded annual growth rate (CAGR) for more than a decade.</p><p>To put this into perspective, at a 20% CAGR, an investment doubles in roughly 3.5 years. Companies like Bosch India have even touched 30% CAGR, showcasing how quickly wealth can grow when investing in fundamentally strong businesses.</p><p><strong>The takeaway:</strong> long-term investments in quality companies can create wealth at an impressive pace.</p><p>Here are long term charts of Bosch India, Eicher Motors, and TCS Limited that can set you thinking about long term wealth creation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BCMxVpge-piYhRouilNYtw.png" /><figcaption>TCS, Eicher, Bosch</figcaption></figure><p>You might wonder if I’m being selective by only showing charts that look impressive. What about companies like <strong>Suzlon Energy, Reliance Power, and Sterling Biotech</strong>? How do their long-term charts stack up? Let’s take a look.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iWUG56QR9WHu42q23HkJpQ.png" /></figure><p>These are just three examples of <strong>wealth destructors</strong> — and there are many more in the Indian markets.</p><p>The real challenge for any investor lies in <strong>distinguishing investment-grade companies that consistently create wealth from those that erode it</strong>. Successful, wealth-creating businesses tend to share certain key attributes, while wealth destructors exhibit telltale warning signs that an astute investor can recognize.</p><p><strong>Fundamental analysis</strong> is the tool that equips you with the conviction to invest for the long term. It helps you identify these defining characteristics, enabling you to choose companies with strong wealth-creating potential while avoiding those that could damage your portfolio.</p><p><strong>Essential Skills to Become a Fundamental Analyst:</strong></p><ul><li>Understanding the basic financial statements</li><li>Understand businesses concerning the industry in which it operates</li><li>Basic arithmetic operations such as addition, subtraction, division, and multiplication</li></ul><h3>TA vs FA</h3><p>Technical Analysis (TA) and Fundamental Analysis (FA) serve different purposes, but together they create a powerful investment strategy.</p><p><strong>TA focuses on timing the market</strong> to capture quick, short-term returns by identifying better entry and exit points. <strong>FA focuses on building wealth</strong> by investing in fundamentally strong companies for the long term.</p><p>Take Eicher Motors as an example. An investor who bought the stock in 2006 based on FA saw minimal returns until 2010, when the real rally began. During those quiet years, using TA for short-term trades could have generated additional returns while waiting for the long-term story to play out.</p><p>This balance forms the <strong>Core-Satellite Strategy</strong>:</p><ul><li><strong>Core portfolio (60%)</strong> — Invested long term in high-quality companies, expected to grow steadily at 12–15% CAGR.</li><li><strong>Satellite portfolio (40%)</strong> — Actively traded using TA in equities, futures, or options, aiming for 10–12% annual absolute returns.</li></ul><p>By combining FA and TA, you get the best of both worlds — steady wealth creation and additional short-term gains.</p><h3>Essential Tools for Fundamental Analysis</h3><p>The tools required to perform effective fundamental analysis are simple — and most are freely available. Here’s what you need:</p><ol><li><strong>Company’s Annual Report</strong> — This is your primary source of information. Everything you need to analyze a business is contained here, and you can download it directly from the company’s website at no cost.</li><li><strong>Industry Data</strong> — Understanding the industry’s performance helps you benchmark the company against its peers. Basic industry statistics are often freely available on industry association websites.</li><li><strong>Access to News</strong> — Staying updated on company- and industry-specific developments is crucial. A reliable business newspaper or tools like Google Alerts can help you track important news in real time.</li><li><strong>Microsoft Excel</strong> — While not free, Excel is extremely useful for performing calculations, building models, and organizing financial data efficiently.</li></ol><p>With just these four tools, you can conduct research comparable to institutional analysts. The key is not the number of tools you use, but how logically and simply you approach the analysis. Even at the institutional level, the best research is straightforward, structured, and focused.</p><h3>References</h3><p>[1] Zerodha Varsity <a href="https://zerodha.com/varsity/chapter/introduction-fundamental-analysis/">https://zerodha.com/varsity/chapter/introduction-fundamental-analysis/</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f326d1a3888" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Types of RAG: Part 1]]></title>
            <link>https://medium.com/@navinaananthan/types-of-rag-part-1-010799502469?source=rss-b9399f5c1ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/010799502469</guid>
            <category><![CDATA[self-rag]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[types-of-rag]]></category>
            <category><![CDATA[corrective-rag]]></category>
            <dc:creator><![CDATA[Navin Aananthan]]></dc:creator>
            <pubDate>Tue, 26 Aug 2025 09:54:15 GMT</pubDate>
            <atom:updated>2026-07-07T15:29:41.652Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Retrieval-Augmented Generation (RAG) — A Quick Recap</strong><br>RAG is a hybrid approach that combines the strengths of information retrieval and text generation to produce factually accurate, context-aware responses. Instead of relying solely on a model’s internal knowledge, RAG dynamically fetches relevant documents from an external source and fuses them with the query before generating an answer.</p><p><strong>Key idea:</strong></p><ul><li><strong>Retrieval (Finding the Facts):</strong> When you ask a question, the system does not immediately send it to the large language model. Instead, it sends your query to a search engine (usually a <strong>vector database).</strong> This database contains thousands or millions of documents (like company policies, medical records, or up-to-date news articles) that have been converted into numbers (embeddings). The database calculates the mathematical similarity between your question and its documents to find the top most relevant pieces of text.</li><li><strong>Augmentation (Building the Prompt): </strong>Once the system retrieves the relevant facts, it combines them with your original question. It essentially builds a new prompt with the facts. e.g., “Based on this context: <strong>[Retrieved Facts]</strong>, please answer: <strong>[Original Question]</strong>.”</li><li><strong>Generation (Writing the Answer):</strong> Finally, this massive, augmented prompt is sent to the LLM. Because the LLM now has the exact facts it needs right in its prompt, it can generate an accurate, specific answer based on your private or recent data, rather than relying on its outdated or generalized training memory.</li></ul><p>This reduces hallucinations, improves factual accuracy, and makes models adaptable to large or frequently changing datasets.</p><h3><strong>1. Basic RAG</strong></h3><p>In a basic RAG setup, the system takes a user’s question, searches a vector database or knowledge graph for relevant context, and uses those retrieved facts to generate the final answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/0*Z7oDbb4aTBafqOuK.jpeg" /></figure><h3>2. Corrective RAG</h3><p><strong>Corrective Retrieval-Augmented Generation (CRAG)</strong> is a RAG variant that adds a <em>quality-control and repair loop</em> around retrieval.</p><p>Instead of blindly passing the top-k chunks to the LLM, CRAG first <strong>evaluates whether the retrieved evidence is good enough</strong>; if it isn’t, it <strong>takes corrective actions</strong> (e.g., rewrite the query, run a web search, expand sources) before the LLM answers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tn_-Mu6x6DLXtyXm2sy9eQ.png" /></figure><h4>Why do we need it?</h4><p>Basic RAG fails when:</p><ul><li>The <strong>index is stale or incomplete</strong> (new facts aren’t in your KB),</li><li>the <strong>retriever returns off-topic chunks</strong> for ambiguous or long-tail questions,</li><li>long documents include <strong>lots of irrelevant text</strong> that distracts the LLM.</li></ul><h4>Implementation</h4><ul><li><strong>Retrieve</strong> — Get top-k chunks from your vector/hybrid search.</li><li><strong>Grade</strong> — Use a small LLM to check if the chunks are relevant (yes/no/score).</li><li><strong>Correct</strong> — If score is low: rewrite query, fetch more results, or add web search.</li><li><strong>Refine</strong> — Split passages into smaller “knowledge strips” and filter irrelevant ones.</li><li><strong>Generate</strong> — Pass only the high-quality strips to your main LLM for answering.</li><li><strong>Tune</strong> — Adjust relevance threshold and balance accuracy vs. latency/cost.</li></ul><h3>3. Self RAG</h3><p><strong>Self-RAG (Self-Reflective Retrieval-Augmented Generation)</strong> is a variant of RAG where the <strong>LLM itself decides whether it needs retrieval, what to retrieve, and how to use the retrieved information</strong>.<br>Instead of a fixed pipeline (retrieve → feed → generate), the model:</p><ul><li><strong>Reflects</strong> on its own knowledge vs. what’s missing,</li><li><strong>Generates retrieval queries</strong> if needed,</li><li><strong>Assesses and integrates retrieved evidence</strong>, and</li><li><strong>Produces an answer while citing sources or reasoning about confidence.</strong></li></ul><p>This approach was introduced by Meta AI (Shuster et al., 2023).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*lNIt5VqiLyC5ix2t.png" /></figure><h4>Why do we need it?</h4><ul><li><strong>Reduces hallucinations</strong>: model learns when to trust external docs over its own memory.</li><li><strong>Improves efficiency</strong>: skips retrieval when it isn’t needed.</li><li><strong>Better control</strong>: can provide citations, uncertainty scores, or multi-source verification.</li></ul><h4>Implementation</h4><ul><li><strong>Retriever</strong> — fetch documents relevant to the user query.</li><li><strong>Relevance Check</strong> — confirm if the retrieved documents actually match the query.</li><li><strong>Initial Answer Generation</strong> — produce an answer using both the query and the documents.</li><li><strong>Hallucination Check</strong> — verify that the answer is fully grounded in the retrieved evidence.</li><li><strong>Rewrite Query (if needed)</strong> — improve the query if results are poor or hallucinated.</li><li><strong>Correctness Check</strong> — ensure the answer completely solves the original question.</li><li><strong>Final Result</strong> — return the answer only after it passes both hallucination and correctness checks.</li></ul><p>We’ve explored these types of RAGs so far — the remaining ones will be covered in the next part of this series.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=010799502469" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retrieval-Augmented Generation Overview]]></title>
            <link>https://medium.com/@navinaananthan/retrieval-augmented-generation-overview-937c852cfbf3?source=rss-b9399f5c1ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/937c852cfbf3</guid>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[retrieval-generation]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[information-retrieval]]></category>
            <dc:creator><![CDATA[Navin Aananthan]]></dc:creator>
            <pubDate>Wed, 13 Aug 2025 04:55:03 GMT</pubDate>
            <atom:updated>2025-08-13T04:57:40.604Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Origins of Retrieval-Augmented Generation (RAG)</h3><p>In 2020, Meta researchers, led by Patrick Lewis, introduced a technique called Retrieval-Augmented Generation (RAG). This approach combines the strengths of natural language generation (NLG) with information retrieval (IR) to enhance language model performance.</p><p>RAG is designed to address a key limitation of large language models: while they possess broad general knowledge, they can fall short when tasks require real-time information, domain-specific insights, or rigorous fact-checking. By integrating an external retrieval mechanism, RAG ensures responses are more precise, contextually relevant, and up to date, effectively bridging the gap between static model knowledge and dynamic information needs. You can refer to the original paper <a href="https://arxiv.org/abs/2005.11401">here</a> for a detailed explanation of the method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z9e_g2q7S9hNXAxVlrT-Jw.png" /></figure><h3>What is Retrieval-Augmented Generation (RAG)?</h3><p>Retrieval augmented generation (RAG) combines the advanced text-generation capabilities of GPT and other large language models with information retrieval functions to provide precise and contextually relevant information. This approach improves language models’ ability to understand and process user queries by integrating the latest and most relevant data.</p><p>In general, <a href="https://medium.com/@navinaananthan/transformers-part-1-a-high-level-overview-de0c8d47f684">large language models</a> are really good at doing many natural language processing tasks. Their generated text is sometimes straight to the point, accurate, and just what the user needs. But a lot of the time, this isn’t the case.</p><p>You’ve most probably encountered a situation where you ask a question to ChatGPT, and you feel like something is wrong with the generated output, no matter how confident the model seems to be. Then you go to check the information yourself and find out that GPT actually “lied.” This phenomenon of large language models is called <a href="https://machinelearningmastery.com/a-gentle-introduction-to-hallucinations-in-large-language-models/">hallucination</a>. Let’s think about why this happens.</p><p>General-purpose language models are pre-trained on vast amounts of data from everywhere. But this doesn’t mean that the model knows the answer to every question. General LLMs fall short in cases like up-to-date or relevant information, domain-specific context, fact-checking, etc. That’s why they’re called general-purpose and need the assistance of other techniques that are widely implemented to make LLMs more versatile.</p><h3>How does Retrieval-Augmented Generation (RAG) work?</h3><p>RAG is about feeding language models with necessary information. Instead of asking LLM directly (like in general-purpose models), we first retrieve the very accurate data from our knowledge library that is well maintained and then use that context to return the answer. When the user sends a query (question) to the retriever, we use vector embeddings (numerical representations) to retrieve the requested document. Once the needed information is found from the vector databases, the result is returned to the user. This largely reduces the possibility of hallucinations and updates the model without retraining the model, which is a costly process. Here’s a very simple diagram that shows the process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EO_LGT_D-JI84XGN89q21g.jpeg" /></figure><p>RAG brings together four key components:</p><ul><li><strong>Embedding model:</strong> This is where documents are turned into vectors, or numerical representations, which make it easier for the system to manage and compare large amounts of text data.</li><li><strong>Retriever:</strong> Think of this as the search engine within RAG. It uses the embedding model to process a question and fetch the most relevant document vectors that match the query.</li><li><strong>Reranker (optional):</strong> This component takes things a step further by evaluating the retrieved documents to determine how relevant they are to the question at hand, providing a relevance score for each one.</li><li><strong>Language model:</strong> Finally, this part of the system takes the top documents provided by the retriever or reranker, along with the original question, and crafts a precise answer.</li></ul><p>In practical terms, RAG is mainly famous in applications that require up-to-date and contextually accurate content. It bridges the gap between general language models and external knowledge sources, paving the way for improved content generation, question-answering, personalized recommendations, and more.</p><h3><strong>Applications of RAG</strong></h3><p><strong>1. Conversational AI</strong></p><ul><li><strong>Customer Support:</strong> Retrieves manuals and FAQs for precise answers.</li><li><strong>Personal Assistants:</strong> Pulls real-time data like news or weather.</li></ul><p><strong>2. Automated Content Generation</strong></p><ul><li><strong>Journalism:</strong> Produces articles with current facts and figures.</li><li><strong>Marketing:</strong> Creates product copy grounded in accurate specs.</li></ul><p><strong>3. Question-Answering and Research</strong></p><ul><li><strong>Education:</strong> Delivers detailed explanations backed by reliable sources.</li><li><strong>Science:</strong> Summarizes research papers for quick insights.</li></ul><p><strong>4. Domain-Specific Uses</strong></p><ul><li><strong>Healthcare:</strong> Suggests research-backed diagnoses or treatments.</li><li><strong>Customer Service:</strong> Gives agents quick access to policies and histories.</li><li><strong>Education:</strong> Helps teachers build richer learning content.</li></ul><p><strong>5. Specialized Applications</strong></p><ul><li><strong>Legal Research:</strong> Fetches case law and statutes for drafting.</li><li><strong>Translation:</strong> Provides context-aware, culturally nuanced output.</li></ul><h3>Limitations and Challenges of RAG</h3><p>While powerful, RAG faces <strong>technical and data-driven hurdles</strong>:</p><p><strong>1. Complexity</strong></p><ul><li>Integrating retrieval and generation increases development effort.</li><li>Scaling large document stores is challenging.</li></ul><p><strong>2. Performance Issues</strong></p><ul><li><strong>Latency:</strong> Retrieval slows real-time responses.</li><li><strong>Context Limits:</strong> Can’t always fit all required information.</li><li><strong>Retrieval Errors:</strong> Poor search results harm output quality.</li></ul><p><strong>3. Data Risks</strong></p><ul><li><strong>Bias:</strong> Sources may introduce or amplify bias.</li><li><strong>Data Quality:</strong> Outdated or unreliable sources reduce accuracy.</li><li><strong>Privacy:</strong> Sensitive data needs strict safeguards.</li></ul><p><strong>4. Needed Improvements</strong></p><ul><li><strong>Better Integration:</strong> Smoother link between retrieval and generation.</li><li><strong>Smarter Retrieval:</strong> More accurate, relevant context.</li><li><strong>Adaptive Learning:</strong> Learns from retrieval successes/failures.</li><li><strong>Efficient Indexing:</strong> Faster search with lower cost.</li></ul><p><strong>5. Future Directions</strong></p><ul><li><strong>Cross-Modal Retrieval:</strong> Combining text with images or video.</li><li><strong>Continuous Learning:</strong> Self-improvement without retraining.</li><li><strong>Interactive Retrieval:</strong> Allows clarification during search.</li><li><strong>Domain Adaptation:</strong> Specialized models for medical or legal use.</li><li><strong>Personalization &amp; Grounding:</strong> Tailored and fact-checked responses.</li></ul><h3>Potential Future Enhancements:</h3><ul><li><strong>Personalization: </strong>Integrating user profiles and historical interactions to personalize responses, making RAG models more effective in customer service and recommendation systems.</li><li><strong>Knowledge Grounding:</strong> Using external knowledge bases not just for retrieval but also for grounding the responses in verifiable facts, which is crucial for educational and informational applications.</li><li><strong>Efficient Indexing: </strong>Employing more efficient data structures and algorithms for indexing the database to speed up retrieval and reduce computational costs.</li></ul><p>In conclusion, Retrieval Augmented Generation (RAG) is a powerful technique that harnesses the synergy between vector databases and large language models (LLMs) to generate human-like, knowledge-grounded responses. By seamlessly integrating information retrieval from vector databases with the generative capabilities of LLMs, RAG enables the creation of accurate and relevant content that incorporates both factual knowledge and contextual understanding.</p><h3>References</h3><p>[1] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … Kiela, D. (2020). <em>R</em><a href="https://arxiv.org/abs/2005.11401"><em>etrieval-Augmented Generation for Knowledge-Intensive NLP Tasks</em></a> [Preprint]. <strong>arXiv</strong>. Accepted at NeurIPS 2020. arXiv:2005.11401. Originally submitted on 22 May 2020; revised version on 12 April 2021.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=937c852cfbf3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformers Part 4: Decoders]]></title>
            <link>https://medium.com/@navinaananthan/transformers-part-3-decoders-aaac049be771?source=rss-b9399f5c1ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/aaac049be771</guid>
            <category><![CDATA[generative-model]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <category><![CDATA[transformer-decoder]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[encoder-decoder]]></category>
            <dc:creator><![CDATA[Navin Aananthan]]></dc:creator>
            <pubDate>Fri, 25 Jul 2025 10:21:02 GMT</pubDate>
            <atom:updated>2025-07-27T18:05:19.579Z</atom:updated>
            <content:encoded><![CDATA[<p>In the Transformer architecture, the <strong>decoder</strong> plays a critical role in generating outputs, particularly in <strong>sequence-to-sequence tasks</strong> like machine translation, text generation, and summarization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*cdzo_0ItQ35cfr7A.gif" /></figure><p>A <strong>Transformer decoder</strong> takes the encoded information from the encoder (which has processed the input sequence) and generates the output sequence <strong>step-by-step</strong>, attending both to previously generated outputs and the encoder’s representations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*FTQXCiHxxsKZBtT7" /></figure><h3>1. Output Embeddings in the Decoder</h3><p>Just like the encoder, the decoder uses <strong>embeddings</strong> to convert each token (from the <strong>shifted target sequence</strong>) into a vector representation.</p><h4>Process:</h4><ul><li>The decoder takes the <strong>shifted target tokens</strong> (e.g., [&quot;&lt;start&gt;&quot;, &quot;I&quot;, &quot;am&quot;, ...]).</li><li>These tokens are passed through an <strong>embedding layer</strong> (learned during training), converting them to dense vectors.</li><li>This gives the decoder a rich, learned representation of each token.</li></ul><p>So if your vocabulary has 10,000 words and you’re using 512-dimensional embeddings, each token becomes a 512-dimvector.</p><h3>2. Positional Encoding</h3><p>Since the Transformer has <strong>no recurrence or convolution</strong>, it needs a way to understand the <strong>order of tokens</strong>. That’s where <strong>positional encodings</strong> come in.</p><p>The decoder uses <strong>exactly the same kind of positional encoding</strong> as the encoder:</p><ul><li>Can be <strong>fixed</strong> (e.g., sinusoidal) or <strong>learned</strong>.</li><li>Added to the output embeddings <strong>before</strong> being fed into the first decoder layer.</li></ul><h3>3. Masked Multi-Head Self-Attention</h3><p>This is similar to the self-attention mechanism in the encoder but with a crucial difference: it prevents positions from attending to subsequent positions, which means that each word in the sequence isn’t influenced by future tokens.</p><p>For instance, when the attention scores for the word “are” are being computed, it’s important that “are” doesn’t get a peek at “you”, which is a subsequent word in the sequence.</p><p>Masked Self-Attention just keeps calculating similarities by looking at itself and the words that came before it, but not after, until it gets to the end of the input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gXwBW7zYHJhWsrSU.png" /></figure><p>This masking ensures that the predictions for a particular position can only depend on known outputs at positions before it. The mask is used because, when the masked scores are softmaxed, the negative infinities are zeroed out, leaving no attention scores for subsequent tokens.</p><p>The first multi-headed attention layer’s calculation of the attention scores only differs in this masking. The mask is still being applied to several heads in this layer before being concatenated and passed through a linear layer for additional processing. A masked output vector with instructions on how the model should pay attention to the decoder’s input is the result of the initial multi-headed attention.</p><h3>3. Encoder-Decoder Multi-Head Attention (Cross-Attention)</h3><p>In the second multi-headed attention layer of the decoder, we see a unique interplay between the encoder and decoder’s components. Here, the outputs from the encoder take on the roles of both queries and keys, while the outputs from the first multi-headed attention layer of the decoder serve as values.</p><p>This setup effectively aligns the encoder’s input with the decoder’s, empowering the decoder to identify and emphasize the most relevant parts of the encoder’s input.</p><p>Following this, the output from this second layer of multi-headed attention is then refined through a pointwise feedforward layer, enhancing the processing further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FHdrskjsstoAXvdn.png" /></figure><h3>4. Linear Classifier and Softmax for Generating Output Probabilities</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hd8DxSoEZk212Ytt.png" /></figure><p>The journey of data through the transformer model culminates in its passage through a final linear layer, which functions as a classifier.</p><p>The size of this classifier corresponds to the total number of classes involved (number of words contained in the vocabulary). For instance, in a scenario with 1000 distinct classes representing 1000 different words, the classifier’s output will be an array with 1000 elements.</p><p>This output is then introduced to a softmax layer, which transforms it into a range of probability scores, each lying between 0 and 1. The highest of these probability scores is key, its corresponding index directly points to the word that the model predicts as the next in the sequence.</p><h3>4. Predict the Next Token</h3><p>At each position t, the decoder predicts the <strong>next token</strong> based on:</p><ul><li>All previous tokens (positions &lt; t)</li><li>Encoder outputs (in case of encoder-decoder models like T5 or GPT-style with context)</li></ul><pre>| Step | Decoder Input                           | Predicts   | Output Token        |<br>| ---- | --------------------------------------- | ---------- | ------------------- |<br>| 1    | `[&lt;start&gt;]`                             | `&quot;I&quot;`      | → append `&quot;I&quot;`      |<br>| 2    | `[&lt;start&gt;, &quot;I&quot;]`                        | `&quot;am&quot;`     | → append `&quot;am&quot;`     |<br>| 3    | `[&lt;start&gt;, &quot;I&quot;, &quot;am&quot;]`                  | `&quot;hungry&quot;` | → append `&quot;hungry&quot;` |<br>| 4    | `[&lt;start&gt;, &quot;I&quot;, &quot;am&quot;, &quot;hungry&quot;]`        | `&quot;now&quot;`    | → append `&quot;now&quot;`    |<br>| 5    | `[&lt;start&gt;, &quot;I&quot;, &quot;am&quot;, &quot;hungry&quot;, &quot;now&quot;]` | `&quot;.&quot;`      | → append `&quot;.&quot;`      |<br><br>The next predicted token is &lt;eos&gt; → stop generation</pre><p>The Transformer <strong>decoder</strong> is a powerful module designed to <strong>generate sequences one token at a time</strong>, such as in translation, text generation, or summarization. It takes the <strong>partially generated output</strong> and learns how to <strong>predict the next token</strong> using attention mechanisms and learned language patterns.</p><p>I’d also like to mention that I’m happy to answer any questions you may have about Transformers. In my next blog, I plan to address some common questions readers often have — such as how Transformer outputs are generated and how backpropagation works. Feel free to let me know if there are other topics you’d like me to cover.</p><h3>References</h3><p>[1] Ashish Vaswani, et al. and team, <a href="https://arxiv.org/abs/1706.03762">Attention is all you need</a>, 2017.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aaac049be771" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformers Part 3: Encoders]]></title>
            <link>https://medium.com/@navinaananthan/transformers-part-3-encoders-12a952c55112?source=rss-b9399f5c1ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/12a952c55112</guid>
            <category><![CDATA[feed-forward-networks]]></category>
            <category><![CDATA[multi-head-attention]]></category>
            <category><![CDATA[self-attention-mechanism]]></category>
            <category><![CDATA[normalization]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Navin Aananthan]]></dc:creator>
            <pubDate>Mon, 14 Jul 2025 05:24:08 GMT</pubDate>
            <atom:updated>2025-07-25T10:21:28.144Z</atom:updated>
            <content:encoded><![CDATA[<p>Now, it’s time to zoom in on the <strong>encoder</strong>, the core component that helps transformers understand and represent input sequences.</p><p>If you’re wondering how machines turn words into meaning, how attention actually works, and why encoders are so good at capturing context — this post is for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*96XyEIAJWGPA9s70.png" /></figure><h3>What Does the Encoder Do?</h3><p>In the Transformer model, the <strong>encoder</strong> is responsible for transforming the input sequence into a rich, high-dimensional representation that captures both local and global context.</p><p>For example, in a translation task, the sentence “The bat flew out of the cave” is processed by the encoder into a set of vectors that reflect the meanings of the words, their relationships, and the sentence structure. The decoder then uses this to generate the translation.</p><p>But how does the encoder build such a meaningful representation?</p><h3><strong>1. Multi‑Headed Self‑Attention</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/0*EJApVPd-czcTh60J.png" /></figure><p>To compute <strong>contextual representations</strong> for each token in the sequence by <strong>looking at all other tokens</strong>, using attention weights.</p><p>So each word’s final embedding becomes aware of surrounding words — not just its own position.</p><h4><strong>What Happens Inside Self-Attention</strong></h4><blockquote>Self-attention allows a model to look at other <strong>positions in the same sequence</strong> to gather context before deciding how to represent a word.</blockquote><p>In simple terms:</p><ul><li>For each word in a sentence, self-attention helps determine <strong>how much focus it should give to every other word</strong> (including itself).</li><li>It outputs a <strong>contextualized vector</strong> for each word that blends information from the whole sentence.</li></ul><p>Take the sentence:<br> <strong>“The animal didn’t cross the street because it was too tired.”</strong></p><p>When interpreting the word <strong>“it”</strong>, we need context. <strong>“It”</strong> likely refers to <strong>“the animal”</strong>.</p><p>Self-attention helps the model figure out that “it” is <strong>most related to “animal”</strong>, not “street”—by assigning more weight to “animal” when encoding “it”.</p><h3>Components of the Self-Attention Block</h3><p>At its core, the self-attention mechanism operates on three fundamental components: <strong>Queries (Q), Keys (K), and Values (V)</strong>.</p><p><strong>Queries (Q)</strong><br>The <strong>Query</strong> is essentially the representation of the word (or token) that the model is currently focusing on. Think of it as the model’s way of asking, <strong>How relevant is this word in the context of the entire sequence?</strong> For each word in the sequence, the model generates a query vector, which is then used to evaluate its relationship with other words in the sequence.</p><p><strong>Keys (K)</strong><br>The <strong>Keys</strong> represent all the words in the sequence, including the word currently being focused on. Each word in the sequence has a corresponding key vector. These keys serve as reference points that the query vector is compared against. In essence, the key vectors help the model determine how closely related each word in the sequence is to the word currently under focus.</p><p><strong>Values (V)</strong><br>The <strong>Values</strong> are what the model ultimately uses to construct its understanding of the sequence. Each word in the sequence is associated with a value vector, which holds the contextual information or the “<strong>meaning</strong>” of the word. Once the model has determined how much attention to give to each word (based on the comparison between the queries and keys), it uses the value vectors to build a weighted representation of the context.</p><p>In summary, the queries, keys, and values are the essential building blocks of the self-attention mechanism. They allow the model to dynamically evaluate the importance of each word in the context of the entire input sequence, leading to a more nuanced and contextually aware understanding of language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/0*OkwLHjTuJ38PtQuJ.png" /></figure><p><strong>Step 1: Compute the Query, Key, and Value Matrices</strong></p><p>The first step in the self-attention process is to generate the <strong>Query (Q), Key (K), and Value (V)</strong> matrices.<br>Each word in the input sequence is transformed into three different vectors: a query vector, a key vector, and a value vector.<br>These vectors are computed by multiplying the input matrix X (which contains the word embeddings) by three different weight matrices<strong> W_Q , W_K , and W_V</strong> , which are learnable parameters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/0*S0Fmsmg6Ov9zrBUE.png" /></figure><p><strong>Step 2: Calculate Attention Scores</strong></p><p>Next, the model calculates how much attention each word should pay to every other word in the sequence. This is done by taking the dot product of the query matrix Q with the transpose of the key matrix K :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/0*AR7dTGP1VHJs8OJb.png" /></figure><p>This operation produces a matrix of scores that reflect the similarity or compatibility between words, indicating how much one word should influence another in the context of the input sequence.</p><p><strong>Step 3: Scaling the Scores</strong></p><p>In scenarios involving high-dimensional matrices, the raw dot product scores can result in large variances, which may destabilize the training process. To address this, the scores are scaled by the square root of the dimensionality of the key vectors sqrt{d_k}:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/0*nd9iafYJlMp7r7fw.png" /></figure><p>This scaling helps prevent the gradients from becoming too large during backpropagation, mitigating the risk of the vanishing gradient problem.</p><p><strong>Step 4: Apply the Softmax Function</strong></p><p>The scaled scores are then passed through a softmax function to normalize them into a probability distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/0*3daXJ3AGlxiPYww0.png" /></figure><p>The softmax function ensures that all the attention weights sum to 1, making it easier to interpret these values as probabilities that dictate how much focus should be placed on each word in the sequence.</p><p><strong>Step 5: Weight the Values</strong></p><p>Finally, the attention weights are used to compute a weighted sum of the value vectors V :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/0*2eYZ84CVSUOGKXAh.png" /></figure><p>This step results in a set of output vectors, each representing a word in the sequence but now enriched with contextual information from the entire input sequence.</p><h4>What Happens Inside Multi-Head Attention</h4><p>Multi-Head Attention is an advanced extension of the Self-Attention mechanism used within the Transformer architecture. This mechanism enhances the model’s ability to focus on different parts of an input sequence simultaneously, thereby capturing a variety of perspectives and relationships within the data.</p><p>In essence, Multi-Head Attention involves <strong>repeating the self-attention</strong> process multiple times, with each repetition using different linear projections of the input data. This allows the model to attend to different aspects of the sequence in parallel, making the final representation more robust and contextually rich.</p><p>In <strong>multi-head attention</strong>, we divide the input embedding vector (d_model) across multiple heads so that each head works on a smaller part of the information.<br>If there are h heads, each head gets a subvector of size d_k = d_model / h.</p><p>This allows each head to learn different types of relationships (like grammar, meaning, or position) in <strong>parallel</strong>, without increasing the overall size or cost.</p><p>After all heads compute attention, their outputs are <strong>concatenated</strong> stacked horizontally back to size d_model and passed through a final linear layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/0*E5yiHfhuuqputK0g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/0*LYOzEKMjMB7iMonj.png" /></figure><p>After processing through the individual attention heads, the outputs of all heads are concatenated together. This step combines the multiple perspectives that each head has focused on</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/0*SCyahDNxsnk-A7aV.png" /></figure><p>Finally, the concatenated output is passed through another linear projection using a weight matrix to produce the final output. This step integrates all the different perspectives into a single, unified representation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/0*xEf7FRZxzQENVYmd.png" /></figure><h3>2. Add &amp; Normalization</h3><p>The <strong>Add &amp; Norm</strong> layer, as the name suggests, consists of two parts: add and norm. It is also known as Residual Connection followed by Layer Normalization.</p><p><strong>Residual connection</strong>, also known as skip connection, is a technique used in deep neural networks to facilitate the training of very deep architectures. This technique was introduced in ResNet (Residual Networks) by Kaiming He et al. in 2015 [1]. This network was based on a concept called as Residual Learning. The main idea of this concept is to allow the input to a layer to bypass the layer’s operations and be added directly to the layer’s output.</p><blockquote>Residual=X+MHA(X)</blockquote><p>This helps:</p><ul><li>Preserve the original information (important in deep networks)</li><li>Improve gradient flow during training (solves vanishing gradients)</li></ul><p><strong>Layer Normalization</strong> normalizes the output of the previous operation across the <strong><em>features.</em></strong></p><p>During the layer normalization each item/word in the batch is normalized between the range of 0–1. These item can be anything, in our case of transformers it can be a vector embeddings of 512 dimensions of a word. We compute mean and variance of these items and replace each value in the matrix by,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/161/0*GIhSeSLOnjNKnBfw.png" /></figure><h3>3. Position-wise Feed Forward</h3><p>The <strong>FFN is applied independently to each token</strong> (i.e., each row of the matrix), <strong>Each token is processed independently</strong>, even though we used a single matrix multiplication. This is what allows Transformers to efficiently process sequences in parallel <strong>without losing per-token independence</strong> in the FFN.</p><p>It transforms each token’s vector to a new vector of the same size, but with a non-linear transformation in between.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*yUDeEqgliN0--fu46mnvMg.png" /></figure><pre>| Layer                    | Shape        |<br>| ------------------------ | ------------ |<br>| Input                    | [N, 768]     |(These size are an example)<br>| First Linear Layer (W₁)  | [768 → 3072] |<br>| ReLU                     | [N, 3072]    |<br>| Second Linear Layer (W₂) | [3072 → 768] |<br>| Output                   | [N, 768]     |</pre><p>Then again we have a Add &amp; Normalization layer then the output is passed to the Decoder.</p><p>Please refer my next blog for <a href="https://medium.com/@navinaananthan/transformers-part-3-decoders-aaac049be771">Decoders</a>.</p><h3>References</h3><p>[1] Ashish Vaswani, et al. and team, <a href="https://arxiv.org/abs/1706.03762">Attention is all you need</a>, 2017.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12a952c55112" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformers Part 2: Input and Positional Encoding]]></title>
            <link>https://medium.com/@navinaananthan/transformers-part-2-encoders-6c5adbc9caad?source=rss-b9399f5c1ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c5adbc9caad</guid>
            <category><![CDATA[encoder-decoder-model]]></category>
            <category><![CDATA[positional-encoding]]></category>
            <category><![CDATA[transformer-architecture]]></category>
            <category><![CDATA[input-embedding]]></category>
            <category><![CDATA[language-model]]></category>
            <dc:creator><![CDATA[Navin Aananthan]]></dc:creator>
            <pubDate>Mon, 23 Jun 2025 11:54:16 GMT</pubDate>
            <atom:updated>2025-07-25T05:02:39.200Z</atom:updated>
            <content:encoded><![CDATA[<p>In Part 1, we explored the high-level concept of transformers—why they revolutionized deep learning and how the attention mechanism made recurrence obsolete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*ncsFiZOzfmAA10zP.png" /></figure><h3>1. Input Embeddings</h3><p><strong>Input embeddings</strong> are the <strong>numerical vector representations</strong> of words (or subword tokens) in a way that captures their meaning and relationships with other words.</p><h4>Why do we need them?</h4><p>Neural networks can’t process raw text like “apple” or “robot” directly—they require numerical input. To achieve this, each token (typically a word or subword) is converted into a fixed-size vector of real numbers, known as an <strong>embedding vector</strong>. In Transformer models, the input sequence is usually limited to a maximum of <strong>512 tokens</strong>, with each token represented by its corresponding embedding.</p><p>Steps:</p><p>1. Using a typical BERT tokenizer (e.g., WordPiece), the above sentence would be split into tokens like</p><pre>transformers, have, revolutionized, natural, language, processing, by, enabling, models, to, learn, contextual, representations, .</pre><p>2. Either we do padding or truncating if the length of the token is too large. Here it is 15, so we do padding. We also add special tokens [CLS] at the beginning and [SEP] at the end.</p><pre>[CLS] transformers have revolutionized natural ... representations . [SEP] [PAD] [PAD] ... [PAD]</pre><p>In a Transformer model, the <strong>input embedding size</strong> refers to the dimensionality of the vector representation for each input token (such as a word or subword). This size is typically denoted as <strong>d_model</strong>, which in the original Transformer paper (&quot;Attention is All You Need&quot; by Vaswani et al., 2017) is set to <strong>512</strong>. Each token is mapped to a <strong>512-dimensional vector</strong>, and this dimension remains consistent throughout the model. As a result, an input sequence with N tokens is represented as a matrix of shape <strong>[N, d_model]</strong>.</p><pre>[Token_1_embedding]<br>[Token_2_embedding]<br>...<br>[Token_512_embedding]   ← final matrix of shape (512 × 512)</pre><h3>2. Positional Encoding</h3><p>Transformers process input tokens <strong>in parallel</strong>, unlike RNNs or LSTMs, which process tokens <strong>sequentially</strong>. This means transformers <strong>have no sense of order—</strong>so we need to <strong>inject positional information</strong> manually. It <strong>adds information about the position</strong> of each token in the sequence to its embedding.</p><p>Positional encoding is a set of vectors—<strong>one for each position</strong> in the sequence. Each vector is of the same dimension as the embedding (e.g.,<strong>d_model</strong> 512). These vectors are added to the <strong>input embedding</strong> matrix—element-wise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*AzL_a5GP2ywGRcW62-4yXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*hPHUbj_lH4l-4rBEgO1ziA.png" /></figure><p>This means</p><ul><li>Even indices (0, 2, 4, …) → sine functions</li><li>Odd indices (1, 3, 5, …) → cosine functions</li></ul><p>This allows the model to <strong>learn relative and absolute positions</strong> through periodic patterns.</p><ul><li>pos: The <strong>position</strong> of the token in the sequence (0, 1, 2, …, max_seq_length)</li><li>i: The <strong>dimension index</strong> (0 to d_model−1)</li><li>d_model: The <strong>embedding size</strong> (e.g., 512)</li></ul><p>Assume:</p><pre>[&quot;apple&quot;, &quot;ball&quot;]<br><br>| Token | Word Embedding (4D)    |<br>| ----- | -----------------------|<br>| apple | [0.4, -0.2, 0.6, 0.1]  |<br>| ball  | [0.7,  0.3, -0.1, 0.8] |<br><br>| Position | Positional Encoding                                                |<br>| -------- | -------------------------------------------------------------------|<br>| 0        | [sin(0), cos(0), sin(0), cos(0)] = [0.0, 1.0, 0.0, 1.0]            |<br>| 1        | [sin(1), cos(1), sin(0.01), cos(0.01)] ≈ [0.84, 0.54, 0.01, 0.999] |<br><br><br>&quot;apple&quot;<br>[0.4, -0.2, 0.6, 0.1] + [0.0, 1.0, 0.0, 1.0] = [0.4, 0.8, 0.6, 1.1]<br><br>&quot;ball&quot;<br>[0.7, 0.3, -0.1, 0.8] + [0.84, 0.54, 0.01, 0.999] ≈ [1.54, 0.84, -0.09, 1.799]</pre><p>Now &quot;apple&quot; has one combined vector (semantic + positional), and &quot;ball&quot; has another. Even if &quot;ball&quot; appears later in another sentence, its <strong>positionally encoded embedding</strong> will be <strong>different</strong>, allowing the Transformer to distinguish &quot;apple ball&quot; from &quot;ball apple&quot;.</p><ul><li>Input embedding: captures <strong>what</strong> the token is.</li><li>Positional encoding: captures <strong>where</strong> the token is.</li></ul><p>Please check into my next part about <a href="https://medium.com/@navinaananthan/transformers-part-3-encoders-12a952c55112">Encoders</a>.</p><h3>References</h3><p>[1] Ashish Vaswani, et al. and team, <a href="https://arxiv.org/abs/1706.03762">Attention is all you need</a>, 2017.</p><p>[2] <a href="https://towardsdatascience.com/transformers-explained-visually-not-just-how-but-why-they-work-so-well-d840bd61a9d3/">https://towardsdatascience.com/transformers-explained-visually-not-just-how-but-why-they-work-so-well-d840bd61a9d3/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c5adbc9caad" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>