<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sujangyawali on Medium]]></title>
        <description><![CDATA[Stories by Sujangyawali on Medium]]></description>
        <link>https://medium.com/@sujangyawali177?source=rss-23434c2aa8d1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*emVrLwNKftgnjDoq</url>
            <title>Stories by Sujangyawali on Medium</title>
            <link>https://medium.com/@sujangyawali177?source=rss-23434c2aa8d1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 17:38:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sujangyawali177/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How DeepSeek’s Harness Rethinks AI Agent Memory]]></title>
            <link>https://medium.com/@sujangyawali177/how-deepseeks-harness-rethinks-ai-agent-memory-ed342fc975c5?source=rss-23434c2aa8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed342fc975c5</guid>
            <category><![CDATA[harness-engineering]]></category>
            <category><![CDATA[agent-memory-systems]]></category>
            <category><![CDATA[deepseek-harness]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sujangyawali]]></dc:creator>
            <pubDate>Sun, 23 Aug 2026 12:21:54 GMT</pubDate>
            <atom:updated>2026-08-23T12:21:54.142Z</atom:updated>
            <content:encoded><![CDATA[<p>AI agents are often described as systems that can <strong>remember</strong> users, conversations, tools, and past experiences.</p><p>But there is a deeper question:</p><blockquote><strong><em>What does it actually mean for an AI agent to remember?</em></strong></blockquote><p>A common architecture treats memory as a collection of separate systems: working memory for the current conversation, semantic memory for facts, episodic memory for past events, and procedural memory for skills.</p><p>DeepSeek’s harness design suggests a different way to think about this.</p><p>Instead of making a separate memory store the center of the architecture, the <strong>session event log becomes the source of truth</strong>, while the context sent to the LLM is <strong>derived from those events</strong>.</p><h3>The Traditional Way: Memory Is Retrieved</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jtcuPorjgPITSwvD_sunSQ.jpeg" /></figure><p>An AI agent doesn’t simply “remember” everything like a human.</p><p>Instead, it uses a <strong>memory architecture</strong> to bring the right information into the LLM’s context when needed.</p><p>The <strong>Working Memory / Context RAM</strong> acts as the agent’s temporary workspace, containing the current user prompt, recent conversation, system instructions, tool results, and relevant memories.</p><p>Behind it are three types of long-term memory:</p><p><strong>Procedural Memory</strong> stores <em>how to do things</em> such as skills and rules.</p><p><strong>Semantic Memory</strong> stores <em>what the agent knows</em> such as user preferences and facts.</p><p><strong>Episodic Memory</strong> stores <em>what happened</em> such as previous conversations and events.</p><p>When a new request arrives, <strong>RAG retrieves only the relevant memories</strong> and injects them into the Working Memory before sending the complete context to the LLM.</p><p>After the response, the interaction can be saved as an episode, and after enough conversations, a <strong>Summarizer Agent</strong> can consolidate important information into durable semantic facts.</p><h3><strong>How DeepSeek Harness Manages Agent Memory</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rmFUdh1grqfMn_Wofno59g.png" /><figcaption>How DeepSeek Harness Manages Agent Memory</figcaption></figure><p>The key difference in this architecture is that the <strong>Session Event Log acts as the source of truth</strong>.</p><p>Instead of treating memory as a single storage system, the harness records what happens during an agent session as an <strong>ordered sequence of events</strong> — such as user messages, assistant messages, tool calls, tool results, errors, and system events.</p><p>When the agent makes an LLM request, <strong>deriveMessages() derives the model-visible message history from the session events</strong>. Separately, the harness assembles the <strong>system prompt, tool schemas, and other request-specific information</strong>, while the agent/pre-step stage can modify or reject the messages before the model request is sent.</p><p>The resulting <strong>model context</strong> is then sent to the LLM. If the model calls a tool, the tool call and its result are recorded as new events in the session log, and the cycle continues.</p><p>A <strong>Persistence Layer</strong>, such as JSONL or SQLite, can make the session event log durable so that the session can later be reloaded, resumed, or reconstructed.</p><p>Optional memory systems, such as semantic memory or vector-based retrieval, can provide additional information, but they are <strong>not the primary source of truth for the session history</strong>.</p><p>Events record what happened → deriveMessages() derives the model history → prompts and tools are assembled → agent/pre-step controls the request → LLM responds → tool interactions create new events → the cycle continues.</p><h3>The Bigger Architectural Shift</h3><p>The interesting idea isn’t simply:</p><p><em>“DeepSeek has another type of memory.”</em></p><p>The deeper idea is:</p><p><strong><em>The session event log serves as the source of truth, while the context presented to the LLM is a derived view of that event history.</em></strong></p><p>This changes the mental model from:</p><p><strong>Store Memory → Retrieve Memory → Send to LLM</strong></p><p>to:</p><p><strong>Record Events → Derive Model History → Assemble Context → Send to LLM</strong></p><p>As the session grows, <strong>compaction can manage the model-visible history without removing the underlying event record</strong>.</p><p>That is the key architectural distinction: <strong>the system preserves the complete history as events and dynamically derives what the model needs to see for each request.</strong></p><h3>Final Mental Model</h3><p>Think of the <strong>Session Event Log as the agent’s diary</strong>.</p><p>The <strong>Persistence Layer</strong> keeps that diary durable.</p><p><strong>deriveMessages()</strong> derives the model-visible message history from the session events, while the <strong>system prompt and tool layers</strong> add the instructions and capabilities required for the current request.</p><p>The <strong>agent/pre-step</strong> stage can then modify or reject what is sent to the model.</p><p><strong>Compaction</strong> can manage older history when the session grows too large, while optional <strong>RAG or memory systems</strong> can provide additional knowledge when needed.</p><p>The <strong>LLM</strong> receives the resulting model context and reasons over it.</p><p>Then every new action becomes another event in the diary.</p><p>So the complete loop is:</p><blockquote><strong>Events record what happened → model history is derived → prompts and tools are assembled → pre-step controls the request → LLM reasons → tools execute actions → new events are recorded → the next context is derived.</strong></blockquote><p>The agent isn’t carrying its entire history inside the model.</p><p><strong>Its history lives in the event log, while the model’s current context is derived from that history for each request.</strong></p><p><em>Note in Nepali:</em></p><p><strong><em>Problem:</em></strong><em><br> Traditional agent मा long-running execution को </em><strong><em>state, conversation, tool interactions र memory विभिन्न mechanisms मा distributed भएर manage गर्नुपर्ने</em></strong><em> हुन सक्छ, जसले reconstruction, debugging, replay र auditing जटिल बनाउन सक्छ।</em></p><p><strong><em>DeepSeek solution:</em></strong><em><br> </em><strong><em>Session Event Log लाई canonical history बनाउने र त्यसबाट model-visible state/history derive गर्ने।</em></strong></p><blockquote><strong><em>Traditional:</em></strong><em> Multiple Memory Stores → Retrieve/Assemble → Context → LLM</em></blockquote><blockquote><strong><em>DeepSeek:</em></strong><em> Event History → Derive Model History → Context → LLM</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed342fc975c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Policy Gradients to GRPO: A Practical Journey from Reinforcement Learning to LLM Reasoning]]></title>
            <link>https://medium.com/@sujangyawali177/from-policy-gradients-to-grpo-a-practical-journey-from-reinforcement-learning-to-llm-reasoning-ad80660d5d29?source=rss-23434c2aa8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad80660d5d29</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[policy-gradient]]></category>
            <category><![CDATA[grpo]]></category>
            <category><![CDATA[llm-reasoning]]></category>
            <dc:creator><![CDATA[Sujangyawali]]></dc:creator>
            <pubDate>Sat, 15 Aug 2026 02:25:24 GMT</pubDate>
            <atom:updated>2026-08-15T02:25:24.562Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Reinforcement Learning (RL) provides a powerful framework for learning optimal behavior through <strong>interaction, rewards, and feedback</strong>. Traditional RL often begins with value-based methods that estimate how good states or actions are, but as environments become larger and more complex, <strong>function approximation and policy-gradient methods</strong> provide a more scalable approach. This progression leads from <strong>Monte Carlo and Temporal-Difference learning</strong> to <strong>REINFORCE, Actor–Critic, GAE, TRPO, and PPO</strong>, each addressing limitations such as high variance, unstable updates, and inefficient optimization.</p><p>The same ideas can be applied to <strong>Large Language Models (LLMs)</strong> by treating the LLM as an RL agent that generates text token by token. Rewards can then guide the model toward better responses, leading to methods such as <strong>PPO and GRPO</strong>. In particular, GRPO uses multiple responses to the same prompt and their relative rewards to improve the policy without requiring a separate value model.</p><p>This blog follows that journey <strong>from policy gradients to GRPO</strong>, explaining the key concepts, mathematical intuition, training pipelines, and how methods such as <strong>SFT, GRPO, and knowledge distillation</strong> can be combined to build stronger LLM reasoning models.</p><h3>Table of Contents</h3><ol><li><strong>RL Fundamentals</strong> — Function Approximation, MC &amp; TD</li><li><strong>Policy Gradients</strong> — REINFORCE, Baseline, Actor–Critic, GAE</li><li><strong>TRPO &amp; PPO</strong> — Trust Regions, Clipping &amp; Stable Updates</li><li><strong>RL for LLMs</strong> — LLM as Agent, Rewards &amp; RLHF</li><li><strong>GRPO</strong> — Group Rewards, Relative Advantages &amp; PPO Clipping</li><li><strong>GRPO for Reasoning</strong> — Verifiable Rewards &amp; Reasoning Tasks</li><li><strong>Training Pipelines</strong> — GRPO, SFT + GRPO &amp; Knowledge Distillation</li><li><strong>Choosing the Right Method</strong> — SFT vs GRPO vs Distillation</li><li><strong>Conclusion</strong> — From Policy Gradients to LLM Reasoning</li></ol><h3>𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧 𝐀𝐩𝐩𝐫𝐨𝐱𝐢𝐦𝐚𝐭𝐢𝐨𝐧</h3><p><strong>Function approximation</strong> replaces a large value table with a parameterized function <strong>V̂(s,w)</strong> that learns value patterns from experienced states and <strong>generalizes to unseen states</strong>.</p><p><strong>Tabular:</strong> State → Stored Value<br> <strong>Function Approximation:</strong> State → Function → Predicted Value</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*usa1d-Rckzhbr-6EbjbbXw.png" /><figcaption>𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧 𝐀𝐩𝐩𝐫𝐨𝐱𝐢𝐦𝐚𝐭𝐢𝐨𝐧</figcaption></figure><h4>𝟏. 𝐏𝐫𝐞𝐯𝐢𝐨𝐮𝐬 𝐢𝐝𝐞𝐚</h4><p>In <strong>Tabular RL</strong>, we store every state’s value:</p><p><strong>s → V(s)</strong></p><p>Problem: Huge environments can have too many states to store.</p><h4>𝟐. 𝐒𝐨𝐥𝐮𝐭𝐢𝐨𝐧</h4><p>Instead of a table, learn a function:</p><p><strong>V̂(s,w)</strong></p><p><strong>State → Function → Predicted Value</strong></p><p>The function learns from <em>states we </em><strong><em>have seen</em></strong><em> and can estimate values for </em><strong><em>unseen states</em></strong><em>.</em></p><p><strong>How can we learn from a small number of states and still predict values for many unseen states?</strong></p><ul><li>Use <strong>function approximation</strong> to learn from the states we have experienced and <strong>generalize their values to unseen states</strong>.</li><li>By <strong>learning the underlying pattern</strong> from the states it has experienced, a function can use those patterns to <strong>estimate values for similar unseen states</strong>.</li></ul><h4>𝟑. 𝐇𝐨𝐰 𝐝𝐨 𝐰𝐞 𝐥𝐞𝐚𝐫𝐧?</h4><p>Compare the true and predicted values:</p><p><strong>Error = [V(s) − V̂(s,w)]²</strong></p><p>Then minimize the error using gradient descent.</p><h4>𝟒. 𝐖𝐡𝐲 𝐖𝐞𝐢𝐠𝐡𝐭𝐞𝐝 𝐌𝐒𝐄?</h4><p>Not all states are visited equally.</p><p><strong>μ(s)</strong> = how frequently the agent visits state <strong>s</strong>.</p><p>So:</p><p><strong>VE(w) = Σₛ μ(s)[V(s) − V̂(s,w)]²</strong></p><p>Intuition:</p><p><strong>Frequently visited → high μ(s) → more attention</strong></p><p><strong>Rarely visited → low μ(s) → less attention</strong></p><h4>𝐌𝐚𝐢𝐧 𝐈𝐝𝐞𝐚</h4><blockquote><strong><em>Don’t memorize every state. Learn a function that captures the pattern of values, focusing more on states the agent actually encounters.</em></strong></blockquote><p>𝐌𝐂(Monte Carlo) 𝐯𝐬 𝐓𝐃(Temporal Difference)</p><p>In RL, the <strong>true value V(s) is unknown</strong>, so we create a target <strong>Uₜ</strong> and move our prediction toward it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y1WrPpEa75rroJITDzVjwQ.png" /></figure><p><strong>Error:</strong><br> <strong>δ = Uₜ − V̂(Sₜ)</strong></p><h4>𝐓𝐞𝐫𝐦𝐬 𝐢𝐧 δ = Uₜ − V̂(Sₜ)</h4><ul><li><strong>δ (delta)</strong> → <strong>Temporal-Difference (TD) Error</strong></li><li><strong>Uₜ</strong> → <strong>Target Value / TD Target</strong></li><li><strong>V̂(Sₜ)</strong> → <strong>Current Estimated Value</strong> of state Sₜ</li><li><strong>Sₜ</strong> → <strong>Current State</strong></li><li><strong>t</strong> → <strong>Current time step</strong></li></ul><h3>𝐌𝐂</h3><p>Wait until the episode ends:</p><p><strong>Uₜ = Gₜ</strong></p><p>→ Uses actual total return.</p><h3>𝐓𝐃</h3><p>Update every step:</p><p><strong>Uₜ = Rₜ₊₁ + γV̂(Sₜ₊₁)</strong></p><p>→ Uses the next state’s <strong>estimated value</strong>.</p><p><strong>MC → Wait → Actual return</strong><br> <strong>TD → Update immediately → Estimated next value</strong></p><blockquote><strong><em>TD learns faster because it doesn’t wait for the episode to finish.</em></strong></blockquote><h3>𝐌𝐂 𝐚𝐧𝐝 𝐓𝐃 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐰𝐢𝐭𝐡 𝐆𝐫𝐚𝐝𝐢𝐞𝐧𝐭 𝐃𝐞𝐬𝐜𝐞𝐧𝐭</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z_QBB0fWYmariZDy8soTNw.png" /><figcaption>𝐓𝐃 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐰𝐢𝐭𝐡 𝐆𝐫𝐚𝐝𝐢𝐞𝐧𝐭 𝐃𝐞𝐬𝐜𝐞𝐧𝐭</figcaption></figure><ul><li><strong>Problem:</strong> True value <strong>V(s)</strong> is unknown.</li><li><strong>MC:</strong> Wait until episode ends and use actual return:<br> <strong>Uₜ = Gₜ</strong></li><li><strong>TD:</strong> Update every step using next-state estimate:<br> <strong>Uₜ = Rₜ₊₁ + γV̂(Sₜ₊₁)</strong></li><li><strong>Error for both:</strong><br> <strong>δₜ = Uₜ − V̂(Sₜ)</strong></li><li><strong>Gradient Descent:</strong> Uses this error to update the weights:<br> <strong>wₜ₊₁ = wₜ + αδₜ∇V̂(Sₜ,wₜ)</strong></li></ul><h4>𝐏𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫 𝐔𝐩𝐝𝐚𝐭𝐞 𝐅𝐨𝐫𝐦𝐮𝐥𝐚</h4><p><strong>General Gradient Descent:</strong></p><p><strong>θ ← θ − η∇L(θ)</strong></p><ul><li><strong>θ (theta)</strong> → parameters / weights</li><li><strong>η (eta)</strong> → learning rate</li><li><strong>∇ (nabla)</strong> → gradient</li><li><strong>L(θ)</strong> → loss function</li></ul><p>For our RL problem:</p><p><strong>θ = w</strong></p><ul><li><strong>w</strong> → weight vector</li><li><strong>VE(w)</strong> → value-error / loss</li><li><strong>α (alpha)</strong> → step-size / learning-rate parameter</li></ul><p>So:</p><p><strong>wₜ₊₁ = wₜ − α⁄2 ∇[V̂ᵧ(Sₜ) − V̂(Sₜ,wₜ)]²</strong></p><ul><li><strong>wₜ</strong> → current weights</li><li><strong>wₜ₊₁</strong> → updated weights</li><li><strong>V̂ᵧ(Sₜ)</strong> → target value</li><li><strong>V̂(Sₜ,wₜ)</strong> → current predicted value</li><li><strong>Sₜ</strong> → current state</li><li><strong>t</strong> → time step</li><li><strong>[Target − Prediction]²</strong> → squared error</li></ul><h4>𝐄𝐱𝐩𝐚𝐧𝐝𝐞𝐝 𝐅𝐨𝐫𝐦</h4><p><strong>wₜ₊₁ = wₜ + α[V̂ᵧ(Sₜ) − V̂(Sₜ,wₜ)]∇V̂(Sₜ,wₜ)</strong></p><p>Think of it as:</p><p><strong>New weights = Old weights + α × Error × Gradient</strong></p><p>where:</p><ul><li><strong>α</strong> → how big a step we take</li><li><strong>Error</strong> → how wrong our prediction is</li><li><strong>Gradient</strong> → how the prediction changes with the weights</li></ul><blockquote><strong><em>Goal: adjust w so that V̂(Sₜ,w) gets closer to the target.</em></strong></blockquote><h3>𝐅𝐥𝐨𝐰</h3><p><strong>Predict → TD Target → Calculate Error → Update Weights → Repeat</strong></p><h3>𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧 𝐀𝐩𝐩𝐫𝐨𝐱𝐢𝐦𝐚𝐭𝐢𝐨𝐧 𝐢𝐧 𝐑𝐋</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SKwcuYx0XHR8l0DSagVLUA.png" /><figcaption>𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧 𝐀𝐩𝐩𝐫𝐨𝐱𝐢𝐦𝐚𝐭𝐢𝐨𝐧 𝐢𝐧 𝐑𝐋</figcaption></figure><p>Earlier, <strong>Tabular RL</strong> stored the value of every state as <strong>s → V(s)</strong>, but this becomes impossible when the state space is huge.</p><p>So, we use <strong>function approximation</strong> to learn <strong>V̂(s,w)</strong>, which learns from visited states and generalizes to unseen states.</p><p>We train the function by minimizing the prediction error, giving more importance to frequently visited states using <strong>μ(s)</strong>.</p><p>Since the true value <strong>V(s)</strong> is unknown, we need an estimated target <strong>Uₜ</strong>. There are two ways to get it:</p><p><strong>Monte Carlo</strong>, which waits until the episode ends and uses the total return <strong>Uₜ = Gₜ</strong>, and</p><p><strong>Temporal Difference</strong>, which updates every step using <strong>Uₜ = Rₜ₊₁ + γV̂(Sₜ₊₁,wₜ)</strong>. The difference between the target and current prediction is the <strong>TD error</strong>, <strong>δₜ = Uₜ − V̂(Sₜ,wₜ)</strong>.</p><p>We then use <strong>gradient descent/SGD</strong> to update the weights: <strong>wₜ₊₁ = wₜ + αδₜ∇V̂(Sₜ,wₜ)</strong>.</p><p>Once the value function is learned, we can extend it to <strong>Q̂(s,a,w)</strong> to evaluate actions and use <strong>ε-greedy</strong> to balance exploration and exploitation.</p><p>Thus, the overall idea is: <strong>limited experience → function approximation → MC/TD target → error → gradient descent → generalization → better action selection</strong>.</p><h3>𝐏𝐨𝐥𝐢𝐜𝐲 𝐆𝐫𝐚𝐝𝐢𝐞𝐧𝐭 𝐌𝐞𝐭𝐡𝐨𝐝𝐬</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MxaEjZLcVYR5IM1U8wISAQ.png" /><figcaption>𝐏𝐨𝐥𝐢𝐜𝐲 𝐆𝐫𝐚𝐝𝐢𝐞𝐧𝐭 𝐌𝐞𝐭𝐡𝐨𝐝𝐬</figcaption></figure><p>Earlier, <strong>value-based RL</strong> learned <strong>Q(s,a)</strong> and chose the best action using</p><p><em>*a = arg maxₐ Q(s,a)**.</em></p><p>In <strong>policy-based RL</strong>, we directly learn a parameterized policy <strong>πθ(a|s)</strong>, where <strong>θ</strong> controls the policy.</p><p>The policy takes a <strong>state → action</strong>, using <strong>Softmax</strong> for discrete actions or <strong>Gaussian</strong> for continuous actions.</p><p>Our goal is to maximize the expected return <strong>J(θ) = E[G₀]</strong>. The agent follows the policy, interacts with the environment, receives rewards, and generates a trajectory <strong>τ = (s₀,a₀,r₀,…)</strong>.</p><p>We calculate the return <strong>Gₜ = Σₖ₌ₜ∞ γᵏ⁻ᵗrₖ</strong>, then <em>estimate the policy gradient</em> <strong>∇J(θ) = E[Σₜ ∇log πθ(aₜ|sₜ)Gₜ] where,</strong></p><ul><li><strong>∇J(θ)</strong> → Direction to improve the policy</li><li><strong>πθ(aₜ|sₜ)</strong> → Probability of taking action <strong>aₜ</strong> in state <strong>sₜ</strong></li><li><strong>∇log πθ(…)</strong> → How to change that action’s probability</li><li><strong>Gₜ</strong> → Return/reward obtained after that action</li><li><strong>E[ ]</strong> → Average over sampled experiences</li><li><strong>Σₜ</strong> → Across all time steps</li></ul><p>When <strong>θ</strong> changes, both the <strong>action probabilities</strong> and the <strong>future states</strong> can change. But the <strong>Policy Gradient Theorem</strong> shows that the effect of the changing state distribution cancels out.</p><p>So we only need:</p><p><strong>∇J(θ) = E[Σₜ ∇log πθ(aₜ|sₜ)Gₜ]</strong></p><p>Here:</p><ul><li><strong>∇log πθ(aₜ|sₜ)</strong> → how changing <strong>θ</strong> changes the action probability</li><li><strong>Gₜ</strong> → how good the return was</li></ul><blockquote><strong><em>In simple words:</em></strong><em> We only need to see how changing the policy affects the chosen action and how good that action turned out to be. We don’t need to calculate how the environment changes.</em></blockquote><p>To reduce variance, we can use a baseline <strong>b(sₜ)</strong>, commonly <strong>Vπ(sₜ)</strong>, giving</p><p><strong>Gₜ − b(sₜ),</strong> where Gₜ − Vπ(sₜ) = How much better/worse the action was than expected?</p><p>Finally, we use <strong>gradient ascent</strong>, <strong>θₜ₊₁ = θₜ + α∇J(θₜ)</strong>, which increases the probability of actions that produce good returns.</p><p>We repeat this process:</p><p><strong>Policy → Action → Reward → Return → Policy Gradient → Update θ → Better Policy</strong>.</p><p>This is the foundation of methods such as <strong>PPO, GRPO, and RLHF</strong>.</p><h3>𝐏𝐨𝐥𝐢𝐜𝐲 𝐆𝐫𝐚𝐝𝐢𝐞𝐧𝐭 𝐌𝐞𝐭𝐡𝐨𝐝𝐬</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PblNFPvHLIJR5qXxaIcTIQ.png" /></figure><p>So far, to find the optimal policy, we used the <strong>action-value function qπ(s,a)</strong> and asked which action gives the maximum q-value, but now we look at methods that learn an optimal policy <strong>without considering the value function directly</strong>.</p><p>The policy is represented as <strong>πθ(a|s)</strong>, the probability of taking action <strong>a</strong> in state <strong>s</strong>, with <strong>θ</strong> as the policy parameter.</p><p>We use a preference function <strong>h(s,a,θ)</strong>, which can be a function approximator such as a neural network, and for discrete actions Softmax converts preferences into probabilities:</p><p><strong>πθ(a|s) = exp(h(s,a,θ)) / Σᵦ exp(h(s,b,θ))</strong>.</p><p>The action is sampled according to <strong>a ∼ πθ(·|s)</strong>.</p><p>We define a performance measure <strong>J(θ)</strong>, such as expected return, average reward, or discounted return, and our goal is to <strong>maximize J(θ) with respect to θ</strong>.</p><p>The Policy Gradient Theorem gives <strong>∇J(θ) = Σₛ μ(s) Σₐ qπ(s,a)∇θπθ(a|s)</strong>, where <strong>μ(s)</strong> is the state distribution under policy <strong>π</strong>, <strong>qπ(s,a)</strong> is the action-value function under <strong>π</strong>, and <strong>∇θπθ(a|s)</strong> describes how the policy changes with <strong>θ</strong>.</p><p>We then use <strong>gradient ascent</strong> to update the parameters:</p><p><strong>θₜ₊₁ = θₜ + α∇J(θₜ)</strong>, where <strong>α</strong> is the step size (learning rate).</p><p>The gradient tells us the direction in which to move <strong>θ</strong> to maximize <strong>J(θ)</strong>.</p><p>The above diagram also shows that the state distribution does not depend directly on <strong>θ</strong> in the required derivative, so only <strong>∇θπθ(a|s)</strong> is needed.</p><p>In practice, the flow is</p><p><strong>Current state Sₜ → Choose action according to πθ(a|s) → Take action → Get reward Rₜ₊₁ and next state → Compute gradient ∇J(θ) → Update θ using gradient ascent → Repeat to improve the policy</strong>.</p><p><em>Here, there is </em><strong><em>no direct error like Target − Prediction</em></strong><em>. </em><strong><em>Gₜ</em></strong><em> is the learning signal: a good return increases the probability of the action, while a poor return decreases it. With a baseline, </em><strong><em>Aₜ = Gₜ − Vπ(sₜ)</em></strong><em> tells us how much better or worse the action was than expected.</em></p><p>The listed policy-gradient algorithms include <strong>REINFORCE, Actor-Critic Methods, TRPO, PPO, and DDPG</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E1V9_VVDxom0Jp15o1IHmQ.png" /></figure><p>The image starts with the <strong>Policy Gradient Theorem</strong>, which says <strong>∇J(θ) = E[Σₜ ∇log πθ(aₜ|sₜ)Gₜ]</strong>.</p><p>The idea is simple:</p><p><strong>Gₜ</strong> tells us how good the outcome was, while</p><p><strong>∇log πθ(aₜ|sₜ)</strong> tells us how to change the probability of the chosen action.</p><p>The <strong>score function</strong> is <strong>∇log πθ(a|s) = ∇πθ(a|s) / πθ(a|s)</strong>, which makes this gradient easier to calculate from sampled actions.</p><p>From this theorem, the image introduces three practical methods.</p><p><strong>REINFORCE</strong> uses the actual Monte Carlo return <strong>Gₜ</strong> after the trajectory finishes and updates the policy using <strong>θ ← θ + α∇log πθ(aₜ|sₜ)Gₜ</strong>; it is simple and unbiased but has high variance.</p><p><strong>REINFORCE with a Baseline</strong> subtracts a baseline <strong>b(sₜ)</strong> from the return, using <strong>Gₜ − b(sₜ)</strong>, which reduces variance while keeping the expected gradient unchanged; a common baseline is <strong>Vπ(sₜ)</strong>.</p><p><strong>Actor-Critic</strong> learns this baseline using a <strong>Critic</strong>, while the <strong>Actor</strong> learns the policy. The Critic estimates <strong>V(sₜ)</strong>, and the Actor uses the <strong>Advantage</strong>, <strong>Aₜ = Gₜ − V(sₜ)</strong>, to decide whether the action was better or worse than expected. The Actor update is <strong>θ ← θ + α∇log πθ(aₜ|sₜ)Aₜ</strong>, while the Critic learns the value estimate.</p><p>The image then compares the methods:</p><p><strong>REINFORCE → high variance, no baseline;</strong></p><p><strong>REINFORCE + Baseline → lower variance;</strong></p><p><strong>Actor-Critic → learned baseline, low variance and more practical</strong>.</p><p>The overall pipeline is:-&gt;</p><p><strong>Parameterize Policy → Collect Trajectories → Calculate Return/Advantage → Estimate Gradient → Update θ → Better Policy</strong>.</p><h3>Policy Gradient Theorem &amp; REINFORCE</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kptYbznTEEIuLFjPk5dJ_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ALoxv-K-vK8mJtSk0uurpQ.png" /></figure><p>The above image explains how <strong>REINFORCE improves a policy using rewards</strong>.</p><p>The agent generates trajectories, calculates the return <strong>Gₜ</strong>, and uses it to estimate the policy gradient: <strong>∇J(θ) ≈ Gₜ∇θ log πθ(aₜ|sₜ)</strong>.</p><p>The policy parameters are then updated using gradient ascent:</p><p><strong>θ ← θ + αĝ</strong>.</p><p>In simple terms, <strong>actions that produce high returns become more likely, while actions that produce low returns become less likely</strong>, gradually improving the policy.</p><h3>REINFORCE with Baseline &amp; Actor–Critic</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KkYMeVB2JQhFEn1eL3byIg.png" /></figure><p>This above image explains how <strong>REINFORCE can be improved using a baseline and how this leads to Actor–Critic methods</strong>.</p><p>In basic REINFORCE, the policy is updated using the return <strong>Gₜ</strong>, but this can have high variance.</p><p>By subtracting a baseline, usually the state-value estimate <strong>V̂(sₜ, ω)</strong>, we get the <strong>advantage</strong> <strong>Aₜ = Gₜ − V̂(sₜ, ω)</strong>, which tells us whether an action performed better or worse than expected.</p><p>The actor then updates the policy using <strong>θ ← θ + αAₜ∇θ log πθ(aₜ|sₜ)</strong>.</p><p>In Actor–Critic, instead of waiting for the complete return <strong>Gₜ</strong>, the critic uses the <strong>TD target Rₜ₊₁ + γV̂(sₜ₊₁,ω)</strong> and computes the <strong>TD error δₜ = Rₜ₊₁ + γV̂(sₜ₊₁,ω) − V̂(sₜ,ω)</strong>.</p><p>The <strong>critic learns the value function</strong>, while the <strong>actor improves the policy</strong>, making learning faster and reducing variance.</p><h3>Policy Gradient with Advantage Function &amp; Generalized Advantage Estimation (GAE)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1-4Ah_sdOHnqVBrJyrXdEw.png" /></figure><p>This above image explains how <strong>Advantage Functions and GAE improve policy-gradient learning</strong>.</p><p>Instead of using the full return <strong>Gₜ</strong> directly,</p><p>the advantage function <strong>Aᵗ(sₜ, aₜ) = Qᵗ(sₜ, aₜ) − Vᵗ(sₜ)</strong> measures whether an action is better or worse than the policy’s average behavior.</p><p>If <strong>Aᵗ &gt; 0</strong>, the action is good and its probability is increased;</p><p>if <strong>Aᵗ &lt; 0</strong>, its probability is decreased.</p><p>Since the<em> true advantage</em> is difficult to calculate,</p><p><strong>GAE</strong> estimates it using a <em>weighted sum of TD errors,</em></p><p><strong>Âₜᴳᴬᴱ = Σₗ₌₀∞ (γλ)ˡδₜ₊ₗ</strong>, where <strong>λ</strong> controls the bias–variance trade-off.</p><p>Finally, the policy is updated using <strong>θ ← θ + αE[∇θ log πθ(aₜ|sₜ)Âₜᴳᴬᴱ]</strong>, giving a more stable and lower-variance learning signal.</p><h3>Trust Region Policy Optimization (TRPO)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3gwcIkOJeb6v5O_VT2PPYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l0T0SOIPaRiKmZaFmT3Wog.png" /></figure><p>TRPO is a <strong>policy-gradient algorithm designed to improve a policy safely and stably</strong>.</p><p>It starts with the <strong>advantage function</strong> <strong>A(s,a) = Q(s,a) − V(s)</strong>, which tells whether an action is better or worse than the policy’s average behavior.</p><p>The goal is to improve the new policy <strong>πₙₑw</strong> by increasing the probability of actions with positive advantage and decreasing those with negative advantage.</p><p>However, directly optimizing the new policy is difficult because its state distribution is unknown.</p><p>Therefore, TRPO uses trajectories collected from the old policy <strong>πₒₗ𝒹</strong> and constructs a <strong>surrogate objective</strong> using importance sampling:</p><p><strong>L(θ) = 𝔼[πₙₑw(a|s) / πₒₗ𝒹(a|s) · Aᵖⁱᵒˡᵈ(s,a)]</strong>.</p><p>To prevent the new policy from changing too much, TRPO introduces a <strong>trust region</strong> by restricting the KL divergence:</p><p><strong>𝔼[Dₖₗ(πₒₗ𝒹 || πₙₑw)] ≤ δ</strong>.</p><p>Thus, TRPO follows <strong>Collect trajectories → Calculate advantages → Build surrogate objective → Constrain policy change → Optimize and update policy → Repeat</strong>.</p><p>In simple terms, <strong>TRPO improves the policy toward better actions while keeping each update small enough to avoid a sudden performance drop</strong>.</p><h3>Trust Region Policy Optimization (TRPO) — Objective &amp; Solution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-syTvDD825dg-ij-GHEb_A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e1Qs2xOz_Dla2r0cuzlcwg.png" /></figure><p>The above image explain how <strong>TRPO converts the policy-improvement problem into a constrained optimization problem</strong>.</p><p>TRPO starts with the surrogate objective <strong>L(πₙₑw) = 𝔼[ (πₙₑw(a|s) / πₒₗ𝒹(a|s)) Qₒₗ𝒹(s,a) ]</strong>, where the probability ratio is an <strong>importance-sampling correction</strong> that allows us to use trajectories collected from the old policy. S</p><p>ince directly optimizing the new policy can cause a large and unstable change, TRPO restricts the policy using a <strong>KL-divergence trust region</strong>, <strong>Dₖₗ(πₒₗ𝒹 ‖ πₙₑw) ≤ δ</strong>.</p><p>TRPO then uses a <strong>first-order Taylor approximation</strong> for the objective and a <strong>second-order Taylor approximation</strong> for the KL constraint, producing the quadratic problem <strong>maximize gᵀ(θ − θₒₗ𝒹)</strong> subject to <strong>½(θ − θₒₗ𝒹)ᵀA(θ − θₒₗ𝒹) ≤ δ</strong>, where <strong>g</strong> is the policy-gradient direction and <strong>A</strong> is related to the <strong>Fisher Information Matrix</strong>.</p><p>The resulting update moves in the natural-gradient direction <strong>A⁻¹g</strong>, with a step size chosen so that the KL constraint is satisfied: <strong>θₖ₊₁ = θₖ + √(2δ / (gᵀA⁻¹g)) A⁻¹g</strong>.</p><p>In simple terms, <strong>TRPO finds the direction that improves expected reward, but carefully limits how far the new policy can move from the old policy, producing stable and reliable policy updates</strong>.</p><h3>Reinforcement Learning &amp; Policy Gradients — From Agent Interaction to GRPO</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0rUNQRFSZmWS7iX2XkunXA.png" /></figure><p>This image give an overall picture of <strong>policy-gradient Reinforcement Learning</strong>, including how the same idea applies to <strong>LLMs</strong>.</p><p>For LLMs, the <strong>LLM acts as the agent</strong>, the generated token is the action, and a tool such as a Python interpreter can act as the environment that returns feedback.</p><p>A complete sequence of states and actions forms a <strong>trajectory</strong>, <strong>s₀ → a₀ → s₁ → a₁ → … → sₜ → reward</strong>, and the goal is to adjust the policy so that actions leading to higher rewards become more probable.</p><p>The policy is represented by parameters <strong>θ</strong>, and policy-gradient methods update these parameters according to the direction that increases expected reward.</p><p>The basic REINFORCE update is <strong>θₜ₊₁ = θₜ + α Σₜ ∇θ log πθ(aₜ|sₜ) Gₜ</strong>, where <strong>Gₜ</strong> is the cumulative discounted return.</p><p>If an action produces a positive reward, the probability of that action is increased; if it produces a negative reward, its probability is decreased.</p><p>A <strong>baseline b(sₜ)</strong> can be subtracted to reduce variance, giving <strong>θₜ₊₁ = θₜ + α Σₜ ∇θ log πθ(aₜ|sₜ)(Gₜ − b(sₜ))</strong>.</p><p>This leads naturally to the <strong>advantage function</strong>, which measures how much better or worse an action is compared with the expected behavior:</p><p><strong>Aₜ = Q(sₜ,aₜ) − V(sₜ)</strong>.</p><p>Thus, <strong>Aₜ &gt; 0</strong> means the action was better than expected and should become more likely, while <strong>Aₜ &lt; 0</strong> means it was worse and should become less likely.</p><p><em>For example, given the prompt “Roger has 5 tennis balls. He buys 2 more cans, each containing 3 balls”, the LLM generates tokens such as </em><strong><em>Roger → has → eleven → tennis → balls</em></strong><em>.</em></p><p><em>The generated tokens form the actions </em><strong><em>a₀, a₁, …, aₜ</em></strong><em>, while the prompt and previously generated tokens form the states.</em></p><p><em>The final answer is evaluated, producing a reward such as </em><strong><em>r = 1</em></strong><em> for a correct answer or </em><strong><em>r = 0</em></strong><em> for an incorrect answer.</em></p><p><em>Multiple trajectories can be sampled, producing different rewards.</em></p><p><em>These rewards can then be converted into </em><strong><em>normalized advantages</em></strong><em>, which provide the learning signal for the policy.</em></p><p>Finally, the image above introduce <strong>PPO and GRPO</strong> as practical policy-optimization methods.</p><p>PPO uses the probability ratio <strong>rₜ(θ) = πθ(aₜ|sₜ) / πₒₗ𝒹(aₜ|sₜ)</strong> and clips this ratio to prevent the new policy from changing too much:</p><p><strong>Lᴾᴾᴼ = Σₜ min(rₜAₜ, clip(rₜ,1−ε,1+ε)Aₜ)</strong>.</p><p>GRPO follows the same policy-gradient principle but computes <strong>relative advantages from a group of sampled responses</strong>, typically using <strong>Aᵢ = (Rᵢ − mean(R)) / std(R)</strong>.</p><p>The overall progression shown is therefore</p><p><strong>Agent–Environment Interaction → Trajectories → Rewards → Policy Gradient → Baseline/Advantage → Normalized Advantages → PPO/GRPO → Improved LLM Policy</strong>.</p><h3>PPO and GRPO — Complete Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WJfxg2JdmRzOcxfRdGljWA.png" /></figure><p>PO compares the new policy with the old policy using the probability ratio <strong>rₜ = πθ(aₜ|sₜ) / πₒₗ𝒹(aₜ|sₜ)</strong> and uses clipping, <strong>clip(rₜ, 1−ε, 1+ε)</strong>, to prevent excessively large policy updates.</p><p>Its objective is <strong>Lᴾᴾᴼ = Σₜ min(rₜAₜ, clip(rₜ, 1−ε, 1+ε)Aₜ)</strong>, where <strong>Aₜ</strong> represents how good an action was compared with the expected outcome.</p><p>GRPO extends this idea by adding a KL-divergence penalty to keep the current policy close to a reference policy: <strong>Lᴳᴿᴾᴼ = Lᴾᴾᴼ − βDᴷᴸ(πθ‖πᵣₑ𝒻)</strong>.</p><p>In LLM training, GRPO generates multiple responses for the same prompt, evaluates their rewards, calculates relative or normalized advantages, computes the <em>clipped policy loss plus KL penalty</em>, and LM’s policy parameters using policy gradient descent.</p><p>Thus, the overall process is :</p><p><strong>generate multiple responses → calculate rewards → compute advantages → apply PPO-style clipping and KL regularization → update the policy</strong>,</p><p>making the LLM more likely to produce high-reward responses while keeping training stable.</p><h3>Proximal Policy Optimization (PPO)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w4Cwa6nb0eu7pSwt-Dh2zw.png" /></figure><p>PPO is a <strong>policy-gradient algorithm that improves the policy while preventing it from changing too much in a single update</strong>.</p><p>It starts with the vanilla policy-gradient objective <strong>Lᴾᴳ(θ) = 𝔼[log πθ(aₜ|sₜ) Âₜ]</strong>, where <strong>Âₜ</strong> is the advantage.</p><p>The problem with vanilla policy gradients is that a large update can make the new policy unstable.</p><p>TRPO solves this using a KL-divergence constraint, but that constrained optimization is computationally expensive.</p><p>PPO provides a simpler approximation by using the <strong>probability ratio</strong> <strong>rₜ(θ) = πθ(aₜ|sₜ) / πₒₗ𝒹(aₜ|sₜ)</strong> and clipping it to <strong>[1−ε, 1+ε]</strong>.</p><p>Its objective is <strong>Lᴾᴾᴼ(θ) = 𝔼[min(rₜÂₜ, clip(rₜ,1−ε,1+ε)Âₜ)]</strong>.</p><p>If <strong>Âₜ &gt; 0</strong>, PPO increases the probability of the good action but stops benefiting once <strong>rₜ &gt; 1+ε</strong>; if <strong>Âₜ &lt; 0</strong>, it decreases the probability of the bad action but prevents it from decreasing beyond <strong>1−ε</strong>.</p><p>The overall process is <strong>collect trajectories → calculate advantages → optimize the clipped objective → update policy parameters using the policy gradient → repeat</strong>.</p><p>Thus, PPO can be viewed as <strong>vanilla policy gradient + clipping</strong>, giving a simple, efficient, and stable method widely used for LLM reinforcement learning.</p><h3>PPO with GAE, Value Estimation &amp; Entropy Bonus</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GZ44YwL9Co5bHr0Aw0K7Jg.png" /></figure><p>This image explains the <strong>complete PPO training objective</strong>, showing how PPO combines the <strong>policy update, GAE advantage estimation, value-function learning, and entropy-based exploration</strong>.</p><p>First, PPO updates the policy parameters using <strong>gradient ascent</strong>,</p><p><strong>θₜ₊₁ = θₜ + α∇L𝒸ₗᵢₚ(θ)</strong>.</p><p>The advantage <strong>Aₜ</strong> is estimated using GAE,</p><p><strong>Aₜ = Σₗ₌₀∞(γλ)ˡδₜ₊ₗ</strong>,</p><p>where the TD error is <strong>δₜ₊ₗ = rₜ₊ₗ + γV(sₜ₊ₗ₊₁) − V(sₜ₊ₗ)</strong>.</p><p>The value network is trained using the value-function loss</p><p><strong>Lᵥꜰ(θ) = (Vθ(sₜ) − Vₜᵗᵃʳᵍᵉᵗ)²</strong>, so the critic learns to estimate how valuable each state is.</p><p>PPO then uses the <strong>clipped surrogate objective</strong> to prevent the new policy from changing too much, while an</p><p><strong>entropy bonus</strong> <strong>H(πθ) = −𝔼[log πθ(a|sₜ)]</strong> encourages exploration and prevents premature convergence.</p><p>The complete objective is <strong>maxθ 𝔼[L𝒸ₗᵢₚ(θ) − c₁Lᵥꜰ(θ) + c₂H(πθ)]</strong>,</p><p>where <strong>c₁</strong> controls the value-loss contribution and <strong>c₂</strong> controls the exploration bonus.</p><p>Therefore, PPO balances three goals: <strong>improve performance, keep policy updates stable, and maintain sufficient exploration</strong>, with the policy ultimately updated using <strong>gradient ascent</strong>.</p><h3>Proximal Policy Optimization (PPO)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V58VwG6Jy-5jKKT03E80Hw.png" /></figure><p>First, vanilla policy gradient tries to maximize the expected reward using <strong>∇J(θ) = 𝔼[∇log πθ(aₜ|sₜ) Âₜ]</strong>, where <strong>Âₜ</strong> tells whether the selected action was better or worse than expected.</p><p>TRPO introduced a <strong>KL-divergence constraint</strong> to prevent the new policy from moving too far from the old policy, but solving this constrained optimization is computationally expensive.</p><p>PPO provides a simpler solution by defining the probability ratio</p><p><strong>rₜ(θ) = πθ(aₜ|sₜ) / πₒₗ𝒹(aₜ|sₜ)</strong> and</p><p>clipping it to <strong>[1−ε, 1+ε]</strong>.</p><p>The PPO objective is <strong>Lᴾᴾᴼ(θ) = 𝔼[min(rₜÂₜ, clip(rₜ,1−ε,1+ε)Âₜ)]</strong>.</p><p>When <strong>Âₜ &gt; 0</strong>, PPO wants to increase the probability of the good action, but clipping prevents <strong>rₜ</strong> from increasing beyond <strong>1+ε</strong>;</p><p>when <strong>Âₜ &lt; 0</strong>, PPO wants to decrease the probability of the bad action, but clipping prevents <strong>rₜ</strong> from going below <strong>1−ε</strong>.</p><p>Therefore, PPO keeps policy updates within a safe range without requiring the expensive KL-constrained optimization of TRPO.</p><p>The overall training process is <strong>sample prompts → generate responses with the current policy → evaluate rewards → compute advantages → optimize the clipped PPO objective using policy gradients → update the policy → repeat</strong>.</p><p>This stability and simplicity make PPO particularly useful for <strong>LLM alignment and RLHF</strong>.</p><h3>Reinforcement Learning in LLMs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-o3cpV8l_oeAkW_fClUrFg.png" /><figcaption>Reinforcement Learning in LLMs</figcaption></figure><p>An <strong>LLM can be treated as an RL agent</strong> that generates text one token at a time.</p><p>The <strong>state sₜ</strong> is the prompt plus previously generated tokens, and the <strong>action aₜ</strong> is the next token selected by <strong>πθ(aₜ|sₜ)</strong>.</p><p>An evaluator provides a <strong>reward</strong> based on the quality of the final response, such as <strong>r = 1</strong> for correct and <strong>r = 0</strong> for incorrect.</p><p>The goal is to maximize expected reward using the policy gradient <strong>∇J(θ) = 𝔼[R∇θ log πθ(y|x)]</strong>.</p><p>The overall process is <strong>Prompt → Generate tokens → Evaluate → Calculate reward/advantage → Update θ → Better responses</strong>, using methods such as <strong>REINFORCE, PPO, or GRPO</strong>.</p><h3>Reinforcement Learning from Human Feedback (RLHF)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1BnRq8JGj7Zta7XHSYtr2Q.png" /><figcaption>Reinforcement Learning from Human Feedback (RLHF)</figcaption></figure><p>This explains the <strong>RLHF pipeline used to align an LLM with human preferences</strong>.</p><p>First, humans compare two responses to the same prompt and choose which one is better, creating <strong>pairwise preference data</strong>.</p><p>This data is then used to train a <strong>reward model</strong> rϕ​(x,y), which learns to give higher scores to responses that humans prefer.</p><p>Once trained, the reward model evaluates responses generated by the LLM and provides a reward score.</p><p>The LLM policy <strong>πθ</strong> is then optimized using reinforcement learning to maximize the expected reward: <strong>J(θ) = 𝔼[rφ(x,y)]</strong>, with the policy gradient <strong>∇θJ(θ) = 𝔼[rφ(x,y)∇θ log πθ(y|x)]</strong>.</p><p>The overall process is <strong>Human Preferences → Train Reward Model → LLM Generates Responses → Reward Model Scores Them → Policy Gradient Updates LLM → Better Human-Aligned Responses</strong>. In practice, algorithms such as <strong>PPO</strong> can be used for the policy optimization step.</p><h3>PPO and GRPO for LLMs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N95bR0KbCX4IUVM_HDFUXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*clb-6iafFD99bMvE2ec_iQ.png" /></figure><p>An <strong>LLM can be treated as a Reinforcement Learning agent</strong> that generates text one token at a time, where the state <strong>sₜ</strong> is the prompt plus previously generated tokens and the action <strong>aₜ</strong> is the next token chosen by <strong>πθ(aₜ|sₜ)</strong>.</p><p>A reward model evaluates the complete response and provides a reward based on qualities such as correctness, helpfulness, or safety.</p><p>The <strong>advantage Aₜ = Q(sₜ,aₜ) − V(sₜ)</strong> tells whether an action was better or worse than expected, and GAE can provide a more stable estimate of this advantage.</p><p>PPO then compares the new and old policies using <strong>rₜ = πθ(aₜ|sₜ) / πₒₗ𝒹(aₜ|sₜ)</strong> and clips this ratio to <strong>[1−ε, 1+ε]</strong> to prevent excessively large policy updates.</p><p>Its objective is <strong>Lᴾᴾᴼ = 𝔼[min(rₜAₜ, clip(rₜ,1−ε,1+ε)Aₜ)]</strong>. <strong>GRPO</strong> extends this idea by generating multiple responses for the same prompt and calculating relative or normalized advantages,</p><p>for example <strong>Aᵢ = (Rᵢ − mean(R)) / std(R)</strong>, while also using a KL penalty to keep the policy close to a reference model.</p><p>Overall, the process is <strong>Generate responses → Evaluate rewards → Calculate advantages → Compute PPO/GRPO objective → Update policy parameters using policy gradient → Generate better responses</strong>.</p><h3>Group Relative Policy Optimization (GRPO) for LLMs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LDng7oqKnSkGPtdU61UpmQ.png" /></figure><p><strong>GRPO</strong> is a reinforcement-learning method for training LLMs by generating <strong>multiple responses to the same prompt</strong>, evaluating their rewards, and using their <strong>relative performance</strong> to calculate advantages.</p><p>The LLM acts as the policy <strong>πθ</strong>, the prompt and previously generated tokens form the state <strong>sₜ</strong>, and the next token <strong>yₜ</strong> is the action.</p><p>For a prompt <strong>x</strong>, the LLM generates a completion <strong>y = (y₁,…,yₜ)</strong> according to <strong>πθ(yₜ | x,y₍₍ₜ₎)</strong>.</p><p>PPO can then be applied at the token level using the probability ratio <strong>rₜ(θ) = πθ(yₜ|x,y₍₍ₜ₎) / πₒₗ𝒹(yₜ|x,y₍₍ₜ₎)</strong> and the clipped objective <strong>Lᴾᴾᴼ = 𝔼[min(rₜAₜ, clip(rₜ,1−ε,1+ε)Aₜ)]</strong>, which prevents excessively large policy updates.</p><p>The key idea in <strong>GRPO</strong> is to generate a <strong>group G of multiple outputs</strong> for the same prompt, obtain a reward for each response, and calculate a <strong>normalized relative advantage</strong> rather than requiring a separate value model.</p><p>Conceptually, for response <em>i</em>, the advantage can be normalized as <strong>Aᵢ = (Rᵢ − μᴳ) / (σᴳ + ε)</strong>, where <strong>μᴳ</strong> and <strong>σᴳ</strong> are the mean and standard deviation of rewards within the group.</p><p>A response with above-average reward gets <strong>Aᵢ &gt; 0</strong>, so the policy increases its probability; a below-average response gets <strong>Aᵢ &lt; 0</strong>, so its probability is decreased.</p><p>GRPO also adds a <strong>KL-divergence penalty</strong>, <strong>−βDᴷᴸ(πθ‖πᵣₑ𝒻)</strong>, to prevent the updated LLM from moving too far from a reference policy.</p><p>Therefore, the complete GRPO process is <strong>Prompt → Generate multiple responses → Calculate rewards → Compare rewards within the group → Normalize advantages → Calculate token-level probability ratios → Apply PPO clipping → Add KL penalty → Compute GRPO objective → Gradient ascent/update θ → Generate better responses</strong>.</p><p>The main advantage of GRPO is that it can use <strong>relative rewards among multiple responses without requiring a separate critic/value model</strong>, making it particularly useful for LLM reasoning tasks where rewards such as <strong>correctness, mathematical accuracy, or verifiable outcomes</strong> can be computed for several candidate responses.</p><h3>PPO vs GRPO — Pipeline &amp; Core Difference</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9CJyk7iVbpH2ltyHXszGaQ.png" /></figure><p><strong>PPO:</strong><br> <strong>LLM → Response → Reward → Value Model → Advantage → Clipping → Update LLM</strong></p><p><strong>GRPO:</strong><br> <strong>LLM → Multiple Responses → Rewards → Group-relative Advantage → Clipping + KL → Update LLM</strong></p><p>The biggest difference is <strong>how the advantage is calculated</strong>: PPO typically needs a separate <strong>value/critic model</strong> to estimate <strong>V(s)</strong>,</p><p>while GRPO uses the <strong>relative rewards of multiple responses from the same prompt</strong>, so it can avoid a separate value model.</p><h3>GRPO with Outcome Supervision</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pLWqzWb3AuVexlolnMcsJQ.png" /><figcaption>GRPO with Outcome Supervision</figcaption></figure><p><strong>GRPO</strong> generates <strong>multiple responses for the same prompt</strong>, gives each a <strong>final reward</strong>, and compares their rewards within the group.</p><p>It calculates the normalized advantage as <strong>Âᵢ = (rᵢ − μ) / σ</strong>, where <strong>μ</strong> is the group’s mean reward.</p><p>Positive advantage means the response is better than average, so its probability is increased; negative advantage means it is worse, so its probability is decreased.</p><p>Unlike PPO, GRPO <strong>does not need a separate value/critic model</strong> — the group statistics provide the relative baseline.</p><p><strong>Pipeline:</strong> <strong>Prompt → N responses → Final rewards → Mean/variance → Advantages → Policy gradient → Update LLM.</strong></p><h3>GRPO and Reasoning Models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MUE0jmcSYhV_xg9Admms6Q.png" /><figcaption>GRPO and Reasoning Models</figcaption></figure><p>This explains how <strong>GRPO can be used to turn a normal LLM into a reasoning model</strong> through reinforcement learning.</p><p>The LLM generates <strong>multiple responses for the same question</strong>, and each response is evaluated using a <strong>verifier or reward function</strong>;</p><p>for example, a correct math answer gets <strong>r = 1</strong> and an incorrect answer gets <strong>r = 0</strong>.</p><p>GRPO compares these rewards within the group and calculates a relative advantage, <strong>Âᵢ = (rᵢ − μ) / (σ + ε)</strong>, so better-than-average responses become more likely and worse responses become less likely.</p><p>Unlike PPO, GRPO does not require a separate value/critic model; it uses <strong>group reward statistics</strong> to estimate advantages while still using PPO-style clipping and policy-gradient updates.</p><p>For reasoning tasks, the reward can be <strong>verifiable</strong>, such as mathematical correctness or passing coding tests, rather than requiring a learned reward model.</p><p>During training, the model can gradually discover useful behaviors such as <strong>rechecking its answer, recognizing mistakes, reconsidering an approach, and trying alternative reasoning paths</strong>.</p><p>Format rewards can also encourage structured outputs such as &lt;think&gt;...&lt;/think&gt; and &lt;answer&gt;...&lt;/answer&gt;.</p><p>Thus, the overall process is :</p><p><strong>Question → Generate multiple responses → Verify/reward → Calculate relative advantages → GRPO policy update → Repeat</strong>,</p><p>allowing the LLM to autonomously develop stronger reasoning behavior.</p><h3>GRPO Task &amp; Training Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KrQ79jfgGlCtOdOzQAzKVQ.png" /><figcaption>GRPO Task &amp; Training Flow</figcaption></figure><p><strong>Prompt → Generate multiple responses → Evaluate rewards → Compare responses → Calculate advantages → PPO-style update → Improve LLM.</strong></p><ul><li><strong>Math/Reasoning:</strong> verify correctness.</li><li><strong>Coding:</strong> run tests.</li><li><strong>Factual:</strong> verify facts.</li><li><strong>Preference:</strong> reward model/human preference.</li><li><strong>Format:</strong> check required format.</li><li><strong>Tool/Agent:</strong> check task success.</li></ul><p><strong>Key idea:</strong> Higher-reward responses get higher probability; lower-reward responses get lower probability. GRPO can use <strong>verifiable rewards</strong> without a separate value model.</p><h3>GRPO Fine-Tuning of Qwen 2.5–3B: An 8-Step End-to-End Workflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YX6OXvnaoe-Lb2Oxn-xNfQ.png" /></figure><p><strong>Group Relative Policy Optimization (GRPO)</strong> is a reinforcement-learning method for improving LLMs using <strong>multiple generated responses and their relative rewards</strong>.</p><p>Instead of giving the model supervised reasoning traces, we can define rewards that are automatically verifiable — for example, whether a mathematical answer is correct or whether the response follows a required format.</p><p>A practical GRPO workflow can be built using <strong>Qwen 2.5–3B, Unsloth, and vLLM</strong>.</p><h4>1. Install Unsloth and vLLM</h4><p>The first step is to install <strong>Unsloth</strong> for memory-efficient LLM fine-tuning and <strong>vLLM</strong> for fast inference. vLLM is particularly useful in GRPO because the model needs to generate multiple responses for the same prompt.</p><h4>2. Load Qwen 2.5–3B</h4><p>Next, load the pretrained <strong>Qwen 2.5–3B Instruct</strong> model.</p><p>The example uses <strong>4-bit quantization</strong> to reduce GPU memory requirements. Unsloth handles efficient training while vLLM accelerates response generation.</p><h4>3. Define the Training Template</h4><p>We then specify how the model should structure its response, for example:</p><p><strong>&lt;reasoning&gt; ... &lt;/reasoning&gt;</strong><br> <strong>&lt;answer&gt; ... &lt;/answer&gt;</strong></p><p>This gives the reward functions a predictable structure to evaluate. The goal is not simply to make the model produce these tags, but to establish a consistent format for reasoning and final answers.</p><h4>4. Prepare the Training Dataset</h4><p>The next step is to prepare a dataset such as <strong>GSM8K</strong>, containing mathematical questions and their expected answers. Each example is converted into a prompt that the LLM can solve.</p><p>The important point is that the dataset provides the <strong>problem and verification target</strong>, rather than necessarily providing a complete reasoning trace that the model must imitate.</p><h4>5. Define the Reward Functions</h4><p>This is the core of GRPO training. The model generates several answers for the same question, and each response receives numerical rewards.</p><p>For example:</p><ul><li><strong>Correctness reward:</strong> Is the final mathematical answer correct?</li><li><strong>Format reward:</strong> Does the response follow the required format?</li><li><strong>Strict-format reward:</strong> Does it exactly match the expected structure?</li><li><strong>Soft-format reward:</strong> Does it approximately follow the structure?</li></ul><p>Thus:</p><p><strong>Prompt → Multiple responses → Reward each response</strong></p><p>For example, if four responses receive:</p><p><strong>1.0, 0.2, 0.8, 0.0</strong></p><p>GRPO compares these responses and determines which behaviors should become more or less likely.</p><h4>6. Configure the GRPO Trainer</h4><p>The GRPO trainer is then configured with parameters such as the <strong>learning rate, number of generations per prompt, batch size, sequence length, optimizer, epochs, and vLLM settings</strong>.</p><p>Unlike traditional PPO, GRPO does not require a separate <strong>value model</strong> to estimate the baseline. Instead, it uses the rewards from the group of generated responses to calculate <strong>relative advantages</strong>.</p><p>Conceptually:</p><p><strong>Group rewards → Mean/std → Relative advantages → GRPO objective</strong></p><h4>7. Start the GRPO Training Loop</h4><p>Now the actual reinforcement-learning process begins:</p><p><strong>Question → Qwen generates multiple responses → Evaluate rewards → Calculate group-relative advantages → Compute GRPO loss → Gradient update → Updated Qwen</strong></p><p>Suppose the same question produces four responses with rewards:</p><p><strong>1.0, 0.7, 0.1, 0.0</strong></p><p>The response with a high relative reward receives a <strong>positive advantage</strong>, so the policy is encouraged to increase its probability. Responses performing worse than the group receive lower or negative advantages.</p><p>The model therefore gradually learns behaviors that lead to higher rewards.</p><h4>8. Test the Model Before and After GRPO</h4><p>Finally, evaluate the original model and the GRPO-trained model on new questions.</p><p>The comparison is essentially:</p><p><strong>Before GRPO:</strong><br> Question → Original Qwen → Response</p><p><strong>After GRPO:</strong><br> Question → GRPO-trained Qwen → Improved response</p><p>For reasoning tasks, the improvement may appear as better <strong>answer correctness, verification, rechecking, alternative approaches, and structured reasoning</strong>.</p><h4>The Complete Picture</h4><p>The entire workflow can be summarized as:</p><p><strong>Qwen 2.5–3B → Generate multiple answers → Calculate verifiable rewards → Compute group-relative advantages → GRPO loss → Gradient update → Repeat → Better reasoning behavior</strong></p><p>The key idea is that <strong>GRPO does not directly teach the model what reasoning steps to write</strong>. Instead, it gives the model feedback about <strong>which generated behaviors lead to better outcomes</strong>, and the policy gradually shifts toward those behaviors.</p><h3>SFT + GRPO: Complete Reasoning-Model Training Pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vwOz-WPHd3oLWKOufO7uIA.png" /></figure><p>This workflow combines <strong>Supervised Fine-Tuning (SFT)</strong> with <strong>GRPO reinforcement learning</strong> to turn a pretrained LLM into a stronger reasoning model.</p><h4>1. Choose a Base Model</h4><p>Start with a pretrained model such as <strong>Qwen 2.5–3B</strong>. The base model already understands language but may not reliably perform structured, step-by-step reasoning.</p><h4>2. Build the SFT Dataset</h4><p>Create high-quality examples containing:</p><p><strong>Question → Reasoning → Final Answer</strong></p><p>For example:</p><p>What is 27 × 43?</p><p>The dataset might contain the reasoning steps and the correct answer. Human-written or high-quality generated reasoning examples are used here.</p><h4>3. Supervised Fine-Tuning (SFT)</h4><p>Train the base model to <strong>imitate these good reasoning examples</strong>.</p><p>So SFT teaches:</p><blockquote><strong><em>“This is what a good reasoning response looks like.”</em></strong></blockquote><p>After SFT, the model becomes better at producing structured reasoning, but it is still mainly learning by imitation.</p><h4>4. Define Reward Functions</h4><p>Now switch from imitation to <strong>evaluation</strong>. Define automatic rewards for generated responses, such as:</p><ul><li><strong>Accuracy reward:</strong> Is the final answer correct?</li><li><strong>Format reward:</strong> Does it follow &lt;reasoning&gt;...&lt;/reasoning&gt;&lt;answer&gt;...&lt;/answer&gt;?</li><li><strong>Length reward:</strong> Is the response reasonably sized?</li><li><strong>Math/logic reward:</strong> Are intermediate steps valid?</li><li><strong>Code reward:</strong> Do all tests pass?</li></ul><p>These rewards can be combined:</p><p><strong>Total Reward = Accuracy + Format + Logic + Code + …</strong></p><p>The important point is that the reward should measure <strong>how good the generated response is</strong>, rather than providing the reasoning itself.</p><h4>5. Train with GRPO</h4><p>For each question, GRPO generates <strong>multiple responses</strong>:</p><p><strong>Question → Response 1, Response 2, …, Response N</strong></p><p>Each response receives a reward. GRPO then compares the responses <strong>relative to one another</strong>.</p><p>For example:</p><p><strong>R₁ = 0.2, R₂ = 1.5, R₃ = 2.0, R₄ = 0.1</strong></p><p>The better responses receive positive relative advantages, while worse responses receive negative advantages.</p><p>GRPO then optimizes the policy:</p><p><strong>Higher advantage → increase probability</strong><br> <strong>Lower advantage → decrease probability</strong></p><p>Unlike PPO, GRPO can obtain the relative baseline from the <strong>group of responses</strong>, so it does not require a separate value model.</p><h4>6. Repeat and Improve</h4><p>The process can be repeated:</p><p><strong>SFT → GRPO → Evaluate → GRPO/SFT again → Evaluate</strong></p><p>The model gradually becomes better at producing responses that receive high rewards.</p><h4>The Key Difference Between SFT and GRPO</h4><p><strong>SFT = learning by imitation</strong></p><blockquote><em>“Here is a good reasoning example. Learn to produce responses like this.”</em></blockquote><p><strong>GRPO = learning by feedback</strong></p><blockquote><em>“Here are several attempts. These ones performed better. Increase the probability of behaviors that produced them.”</em></blockquote><p>Therefore, the complete reasoning-model pipeline is:</p><p><strong>Base LLM → SFT → Reasoning Model → Generate multiple solutions → Verify/reward → GRPO → Better reasoning model → Evaluate → Repeat</strong></p><h4>The main idea</h4><p><strong>SFT teaches the model how to reason; GRPO teaches the model which reasoning behaviors actually work.</strong></p><p>This is why combining the two is powerful:</p><p><strong>SFT provides a good starting policy, while GRPO further optimizes it using measurable outcomes such as mathematical correctness, code execution, logical validity, or other verifiable rewards.</strong></p><h3>SFT + Knowledge Distillation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jpTk5sGMY0wJ1AyZLntBuQ.png" /></figure><p>This workflow <strong>takes a powerful Teacher LLM and transfers its reasoning ability to a smaller Student model using supervised fine-tuning</strong>.</p><ol><li><strong>Choose Teacher Model</strong> → Use a strong model such as DeepSeek-R1, GPT-o3, Claude, or Gemini to generate high-quality reasoning.</li><li><strong>Generate Reasoning Dataset</strong> → Give the Teacher many questions and collect its <strong>reasoning + final answers</strong>. This becomes the distilled training dataset.</li><li><strong>Prepare Dataset</strong> → Convert the Teacher’s outputs into an SFT format such as <strong>prompt → reasoning → answer</strong>, usually stored in JSONL/Parquet.</li><li><strong>Fine-Tune Student (SFT)</strong> → Train a smaller model such as <strong>Qwen 2.5–3B, Llama 3.2–3B, or Phi-3 Mini</strong> to imitate the Teacher’s reasoning patterns using standard next-token prediction.</li><li><strong>Evaluate &amp; Deploy</strong> → Test the Student on benchmarks such as <strong>GSM8K, MATH, AIME, GPQA, and coding tasks</strong>, then deploy it if performance is sufficient.</li></ol><h4>Core idea</h4><p><strong>Teacher → generates high-quality reasoning → reasoning dataset → SFT → Student</strong></p><p>Unlike <strong>GRPO</strong>, there is <strong>no reward optimization or group comparison</strong> here. The Student simply learns to <strong>imitate the Teacher’s generated reasoning</strong>.</p><p><strong>Why use it?</strong> It is cheaper and faster than training a large model with RL and allows a small model to inherit useful reasoning behavior from a much stronger model.</p><p><strong>Main limitation:</strong> the Student can inherit the Teacher’s <strong>mistakes and biases</strong>, and it may not discover reasoning strategies beyond what the Teacher demonstrates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQiXqhIuhAzHUnRs5fJt4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FauHx1ufWhyxjgqWaW9msg.png" /></figure><h3>Conclusion</h3><p>The blog follows the evolution from <strong>traditional value-based RL to modern LLM reinforcement learning</strong>:</p><p>function approximation → MC/TD learning → policy gradients → REINFORCE → Actor–Critic → GAE → TRPO → PPO → GRPO → reasoning models.</p><p>The central idea is that <strong>SFT teaches an LLM through examples, while GRPO improves it through feedback from rewards</strong>. GRPO is especially useful when outcomes can be verified, such as mathematics and coding, because multiple responses can be compared using their relative rewards without requiring a separate value model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad80660d5d29" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Diffusion Language Models Explained: From Theory to Training Pipeline]]></title>
            <link>https://medium.com/@sujangyawali177/building-duffusion-language-model-from-scratch-1d5d7d2614e5?source=rss-23434c2aa8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d5d7d2614e5</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[diffusion-model-text]]></category>
            <category><![CDATA[stable-diffusion-model]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <dc:creator><![CDATA[Sujangyawali]]></dc:creator>
            <pubDate>Sun, 26 Jul 2026 01:07:52 GMT</pubDate>
            <atom:updated>2026-07-30T22:09:04.628Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Think of solving a jigsaw puzzle. Traditional language models place one puzzle piece after another until the picture is complete. Diffusion Language Models, on the other hand, start with a messy, blurry picture and slowly make it clearer until the final image appears.</strong> This article explains how that process works in simple steps before diving into the mathematics and algorithms.</p><p>By the End of This Article, You Will Understand</p><ol><li><strong>Introduction to Diffusion Models</strong></li><li><strong>History and Evolution of Generative AI</strong></li><li><strong>Probabilistic Foundations of Diffusion Models</strong></li><li><strong>Forward Diffusion (Noising) Process</strong></li><li><strong>Reverse Diffusion (Denoising) Process</strong></li><li><strong>Autoregressive Language Models (ARMs)</strong></li><li><strong>Limitations of Autoregressive Models</strong></li><li><strong>Birth of Diffusion Language Models (DLMs)</strong></li><li><strong>Masked Language Modeling (MLM)</strong></li><li><strong>Training Objectives of Diffusion Language Models</strong></li><li><strong>Noise Prediction and Denoising Algorithms</strong></li><li><strong>Noise Schedules and Masking Strategies</strong></li><li><strong>Autoregressive Masked Sampling</strong></li><li><strong>Conditional Text Generation</strong></li><li><strong>Training Pipeline of Diffusion Language Models</strong></li><li><strong>Inference and Sampling Process</strong></li><li><strong>ARMs vs Diffusion Language Models</strong></li><li><strong>Real-World Applications of Diffusion Language Models</strong></li><li><strong>Building a Diffusion Language Model from Scratch</strong></li><li><strong>Future of Diffusion-Based Language Models</strong></li></ol><h3>Introduction</h3><p><strong>Diffusion Models for Image Generation</strong> are a powerful class of generative AI models that create realistic images by learning to reverse the process of adding noise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JdJvWaJUYRc8jyigDeHA1g.png" /></figure><p>Unlike autoregressive models, which generate outputs one token or pixel at a time, diffusion models begin with pure random noise and gradually remove it through a series of denoising steps until a clear and meaningful image is produced.</p><p>This coarse-to-fine generation process allows the model to first capture the overall structure of the image and then progressively refine textures and fine details.</p><p>Compared to earlier approaches such as Generative Adversarial Networks (GANs), which often suffered from problems like mode collapse and unstable training, diffusion models are more stable and capable of generating diverse, high-quality images.</p><p>During training, the model learns to predict and remove Gaussian noise that has been added to real images.</p><p>During inference, it starts from random noise and repeatedly applies the learned denoising process, guided by a text prompt, to synthesize a new image.</p><p>This approach forms the foundation of modern text-to-image systems such as <strong>Stable Diffusion, SDXL, FLUX, and DALL·E</strong>, making diffusion models one of the most significant advances in generative AI.</p><h3>History of Diffusion Models</h3><p>The evolution of image generation has progressed through several important milestones. Early generative models, such as <strong>Variational Autoencoders (VAEs)</strong> introduced in <strong>2013</strong>, learned compact latent representations of images but often produced blurry outputs due to their probabilistic reconstruction process.</p><p>In <strong>2014</strong>, <strong>Generative Adversarial Networks (GANs)</strong> revolutionized image synthesis by training a <strong>Generator</strong> and a <strong>Discriminator</strong> in an adversarial setting, enabling the creation of highly realistic images.</p><p>However, GANs suffered from several limitations, including unstable training, mode collapse (where the model generates only a limited variety of images), and difficulties in capturing the full data distribution.</p><p>A major breakthrough came in <strong>2020</strong> with the introduction of <strong>Denoising Diffusion Probabilistic Models (DDPMs)</strong> by Jonathan Ho and colleagues.</p><p>Instead of generating images directly, diffusion models learn to reverse a gradual noising process by starting with random Gaussian noise and iteratively removing it until a realistic image is reconstructed.</p><p>This denoising framework provides stable training, high-quality image generation, and greater diversity than GANs.</p><p>Since then, diffusion models have become the dominant approach for generative image synthesis and serve as the foundation for state-of-the-art text-to-image systems such as <strong>Stable Diffusion</strong>, <strong>DALL·E 2/3</strong>, <strong>Imagen</strong>, <strong>SDXL</strong>, and <strong>FLUX</strong>, significantly advancing the field of generative AI.</p><h3>Probabilistic Foundation of Diffusion Models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r7zPL3OxHzwX-V1JR0VuMA.png" /></figure><p>Diffusion models are fundamentally <strong>probabilistic generative models</strong> that aim to learn the underlying probability distribution of real-world data rather than memorizing individual images.</p><p>Every image can be represented as a high-dimensional vector in pixel space, where each pixel corresponds to a feature.</p><p>Although the pixel space contains an enormous number of possible image combinations, only a very small region represents realistic images, forming the <strong>true data distribution (P₍data₎)</strong>.</p><p>The objective of a diffusion model is to learn a <strong>model distribution (P₍model₎)</strong> that closely approximates <strong>P₍data₎</strong>.</p><p>To achieve this, the training process gradually corrupts real images by adding Gaussian noise through a forward diffusion process until they become pure noise.</p><p>The model is then trained to reverse this process by predicting and removing the added noise step by step, thereby learning how to transform random noise back into realistic images.</p><p>During inference, the model starts from pure Gaussian noise and iteratively denoises it, progressively recovering meaningful structures, textures, and fine details that follow the learned data distribution.</p><p>The training objective is to minimize the difference between <strong>P₍model₎</strong> and <strong>P₍data₎</strong>, enabling the model to generate diverse, high-quality, and realistic images.</p><p>This probabilistic framework forms the foundation of modern text-to-image systems such as <strong>Stable Diffusion, SDXL, FLUX, and DALL·E</strong>.</p><ul><li><strong>P₍data₎</strong> — <strong>True Data Distribution:</strong> The actual probability distribution of real images in the training dataset.</li><li><strong>P₍model₎</strong> — <strong>Model Distribution:</strong> The probability distribution learned by the diffusion model, which aims to approximate <strong>P₍data₎</strong>.</li></ul><p>The goal of training is to make <strong>P₍model₎ ≈ P₍data₎</strong>, so that the generated images become indistinguishable from real images.</p><h3>Training and Generation Process in Diffusion Models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NNTgwm7C4zGHLQr75ZiAtA.png" /><figcaption>Training and Generation Process in Diffusion Models</figcaption></figure><p>This figure illustrates the complete workflow of diffusion models, including both the <strong>forward diffusion (noising)</strong> process used during training and the <strong>reverse diffusion (denoising)</strong> process used during image generation.</p><p>In the forward diffusion process, a real image <strong>x₀</strong> from the training dataset is progressively corrupted by adding small amounts of Gaussian noise according to a predefined noise schedule until it becomes nearly pure noise <strong>xₜ</strong> (or <strong>xₜ</strong> at the final timestep).</p><p>Rather than learning to reconstruct the original image directly, the model learns to predict the exact Gaussian noise <strong>ε</strong> that was added.</p><p>During each training iteration, a random timestep <strong>t</strong> is selected, and the corresponding noisy image <strong>xₜ</strong>, along with the timestep <strong>t</strong>, is provided as input to a neural network, typically a <strong>U-Net</strong>.</p><p>The network predicts the added noise <strong>ε̂</strong>, which is then compared with the true noise <strong>ε</strong> used to generate <strong>xₜ</strong>.</p><p>The prediction error is computed using the Mean Squared Error (MSE) loss, <strong>L = ‖ε − ε̂‖²</strong>, and the network parameters are updated through backpropagation.</p><p>By repeating this process over millions of images and randomly selected timesteps, the model learns how images are progressively corrupted and, more importantly, how to reverse this corruption.</p><p>During inference, the model starts from pure Gaussian noise <strong>xₜ ∼ N(0, I)</strong> and repeatedly predicts and removes the noise at each timestep, gradually transforming the noisy sample into a clean and realistic image <strong>x₀</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AsuW4elmOFdSImvYf-orSQ.png" /><figcaption>Diffusion Model</figcaption></figure><p>Thus, <strong>noise prediction serves as the bridge between the forward noising process and the reverse denoising process</strong>, enabling the model to learn the underlying data distribution <strong>P₍data₎</strong> and generate high-quality images that closely follow the learned model distribution <strong>P₍model₎</strong>.</p><h3>Notes:</h3><h4>General Generative Models</h4><ul><li><strong>Goal:</strong> Learn the underlying probability distribution of real data (<strong>P₍data₎</strong>) and generate new samples by sampling from it.</li></ul><h4>Conditional Generative Models (Text-to-Image)</h4><ul><li><strong>Goal:</strong> Learn the conditional probability distribution <strong>P₍data₎(x | c)</strong>, where <strong>c</strong> is the input prompt.</li><li>The model generates images by sampling from the distribution of images that match the given prompt.</li><li>The text prompt guides the generation process so that the output is both realistic and semantically aligned with the input.</li></ul><h3>From Autoregressive Models to Diffusion Models: Motivation, Training, and Advantages</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SNkI-ZRU4HLZqecDwNZpzA.png" /><figcaption>From Autoregressive Models to Diffusion Models: Motivation, Training, and Advantages</figcaption></figure><p>This figure illustrates the evolution from <strong>Autoregressive Models (ARMs)</strong> to <strong>Diffusion Models</strong>, explaining why diffusion-based generative models emerged as a powerful alternative for text and image generation.</p><p>Autoregressive models generate data sequentially by predicting one token at a time from left to right, learning the probability distribution <strong>P₍data₎(x)</strong> through factorization as <strong>P(x₁, x₂, …, xₜ) = ∏ₜ P(xₜ | x₍&lt;ₜ₎)</strong>.</p><p>While this approach performs well for forward text generation, it suffers from several limitations, including slow sequential inference, inability to naturally model bidirectional dependencies, exposure bias, and difficulty performing tasks such as infilling, editing, and reverse reasoning.</p><p>To address these challenges, diffusion models adopt a fundamentally different learning strategy.</p><p>During training, a forward diffusion process gradually corrupts real data by adding Gaussian noise until it becomes pure noise, while the model learns the reverse diffusion process by predicting and removing the added noise step by step.</p><p>Rather than directly generating data, the model learns to reverse the corruption process, enabling it to recover meaningful samples from random noise.</p><p>For conditional generation, such as text or image generation from a prompt, the model learns the conditional probability distribution <strong>P₍θ₎(x | c)</strong>, where <strong>c</strong> represents the input condition or prompt.</p><p>During inference, the prompt guides the denoising process so that the generated output is both realistic and semantically aligned with the given condition.</p><p>By learning the underlying data distribution through reverse diffusion, diffusion models naturally capture bidirectional relationships, support controllable generation, reduce exposure bias, and produce high-quality outputs.</p><p>Consequently, diffusion models have become the foundation of modern generative AI systems such as <strong>Stable Diffusion, SDXL, FLUX, DALL·E</strong>, and emerging diffusion-based language models.</p><h3>From Autoregressive Models to Diffusion Language Models</h3><h4>Diffusion Intuition</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vn9qORplHH46KJf4zDE0dg.png" /><figcaption>Diffusion Intuition</figcaption></figure><p>At each training step, a few tokens in the sentence are <strong>randomly masked (noised)</strong>, while the remaining tokens are kept unchanged. The diffusion model receives this <strong>partially corrupted sentence</strong> as input and learns to predict the <strong>original masked tokens</strong> using the surrounding context. Since the model can attend to both the left and right context, it performs <strong>bidirectional learning</strong>, enabling it to recover any missing token from all the other tokens in the sequence. During inference, this denoising process is repeated iteratively until the complete, meaningful text is reconstructed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wagVzqkX5CjWEWRjujSi1w.png" /><figcaption>Autoregressive Models to Diffusion Language Models</figcaption></figure><h3>① What are Autoregressive Models (ARMs)?</h3><ul><li>① Generate text <strong>one token at a time</strong> from left to right.</li><li>② Each token depends on previously generated tokens.</li><li>③ Learn the probability distribution <strong>P(x₁, x₂, …, xₜ)</strong>.</li><li>④ Repeat until the complete sequence is generated.</li><li><strong>Key Idea:</strong> Predict the <strong>next token</strong> using previous tokens.</li></ul><h3>② How ARMs Work (Transformer LLMs)</h3><ul><li>① Input text is tokenized and embedded.</li><li>② Tokens pass through Transformer layers.</li><li>③ Self-attention captures token relationships.</li><li>④ Output layer predicts the next-token probabilities.</li><li>⑤ Select the next token and repeat.</li><li><strong>Key Idea:</strong> Generate text <strong>sequentially</strong>.</li></ul><h3>③ ARMs Intuition &amp; Example</h3><ul><li>① Predict one token at each step.</li><li>② Use only previously generated tokens.</li><li>③ Work well for forward text generation.</li><li>④ Struggle with reverse reasoning and missing-token prediction.</li><li><strong>Key Idea:</strong> Generation follows <strong>Left → Right</strong>.</li></ul><h3>④ Why are ARMs Limited?</h3><ul><li>① Sequential decoding is slow.</li><li>② Cannot naturally use bidirectional context.</li><li>③ Suffer from exposure bias.</li><li>④ Limited for editing, infilling, and reasoning.</li><li><strong>Key Idea:</strong> Need a model that understands <strong>all tokens together</strong>.</li></ul><h3>⑤ Birth of Diffusion Models</h3><ul><li>① Start with clean text (<strong>x₀</strong>).</li><li>② Gradually add noise (forward diffusion).</li><li>③ Learn to remove noise (reverse diffusion).</li><li>④ Recover the original meaningful text.</li><li><strong>Key Idea:</strong> Learn <strong>Noise → Text</strong> instead of next-token prediction.</li></ul><h3>⑥ Diffusion Intuition</h3><ul><li>① Randomly mask (noise) some tokens.</li><li>② Input the noisy text to the model.</li><li>③ Predict the original masked tokens.</li><li>④ Use both left and right context for prediction.</li><li><strong>Key Idea:</strong> Learn <strong>bidirectional</strong> token prediction.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IgglJ1pi6vPpWh-o1Yvh8g.png" /></figure><h3>⑦ Training Objective (Diffusion Language Model)</h3><ul><li>① Start with clean text (<strong>x₀</strong>).</li><li>② Add random noise to create <strong>xₜ</strong>.</li><li>③ Model predicts the added noise (or original text).</li><li>④ Compute loss and update parameters.</li><li><strong>Key Idea:</strong> Learn to remove noise and recover meaningful text.</li></ul><h3>⑧ Conditional Generation (Given Prompt)</h3><ul><li>① User provides a prompt ©.</li><li>② Start from random noise.</li><li>③ Prompt guides each denoising step.</li><li>④ Generate text matching the prompt.</li><li><strong>Key Idea:</strong> Prompt controls the generated output.</li></ul><h3>⑨ ARMs vs Diffusion Models</h3><p><strong>ARMs</strong></p><ul><li>➜ One token at a time</li><li>➜ Sequential generation</li><li>➜ Hard to edit</li></ul><p><strong>Diffusion Models</strong></p><ul><li>➜ Refine all tokens together</li><li>➜ Parallel generation</li><li>➜ Easy editing and infilling</li></ul><h3>⑩ Diffusion in LLMs</h3><ul><li>① Start with random noise.</li><li>② Predict and remove noise.</li><li>③ Repeat until <strong>t = 0</strong>.</li><li>④ Output meaningful text.</li><li><strong>Key Idea:</strong> Gradually convert noise into text.</li></ul><h3>⑪ Why Diffusion for Text?</h3><ul><li>✔ Bidirectional understanding</li><li>✔ Better editing &amp; infilling</li><li>✔ Lower exposure bias</li><li>✔ Prompt-controlled generation</li><li>✔ Parallel refinement</li></ul><p><strong>Key Idea:</strong> Learn <strong>Noise → Meaningful Text</strong>.</p><h3>⑫ Real-World Applications</h3><ul><li>📝 Text editing</li><li>✍️ Story generation</li><li>🧩 Text infilling</li><li>📄 Sentence completion</li><li>🔄 Text rewriting</li><li>🧠 Reasoning &amp; Question Answering</li></ul><h3>Masked Language Modeling (MLM): Complete Overview</h3><p>Masked Language Modeling (MLM) is a self-supervised learning technique used by encoder-based Transformer models, such as <strong>BERT</strong>, to learn rich contextual representations of language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Qs5f-oshY9oQKmUZ8Ikrg.png" /><figcaption>Masked Language Modeling (MLM): Complete Overview</figcaption></figure><p>Instead of predicting the next token sequentially, MLM randomly masks a subset of tokens in a sentence and trains the model to recover the original tokens using both the left and right context.</p><p>During training, the input sentence is first tokenized and converted into token embeddings, positional embeddings, and, in some approaches, time embeddings before being processed by multiple Transformer encoder layers with multi-head self-attention and feed-forward networks.</p><p>At each training step, a masking schedule <strong>r(t) = t/T</strong> determines the probability of masking tokens, where the masking ratio gradually increases from a small fraction to a fully masked sequence.</p><p>Each token is independently selected for masking using a <strong>Bernoulli(r(t))</strong> distribution, producing increasingly noisy inputs over time.</p><p>The Transformer processes the masked sentence and outputs a probability distribution (logits) over the entire vocabulary for every masked position.</p><p>The training objective is to predict the original tokens only at the masked positions, and the prediction error is computed using the <strong>Cross-Entropy Loss</strong>, <strong>L = −∑ᵢ log P(xᵢᵗʳᵘᵉ | xₙₒᵢₛʸ)</strong>, where the loss is calculated only for the masked tokens.</p><p>The model parameters are then updated through backpropagation over many batches until convergence.</p><p>By repeatedly learning to reconstruct masked words from partially corrupted sentences, MLM develops a deep bidirectional understanding of language, enabling it to capture semantic and syntactic relationships, making it highly effective for tasks such as text classification, question answering, named entity recognition, sentence understanding, and information retrieval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D8aZU8oDZgxMfC0LC9vDWQ.png" /></figure><h3>Autoregressive Masked Sampling</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xD9pOmjopmgbWvofsp7J6w.png" /><figcaption>Autoregressive Masked Sampling</figcaption></figure><p>Autoregressive Masked Sampling is a decoding strategy used in diffusion language models to generate text by <strong>gradually revealing tokens from a fully masked sequence</strong>.</p><p>The process starts with all tokens masked, and at each sampling step the model predicts all masked tokens along with their confidence scores.</p><p>The most confident predictions are kept, while the remaining tokens are masked again for the next iteration.</p><p>This process is repeated until only the final token predictions remain, progressively converting a fully masked sequence into meaningful text.</p><p>By refining the sequence step by step based on prediction confidence, the model produces stable, accurate, and coherent text generation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TBGxzAyCk4XZgaz49fX8_w.png" /></figure><h3>Denoising Algorithm in Diffusion Language Models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rRvV0oswB2NcsmpOBZABsA.png" /><figcaption>Denoising Algorithm in Diffusion Language Models</figcaption></figure><p>This figure illustrates the <strong>step-by-step denoising algorithm</strong> used in diffusion language models to generate text from a fully masked sequence.</p><p>The generation process begins with all tokens masked, representing pure noise with no meaningful information.</p><p>At each timestep, the noisy sequence is passed through the diffusion model, which produces logits and probability distributions for every masked position.</p><p>These probabilities are used to sample token predictions and compute a confidence score for each predicted token, typically based on the maximum predicted probability.</p><p>Instead of revealing all predicted tokens at once, only the <strong>top-k most confident tokens</strong> are unmasked, while the remaining positions stay masked for the next iteration.</p><p>This iterative denoising process continues over multiple timesteps, gradually revealing more tokens as the model becomes increasingly confident in its predictions.</p><p>The sequence progressively evolves from a fully masked state to a partially denoised sequence and finally to a complete, meaningful sentence when no masked tokens remain.</p><p>By revealing only the most confident predictions at each step, the denoising algorithm reduces early prediction errors, improves generation stability, and produces coherent, high-quality text.</p><h3>Forward Diffusion (Noising Process) in Diffusion Language Models</h3><p>This figure illustrates the <strong>forward diffusion (noising) process</strong>, which is used during training to gradually corrupt a clean text sequence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P0jaqilmUgzzc_vheaDf3A.png" /><figcaption>Forward Diffusion (Noising Process) in Diffusion Language Models</figcaption></figure><p>The process starts with a clean sequence <strong>x₀</strong> containing the original tokens and no noise.</p><p>At each timestep <strong>t</strong>, a small amount of noise is added by randomly replacing selected tokens with <strong>[MASK]</strong> tokens according to a predefined <strong>noise schedule (βₜ)</strong>.</p><p>Initially, only a few tokens are masked, preserving most of the original information, while later timesteps introduce more masking until the sequence becomes almost entirely masked, representing pure noise <strong>xₜ</strong> at <strong>t = T</strong>.</p><p>The amount of noise added at each step is controlled by the noise schedule, where smaller values of <strong>βₜ</strong> add less noise in the early stages and larger values add more noise in the later stages.</p><p>This gradual corruption exposes the model to different noise levels, allowing it to learn how meaningful text becomes increasingly corrupted over time.</p><p>Although the forward noising process itself is fixed and does not require learning, it provides the training data for the reverse diffusion process.</p><p>During reverse diffusion, the model learns to progressively remove the added noise and reconstruct the original sequence from partially or fully masked inputs.</p><p>Thus, the forward diffusion process forms the foundation for training diffusion language models to recover meaningful text from noisy sequences.</p><h3>Diffusion Language Model (DLM) from Scratch — Step-by-Step Pipeline</h3><p>Unlike autoregressive language models (e.g., GPT), which generate one token at a time, a DLM learns by <strong>corrupting text with masks during training</strong> and <strong>iteratively denoising masked text during inference</strong>. The following pipeline summarizes the entire workflow.</p><h3>Step 1: Load the Dataset</h3><p>Load the <strong>TinyStories</strong> dataset, which provides the text corpus used for training.</p><pre>TinyStories Dataset<br>        │<br>        ▼<br>Training Stories</pre><p><strong>Purpose</strong></p><ul><li>Read the training corpus.</li><li>Prepare data for tokenizer training and model training.</li></ul><h3>Step 2: Train a Byte-Level BPE Tokenizer</h3><p>Instead of using a pretrained tokenizer, train a tokenizer from scratch.</p><pre>Raw Text<br>     │<br>     ▼<br>Byte-Level BPE<br>     │<br>     ▼<br>Vocabulary<br>     │<br>     ▼<br>Token IDs</pre><p><strong>Purpose</strong></p><ul><li>Learn the vocabulary automatically.</li><li>Convert text into numerical token IDs.</li></ul><h3>Step 3: Build the Diffusion Transformer</h3><p>Create a bidirectional Transformer that predicts masked tokens.</p><pre>Token IDs<br>     │<br>     ▼<br>Token Embedding<br>     │<br>     ▼<br>Position Embedding<br>     │<br>     ▼<br>Time-Step Embedding<br>     │<br>     ▼<br>Transformer Encoder<br>     │<br>     ▼<br>Vocabulary Logits</pre><p><strong>Purpose</strong></p><ul><li>Learn to reconstruct corrupted text.</li></ul><h3>Step 4: Prepare the Training Dataset</h3><p>Convert every story into a chat-style format before tokenization.</p><pre>Story<br>   │<br>   ▼<br>Chat Format<br>   │<br>   ▼<br>Tokenizer<br>   │<br>   ▼<br>Fixed-Length Token Blocks</pre><p>Example</p><pre>&lt;|user|&gt;<br>Write a short story.<br>&lt;|assistant|&gt;<br>Once upon a time...</pre><p><strong>Purpose</strong></p><ul><li>Create consistent training samples.</li><li>Stream tokens into fixed-length sequences.</li></ul><h3>Step 5: Forward Diffusion (Mask Corruption)</h3><p>Randomly replace tokens with <strong>[MASK]</strong>.</p><pre>Original Sentence<br>        │<br>        ▼<br>Random Masking<br>        │<br>        ▼<br>Noisy Sentence (xₜ)</pre><p>Example</p><pre>Original<br>The cat sat on the mat<br>        │<br>        ▼<br>Masked<br>The [MASK] sat on [MASK] mat</pre><p><strong>Purpose</strong></p><ul><li>Simulate the diffusion process by gradually removing information.</li></ul><h3>Step 6: Feed the Noisy Sequence to the Transformer</h3><p>The Transformer receives the corrupted sequence and predicts the missing tokens.</p><pre>Masked Sentence<br>        │<br>        ▼<br>Diffusion Transformer<br>        │<br>        ▼<br>Predicted Tokens</pre><p><strong>Purpose</strong></p><ul><li>Recover the original sentence from noisy input.</li></ul><h3>Step 7: Compute the Diffusion Loss</h3><p>Compare predicted tokens with the original tokens.</p><pre>Predictions<br>      │<br>      ▼<br>Cross-Entropy Loss<br>      │<br>      ▼<br>Masked Positions Only</pre><p><strong>Purpose</strong></p><ul><li>Learn only from masked tokens.</li><li>Ignore unmasked positions during training.</li></ul><h3>Step 8: Update Model Parameters</h3><p>Use backpropagation and the optimizer to improve the model.</p><pre>Loss<br>   │<br>   ▼<br>Backward Pass<br>   │<br>   ▼<br>Gradient Update<br>   │<br>   ▼<br>Improved Model</pre><p><strong>Purpose</strong></p><ul><li>Minimize prediction errors.</li><li>Learn better token representations.</li></ul><h3>Step 9: Save the Trained Model</h3><p>Save the trained model and tokenizer.</p><pre>Trained Model<br>      │<br>      ▼<br>model.pt<br>config.json<br>tokenizer/</pre><p><strong>Purpose</strong></p><ul><li>Reuse the model without retraining.</li></ul><h3>Inference (Reverse Diffusion)</h3><p>After training, the model generates text using <strong>iterative denoising</strong>.</p><h3>Step 10: Initialize with MASK Tokens</h3><p>Keep the prompt fixed and initialize the response with masks.</p><pre>Prompt<br>↓<br>[MASK] [MASK] [MASK] [MASK]</pre><p><strong>Purpose</strong></p><ul><li>Create the initial noisy sequence.</li></ul><h3>Step 11: Predict Missing Tokens</h3><p>Predict all masked positions simultaneously.</p><pre>████ ████ ████<br>↓<br>The ████ sat ████</pre><p><strong>Purpose</strong></p><ul><li>Generate candidate tokens for every masked position.</li></ul><h3>Step 12: Keep High-Confidence Predictions</h3><p>Accept predictions with high confidence.</p><pre>Predicted Token<br>       │<br>       ▼<br>Confidence Score<br>       │<br>       ▼<br>Keep Token</pre><p><strong>Purpose</strong></p><ul><li>Preserve reliable predictions.</li></ul><h3>Step 13: Re-mask Low-Confidence Tokens</h3><p>Replace uncertain predictions with <strong>[MASK]</strong> again.</p><pre>Low Confidence<br>       │<br>       ▼<br>[MASK]</pre><p><strong>Purpose</strong></p><ul><li>Allow uncertain positions to be refined in later iterations.</li></ul><h3>Step 14: Repeat Iterative Denoising</h3><p>Repeat prediction and re-masking until the sequence converges.</p><pre>████ ████ ████<br>↓<br>The ████ sat ████<br>↓<br>The cat sat ████<br>↓<br>The cat sat on the mat</pre><p><strong>Purpose</strong></p><ul><li>Gradually refine the generated text.</li></ul><h3>Step 15: Decode the Final Tokens</h3><p>Convert token IDs back into readable text.</p><pre>Token IDs<br>      │<br>      ▼<br>Tokenizer<br>      │<br>      ▼<br>Generated Story</pre><p><strong>Purpose</strong></p><ul><li>Produce the final natural-language output.</li></ul><h3>Step 16: Visualize the Diffusion Process</h3><p>Record every denoising step and create an animated GIF.</p><pre>Step 64<br>↓<br>Step 63<br>↓<br>Step 62<br>↓<br>...<br>↓<br>Step 1<br>↓<br>Final Output</pre><p><strong>Purpose</strong></p><ul><li>Visualize how the model gradually transforms masked text into coherent language.</li></ul><h3>Step 17: Display and Download the GIF</h3><pre>Generated GIF<br>      │<br>      ├────────► Display in Notebook<br>      │<br>      └────────► Download as File</pre><p><strong>Purpose</strong></p><ul><li>View the diffusion process interactively.</li><li>Save the animation for presentations or blogs.</li></ul><h3>️ Complete Diffusion Language Model Pipeline</h3><pre>TinyStories Dataset<br>        │<br>        ▼<br>Train Byte-Level BPE Tokenizer<br>        │<br>        ▼<br>Build Diffusion Transformer<br>        │<br>        ▼<br>Prepare Chat Dataset<br>        │<br>        ▼<br>Forward Diffusion (Mask Tokens)<br>        │<br>        ▼<br>Train Transformer<br>        │<br>        ▼<br>Compute Diffusion Loss<br>        │<br>        ▼<br>Update Model Parameters<br>        │<br>        ▼<br>Save Model<br>        │<br>        ▼<br>Prompt + MASK Tokens<br>        │<br>        ▼<br>Predict Missing Tokens<br>        │<br>        ▼<br>Keep High-Confidence Tokens<br>        │<br>        ▼<br>Re-mask Low-Confidence Tokens<br>        │<br>        ▼<br>Repeat Iterative Denoising<br>        │<br>        ▼<br>Decode Final Text<br>        │<br>        ▼<br>Generate Inference GIF<br>        │<br>        ▼<br>Display / Download GIF<br>        │<br>        ▼<br>Compare with Official dLLM</pre><blockquote><strong><em>Summary:</em></strong><em> A Diffusion Language Model learns by repeatedly masking tokens during training and reconstructing them, then generates text by starting from masked tokens and iteratively denoising the sequence until a coherent response is produced.</em></blockquote><h3>Conclusion</h3><p>In this article, we built a <strong>Diffusion Language Model (DLM)</strong> from scratch and explored its complete workflow, from tokenizer training and forward diffusion to model training and iterative denoising during inference.</p><p>Unlike autoregressive language models, DLMs learn to reconstruct masked text and generate responses by progressively refining an entire sequence.</p><p>Although this implementation is simplified for educational purposes, it captures the core ideas behind modern diffusion language models and provides a strong foundation for understanding and building more advanced diffusion-based generative AI systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d5d7d2614e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Complete RAG Handbook: From Fundamentals to Production-Ready AI Systems]]></title>
            <link>https://medium.com/@sujangyawali177/the-complete-rag-handbook-from-fundamentals-to-production-ready-ai-systems-7f2d5498dec8?source=rss-23434c2aa8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f2d5498dec8</guid>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Sujangyawali]]></dc:creator>
            <pubDate>Mon, 20 Jul 2026 21:25:09 GMT</pubDate>
            <atom:updated>2026-07-20T21:25:09.735Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Learn Retrieval-Augmented Generation from the ground up, including indexing, embeddings, vector databases, hybrid search, production optimization, GraphRAG, Agentic RAG, Multimodal RAG, security, observability, and deployment using LangChain.</em></p><h3>What You’ll Learn</h3><p>After reading this blog, you’ll be able to:</p><ul><li>Build a complete <strong>Retrieval-Augmented Generation (RAG)</strong> pipeline from scratch.</li><li>Load, process, chunk, and embed documents for efficient retrieval.</li><li>Choose and optimize <strong>vector databases</strong> for production use.</li><li>Understand <strong>semantic search</strong>, <strong>BM25</strong>, and <strong>hybrid search</strong> strategies.</li><li>Optimize RAG using <strong>semantic chunking</strong>, <strong>late chunking</strong>, and <strong>contextual retrieval</strong>.</li><li>Build advanced systems with <strong>Agentic RAG</strong>, <strong>GraphRAG</strong>, and <strong>Multimodal RAG (ColPali)</strong>.</li><li>Improve performance through <strong>index tuning</strong>, <strong>cost optimization</strong>, and <strong>caching</strong>.</li><li>Secure LLM applications using <strong>input validation</strong>, <strong>PII protection</strong>, and <strong>guardrails</strong>.</li><li>Monitor and debug AI applications with <strong>LangSmith</strong>, logging, metrics, and observability.</li><li>Deploy <strong>production-ready, scalable, secure, and reliable AI applications</strong> using LangChain.</li></ul><p><strong>By the end of this guide, you’ll have the knowledge to design, build, optimize, and deploy modern production-ready RAG systems from fundamentals to advanced techniques.</strong></p><h3>RAG (Retrieval-Augmented Generation)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WaV1UBLN7q5fvlnBNjFbhQ.png" /><figcaption>RAG (Retrieval-Augmented Generation)</figcaption></figure><p><strong>RAG</strong> stands for <strong>Retrieval-Augmented Generation</strong>, a technique that combines <strong>information retrieval</strong> with <strong>LLMs</strong> to produce accurate and grounded responses.</p><ul><li><strong>Retrieval:</strong> Searches a knowledge base or vector database to find the most relevant documents for the user’s query.</li><li><strong>Augmented:</strong> Combines the retrieved documents with the user’s query to create enriched context for the LLM.</li><li><strong>Generation:</strong> The LLM uses the augmented context to generate a relevant, accurate, and source-grounded response.</li></ul><h3>Key Terms</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7kxaSJfNkQtSQNaacDShOQ.png" /><figcaption>Key Terms</figcaption></figure><ul><li><strong>Context:</strong> Retrieved documents or information relevant to the user’s query.</li><li><strong>Question:</strong> The user’s input query.</li><li><strong>Prompt:</strong> A template that combines the <strong>context</strong> and <strong>question</strong> into instructions for the LLM.</li><li><strong>LLM:</strong> The Large Language Model that generates the answer using the provided context.</li><li><strong>Parser:</strong> Processes and formats the LLM’s output into the final response.</li></ul><h3>Parallel Input Processing</h3><ul><li><strong>Retriever:</strong> Retrieves relevant documents and provides them as the <strong>context</strong>.</li><li><strong>RunnablePassthrough:</strong> Passes the user’s <strong>question</strong> unchanged into the chain.</li><li>Both <strong>context</strong> and <strong>question</strong> are combined in the prompt before calling the LLM.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FJ38RJ5yZ6LnzqOxLWFyFQ.png" /></figure><p>If the retrieved documents don’t contain the answer, the LLM should say “I don’t know” instead of making something up.</p><h3>Handling “I Don’t Know”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n3PNKu-CsAAtEbWMqfd6gw.png" /><figcaption>Handling “I Don’t Know”</figcaption></figure><ul><li><strong>Purpose:</strong> Prevent the LLM from hallucinating when the answer is not in the retrieved context.</li><li><strong>Without Instruction:</strong> The LLM may make up an answer.</li><li><strong>With Instruction:</strong> The LLM replies <strong>“I don’t have information about that.”</strong> if the context doesn’t contain the answer.</li><li><strong>Benefit:</strong> Produces more accurate, reliable, and grounded RAG responses.</li></ul><h3>RAG with Sources</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J_4Y0ZBIZowQC4A7Vhc6vw.png" /><figcaption>RAG with Sources</figcaption></figure><p><strong>Purpose:</strong> Attach the source of each retrieved document so the LLM can generate answers with citations, improving transparency and trust.</p><h3>Terms Used</h3><ul><li><strong>Retriever Output:</strong> The documents retrieved from the vector store based on the user’s query.</li><li><strong>Page Content:</strong> The actual text/content retrieved from each document.</li><li><strong>Source:</strong> Metadata indicating where the content came from (e.g., doc.pdf, guide.txt, faq.md).</li><li><strong>format_docs_with_sources:</strong> A function that formats retrieved documents by adding their source tags before sending them to the LLM.</li><li><strong>Formatted Context:</strong> The final context passed to the LLM, where each document includes its source information.</li><li><strong>Citation:</strong> References to the original documents used to generate the answer, allowing users to verify the information.</li></ul><h3>Project Initialization</h3><p>We use <strong>uv</strong> to create a clean Python project, manage dependencies, and isolate the environment.</p><h3>1. Initialize the project</h3><pre>uv init</pre><h3>2. Create and activate a virtual environment</h3><pre>uv venv<br>source .venv/bin/activate      # macOS/Linux<br># .venv\Scripts\activate       # Windows</pre><h3>3. Install required packages</h3><pre>uv add langchain langchain-core langgraph langchain-openai python-dotenv</pre><h3>4. Create a .env file</h3><pre>OPENAI_API_KEY=your_openai_api_key</pre><h3>5. Verify the setup</h3><pre>from dotenv import load_dotenv<br>from langchain_openai import ChatOpenAI<br>load_dotenv()<br>llm = ChatOpenAI(<br>    model=&quot;gpt-4o-mini&quot;,<br>    temperature=0<br>)<br>response = llm.invoke(&quot;Say &#39;Setup complete!&#39;&quot;)<br>print(response.content)</pre><p><strong>Output</strong></p><pre>Setup complete!</pre><blockquote><em>At this point, the development environment is ready, and we can start building the Retrieval-Augmented Generation (RAG) pipeline.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5uEmLxzv1gkO3kCqTSC9qg.png" /></figure><p>A <strong>Document Loader</strong> is the entry point of a RAG pipeline that reads documents from various file formats such as <strong>PDF, TXT, HTML, DOCX, and CSV</strong> and converts them into a standardized <strong>Document</strong> object.</p><p>Each document contains the extracted text (page_content) along with useful metadata (metadata) such as the source file, page number, author, or title.</p><p>This standardized format allows the downstream components of the RAG pipeline, such as text splitters, embedding models, and vector databases, to process documents consistently regardless of their original format.</p><h3>Core Document Loaders</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oYTFqEPtNjcBKrQhhOpMWw.png" /><figcaption>Core Document Loaders</figcaption></figure><ul><li><strong>PyPDFLoader</strong> — Loads and extracts text from <strong>PDF (</strong><strong>.pdf)</strong> files.</li><li><strong>TextLoader</strong> — Reads content from <strong>plain text (</strong><strong>.txt)</strong> files.</li><li><strong>DirectoryLoader</strong> — Loads <strong>multiple files</strong> from a directory, making it easy to process an entire document collection.</li><li><strong>WebBaseLoader</strong> — Fetches and extracts text from <strong>web pages</strong> using their URLs.</li><li><strong>UnstructuredLoader</strong> — Handles <strong>complex or mixed-format documents</strong> (e.g., PDF, DOCX, HTML, images) while preserving document structure.</li></ul><pre>loader = TextLoader(source) or WebBaseLoder(source) etc<br>docs = loader.load()</pre><ul><li>Create a loader by specifying the document source.</li><li>Call load() to convert the source into a <strong>List[Document]</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yPkyn4A9prFy16yRNZmYVg.png" /><figcaption>Core Document Loaders</figcaption></figure><h3>Benefits of List[Document]</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vqkK6ajZMw0yX8dSD-Zk4Q.png" /><figcaption>List[Document]</figcaption></figure><ul><li><strong>Supports multiple documents</strong> from any source.</li><li><strong>Preserves metadata</strong> (page number, source, author, URL, etc.).</li><li><strong>Works seamlessly</strong> with text splitters, embedding models, and vector databases.</li><li><strong>Provides a standard format</strong> across all document loaders.</li></ul><h3>PDF Loading Options</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*smzvB68QNSC_O-JTfaHNNg.png" /><figcaption>PDF Loading Options</figcaption></figure><ul><li><strong>PyPDFLoader</strong> — Fast and lightweight loader for <strong>simple text-based PDFs</strong>. Best choice for most RAG applications.</li><li><strong>PyMuPDFLoader</strong> — <strong>Fastest</strong> PDF loader with <strong>rich metadata</strong>. Ideal for large-scale and production RAG systems.</li><li><strong>UnstructuredPDFLoader</strong> — Best for <strong>complex PDFs</strong> containing tables, images, or multi-column layouts. Slower but preserves document structure.</li></ul><blockquote><strong><em>Recommendation:</em></strong><em> Start with </em><strong><em>PyPDFLoader</em></strong><em>. Use </em><strong><em>PyMuPDFLoader</em></strong><em> for better performance and </em><strong><em>UnstructuredPDFLoader</em></strong><em> for complex document layouts.</em></blockquote><h3>Web Loading (WebBaseLoader)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hq9XzlW7nT_IckqwVE2yMw.png" /><figcaption>Web Loading</figcaption></figure><ul><li><strong>WebBaseLoader</strong> is used to load and extract text from <strong>web pages</strong> using their URLs.</li><li>Supports both <strong>single</strong> and <strong>multiple URLs</strong>.</li><li>Downloads the webpage, extracts the readable text, and removes unnecessary HTML content.</li><li>Converts each webpage into a LangChain <strong>Document</strong> object.</li><li>For multiple URLs, it returns a <strong>List[Document]</strong>, where each webpage becomes a separate Document.</li><li>Each Document contains:</li><li><strong>page_content</strong> – The extracted webpage text.</li><li><strong>metadata</strong> – Information such as the source URL, page title, and other webpage details.</li><li>The resulting documents are ready for <strong>text splitting, embedding generation, and vector database storage</strong> in the RAG pipeline.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tOFf3c_spqZc_BjyrBqE_A.png" /><figcaption>Directory Loading</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gVla6Wrz61S_7_x_HF9zUg.png" /><figcaption>Glob Pattern</figcaption></figure><pre>from langchain_community.document_loaders import DirectoryLoader, PyPDFLoader<br><br>loader = DirectoryLoader(<br>    path=&quot;docs/&quot;,<br>    glob=&quot;**/*.pdf&quot;,<br>    loader_cls=PyPDFLoader<br>)<br><br>docs = loader.load()<br><br>print(f&quot;Total Documents: {len(docs)}&quot;)<br><br># Display first few documents<br>for doc in docs[:3]:<br>    print(doc.metadata)</pre><p>Output</p><pre>Total Documents: 3<br><br>{&#39;source&#39;: &#39;docs/report.pdf&#39;, &#39;page&#39;: 0}<br>{&#39;source&#39;: &#39;docs/guide.pdf&#39;, &#39;page&#39;: 0}<br>{&#39;source&#39;: &#39;docs/summary.pdf&#39;, &#39;page&#39;: 0}</pre><h3>RAG Indexing Pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XXYmI0w2Y10x_ckgZ6bd6A.png" /><figcaption>RAG Indexing Pipeline</figcaption></figure><p><strong>1. Document Loaders</strong></p><ul><li>Load documents from different sources.</li><li>Extract text and metadata.</li><li>Convert data into Document objects.</li></ul><p><strong>2. Text Splitters</strong></p><ul><li>Split large documents into smaller chunks.</li><li>Preserve context between chunks.</li><li>Add overlap to avoid losing information.</li></ul><p><strong>3. Embedding Generation</strong></p><ul><li>Convert each text chunk into a numerical vector.</li><li>Capture the semantic meaning of the text.</li><li>Use an embedding model to generate vectors.</li></ul><p><strong>4. Vector Storage</strong></p><ul><li>Store embeddings in a vector database.</li><li>Create an index for efficient similarity search.</li><li>Enable fast retrieval of relevant chunks.</li></ul><p><strong>5. Ready for Queries</strong></p><ul><li>The indexed documents are ready for retrieval.</li><li>User queries are matched with the most relevant chunks.</li><li>Retrieved context is provided to the LLM to generate accurate responses.</li></ul><h3>Chunking</h3><p><strong>Chunking</strong> is the process of splitting large documents into <strong>smaller, meaningful pieces (chunks)</strong> before creating embeddings and storing them in a vector database for retrieval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7A3zvO5pUbIQW6ABrJ8RaA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bw9alDuoky7jAuORPjeQjw.png" /><figcaption>Chunking</figcaption></figure><h4>Key Idea</h4><ul><li><strong>Bad Chunking:</strong> Breaks sentences or context → Poor retrieval → Incorrect RAG answers.</li><li><strong>Good Chunking:</strong> Preserves complete ideas and context → Better retrieval → Accurate, grounded answers.</li></ul><h3>Fixed Chunking</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eCTn-uZmJNoD-ALTVGXBzA.png" /><figcaption>Fixed Vs Semantic Chunking</figcaption></figure><ul><li>Splits by fixed character size.</li><li>Can break sentences and context.</li><li>May produce incomplete answers.</li></ul><h3>Semantic Chunking</h3><ul><li>Splits by meaning and context.</li><li>Keeps related information together.</li><li>Produces more accurate and complete answers.</li></ul><h3>Why Chunking matters so much ?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cXkGI8kKzrEwQD3j5Hgfog.png" /><figcaption>Why Chunking matters so much ?</figcaption></figure><ul><li>Each chunk is embedded <strong>independently</strong>.</li><li>The embedding model only sees that <strong>single chunk</strong>, not the full document.</li><li>If a chunk contains <strong>incomplete information</strong>, its embedding captures <strong>incomplete meaning</strong>.</li><li>Poor embeddings lead to <strong>poor retrieval</strong>.</li><li><strong>Good chunking preserves complete context</strong>, resulting in accurate retrieval and better RAG responses</li></ul><h3>Four Chunking Variables That Affect RAG Quality</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GsSSC8RAb2MTDgzm-yVO6w.png" /><figcaption>Four Chunking Variables That Affect RAG Quality</figcaption></figure><p><strong>1. Chunk Size</strong></p><ul><li>Too small → Loses context.</li><li>Too large → Dilutes meaning.</li><li>Best range: <strong>200–1000 tokens</strong> (depends on use case).</li></ul><p><strong>2. Overlap</strong></p><ul><li>Keep <strong>10–20% overlap</strong> between chunks.</li><li>Preserves context across chunks.</li></ul><p><strong>3. Split Boundaries</strong></p><ul><li><strong>Fixed:</strong> Splits at fixed size (simple).</li><li><strong>Recursive:</strong> Splits at paragraphs/sentences (better).</li><li><strong>Semantic:</strong> Splits by meaning (best quality).</li></ul><p><strong>4. Content Type</strong></p><ul><li>Different content needs different chunking.</li><li>Examples: <strong>Code, Legal documents, Markdown, Text</strong>.</li></ul><h3>Chunking Strategies Compared</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vIzLe96diT2AQjGWIGAllw.png" /><figcaption>Chunking Strategies Compared</figcaption></figure><h4>1. Fixed Chunking (Basic)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dLhtPlHLmmYMOBdpfvlsjQ.png" /><figcaption>Fixed Chunking (Basic)</figcaption></figure><ul><li>Splits text by a fixed size (characters/tokens).</li><li>Simple and fast.</li><li>May break context.</li></ul><h4>2. Recursive Chunking (Intermediate)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o955DfE6oNDKCMTEQ4SRzA.png" /><figcaption>Recursive Chunking (Intermediate)</figcaption></figure><ul><li>Splits at paragraphs, then sentences, then words if needed.</li><li>Preserves structure better.</li><li>Common choice for most RAG applications.</li></ul><h4>3. Semantic Chunking (Advanced)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*15tKvpmK1rcqBPApQFAfTg.png" /><figcaption>Semantic Chunking (Advanced)</figcaption></figure><ul><li>Splits based on meaning and context.</li><li>Keeps related information together.</li><li>Provides the best retrieval quality.</li></ul><h4>4. Late Chunking (Advanced)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TCglp3OrJJwnysHSO3rvXA.png" /><figcaption>Late Chunking (Advanced)</figcaption></figure><ul><li>Create embeddings for the full document first.</li><li>Split into chunks afterward.</li><li>Preserves global context and can improve retrieval.</li></ul><h3>80/20 Rule</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P6FTO5C6DOTa5n89ttnZZw.png" /><figcaption>80/20 Rule</figcaption></figure><ul><li><strong>Recursive Chunking</strong> solves <strong>~80%</strong> of RAG use cases.</li><li><strong>Semantic Chunking</strong> provides the remaining <strong>20%</strong> improvement.</li><li><strong>Start with Recursive</strong> and switch to <strong>Semantic</strong> only if needed.</li></ul><h3>Embedding Dimensions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rQoBQVq0UqLwoutC5m10gA.png" /><figcaption>Embedding Dimensions</figcaption></figure><ul><li><strong>Embedding dimension</strong> is the size (length) of the vector representing a text.</li><li><strong>Higher dimensions</strong> capture more semantic information but require more storage and computation.</li><li><strong>Lower dimensions</strong> are faster and cheaper but may lose some detail.</li></ul><h3>Chat Model (Text → Text)</h3><p>Used for conversations and content generation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ye63hZ7fh5ALEaxqmIHehA.png" /><figcaption>Chat Model vs Embedding Models</figcaption></figure><pre>import os<br>from dotenv import load_dotenv<br>from openai import OpenAI<br><br>load_dotenv()<br><br>client = OpenAI(api_key=os.getenv(&quot;OPENAI_API_KEY&quot;))<br><br>response = client.chat.completions.create(<br>    model=&quot;gpt-4o-mini&quot;,<br>    messages=[<br>        {<br>            &quot;role&quot;: &quot;system&quot;,<br>            &quot;content&quot;: &quot;You are a helpful assistant.&quot;<br>        },<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: &quot;What is the capital of France?&quot;<br>        }<br>    ]<br>)<br><br>print(response.choices[0].message.content)</pre><p>Output</p><pre>Paris is the capital of France.</pre><h3>Embedding Model(Text → Vector)</h3><p>Used for search, similarity, retrieval, and RAG</p><pre>import os<br>from dotenv import load_dotenv<br>from openai import OpenAI<br><br>load_dotenv()<br><br>client = OpenAI(api_key=os.getenv(&quot;OPENAI_API_KEY&quot;))<br><br>response = client.embeddings.create(<br>    model=&quot;text-embedding-3-small&quot;,<br>    input=&quot;Your text string goes here&quot;<br>)<br><br>print(response)<br>print(len(response.data[0].embedding))</pre><p>Output</p><pre>CreateEmbeddingResponse(<br>    data=[<br>        Embedding(<br>            embedding=[<br>                0.00168,<br>                -0.02760,<br>                -0.03390,<br>                0.02447,<br>                ...<br>            ],<br>            index=0,<br>            object=&#39;embedding&#39;<br>        )<br>    ],<br>    model=&#39;text-embedding-3-small&#39;,<br>    object=&#39;list&#39;,<br>    usage=Usage(prompt_tokens=5, total_tokens=5)<br>)<br><br>1536</pre><h3><strong>RAG PIPELINE : Indexing and Querying</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BjiZ7I_gAx0hO7f3jqX8ww.png" /><figcaption>Phase 1: Indexing</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gvBWB3qi5Wm2OsX55YjQ9A.png" /><figcaption>Phase 2: Querying</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MTPjDMzg-XhxCWr0rEvrsQ.png" /><figcaption>Complete Pipeline</figcaption></figure><h3>Three Rules for Production RAG</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9yKVUgj3UfdXXO9xvHIxaw.png" /><figcaption>Three Rules for Production RAG</figcaption></figure><h3>1. Use the Same Embedding Model Everywhere</h3><ul><li>Use the <strong>same embedding model</strong> for documents and user queries.</li><li>This keeps embeddings consistent and improves search accuracy.</li></ul><h3>2. Focus on Quality, Not Quantity</h3><ul><li>Store <strong>high-quality and relevant</strong> documents.</li><li>More documents do not always mean better results.</li><li>Clean, useful data gives better answers.</li></ul><h3>3. Test Retrieval Separately</h3><ul><li>First check if the retriever finds the <strong>right documents</strong>.</li><li>Then test the LLM’s answer.</li><li>This helps identify whether the problem is with <strong>retrieval</strong> or <strong>generation</strong></li></ul><h3>Chroma Database Workflow (RAG)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S9j8JurDJfuWicCQmYUBxw.png" /><figcaption>Chroma Database Workflow (RAG)</figcaption></figure><h3>1. Application Layer</h3><ul><li>Receives the user’s query.</li><li>Returns the final answer to the user.</li></ul><h3>2. Embedding Generation</h3><ul><li>Converts the user’s query into a <strong>vector (embedding)</strong>.</li><li>Uses the <strong>same embedding model</strong> that was used for indexing documents.</li></ul><h3>3. Chroma Database</h3><ul><li>Stores document chunks and their embeddings.</li><li>Performs <strong>semantic similarity search</strong> to find the most relevant chunks.</li></ul><h3>Workflow</h3><ol><li>User submits a question.</li><li>Query is converted into an embedding.</li><li>Chroma compares the query embedding with stored document embeddings.</li><li>Retrieves the most relevant document chunks.</li><li>Retrieved chunks and the user’s query are sent to the LLM.</li><li>The LLM generates a context-aware, grounded answer.</li></ol><h3>Why Chroma?</h3><ul><li>Performs <strong>semantic search</strong>, not just keyword matching.</li><li>Finds documents based on <strong>meaning and context</strong>.</li><li>Improves retrieval accuracy and RAG response quality.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ll7zep4sei1OZvmSbBaZSw.png" /><figcaption>Similarity Search with Scores</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hUfpyffQM381QthkjIF9WQ.png" /><figcaption>Similarity Formula</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nzKHQKVehS1G4ZxbLQHkGg.png" /><figcaption>Metadata Filtering</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t3N7kAK8bb_bpdI9n-J11A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KuClKPk3ssUxb03A8GJqAA.png" /></figure><ul><li><strong>Embedding:</strong> Converts text into a numerical vector that represents its semantic meaning.</li><li><strong>Dimensions:</strong> The number of values in an embedding (e.g., 1536). Higher dimensions can capture more semantic information.</li><li><strong>Normalization:</strong> Scales an embedding so its <strong>vector norm equals 1</strong>, ensuring fair similarity comparison based on meaning rather than vector size.</li><li><strong>Vector Norm:</strong> The length (magnitude) of an embedding vector. A norm of <strong>1.0</strong> means the vector is normalized.</li><li><strong>Batch Embedding:</strong> Generates embeddings for multiple texts in a single API call, improving speed and efficiency.</li><li><strong>Similarity Search:</strong> Compares a query embedding with stored embeddings to retrieve the most semantically similar documents.</li><li><strong>Similarity Score:</strong> Indicates how closely a document matches the query. <strong>Higher score = more similar</strong> (or <strong>lower distance</strong>, depending on the metric used).</li><li><strong>Distance:</strong> Measures the difference between two embeddings. <strong>Smaller distance = greater similarity</strong>.</li><li><strong>Ranking:</strong> Orders retrieved documents from the most relevant to the least relevant based on similarity score or distance.</li></ul><h3>When Vector Search Fails ?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*18PxEosaIlfeX05G4lfV5Q.png" /><figcaption>When Vector Search Fails ?</figcaption></figure><ul><li><strong>Product Codes:</strong> Models do not understand product IDs like <strong>SKU-7742X</strong> because they have no meaning.</li><li><strong>Error Codes:</strong> Codes like <strong>E_CONN_REFUSED</strong> are just text, so the model cannot understand what they mean.</li><li><strong>Acronyms:</strong> Short forms like <strong>WCAG</strong> or <strong>JWT</strong> may not be recognized unless the model has learned them.</li><li><strong>Exact Names:</strong> Vector search looks for similar meaning, so it may return the wrong person or item with a similar name.</li></ul><p><strong>Key takeaway:</strong> Vector search is good at understanding <strong>meaning</strong>, but not <strong>codes, IDs, acronyms, or exact names</strong>. For these, use <strong>keyword search</strong> or <strong>hybrid search (keyword + vector search)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hCWnygu8YkDsUOnjZJtBCA.png" /><figcaption>Vector Search vs BM25 Search</figcaption></figure><h3>Vector Search</h3><ul><li>Finds documents based on <strong>meaning</strong>.</li><li>Good for semantic search, synonyms, and natural language.</li><li>Poor for exact matches, codes, IDs, and acronyms.</li></ul><h3>BM25 (Keyword Search)</h3><ul><li>Finds documents based on <strong>exact keywords</strong>.</li><li>Good for exact matches, codes, IDs, and rare terms.</li><li>Does not understand meaning or synonyms.</li></ul><h3>Hybrid Search</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uDMlLqnFTQ3RHlzJMtXQvg.png" /><figcaption>Hybrid Search Pipeline</figcaption></figure><ol><li><strong>User enters a query</strong> (e.g., product code or question).</li><li>The query is sent to <strong>both Vector Search and BM25 Search</strong> at the same time.</li><li><strong>Vector Search</strong> finds documents with <strong>similar meaning</strong>.</li><li><strong>BM25 Search</strong> finds documents with <strong>exact keyword matches</strong>.</li><li><strong>Reciprocal Rank Fusion (RRF)</strong> combines the rankings from both searches.</li><li>Documents that rank well in <strong>both searches</strong> move to the top.</li><li>The system returns the <strong>best combined results</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ps5qjePTPU4riFZ2QlssVQ.png" /><figcaption>When to Use Hybrid Search</figcaption></figure><h3>Production Hybrid Pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2lxNzM_GpavCIqlECSqIhQ.png" /><figcaption>Production Hybrid Pipeline (Simple)</figcaption></figure><ol><li><strong>User sends a query.</strong></li><li>The query is searched by <strong>two retrievers at the same time</strong>: <strong>Vector Retriever</strong> → Finds documents with similar meaning (semantic search). <strong>BM25 Retriever</strong> → Finds documents with exact keyword matches</li><li>Both retrievers return their own ranked results.</li><li><strong>EnsembleRetriever</strong> combines the results using weights (e.g., 0.5 for Vector and 0.5 for BM25).</li><li>Documents that score well in both retrievers are ranked higher.</li><li>The final ranked documents are returned to the LLM to generate the answer.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0b6vfoi8d0uRyT642CQsYA.png" /></figure><pre># 1. Vector Retriever<br># Performs semantic (meaning-based) search using the vector database.<br>vector_retriever = vectorstore.as_retriever(k=3)<br><br># 2. BM25 Retriever<br># Performs keyword-based search using exact word matching.<br>bm25_retriever = BM25Retriever.from_documents(docs, k=3)<br><br># 3. Ensemble Retriever<br># Combines results from both retrievers using Reciprocal Rank Fusion (RRF).<br>ensemble_retriever = EnsembleRetriever(<br>    retrievers=[vector_retriever, bm25_retriever],<br>    weights=[0.5, 0.5]<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gA3V2i097UURgdXCOSwHCg.png" /></figure><p><strong>Rule:</strong> Start with <strong>[0.5, 0.5]</strong> and adjust the weights based on your users&#39; search patterns.</p><p><strong>Test Queries</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EbeDop8d9q6LyGBR0tt0uQ.png" /></figure><pre>test_queries = [<br>    &quot;SKU-7742X specifications&quot;,   # Exact product code<br>    &quot;E_CONN_REFUSED error&quot;,        # Error code<br>    &quot;How do I authenticate?&quot;,      # Semantic query<br>    &quot;WCAG compliance&quot;,             # Acronym<br>    &quot;router configuration&quot;         # General semantic query<br>]</pre><p>These queries help you decide whether to give <strong>more weight to BM25</strong> (exact matches) or <strong>more weight to Vector Search</strong> (semantic matches).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QjhYWuuJzzI6Z50l0cDNIA.png" /><figcaption>Production Consideration for hybrid search</figcaption></figure><h3>Cost Optimization Patterns</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R1Sai2MObadhVOMX6nua-A.png" /><figcaption>Cost Optimization Pattern</figcaption></figure><ul><li><strong>Model Routing:</strong> Use <strong>cheap models</strong> for simple tasks and <strong>powerful models</strong> for complex tasks.</li><li><strong>Semantic Caching:</strong> Reuse previous answers for similar queries to avoid repeated LLM calls.</li><li><strong>Token Budgeting:</strong> Limit and track token usage to control API costs.</li><li><strong>LangSmith Monitoring:</strong> Track token usage, latency, errors, and costs to optimize production performance.</li></ul><h3>The Need for Observability</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eF24btILnUTSiRQxJblJIg.png" /></figure><ul><li><strong>Traditional software:</strong> Bugs produce <strong>stack traces</strong>, making it easy to locate and fix errors.</li><li><strong>LLM applications:</strong> Wrong answers have <strong>no stack trace</strong>, so it’s hard to identify whether the issue is the agent, prompt, tool, or retrieval.</li><li><strong>Multi-agent systems:</strong> Multiple nodes, agents, and LLM calls make debugging much more complex.</li><li><strong>Solution:</strong> Use <strong>observability (e.g., LangSmith)</strong> to trace every step, monitor performance, and find the root cause quickly.</li></ul><h4>Why LLM Debugging is Hard ?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*utwC6DqYi2T1D3UJhEjq7A.png" /><figcaption>Why LLM Debugging is Hard</figcaption></figure><ul><li><strong>Non-deterministic:</strong> The same input can produce different outputs, making bugs hard to reproduce.</li><li><strong>Cascading errors:</strong> One small mistake (e.g., bad search result) can lead to incorrect final answers.</li><li><strong>Silent failures:</strong> LLMs often give confident but wrong answers without any error or crash.</li><li><strong>Cost surprises:</strong> Uncontrolled loops or repeated LLM calls can greatly increase token usage and costs.</li></ul><h3>Observability</h3><p>Observability is the ability to understand how an AI system performs by tracking the entire journey of a request — not just the final output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z96IfBUUTs3LXha84ezUnw.png" /><figcaption>Observability</figcaption></figure><ul><li><strong>Tool:</strong> <strong>LangSmith</strong> provides observability for LangChain and LangGraph applications.</li></ul><h4>Key Features of LangSmith</h4><ul><li><strong>Tracing:</strong> Visualizes every step of the agent workflow, including prompts, tool calls, and decision-making.</li><li><strong>Metrics:</strong> Tracks latency, token usage, API costs, and performance in real time.</li><li><strong>Evaluation (Evals):</strong> Measures output quality, correctness, and detects regressions before they affect users.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xgIw3mF9mtXg6f0SUaM5XA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*82ri3ZMNxYWiUEag4e2BTQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0MSeDx0Lfsf_66HbiCdT4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_AeMiLhJGrhU33vbRf_izA.png" /></figure><h3>Semantic Chunking</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x1S-h0jVol95mckRGoH3vA.png" /><figcaption>Semantic Chunking</figcaption></figure><p><strong>Semantic Chunking</strong> is an advanced chunking technique that splits documents based on <strong>semantic meaning and embedding similarity</strong> instead of fixed character or token limits.</p><p>By preserving complete ideas and topic coherence, it improves retrieval quality, reduces irrelevant context, and enhances RAG accuracy.</p><p>Its effectiveness can be evaluated using <strong>retrieval metrics, execution time, and answer quality</strong> to determine whether it performs better than traditional chunking methods.</p><h3>RAG Optimization Concepts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nChdEPIxltunBgeSdCo5Vw.png" /></figure><p><strong>RAG Optimization</strong> refers to techniques that improve <strong>retrieval quality, reduce noise, optimize context, lower costs, and generate more accurate, grounded responses</strong>. These methods enhance every stage of the RAG pipeline — from document retrieval to context preparation and final answer generation — resulting in faster, more reliable, and trustworthy AI applications.</p><h3>Index Tuning &amp; Scaling for Production</h3><p><strong>Index Tuning &amp; Scaling</strong> focuses on optimizing the <strong>vector index, retrieval parameters, and infrastructure</strong> to improve search accuracy, reduce latency, and efficiently handle large-scale data and user traffic. Proper tuning ensures <strong>fast, scalable, and cost-effective</strong> RAG systems in production environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vjHr0_WwsKGhjmjlYACv1A.png" /></figure><h3>HNSW Index Parameters</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pgju5ntw0H16jk-pQuLr6w.png" /><figcaption>HNSW Index Parameters</figcaption></figure><p><strong>HNSW (Hierarchical Navigable Small World)</strong> is a vector search algorithm used in vector databases. Two important parameters control the balance between <strong>search speed, memory usage, and retrieval accuracy</strong>.</p><ul><li><strong>M (Max Connections):</strong> Controls how many connections each vector has. Higher <strong>M</strong> improves accuracy but uses more memory and creates a larger index.</li><li><strong>ef (Search Effort):</strong> Controls how many vectors are explored during search. Higher <strong>ef</strong> increases search accuracy but makes searches slower.</li></ul><p><strong>Key Idea:</strong> Increase <strong>M</strong> and <strong>ef</strong> for better retrieval accuracy, or lower them for faster searches and lower memory usage.</p><h3>Production Trade-Off (HNSW Tuning)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FEMsbFNeoxnrcU2dKRXq-Q.png" /><figcaption>Production Trade-Off (HNSW Tuning)</figcaption></figure><p>Production HNSW tuning is about finding the right balance between <strong>accuracy, speed, and memory</strong>.</p><p>Increasing <strong>M</strong> and <strong>ef</strong> improves retrieval accuracy but requires more memory and slower searches, while lower values provide faster performance with lower memory usage at the cost of accuracy.</p><p>The best settings depend on whether your application prioritizes speed, balanced performance, or maximum accuracy.</p><h3>HNSW Configuration in pgvector &amp; Chroma</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fu-Twv8GoqooaP8Fx3GGbA.png" /><figcaption>HNSW Configuration in pgvector &amp; Chroma</figcaption></figure><p>This example shows how to configure <strong>HNSW indexes</strong> in <strong>pgvector</strong> and <strong>Chroma</strong> for efficient vector search.</p><p>The key parameters are <strong>M</strong> (controls node connections), <strong>ef_construction</strong> (controls index quality during creation), and <strong>ef_search</strong> (controls search accuracy during queries).</p><p>Increasing these values improves retrieval accuracy but requires more memory and slower indexing/search.</p><h3>Scaling Vector Databases</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5aEzPlhZSyyJXdJZmiZiQw.png" /><figcaption>Scaling Vector Databases</figcaption></figure><p>As the amount of data and user traffic grows, vector databases must scale to maintain fast and accurate retrieval.</p><p>Scaling can be done <strong>vertically</strong> by increasing the resources of a single server or <strong>horizontally</strong> by distributing data across multiple servers (sharding).</p><p>Most applications should start with <strong>vertical scaling</strong> and move to <strong>horizontal scaling</strong> only when necessary.</p><h3>Managed vs Self-Hosted</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kq9umWDVEdXy5BLOjT3Epg.png" /><figcaption>Managed vs Self-Hosted</figcaption></figure><p>Managed databases (e.g., <strong>Pinecone</strong>) are easy to use, automatically scale, and require no infrastructure management.</p><p>Self-hosted databases (e.g., <strong>pgvector</strong>) provide full control and lower costs at scale but require you to manage deployment, scaling, and maintenance.</p><p>Choose based on your team’s expertise, budget, and scalability needs.</p><h3>The Real Costs of Vector Search</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MB82ZBCn1xwUGd-hhK5YNA.png" /><figcaption>The Real Costs of Vector Search</figcaption></figure><p>Vector search costs grow as your data and traffic increase. Choosing the right vector database, optimizing indexes, and tuning search parameters can significantly reduce infrastructure costs while maintaining high retrieval performance.</p><h3>Real-World Vector Search Costs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KSRPDM16N_DofA_G0QMohg.png" /><figcaption>Real-World Vector Search Costs</figcaption></figure><p>For a medium-sized workload (500K vectors and 10K queries/day), managed services like <strong>Pinecone Serverless</strong> are competitively priced, while <strong>self-hosted pgvector</strong> offers lower costs but requires more operational effort. The best choice depends on your budget, performance needs, and DevOps resources.</p><h3>Costs at Scale</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n0QNlVby7gTSfgc82sfxtQ.png" /><figcaption>Costs at Scale</figcaption></figure><p>As the number of vectors grows, <strong>managed vector databases</strong> become much more expensive, while <strong>self-hosted pgvector</strong> scales more cost-effectively.</p><p>At large scale, self-hosting can save <strong>thousands of dollars per month</strong>, making it a better choice for high-volume applications.</p><h3>Cost Optimization Strategies</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hB5jl9cYjKY_j6B9kvtK-g.png" /></figure><ul><li><strong>Reduce Dimensions</strong> — Decrease the embedding size (e.g., <strong>1536 → 512</strong>) to reduce storage and speed up searches.</li><li><strong>Quantization</strong> — Convert vectors from <strong>float32 to int8</strong> to reduce memory usage with minimal accuracy loss.</li><li><strong>Batch Queries</strong> — Process multiple queries together instead of one at a time to improve efficiency and lower costs.</li><li><strong>Caching</strong> — Store answers for frequently asked queries so repeated requests don’t require another vector search or LLM call.</li><li><strong>Right-Sizing</strong> — Allocate only the resources (CPU, RAM, storage) your application actually needs to avoid overpaying.</li></ul><h4>Reducing Embedding Dimensions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mfro4lIn1WILGftRlAu0Yw.png" /><figcaption>Reducing Embedding Dimensions</figcaption></figure><p>This code creates embeddings using <strong>OpenAI’s </strong><strong>text-embedding-3-small</strong> model with <strong>512 dimensions</strong> instead of the default <strong>1536 dimensions</strong>.</p><p>Smaller embeddings reduce storage, memory usage, and vector search costs while maintaining good retrieval quality for many RAG applications.</p><h3>Batch Queries</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KxS7Fkv0vmnnJ3CySk3msw.png" /><figcaption>Batch Queries</figcaption></figure><p>Instead of sending <strong>one API request per query</strong>, send <strong>multiple queries in a single batch request</strong>. Batching reduces API calls, lowers costs, improves throughput, and speeds up large-scale vector searches.</p><p>Instead of sending <strong>one API request per query</strong>, send <strong>multiple queries in a single batch request</strong>. Batching reduces API calls, lowers costs, improves throughput, and speeds up large-scale vector searches.</p><h3>Caching</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lKZBc73C0UqrWIAFcXAzWw.png" /><figcaption>Caching</figcaption></figure><p>This code uses an <strong>LRU (Least Recently Used) cache</strong> to store results of previous similarity searches. If the same query is requested again, the cached result is returned instead of searching the vector database again, reducing latency, API calls, and costs.</p><h3>Security &amp; PII Handling Patterns</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mrFCKs1JE-QbyVTRpTbsvg.png" /><figcaption>Security &amp; PII Handling Patterns</figcaption></figure><p>These are best practices for securing LLM applications by preventing prompt injection, protecting sensitive data, validating outputs, and monitoring the system.</p><p><strong>1. Input Sanitization</strong><br> Clean and validate user input to block prompt injection and malicious instructions before they reach the LLM.</p><p><strong>2. PII Detection &amp; Masking</strong><br> Detect and hide sensitive information such as emails, phone numbers, SSNs, credit card numbers, and IP addresses to protect user privacy.</p><p><strong>3. LLM-as-Guard Pattern</strong><br> Use a separate LLM or guard model to classify requests as safe or unsafe and block harmful or restricted prompts.</p><p><strong>4. Production Security Pipeline</strong><br> A secure workflow: Input Sanitization → PII Masking → Security Guard → Business Logic (RAG/Agents) → Output Validation → Logging &amp; Monitoring.</p><p><strong>5. Output Validation</strong><br> Check the model’s response for harmful, unsafe, or sensitive content before sending it to the user.</p><p><strong>6. Metrics &amp; Logging</strong><br> Record security events, blocked requests, errors, latency, and system activity for monitoring and auditing.</p><p><strong>7. Key Benefits</strong><br> Improves privacy, prevents prompt injection, ensures compliance, builds user trust, and reduces the risk of data leakage.</p><p><strong>Principle</strong><br> <strong>Trust nothing, validate everything, and log every important action.</strong></p><h3>Choosing the Right Vector Database</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nqSyg1vIskJ980AankPqBQ.png" /><figcaption>Choosing the Right Vector Database</figcaption></figure><p>The best vector database depends on your application size and budget. Start with a simple managed solution for small projects, then move to managed or self-hosted databases as your data and traffic grow. Avoid over-optimizing early and scale only when necessary.</p><h3>Production Hosting</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FE368Pee4DJg_rt4eDFbaQ.png" /><figcaption>Production Hosting</figcaption></figure><p>For production deployment, choose a hosting platform based on your budget and scalability needs. <strong>Supabase</strong> is a great all-round choice, <strong>Neon</strong> is ideal for serverless applications, and <strong>AWS RDS</strong> is best for enterprise workloads requiring high performance and reliability.</p><h3>Production Visibility</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ylOc7-ZrO0jC0UYMgFhclQ.png" /><figcaption>Production Visibility</figcaption></figure><p>Production visibility helps you monitor and debug LLM applications in real time.</p><p>It combines <strong>structured logging</strong> (what happened), <strong>metrics collection</strong> (how much happened), and <strong>instrumented LLMs</strong> (tracking every LLM call) to ensure your application is reliable, fast, and cost-effective.</p><h3>Monitoring in Production</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9iuy-YW5z5nxANtgoo9BuA.png" /><figcaption>Monitoring in Production</figcaption></figure><p>Monitoring is the outer layer of an LLM application that observes every component without changing its behavior.</p><p>It tracks <strong>logs, metrics, and traces</strong> across security, cost optimization, and error handling, making it easier to detect issues, measure performance, and maintain a reliable production system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8wo0iNd-qld9hzatk2bsw.png" /></figure><p><strong>1. Structured Logging:</strong> Store events in a structured JSON format with details like timestamps, messages, functions, and metadata for easier debugging and auditing.</p><p><strong>2. Metrics Collection:</strong> Collect key metrics such as request count, latency, token usage, error rate, cache hit rate, and API cost to monitor system health.</p><p><strong>3. Instrumented LLM:</strong> Wrap every LLM call to capture prompts, responses, latency, token usage, model parameters, errors, and metadata for complete observability.</p><p><strong>4. Metrics Summary (Dashboards):</strong> Visualize important metrics like total requests, latency, error rate, cache hit rate, token usage, and cost using real-time dashboards.</p><p><strong>5. Alerts &amp; Notifications:</strong> Automatically notify teams when latency, error rate, cost, cache performance, or model failures exceed predefined thresholds.</p><h3>Monitoring &amp; Logging vs LangSmith</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAoWyw46VZU-OF3IHguEvQ.png" /><figcaption>Monitoring &amp; Logging vs LangSmith</figcaption></figure><p>Monitoring &amp; Logging and LangSmith serve different but complementary purposes.</p><p><strong>Monitoring &amp; Logging</strong> tracks the overall health of your application, including latency, errors, costs, and infrastructure metrics.</p><p><strong>LangSmith</strong> focuses on LLM behavior by tracing prompts, responses, tool calls, retrieval steps, and agent execution.</p><p>Together, they provide complete observability for building reliable, debuggable, and production-ready AI applications.</p><h3>Production-Ready API</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Izg3AxidtlVv9kZ3cE8qeQ.png" /><figcaption>Production-Ready API</figcaption></figure><p>A production-ready API combines essential features like <strong>LangSmith tracing, input sanitization, PII masking, error handling with retries, response caching, rate limiting, structured logging, metrics collection, health checks, and Docker deployment</strong> to build secure, scalable, observable, and reliable AI applications ready for real-world production use.</p><h3>Production LLM API Request Flow</h3><p>A production LLM API processes every request through multiple layers to ensure <strong>security, reliability, performance, and observability</strong>. T</p><p>he request is rate-limited, sanitized, checked for cached responses, validated, logged with metrics, and finally returned as a structured JSON response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*lOizlc8Qmloku_NBpDsFkg.png" /></figure><h3>Fallback Pattern</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y6dOgZhQsb9Z0IPhNZ0bxA.png" /><figcaption>Fallback Pattern</figcaption></figure><p>A fallback pattern improves application reliability by trying an alternative option when the primary process fails.</p><p>If the main process succeeds, the request ends normally. If it fails, the system switches to a backup model or service.</p><p>If the fallback also fails, it returns a clear and user-friendly error instead of crashing.</p><h3>Production AI API Checklist</h3><p>Building a production-ready AI application requires more than just connecting an LLM.</p><p>A robust system must be <strong>secure, reliable, performant, observable, and easy to deploy</strong>.</p><p>By following these production best practices, you can protect user data, improve system stability, optimize response times, monitor application health, and ensure smooth deployment at scale.</p><p>The following checklists cover the essential components every production AI API should include.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TQM7an4LZhSuCyUd7KsglQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eWu6kFU8OpOpy4WPYevFug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y97BwNADf3yF5N5hA5r9ZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oAIUpR8Tb9G1SwZL-tQVzg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A3baY5Wf8qubHvwZeyduxw.png" /></figure><h3>Long Context vs RAG</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RH5Z9vpUQ3SRAKkDx6NTMQ.png" /><figcaption>Long Context vs RAG</figcaption></figure><p>Long Context and RAG are two ways to provide knowledge to an LLM.</p><p><strong>Long Context</strong> sends large documents directly to the model, while <strong>RAG (Retrieval-Augmented Generation)</strong> retrieves only the most relevant information.</p><p>RAG is usually <strong>faster, cheaper, and more scalable</strong>, whereas Long Context is better for tasks that require understanding an entire document.</p><h3>Contextual Retrieval</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RSGqyLHa-vRYpSg5QhSMDg.png" /><figcaption>Contextual Retrieval</figcaption></figure><p>Contextual Retrieval improves RAG by adding meaningful context to each document chunk before indexing.</p><p>Unlike basic chunking, where chunks lose their surrounding information, contextual retrieval makes each chunk more informative, leading to <strong>better retrieval accuracy and fewer retrieval failures</strong>.</p><h3>Late Chunking vs Early Chunking</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-OxG7dd5eZPY-RT5oIZ8ug.png" /><figcaption>Late Chunking vs Early Chunking</figcaption></figure><p><strong>Early Chunking</strong> splits a document before creating embeddings, which can lose important context.</p><p><strong>Late Chunking</strong> first embeds the entire document, then splits it into chunks, preserving more context and improving retrieval accuracy, especially for references across the document.</p><h3>Agentic RAG Workflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4jf5LGzTVRUkYucu6G1hDQ.png" /><figcaption>Agentic RAG Workflow</figcaption></figure><ol><li><strong>User Query</strong> → User asks a question.</li><li><strong>Agent</strong> → Decides whether retrieval is needed.</li><li><strong>Retrieve</strong> → Fetches relevant documents from the vector database.</li><li><strong>Evaluate</strong> → Checks if the retrieved results are relevant.</li><li><strong>Decision</strong> → If results are good, generate the answer. If not, rewrite the query and search again.</li><li><strong>Generate</strong> → Produces the final answer using the best retrieved context.</li></ol><h3>GraphRAG</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lT9-KC20LUTC-26nvgbYcA.png" /><figcaption>GraphRAG</figcaption></figure><p><strong>GraphRAG</strong> enhances traditional RAG by using a knowledge graph to connect related entities and relationships.</p><p>Instead of retrieving isolated documents, it follows connections between entities, making it ideal for <strong>multi-hop reasoning</strong>, where answering a question requires combining information from multiple linked sources.</p><h3>GraphRAG Workflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ljp7HLZtS4EpU8LIzC5LKQ.png" /><figcaption>GraphRAG Workflow</figcaption></figure><ol><li><strong>Extract Entities &amp; Relationships</strong> → Identify entities (people, companies, products) and their relationships from documents.</li><li><strong>Build Knowledge Graph</strong> → Store these entities and connections in a graph database.</li><li><strong>Graph Traversal</strong> → Traverse the graph to connect related entities and perform multi-hop reasoning.</li><li><strong>Hybrid Retrieval</strong> → Combine graph relationships with document retrieval to generate a more accurate answer.</li></ol><h3>ColPali: Vision-Based Document RAG</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5NAITqhfjzf4D2JBkNcDYw.png" /><figcaption>ColPali: Vision-Based Document RAG</figcaption></figure><p><strong>ColPali (Contextualized Late Interaction for Vision-Language Models)</strong> is a <strong>multimodal RAG</strong> approach that retrieves information from <strong>document images rather than only extracted text</strong>.</p><p>In traditional text-based RAG, documents are first converted into text using OCR, which can lose important information such as tables, charts, formulas, and page layouts.</p><p>ColPali avoids this problem by rendering each PDF page as an image and creating <strong>vision embeddings</strong> that preserve the document’s visual structure.</p><p>When a user submits a query, the system retrieves the most relevant page images and provides them to a Vision LLM to generate an accurate answer.</p><p>This makes ColPali highly effective for scanned PDFs, financial reports, research papers, invoices, forms, and any document where visual layout is essential.</p><h3>Conclusion: The Evolution of Modern RAG</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vOpGBR2TEEOQlg5Ls3RbwA.png" /><figcaption>Conclusion: The Evolution of Modern RAG</figcaption></figure><ul><li><strong>Long Context vs RAG</strong> helps choose the most <strong>cost-effective</strong> way to provide context to the LLM.</li><li><strong>Contextual Retrieval</strong> improves search by retrieving information based on the <strong>query’s context and intent</strong>.</li><li><strong>Late Chunking</strong> preserves document semantics, resulting in <strong>better retrieval quality and coherence</strong>.</li><li><strong>Agentic RAG</strong> enables AI agents to <strong>plan, reason, use tools, and perform multi-step retrieval</strong> autonomously.</li><li><strong>GraphRAG</strong> leverages <strong>knowledge graphs</strong> to retrieve connected entities and relationships for deeper reasoning.</li><li><strong>Multimodal RAG (ColPali)</strong> extends retrieval beyond text by understanding <strong>images, tables, charts, formulas, and scanned documents</strong>.</li></ul><h3>Final Takeaway</h3><ul><li><strong>2023:</strong> Naive RAG → Basic document retrieval.</li><li><strong>2024:</strong> Optimized RAG → Better chunking, retrieval, and performance.</li><li><strong>2026:</strong> Agentic + Multimodal RAG → Intelligent reasoning, structured knowledge exploration, and multimodal understanding for production-ready AI systems.</li></ul><p><strong>Together, these techniques create faster, more accurate, scalable, and context-aware RAG applications capable of handling complex real-world tasks.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f2d5498dec8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Engineering OS Explained: React, Convex, FastAPI, Codex, Conductor, Railway, Git Worktrees…]]></title>
            <link>https://medium.com/@sujangyawali177/ai-engineering-os-explained-react-convex-fastapi-codex-conductor-railway-git-worktrees-c8724f89bdf5?source=rss-23434c2aa8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8724f89bdf5</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[developer-tools]]></category>
            <dc:creator><![CDATA[Sujangyawali]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 20:17:47 GMT</pubDate>
            <atom:updated>2026-07-13T20:17:47.094Z</atom:updated>
            <content:encoded><![CDATA[<h3>AI Engineering OS Explained: React, Convex, FastAPI, Codex, Conductor, Railway, Git Worktrees, Shell, Parallel AI Agents, and the Modern Software Engineering Stack</h3><h3>Introduction</h3><p>If you’ve watched demos from modern AI companies, you’ve probably heard people say things like:</p><blockquote><em>“We use React, Convex, Shell, Codex inside Conductor, spin multiple Git worktrees, fix bugs, ship features, run parallel workflows, and deploy on Railway.”</em></blockquote><p>At first, this sounds confusing.</p><p>The truth is that <strong>each technology solves one specific problem</strong>. Together, they create what many people now call an <strong>AI Engineering Operating System</strong> — a platform where AI agents help build, test, review, and deploy software.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uZ7iPxBv5LA720UAVCV36A.png" /><figcaption>AI Engineering OS Explained</figcaption></figure><p>Let’s understand every term.</p><h3>The Complete Architecture</h3><pre><br><br><br>                   USER<br>                      │<br>                      ▼<br>             React Frontend Dashboard<br>                      │<br>              (Beautiful User Interface)<br>                      │<br>                      ▼<br>                  Convex Client<br>                      │<br>          Realtime Sync &amp; Live Updates<br>                      │<br>                      ▼<br>                FastAPI Backend<br>                      │<br>          AI Conductor (Orchestrator)<br>                      │<br>     ┌────────────────┼────────────────┐<br>     ▼                ▼                ▼<br> Planner Agent   Backend Agent   Frontend Agent<br>     │                │                │<br>     ▼                ▼                ▼<br> Git Worktree    Git Worktree    Git Worktree<br>     │                │                │<br>     ▼                ▼                ▼<br> Linux Shell     Linux Shell     Linux Shell<br>     │                │                │<br>     ▼                ▼                ▼<br> Codex / Claude  Codex / Claude  Codex / Claude<br>     │<br>     ▼<br> Tests → Review → GitHub → Railway → Production</pre><p>Every box has a different responsibility.</p><h3>React</h3><h3>What is React?</h3><p>React is a <strong>frontend library</strong>.</p><p>Its job is to build what users see.</p><p>Examples:</p><ul><li>Dashboard</li><li>Login page</li><li>Agent status</li><li>Terminal viewer</li><li>Pull requests</li><li>Charts</li><li>Buttons</li><li>Project list</li></ul><p>React creates the <strong>User Interface (UI)</strong>.</p><h3>Frontend</h3><p>The frontend is everything users interact with.</p><p>Examples:</p><ul><li>Buttons</li><li>Forms</li><li>Navigation</li><li>Dashboards</li><li>Progress bars</li><li>Live logs</li></ul><p>React is one way to build a frontend.</p><h3>Is Convex for Making a Good UI?</h3><p><strong>No.</strong></p><p>This is one of the biggest misconceptions.</p><p>Convex does <strong>not</strong> make your UI prettier.</p><p>These tools create the UI:</p><ul><li>React</li><li>Tailwind CSS</li><li>shadcn/ui</li><li>Framer Motion</li></ul><p>Convex does something completely different.</p><h3>What is Convex?</h3><p>Convex is a <strong>realtime backend platform</strong>.</p><p>It provides:</p><ul><li>Database</li><li>Backend functions</li><li>Live synchronization</li><li>Authentication support</li><li>Automatic updates</li></ul><p>Think of it like this:</p><p>React builds the screen.</p><p>Convex keeps the screen updated.</p><h3>Example</h3><p>Imagine five AI agents working.</p><p>Without Convex:</p><pre>Agent finished</pre><pre>↓</pre><pre>Database changed</pre><pre>↓</pre><pre>User presses Refresh</pre><pre>↓</pre><pre>New status appears</pre><p>With Convex:</p><pre>Agent finished</pre><pre>↓</pre><pre>Database updated</pre><pre>↓</pre><pre>React updates instantly</pre><pre>↓</pre><pre>No Refresh Button</pre><p>That’s why AI dashboards often use Convex.</p><p>Convex improves the <strong>experience</strong>, not the <strong>appearance</strong>.</p><h3>FastAPI</h3><p>FastAPI is the backend.</p><p>It performs heavy work such as:</p><ul><li>Calling AI models</li><li>Reading Git repositories</li><li>Running Docker</li><li>Executing shell commands</li><li>Managing authentication</li><li>Talking to GitHub</li></ul><p>Think of FastAPI as the application’s engine.</p><h3>Backend</h3><p>The backend performs everything users don’t see.</p><p>Examples:</p><ul><li>AI reasoning</li><li>Database operations</li><li>Business logic</li><li>API endpoints</li><li>Authentication</li><li>Scheduling</li></ul><p>The frontend displays.</p><p>The backend computes.</p><h3>Shell</h3><p>The Linux shell is a command-line interface.</p><p>AI coding agents use it exactly like human developers.</p><p>Example:</p><pre>git status</pre><pre>pytest</pre><pre>npm install</pre><pre>docker compose up</pre><p>The shell lets AI interact with the operating system.</p><h3>Why does an AI use the Shell?</h3><p>Because software engineering is more than writing code.</p><p>An AI needs to:</p><ul><li>Install packages</li><li>Run tests</li><li>Build projects</li><li>Start servers</li><li>Read logs</li><li>Commit code</li></ul><p>The shell makes all of that possible.</p><h3>Codex</h3><p>Codex is an AI coding model.</p><p>It can:</p><ul><li>Read code</li><li>Write code</li><li>Explain code</li><li>Fix bugs</li><li>Generate tests</li><li>Refactor projects</li></ul><p>Think of Codex as one software engineer.</p><h3>Git</h3><p>Git stores project history.</p><p>AI agents constantly execute:</p><pre>git commit</pre><pre>git checkout</pre><pre>git status</pre><p>Git tracks every change.</p><h3>Git Worktrees</h3><p>Normally a repository has one working directory.</p><p>Example:</p><pre>project/</pre><p>Git Worktrees allow multiple working directories.</p><pre>project/</pre><pre>project-login/</pre><pre>project-dashboard/</pre><pre>project-tests/</pre><p>Each AI agent receives its own worktree.</p><p>Benefits:</p><ul><li>Parallel development</li><li>Fewer merge conflicts</li><li>Independent testing</li></ul><h3>Parallel Workflow</h3><p>Instead of one AI doing everything:</p><pre>User</pre><pre>↓</pre><pre>One AI</pre><pre>↓</pre><pre>One Branch</pre><pre>↓</pre><pre>Done</pre><p>Modern systems work like this:</p><pre>User</pre><pre>↓</pre><pre>Planner</pre><pre>↓</pre><pre>Frontend Task</pre><pre>Backend Task</pre><pre>Testing Task</pre><pre>Documentation Task</pre><pre>↓</pre><pre>Four AI Engineers</pre><pre>↓</pre><pre>Four Git Worktrees</pre><pre>↓</pre><pre>Parallel Execution</pre><p>This significantly reduces the time to complete independent tasks.</p><h3>Conductor</h3><p>Conductor is <strong>not</strong> an AI model.</p><p>It is an orchestrator.</p><p>Imagine a software company.</p><p>Who assigns work?</p><p>The engineering manager.</p><p>Conductor plays a similar role.</p><p>Responsibilities:</p><ul><li>Create workspaces</li><li>Launch AI agents</li><li>Assign tasks</li><li>Track progress</li><li>Merge results</li><li>Coordinate reviews</li></ul><p>Conductor manages the engineers.</p><h3>Planner</h3><p>The planner breaks one large request into smaller tasks.</p><p>Input:</p><pre>Build authentication</pre><pre>Improve dashboard</pre><pre>Write tests</pre><p>Output:</p><pre>Task 1</pre><pre>Authentication</pre><pre>--------------</pre><pre>Task 2</pre><pre>Dashboard</pre><pre>--------------</pre><pre>Task 3</pre><pre>Testing</pre><p>Each task goes to an appropriate agent.</p><h3>AI Agents</h3><p>Instead of one AI:</p><pre>AI</pre><p>Modern systems use specialized agents.</p><pre>Frontend Agent</pre><pre>Backend Agent</pre><pre>Testing Agent</pre><pre>Documentation Agent</pre><pre>Deployment Agent</pre><p>Each specializes in a different area.</p><h3>Railway</h3><p>Railway is a deployment platform.</p><p>After AI finishes coding:</p><pre>Github<br>↓</pre><pre>Railway</pre><pre>↓</pre><pre>Website Online</pre><p>Railway hosts:</p><ul><li>React frontend</li><li>FastAPI backend</li><li>PostgreSQL</li><li>Redis</li></ul><p>It makes deployment simple.</p><h3>Putting Everything Together</h3><p>Imagine you ask:</p><blockquote><em>Fix the login bug, improve the dashboard, and deploy.</em></blockquote><p>The system works like this.</p><pre>User</pre><pre>↓</pre><pre>React Dashboard</pre><pre>↓</pre><pre>FastAPI</pre><pre>↓</pre><pre>Planner</pre><pre>↓</pre><pre>Conductor</pre><pre>↓</pre><pre>Creates 3 Git Worktrees</pre><pre>↓</pre><pre>Frontend Agent</pre><pre>Backend Agent</pre><pre>Testing Agent</pre><pre>↓</pre><pre>Linux Shell</pre><pre>↓</pre><pre>Codex</pre><pre>↓</pre><pre>pytest</pre><pre>↓</pre><pre>Git Commit</pre><pre>↓</pre><pre>Pull Request</pre><pre>↓</pre><pre>Railway</pre><pre>↓</pre><pre>Production</pre><p>Every technology contributes one piece.</p><h3>Final Analogy</h3><p>Imagine building a software company.</p><pre>Customer<br>↓</pre><pre>Reception</pre><pre>↓</pre><pre>Engineering Manager</pre><pre>↓</pre><pre>Frontend Engineer</pre><pre>Backend Engineer</pre><pre>QA Engineer</pre><pre>↓</pre><pre>Workspaces</pre><pre>↓</pre><pre>Developer Tools</pre><pre>↓</pre><pre>Deployment</pre><pre>↓</pre><pre>Live Product</pre><p>Now replace those roles with technologies:</p><ul><li>React = Reception (what users see)</li><li>Convex = Live communication system</li><li>FastAPI = Operations department</li><li>Conductor = Engineering manager</li><li>Planner = Project planner</li><li>Codex = Software engineer</li><li>Git Worktrees = Separate desks for each engineer</li><li>Shell = Engineer’s toolbox</li><li>Railway = Factory that ships the final product</li></ul><p>Modern AI engineering platforms are not one giant AI model — they are <strong>systems</strong> that combine many specialized tools into one coordinated workflow.</p><p>The most important lesson is this:</p><ul><li><strong>React makes the UI.</strong></li><li><strong>Tailwind makes it beautiful.</strong></li><li><strong>Convex makes it live.</strong></li><li><strong>FastAPI runs the backend.</strong></li><li><strong>Conductor manages AI workers.</strong></li><li><strong>Codex writes code.</strong></li><li><strong>Shell executes commands.</strong></li><li><strong>Git Worktrees enable parallel development.</strong></li><li><strong>Railway deploys the finished application.</strong></li></ul><p>When combined, these technologies create a modern AI Engineering Operating System capable of planning, implementing, testing, reviewing, and deploying software with multiple AI agents working together.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8724f89bdf5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond RAG: 6 Real-World Applications of Vector Databases Every AI Engineer Should Know]]></title>
            <link>https://medium.com/@sujangyawali177/beyond-rag-6-real-world-applications-of-vector-databases-every-ai-engineer-should-know-879c0966148b?source=rss-23434c2aa8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/879c0966148b</guid>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[pinecone]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sujangyawali]]></dc:creator>
            <pubDate>Sun, 12 Jul 2026 21:41:46 GMT</pubDate>
            <atom:updated>2026-07-12T21:41:46.428Z</atom:updated>
            <content:encoded><![CDATA[<p>When people talk about vector databases, the conversation almost always starts — and ends — with Retrieval-Augmented Generation (RAG). After all, RAG has become one of the most popular ways to make Large Language Models (LLMs) more accurate by retrieving relevant information before generating a response.</p><p>But here’s something I didn’t realize when I first started learning about vector databases: <strong>RAG is just one application.</strong></p><p>The same technology that powers AI chatbots can also be used to build semantic search engines, recommendation systems, image search, facial similarity applications, hybrid search, and even anomaly detection systems. Once you understand how embeddings and vector search work, you begin to see the same pattern appear across many different AI products.</p><p>Think about how Spotify recommends songs that match your taste, how Google Photos finds pictures of your dog without you tagging them, or how an e-commerce website can search for products using both an image and a text description. Behind many of these experiences is the idea of representing data as vectors and searching for items based on similarity rather than exact matches.</p><p>In this article, we’ll explore what vector databases are, why they have become a core part of modern AI systems, and how they can be used far beyond RAG. We’ll walk through six practical applications, understand the underlying concepts, and look at where vector databases fit in real-world AI architectures.</p><p>If you’re learning AI engineering, LangChain, LangGraph, or building LLM-powered applications, understanding vector databases is one of the most valuable skills you can add to your toolkit.</p><h3><strong>Introduction</strong></h3><p>Before building real-world AI applications, it’s important to understand the technology that powers them. A vector database is designed to store and search high-dimensional embeddings efficiently. Instead of retrieving information using exact keywords, it finds data based on semantic similarity, making it an essential component of modern AI systems.</p><p>The following figure illustrates how vector databases convert data into embeddings, store them alongside metadata, and retrieve the most similar results using vector similarity search. It also highlights the differences between traditional vector indexes and production-ready vector databases such as Pinecone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ROaT6DvdTbcnSaCVKaqLRw.png" /><figcaption><strong>Figure:</strong> <em>What is a Vector Database and How Does It Work?</em></figcaption></figure><h3><strong>Build a Semantic Search App with Pincone</strong></h3><p>Traditional search engines rely on exact keyword matching, which often fails when users phrase the same idea differently. Semantic search solves this limitation by comparing the meaning of text rather than the words themselves.</p><p>In this project, we’ll build a semantic search application using Sentence Transformers to generate embeddings and Pinecone to perform fast similarity search. The following figure provides a complete overview of the end-to-end workflow, from preparing the dataset to retrieving the most semantically relevant results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FFAi-hV87kuSKcMq8vqcvQ.png" /><figcaption><strong>Figure:</strong> <em>Semantic Search Pipeline using Pinecone</em></figcaption></figure><h3>Build a RAG System with Pinecone and OpenAI</h3><p>Retrieval-Augmented Generation (RAG) extends Large Language Models by allowing them to retrieve external knowledge before generating a response. Instead of relying only on information learned during training, the model first searches a vector database for relevant documents and then uses those documents as context for answer generation.</p><p>In this example, we’ll build a complete RAG pipeline using Pinecone as the retrieval layer and OpenAI as the reasoning engine. The figure below demonstrates how user queries, vector retrieval, and LLM generation work together to produce grounded and context-aware responses.</p><p><strong>Figure:</strong> <em>End-to-End RAG Pipeline with Pinecone and OpenAI</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HtYYAwyeUauHORe4P3LMeA.png" /></figure><h3>Building a Simple Recommender System with Vector Embeddings</h3><p>Recommendation systems are one of the most successful applications of vector similarity search. Rather than recommending items based solely on popularity or manually defined rules, embeddings capture the semantic meaning of content, allowing the system to recommend items that are genuinely similar.</p><p>In this project, we’ll build a simple recommender system that supports both title-based and content-based recommendations. The figure below illustrates how documents are converted into embeddings, indexed in Pinecone, and retrieved based on semantic similarity.</p><p><strong>Figure:</strong> <em>Recommendation System using Vector Embeddings</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U3myFRAuRQJSvdQ3Ez0RHw.png" /></figure><h3>Build a Face Similarity Search System</h3><p>Vector databases are not limited to text — they can also power computer vision applications. Modern facial recognition models transform images into numerical embeddings that capture unique facial features, making it possible to compare faces using vector similarity.</p><p>In this project, we’ll generate facial embeddings using FaceNet, store them in Pinecone, and compare similarity scores to identify the most visually similar faces. The following figure shows the complete workflow from image preprocessing to similarity computation.</p><p><strong>Figure:</strong> <em>Face Similarity Search Pipeline</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YjOkP6QBbIxx2xLSrzz87g.png" /><figcaption>Hybrid Search with Pinecone</figcaption></figure><h3>Hybrid Search with Pinecone: Dense + Sparse Vectors</h3><p>Semantic search excels at understanding meaning, while keyword search is better at matching exact terms. Many production search systems combine these two approaches to achieve the best overall search quality.</p><p>Hybrid search integrates dense embeddings with sparse keyword representations, allowing systems to understand both semantic intent and exact keyword matches.</p><p>Dense vectors encode semantic meaning using deep learning models such as CLIP or Sentence Transformers, while sparse vectors preserve important keyword information using methods such as BM25. Pinecone allows both representations to be stored together and searched simultaneously.</p><p>In this implementation, we’ll explore how the hybrid search algorithm balances semantic understanding and keyword matching through the alpha (α) parameter. The figure below demonstrates the complete hybrid retrieval architecture and how different alpha values influence search results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7QWYDBPkxK8ppbYVNx-4g.png" /></figure><h3>Anomaly Detection in Cisco ASA Log Files using Sentence Transformer and Pinecone</h3><p>Similarity search is useful not only for retrieving relevant information but also for detecting unusual patterns. When log messages are converted into embeddings, normal events naturally cluster together, while abnormal events appear as outliers with low similarity scores.</p><p>In this project, we’ll train a Sentence Transformer model on Cisco ASA log data, store the embeddings in Pinecone, and use similarity search to identify anomalous log entries. The following figure presents the complete pipeline, including data preparation, model training, embedding generation, vector storage, similarity search, and anomaly detection.</p><p><strong>Figure:</strong> <em>Anomaly Detection Pipeline using Sentence Transformers and Pinecone</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X7Q8xTo541Ex58Advqzjvw.png" /></figure><h3>Conclusion</h3><p>Vector databases have become a foundational component of modern AI systems. While RAG has popularized their use, it represents only one of many practical applications. The same vector search principles power semantic search engines, recommendation systems, computer vision applications, hybrid retrieval systems, and anomaly detection pipelines.</p><p>As AI continues to evolve toward multimodal and agentic systems, understanding embeddings, similarity search, and vector databases will become an essential skill for every AI engineer. Whether you’re building LLM applications with LangChain and LangGraph or developing production-ready search and recommendation systems, mastering vector databases will give you a strong foundation for the next generation of AI applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=879c0966148b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Agentic AI and LangGraph: Why LangGraph is the Future of Complex AI Workflows]]></title>
            <link>https://medium.com/@sujangyawali177/understanding-agentic-ai-and-langgraph-why-langgraph-is-the-future-of-complex-ai-workflows-2d8d2dc3c4d9?source=rss-23434c2aa8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d8d2dc3c4d9</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[langgraph]]></category>
            <dc:creator><![CDATA[Sujangyawali]]></dc:creator>
            <pubDate>Sat, 11 Jul 2026 12:18:35 GMT</pubDate>
            <atom:updated>2026-07-11T12:18:35.961Z</atom:updated>
            <content:encoded><![CDATA[<p><em>From autonomous AI agents to human-in-the-loop systems, memory, fault tolerance, and workflow orchestration — here’s everything you need to know about LangGraph.</em></p><p>The AI world is rapidly moving beyond simple chatbots.</p><p>Instead of answering one question at a time, modern AI systems are expected to:</p><ul><li>plan tasks,</li><li>make decisions,</li><li>use external tools,</li><li>remember previous interactions,</li><li>recover from failures,</li><li>collaborate with humans,</li><li>and even coordinate multiple AI agents.</li></ul><p>This new generation of systems is known as <strong>Agentic AI</strong>.</p><p>Frameworks like <strong>LangChain</strong> made it easy to build LLM applications, but as workflows became more complex, developers needed something that could manage state, branching, retries, and long-running processes.</p><p>That’s exactly why <strong>LangGraph</strong> was created.</p><p>In this article, we’ll understand:</p><ul><li>What Agentic AI is</li><li>Characteristics of intelligent agents</li><li>Components of an AI agent</li><li>How LangGraph organizes agent workflows</li><li>Why LangChain alone isn’t enough</li><li>LangChain vs LangGraph</li><li>When to use each framework</li></ul><h3>What is Agentic AI?</h3><p>Agentic AI refers to AI systems capable of making decisions and executing tasks with minimal human intervention while continuously adapting to changing situations.</p><p>Unlike traditional chatbots that simply respond to prompts, an AI agent can:</p><ul><li>Understand goals</li><li>Break them into smaller tasks</li><li>Choose appropriate tools</li><li>Execute actions</li><li>Learn from outcome</li><li>Recover from failures</li><li>Ask humans for approval when necessary</li></ul><p>Think of an AI employee instead of an AI chatbot.</p><h3>Characteristics of Agentic AI</h3><p>The notes identify six major characteristics.</p><h3>Autonomous</h3><p>An AI agent can perform tasks independently.</p><p>Examples include:</p><ul><li>screening resumes</li><li>sending emails</li><li>querying APIs</li><li>retrieving documents</li></ul><p>However, autonomy should always be controlled.</p><p>There are several ways to control autonomous behavior.</p><h3>Permission Scope</h3><p>Define exactly what an agent is allowed to do.</p><p>For example:</p><ul><li>An HR agent may screen candidates automatically.</li><li>Rejecting a candidate still requires human approval.</li></ul><p>This prevents unintended actions.</p><h3>Human-in-the-Loop (HITL)</h3><p>Sometimes the AI should stop and wait.</p><p>Examples include:</p><ul><li>“Should I post this job description?”</li><li>“Should I send this offer letter?”</li></ul><p>LangGraph allows inserting checkpoints where execution pauses until a human approves.</p><h3>Override Controls</h3><p>Humans should always be able to:</p><ul><li>stop</li><li>pause</li><li>modify</li><li>resume</li></ul><p>an AI workflow at any moment.</p><h3>Guardrails and Policies</h3><p>Agents should follow organizational policies.</p><p>Examples:</p><ul><li>Never schedule interviews on weekends.</li><li>Never use informal language with customers.</li><li>Never share confidential information.</li></ul><p>Guardrails define ethical and operational boundaries.</p><h3>Goal-Oriented Planning</h3><p>An AI agent works toward achieving a goal instead of merely responding to prompts.</p><p>Before execution begins, it performs planning.</p><p>Planning includes:</p><ul><li>selecting tools</li><li>estimating resources</li><li>estimating dependencies</li><li>decomposing large goals into smaller tasks</li></ul><p>Instead of asking:</p><blockquote><em>“What should I answer?”</em></blockquote><p>the agent asks:</p><blockquote><em>“How do I complete this objective?”</em></blockquote><h3>Reasoning</h3><p>Reasoning occurs during both planning and execution.</p><h4>During Planning</h4><p>The agent decides:</p><ul><li>Which tools should I use?</li><li>What sequence should I follow?</li><li>How long might execution take?</li><li>Should I split the task?</li></ul><h4>During Execution</h4><p>The agent reasons continuously.</p><p>Examples:</p><ul><li>Should I pause for human approval?</li><li>Did an API fail?</li><li>Should I retry?</li><li>Should I choose another tool?</li></ul><p>Reasoning enables intelligent decision-making throughout execution.</p><h3>Adaptability</h3><p>Real-world environments constantly change.</p><p>An AI agent should modify its behavior whenever circumstances change.</p><p>Examples include:</p><ul><li>Calendar API failures</li><li>Receiving external feedback</li><li>Fewer-than-expected job applications</li><li>New business goals</li></ul><p>Rather than following a rigid workflow, the agent dynamically adjusts its plan.</p><h3>Context Awareness</h3><p>Agents should understand the current context before making decisions.</p><p>Context includes:</p><ul><li>original objective</li><li>conversation history</li><li>current progress</li><li>environment state</li><li>tool responses</li><li>user preferences</li><li>organizational policies</li></ul><p>Without context, an agent behaves like it has amnesia.</p><h3>Memory Enables Context Awareness</h3><p>LangGraph supports two kinds of memory.</p><h3>Short-Term Memory</h3><p>Stores:</p><ul><li>active conversation</li><li>tool outputs</li><li>current workflow state</li></ul><h3>Long-Term Memory</h3><p>Stores:</p><ul><li>previous interactions</li><li>user preferences</li><li>historical knowledge</li></ul><p>Memory allows future decisions to become increasingly personalized.</p><h3>Components of an Agentic AI System</h3><p>An intelligent AI agent consists of five major components.</p><h3>1. Brain</h3><p>The brain is usually an LLM.</p><p>Responsibilities include:</p><ul><li>understanding goals</li><li>planning</li><li>reasoning</li><li>selecting tools</li><li>communicating</li></ul><p>Think of it as the decision-making engine.</p><h3>2. Orchestrator</h3><p>The orchestrator manages execution.</p><p>Responsibilities include:</p><ul><li>task sequencing</li><li>conditional routing</li><li>retry logic</li><li>looping</li><li>delegation</li></ul><p>Delegation determines whether work should be assigned to:</p><ul><li>another LLM</li><li>external tools</li><li>humans</li></ul><p>LangGraph primarily acts as this orchestration layer.</p><h3>3. Tools</h3><p>Agents become useful only after connecting them to tools.</p><p>Examples include:</p><ul><li>APIs</li><li>databases</li><li>search engines</li><li>calendars</li><li>email</li><li>RAG systems</li></ul><p>Tools allow an agent to interact with the outside world.</p><h3>4. Memory</h3><p>Memory consists of:</p><h4>Short-Term Memory</h4><p>Maintains the active session.</p><h4>Long-Term Memory</h4><p>Stores:</p><ul><li>user preferences</li><li>historical interactions</li></ul><h3>State Tracking</h3><p>Tracks workflow progress.</p><p>Examples:</p><ul><li>completed tasks</li><li>pending tasks</li></ul><h3>Supervisor</h3><p>The supervisor provides human oversight.</p><p>Responsibilities include:</p><ul><li>approval requests</li><li>enforcing guardrails</li><li>handling edge cases</li><li>escalating uncertain situations</li></ul><p>This enables safe deployment in production.</p><h3>Example: Automated Hiring Workflow</h3><p>The notes illustrate a complete HR automation workflow.</p><p>The workflow begins with a hiring request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hVJ9rrw6tyY15WAY2395sA.png" /><figcaption>Automated Hiring Workflow</figcaption></figure><p>Why LangChain Alone Becomes Difficult</p><p>LangChain works extremely well for simple pipelines.</p><p>However, several challenges arise as workflows become more sophisticated.</p><h3>1. Complex Control Flow</h3><p>Real workflows include:</p><ul><li>branches</li><li>loops</li><li>jumps</li><li>retries</li></ul><p>Managing these manually creates a large amount of glue code.</p><p>Maintainability quickly decreases.</p><h3>2. State Management</h3><p>Every workflow contains state.</p><p>Examples include:</p><ul><li>goal</li><li>job description</li><li>approval status</li><li>posting status</li><li>number of applicants</li><li>shortlisted candidates</li><li>interview results</li><li>onboarding status</li></ul><p>Traditional LangChain does not automatically manage this state.</p><p>Developers must write manual code.</p><p>LangGraph introduces explicit state objects using structures such as:</p><ul><li>Pydantic models</li><li>TypedDicts</li></ul><p>This makes workflows stateful and easier to manage.</p><h3>3. Event-Driven Execution</h3><p>LangChain primarily executes sequential chains.</p><p>Once execution begins, it typically continues until completion.</p><p>Real-world systems often need to:</p><ul><li>pause</li><li>wait</li><li>resume later</li></ul><p>LangGraph supports event-driven execution using checkpoints.</p><h3>4. Fault Tolerance</h3><p>Long-running workflows inevitably encounter failures.</p><p>Examples include:</p><ul><li>API outages</li><li>server crashes</li><li>network failures</li></ul><p>Without checkpoints, the workflow must restart from the beginning.</p><p>LangGraph continuously stores workflow state.</p><p>This enables:</p><ul><li>retries for minor failures</li><li>recovery for major failures</li><li>resuming from the failure point</li></ul><p>instead of restarting the entire workflow.</p><h3>Checkpointing</h3><p>Checkpointing is one of LangGraph’s most important capabilities.</p><p>Imagine a workflow with ten steps.</p><p>If step eight fails:</p><p>Traditional systems restart from step one.</p><p>LangGraph resumes directly from step eight.</p><p>This dramatically improves reliability and efficiency.</p><h3>Human-in-the-Loop</h3><p>Many business decisions require human approval.</p><p>Examples include:</p><ul><li>approving financial transactions</li><li>publishing job postings</li><li>approving medical recommendations</li><li>sending offer letters</li></ul><p>LangGraph allows workflows to pause, wait for approval, and resume — even days later.</p><p>This is much more reliable than manually splitting workflows into multiple chains.</p><h3>Nested Workflows</h3><p>Large applications often contain reusable workflows.</p><p>LangGraph supports subgraphs.</p><p>Benefits include:</p><ul><li>modular design</li><li>reusable workflows</li><li>multi-agent systems</li><li>cleaner architecture</li></ul><p>Instead of creating one massive workflow, developers compose multiple smaller workflows together.</p><h3>Observability</h3><p>Production AI systems require monitoring.</p><p>Developers should be able to answer questions like:</p><ul><li>Which node failed?</li><li>Which tool was called?</li><li>How long did execution take?</li><li>Why did the workflow branch?</li></ul><p>LangGraph integrates with <strong>LangSmith</strong> to provide complete execution timelines.</p><p>This enables:</p><ul><li>debugging</li><li>tracing</li><li>monitoring</li><li>backtracking</li></ul><p>for complex AI systems.</p><h3>LangChain vs LangGraph</h3><p>LangChainLangGraphLinear workflowsComplex workflowsPrompt chainsConditional routingStatelessStatefulSequential executionEvent-driven executionLimited workflow orchestrationFull orchestrationMinimal retry supportBuilt-in retry and recoveryNo checkpointingCheckpointingDifficult HITLNative Human-in-the-LoopBasic applicationsProduction-grade AI agents</p><h3>When Should You Use LangChain?</h3><p>Choose LangChain if you’re building:</p><ul><li>prompt chains</li><li>RAG pipelines</li><li>simple chatbots</li><li>document Q&amp;A</li><li>lightweight automation</li><li>sequential workflows</li></ul><p>It provides excellent building blocks for LLM applications.</p><h3>When Should You Use LangGraph?</h3><p>Choose LangGraph when your application requires:</p><ul><li>complex workflows</li><li>branching logic</li><li>loops</li><li>retries</li><li>human approvals</li><li>memory</li><li>state management</li><li>event-driven execution</li><li>long-running workflows</li><li>multi-agent coordination</li></ul><p>LangGraph is designed for production-grade AI agents.</p><h3>Final Thoughts</h3><p>LangChain and LangGraph are not competitors — they solve different problems.</p><p><strong>LangChain</strong> provides the building blocks for interacting with LLMs, tools, prompts, and retrievers.</p><p><strong>LangGraph</strong> builds on top of those capabilities to orchestrate complex, stateful, long-running workflows with checkpointing, retries, human-in-the-loop interactions, and multi-agent coordination.</p><p>A simple chatbot may only need LangChain.</p><p>A production AI assistant that manages approvals, calls APIs, remembers context, recovers from failures, and coordinates multiple agents will almost certainly benefit from LangGraph.</p><p>As AI applications continue evolving from prompt-based systems into autonomous software agents, understanding LangGraph is becoming an essential skill for AI engineers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d8d2dc3c4d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From LangGraph to MCP to RAG: A Complete Roadmap to Building Production-Ready AI Agents]]></title>
            <link>https://medium.com/@sujangyawali177/from-langgraph-to-mcp-to-rag-a-complete-roadmap-to-building-production-ready-ai-agents-3bcc456fe2a0?source=rss-23434c2aa8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bcc456fe2a0</guid>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Sujangyawali]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 21:16:49 GMT</pubDate>
            <atom:updated>2026-07-07T21:16:49.169Z</atom:updated>
            <content:encoded><![CDATA[<p>From simple chatbots to enterprise AI agents — learn how LangGraph, LangChain, MCP, Streaming, Memory, and RAG work together to build the next generation of AI applications.</p><p>Artificial Intelligence has evolved far beyond simple chatbots. Modern AI applications can search the web, access databases, remember conversations, execute Python code, call APIs, retrieve information from private documents, and even collaborate with multiple AI agents to solve complex problems.</p><p>This evolution is driven by frameworks like <strong>LangChain</strong>, <strong>LangGraph</strong>, <strong>Model Context Protocol (MCP)</strong>, and <strong>Retrieval-Augmented Generation (RAG)</strong>.</p><p>In this article, we’ll explore the complete journey of building production-ready AI agents — from creating a basic chatbot to deploying intelligent multi-agent systems with external tools and enterprise knowledge bases.</p><h3>Why Traditional LLMs Are Not Enough</h3><p>Large Language Models (LLMs) like GPT, Claude, Gemini, and Llama are incredibly powerful, but they still have limitations.</p><p>A standalone LLM:</p><ul><li>❌ Doesn’t know real-time information</li><li>❌ Cannot access your database</li><li>❌ Doesn’t remember previous conversations</li><li>❌ Cannot call APIs by itself</li><li>❌ Cannot execute workflows</li><li>❌ Cannot search private documents</li></ul><p>This is where <strong>Agentic AI</strong> comes in.</p><p>Instead of relying on only one LLM, modern AI applications combine:</p><ul><li>LangGraph</li><li>LangChain</li><li>External Tools</li><li>Memory</li><li>MCP</li><li>RAG</li><li>Multiple Agents</li></ul><p>Together they create intelligent systems capable of reasoning, planning, retrieving information, and performing actions.</p><h3>The Evolution of AI Applications</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rrmc7k_6m3SgaSJoEL0ifQ.png" /><figcaption>Traditional chatbot vs modern AI agent</figcaption></figure><h4>Traditional Chatbot</h4><p>The model simply predicts the next token.</p><h4><strong>Modern AI Agent</strong></h4><p>Instead of only generating text, the AI agent becomes capable of solving real-world problems.</p><h3>LangGraph: The Brain Behind Agentic AI</h3><p>LangGraph is built on top of LangChain and represents AI applications as <strong>graphs</strong> instead of linear chains.</p><p>A graph consists of:</p><ul><li>Nodes</li><li>Edges</li><li>State</li><li>Execution Flow</li></ul><p>Unlike sequential pipelines, graphs allow:</p><ul><li>Conditional execution</li><li>Loops</li><li>Memory</li><li>Multi-agent communication</li><li>Human approval</li><li>Dynamic routing</li></ul><p>This makes LangGraph ideal for building complex AI systems.</p><h3><strong>Part 1: LangGraph Fundamentals</strong></h3><h4>1. Building Your First Chatbot</h4><p>Every LangGraph application begins with a simple graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jM2enEZEzp8cRT7h41ylnQ.png" /><figcaption>Simple graph</figcaption></figure><p>The chatbot node receives the user’s message, sends it to the LLM, and returns the response.</p><p>Although simple, this introduces the core concept of graph-based execution.</p><h4>2. Understanding the Graph API</h4><p>Instead of writing one long script, LangGraph lets us define workflows using nodes.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l33hAi_1Lm7FyLfVapdm-w.png" /><figcaption>Understanding the Graph API</figcaption></figure><p>Each node performs one specific task.</p><p>Examples include:</p><ul><li>Calling an LLM</li><li>Searching the internet</li><li>Querying a database</li><li>Updating memory</li><li>Executing Python</li><li>Calling an API</li></ul><p>This modular approach makes applications easier to maintain and extend.</p><h4>3. Tool Calling</h4><p>One of the biggest limitations of LLMs is that they cannot perform actions by themselves.</p><p>Instead, they call external tools.</p><p>Example tools include:</p><ul><li>Tavily Search</li><li>Calculator</li><li>Weather API</li><li>SQL Database</li><li>Python Interpreter</li><li>Email Sender</li><li>Custom Business APIs</li></ul><p>The execution flow becomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z_hYu2Ij7zDEIHfsiYh5aQ.png" /><figcaption>too calling</figcaption></figure><p>Instead of hallucinating answers, the model retrieves accurate information from trusted sources.</p><h4>4. Memory</h4><p>Memory transforms a chatbot into a conversational assistant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k1N2-HUXWZPBZkjCfRQWxw.png" /><figcaption>memory</figcaption></figure><p>Memory allows AI agents to remember:</p><ul><li>User preferences</li><li>Names</li><li>Previous tasks</li><li>Conversation history</li><li>Long-term context</li></ul><p>This creates more natural and personalized interactions.</p><h4>5. Human in the Loop</h4><p>Some actions should never happen automatically.</p><p>Examples:</p><ul><li>Sending money</li><li>Deleting files</li><li>Publishing content</li><li>Approving contracts</li></ul><p>Instead, LangGraph allows human approval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P7LWkofpGd4KV-mENdjv2Q.png" /><figcaption>Human in the Loop</figcaption></figure><p>This improves reliability and safety in enterprise systems.</p><h4>6. Streaming Responses</h4><p>Nobody likes waiting for an entire response.</p><p>Streaming sends tokens immediately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jpe6SGDFtjSIi7jFTgEt1Q.png" /><figcaption>Streaming Responses</figcaption></figure><p>Streaming significantly improves user experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNA8Berlw4BlvYq0P2C4JA.png" /><figcaption>Streaming</figcaption></figure><p>LangGraph supports:</p><ul><li>stream()</li><li>astream()</li></ul><p>Streaming modes include:</p><ul><li>values</li><li>updates</li></ul><h4>7. MCP (Model Context Protocol)</h4><p>MCP is becoming the universal standard for connecting LLMs with tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zTN9G72LMksAceOxNxNP5A.png" /><figcaption>MCP</figcaption></figure><p>Think of it as USB-C for AI.</p><p>Instead of building separate integrations for every application, AI systems communicate using one protocol.</p><p>Benefits include:</p><ul><li>Standard communication</li><li>Easy integration</li><li>Better interoperability</li><li>Reusable tools</li></ul><h3>Part 2: Advanced LangGraph</h3><p>Once the basics are understood, we move toward production systems.</p><h4>Workflows vs Agents</h4><p>A workflow follows predefined steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VPdcsfvycVCS1DhumwKqKQ.png" /><figcaption>Workflow vs Agent</figcaption></figure><p>An AI agent decides dynamically what should happen next.</p><p>Instead of fixed logic, it reasons before taking action.</p><p>Instead of fixed logic, it reasons before taking action.</p><h4>Multi-Agent Systems</h4><p>Complex problems often require multiple specialists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V6oWmSQjldziu-puimkuFQ.png" /><figcaption>Multi-Agent Systems</figcaption></figure><p>Each agent specializes in one task while collaborating with others.</p><p>This approach scales much better for enterprise applications.</p><h4>Functional API</h4><p>LangGraph also provides a functional programming interface.</p><p>Each Python function becomes a graph node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MvKCtbYzvkU3csDwASVKYQ.png" /><figcaption>Functional API</figcaption></figure><p>This makes workflows modular, reusable, and easier to debug.</p><h4>Debugging AI Workflows</h4><p>Complex AI systems need visibility.</p><p>Developers monitor:</p><ul><li>State transitions</li><li>Tool calls</li><li>Memory updates</li><li>Node execution</li><li>LLM outputs</li></ul><p>Graph visualization makes debugging significantly easier.</p><h3>Part 3: Production AI Agents</h3><p>Building an AI application is only the beginning.</p><p>Production systems require:</p><ul><li>Deployment</li><li>Monitoring</li><li>Evaluation</li><li>Scaling</li><li>Continuous Improvement</li></ul><h3>LLMOps</h3><p>LLMOps is the equivalent of DevOps for Large Language Models.</p><p>Typical pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TGJ5joDsqe9p9RI_lCeq4Q.png" /><figcaption>LLMOps</figcaption></figure><p>It ensures AI applications remain reliable after deployment.</p><h3>Deployments</h3><p>AI agents can be deployed using:</p><ul><li>FastAPI</li><li>Docker</li><li>Kubernetes</li><li>Hugging Face Spaces</li><li>Cloud Platforms</li></ul><p>Production systems should support:</p><ul><li>Scalability</li><li>Monitoring</li><li>Logging</li><li>High Availability</li></ul><h3>LLM Evaluation</h3><p>A production model should be evaluated continuously.</p><p>Common metrics include:</p><ul><li>Accuracy</li><li>Hallucination Rate</li><li>Faithfulness</li><li>Response Time</li><li>Cost</li><li>Token Usage</li><li>User Satisfaction</li></ul><p>Evaluation ensures that improvements actually benefit users.</p><h3>Understanding ReAct Agents</h3><p>ReAct stands for:</p><p><strong>Reason + Act</strong></p><p>Instead of immediately generating an answer, the model repeatedly follows a cycle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YnCnMio8reS_D8K7qz3a4A.png" /><figcaption>ReAct Agent</figcaption></figure><p>This enables much more intelligent decision-making.</p><h3>Adding Memory in LangGraph</h3><p>Memory is maintained using checkpoints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_3HXU2JyByEY9dPL0ypcAA.png" /><figcaption>Adding Memory in LangGraph</figcaption></figure><p>Each conversation thread maintains its own history.</p><h3>Model Context Protocol (MCP)</h3><p>MCP standardizes communication between AI applications and external tools.</p><p>Architecture:</p><p>Supported transports include:</p><ul><li>stdio</li><li>HTTP</li></ul><p>MCP servers can expose tools such as:</p><ul><li>Weather</li><li>Calculator</li><li>Filesystem</li><li>GitHub</li><li>Databases</li><li>Search APIs</li></ul><p>This removes the need for custom integrations.</p><h3>Retrieval-Augmented Generation (RAG)</h3><p>RAG allows LLMs to answer questions using external knowledge.</p><p>Instead of relying only on training data, the model retrieves relevant information before generating an answer.</p><p>Pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ism2wKjw-4grDYGl4Pfgsw.png" /><figcaption>RAG</figcaption></figure><p>This dramatically improves accuracy</p><h3>Building the Knowledge Base</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_yf3ppETgpVX9AsqUf1UOw.png" /><figcaption>Building the Knowledge Base</figcaption></figure><p>A typical RAG pipeline consists of several stages.</p><h4>Data Ingestion</h4><p>Collect documents from:</p><ul><li>PDFs</li><li>Websites</li><li>SQL</li><li>Excel</li><li>APIs</li><li>Word Documents</li></ul><h4>Parsing</h4><p>Extract meaningful text from raw files.</p><h4>Chunking</h4><p>Large documents are divided into smaller chunks.</p><pre>Document<br>↓</pre><pre>Chunk 1</pre><pre>Chunk 2</pre><pre>Chunk 3</pre><pre>Chunk N</pre><p>Smaller chunks improve retrieval accuracy.</p><h4>Embeddings</h4><p>Each chunk is converted into a numerical vector.</p><pre>Text<br>↓</pre><pre>Embedding Model</pre><pre>↓</pre><pre>Vector</pre><p>Embeddings capture semantic meaning rather than exact wording.</p><h4>Vector Database</h4><p>Embeddings are stored inside specialized databases such as:</p><ul><li>FAISS</li><li>Pinecone</li><li>ChromaDB</li><li>Weaviate</li><li>Milvus</li></ul><h4>Retrieval</h4><p>When the user asks a question:</p><pre>User Query<br>↓</pre><pre>Embedding</pre><pre>↓</pre><pre>Similarity Search</pre><pre>↓</pre><pre>Top-K Chunks</pre><pre>↓</pre><pre>Context</pre><p>Only the most relevant information is retrieved.</p><h4>Augmented Generation</h4><p>The retrieved context is added to the prompt before sending it to the LLM.</p><pre>Retrieved Context</pre><pre>+</pre><pre>User Prompt</pre><pre>↓</pre><pre>LLM</pre><pre>↓</pre><pre>Final Answer</pre><p>This allows the model to answer using private or enterprise knowledge.</p><h3>Traditional Vector RAG vs Vectorless RAG</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eo--HGpUwh6WsPN69KATSg.png" /><figcaption>Traditional Vector RAG vs Vectorless RAG</figcaption></figure><p>Traditional RAG relies on embeddings.</p><pre>Document<br>↓</pre><pre>Chunk</pre><pre>↓</pre><pre>Embedding</pre><pre>↓</pre><pre>Vector Database</pre><pre>↓</pre><pre>Retriever</pre><pre>↓</pre><pre>LLM</pre><p>Vectorless RAG (PageIndex) takes a different approach.</p><p>Instead of embeddings, it builds a hierarchical tree using the document structure.</p><pre>Document<br>↓</pre><pre>Sections</pre><pre>↓</pre><pre>Hierarchy</pre><pre>↓</pre><pre>Tree Navigation</pre><pre>↓</pre><pre>Relevant Section</pre><pre>↓</pre><pre>LLM</pre><h3>Traditional RAG</h3><p>✔ Embeddings</p><p>✔ Vector Database</p><p>✔ Similarity Search</p><p>✔ Semantic Retrieval</p><h3>Vectorless RAG</h3><p>✔ No embeddings</p><p>✔ No vector database</p><p>✔ Tree navigation</p><p>✔ Better for structured documents</p><p>Both approaches have their own strengths depending on the application.</p><h3>Putting Everything Together</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8tAcRvrVhZ1t-0EreyUH8A.png" /><figcaption>Putting Eveything Together</figcaption></figure><p>A modern enterprise AI agent combines all of these technologies into one pipeline.</p><h3>Final Thoughts</h3><p>The future of AI is no longer about prompting a single language model — it is about orchestrating intelligent systems that can <strong>reason, remember, retrieve, and act</strong>.</p><p><strong>LangGraph</strong> provides the orchestration layer for complex workflows, <strong>LangChain</strong> supplies reusable components, <strong>MCP</strong> standardizes communication with external tools, and <strong>RAG</strong> grounds responses in trusted knowledge. Together, these technologies transform LLMs from text generators into capable AI agents that can solve real-world problems.</p><p>Whether you’re building customer support assistants, research copilots, coding agents, document intelligence systems, or enterprise automation platforms, mastering these concepts gives you a strong foundation for developing the next generation of AI applications.</p><p>The era of standalone chatbots is ending. The future belongs to <strong>autonomous, tool-using, memory-enabled, knowledge-grounded AI agents</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bcc456fe2a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reinforcement Learning from Scratch (Part 4): Monte Carlo Methods and Temporal Difference…]]></title>
            <link>https://medium.com/@sujangyawali177/reinforcement-learning-from-scratch-part-4-monte-carlo-methods-and-temporal-difference-5b24243ae844?source=rss-23434c2aa8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b24243ae844</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[monte-carlo-method]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[markov-decision-process]]></category>
            <dc:creator><![CDATA[Sujangyawali]]></dc:creator>
            <pubDate>Sat, 04 Jul 2026 15:36:08 GMT</pubDate>
            <atom:updated>2026-07-04T15:36:08.359Z</atom:updated>
            <content:encoded><![CDATA[<h3>Reinforcement Learning from Scratch (Part 4): Monte Carlo Methods and Temporal Difference Prediction</h3><p>A beginner-friendly guide to Monte Carlo Methods in Reinforcement Learning, covering prediction, control, ε-greedy exploration, importance sampling, and the transition to Temporal Difference Learning.</p><h3>Monte Carlo Methods VS Dynamic Programming</h3><p>Before learning <strong>Monte Carlo (MC) Methods</strong>, let’s recall <strong>Dynamic Programming (DP)</strong>. DP requires a <strong>complete environment model</strong>, including the transition probabilities <strong>P(s′ | s, a)</strong> and reward function <strong>R(s, a, s′)</strong>. However, in most real-world problems, this model is unknown or impossible to obtain.</p><p>Monte Carlo Methods solve this problem by <strong>learning directly from experience</strong>. Instead of using a model, the agent interacts with the environment, observes complete episodes, and estimates values by averaging the returns.</p><p>The return from time step <em>t</em> is:</p><p><strong>Gₜ = Rₜ₊₁ + γRₜ₊₂ + γ²Rₜ₊₃ + ⋯ + γᵀ⁻ᵗ⁻¹Rₜ</strong></p><p>The value of a state is estimated as:</p><p><strong>V(s) ≈ (1 / N(s)) ∑ Gᵢ</strong></p><p>Similarly, the action-value is:</p><p><strong>Q(s, a) ≈ (1 / N(s, a)) ∑ Gᵢ</strong></p><blockquote><strong><em>Key Idea:</em></strong><em> Dynamic Programming learns using a </em><strong><em>known model</em></strong><em>, whereas Monte Carlo learns from </em><strong><em>experience</em></strong><em>. This makes Monte Carlo one of the fundamental </em><strong><em>model-free reinforcement learning</em></strong><em> methods.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gFpiy7xX0d74fCiFbvBQDg.png" /><figcaption>Monte carlo method</figcaption></figure><p>This figure below explains why <strong>Dynamic Programming (DP)</strong> is difficult to apply in many real-world reinforcement learning problems and how <strong>Monte Carlo (MC) methods</strong> overcome this limitation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a8oqZWEBJ350ED24riet-w.png" /><figcaption>Monte carlo</figcaption></figure><p>DP requires a complete <strong>environment model</strong>, including the transition probabilities <strong>P(s′ | s, a)</strong> and reward function <strong>R(s, a, s′)</strong>, to solve the Bellman equation.</p><p>Since these quantities are usually unknown in real-world applications such as self-driving cars, robotics, and recommendation systems, DP cannot be used directly.</p><p>Monte Carlo methods eliminate this requirement by learning <strong>directly from experience</strong>.</p><p>Instead of computing expectations mathematically, the agent interacts with the environment, observes complete episodes, computes the return <strong>Gₜ</strong>, and estimates value functions by averaging sampled returns.</p><p>In short, <strong>Dynamic Programming learns from a known model, whereas Monte Carlo learns from experience</strong>, making it one of the most important <strong>model-free reinforcement learning</strong> methods.</p><h3>Introduction to Monte Carlo Methods</h3><p><strong>Monte Carlo Methods</strong> are one of the fundamental approaches used in Reinforcement Learning to estimate the value of states and learn optimal policies from experience.</p><p>Unlike Dynamic Programming, Monte Carlo methods do <strong>not require a model of the environment</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DuP_xsw0ea7g6VsXub95rg.png" /><figcaption>Monte Carlo Method</figcaption></figure><p>Instead, they learn by running complete episodes, observing the returns from visited states, and updating state values using the average return over multiple episodes.</p><p>As the number of episodes increases, these estimates gradually converge to the true value function.</p><h3><strong>Monte Carlo Prediction</strong></h3><p><strong>Monte Carlo Prediction</strong> estimates the <strong>state-value function Vπ(s)</strong> for a given policy by <strong>learning directly from experience</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JX4ZM96xTQgmGF5HwzbRrA.png" /><figcaption>Monte carlo Prediction</figcaption></figure><p>Instead of using an environment model, the agent generates complete episodes, computes the return <strong>Gₜ</strong> for each visited state, and estimates its value by averaging the observed returns.</p><p>As the number of episodes increases, these estimates gradually converge to the true value function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vgcx6PE_-nkPwle28ULIPg.png" /><figcaption>Monte Carlo Prediction Example</figcaption></figure><h3>Monte Carlo Estimation of Action Values</h3><p>So far, we have learned how Monte Carlo Prediction estimates the <strong>state-value function</strong> <strong>Vπ(s)</strong> by averaging the returns observed from complete episodes. While this tells us how good a state is, it does not tell us <strong>which action should be taken</strong> in that state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tu6ossp_hxRYZVJ3DmJgqA.png" /><figcaption>Monte Carlo Estimation of Action Values</figcaption></figure><p>To solve this problem, Monte Carlo methods extend the same idea to estimate the <strong>action-value function</strong> <strong>Qπ(s, a)</strong>.</p><p>Instead of averaging the returns for each <strong>state</strong>, we now average the returns for each <strong>state-action pair</strong>.</p><p><strong>Q(s, a) ≈ (1 / N(s, a)) Σ Gᵢ</strong></p><p>where:</p><ul><li><strong>N(s, a)</strong> = Number of times the state-action pair <strong>(s, a)</strong> is visited.</li><li><strong>Gᵢ</strong> = Return observed after taking action <strong>a</strong> in state <strong>s</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T43y6byOB9bonwnHUhEpKw.png" /><figcaption>Monte Carlo Estimation of Action Values</figcaption></figure><p>The algorithm remains exactly the same — the only difference is that we estimate values for <strong>(state, action)</strong> pairs instead of just <strong>states</strong>.</p><p>These action-value estimates help the agent compare different actions and form the foundation of <strong>Monte Carlo Control</strong>, where the policy is improved by selecting the action with the highest estimated value.</p><h3>Monte Carlo Control</h3><p>Once we solve the prediction problem, the objective of control problem is to find the policy which is optimal.</p><p>Policy iteration in <em>DP </em><strong><em>uses State value function</em></strong> where as here use <strong><em>action value function.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ayNPk_NTMyzaZ2Z1TT7O3A.png" /><figcaption>Monte-carlo control</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d8ZWGI0xayZB-BK3Tp2iLg.png" /><figcaption>Limitation</figcaption></figure><p>But this raises an important question:</p><ul><li>❓ What if a state is never visited?</li><li>❓ What if a state is visited only a few times?</li><li>❓ How can Monte Carlo estimate the value of rarely explored states?</li></ul><p>These highlight the main limitation of Monte Carlo methods. Since MC learns <strong>only from visited states</strong>, states that are never visited cannot be evaluated, while states visited only a few times produce <strong>high-variance</strong> and unreliable estimates.</p><p>Therefore, <strong>exploration</strong> (e.g., <strong>Exploring Starts</strong> or <strong>ε-greedy policies</strong>) is essential to ensure that all reachable states are visited sufficiently often, allowing accurate value estimation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M_WpEYz6Kw9w-02VfFfQfw.png" /></figure><h3>Monte Carlo Control without Exploring Starts</h3><p>Exploring Starts assumes every state-action pair<br>has a chance of being selected initially.</p><p>Since this assumption is unrealistic,<br>Monte Carlo Control uses ε-greedy exploration instead.</p><h4>Why Do We Need ε-Greedy Exploration?</h4><p>One limitation of Monte Carlo methods is that they can only learn from <strong>visited states</strong>.</p><p>If a state is never visited or is visited very rarely, its estimated value will be poor.</p><p>To overcome this problem, Reinforcement Learning uses <strong>ε-Greedy Exploration</strong>, where the agent usually chooses the best action but occasionally selects a random action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vlq8wvu_TQWGQlW29QlLJA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2by5c23b6qxRXVzARpL7SA.png" /><figcaption>Epsilon Greedy Action selection</figcaption></figure><p>This balances <strong>exploitation</strong> (using current knowledge) and <strong>exploration</strong> (discovering new states), leading to better learning over time.</p><h3>First-Visit Monte Carlo Control (ε-Soft Policy)</h3><p><strong>Goal:</strong> Learn the <strong>optimal action-value function</strong> <strong>Q*(s, a)</strong> and the <strong>optimal policy</strong> <strong>π*(a|s)</strong> <strong>without knowing the environment model</strong>.</p><p>Unlike Dynamic Programming, First-Visit Monte Carlo Control does <strong>not require the transition probabilities</strong> <strong>P(s′|s, a)</strong> <strong>or the reward function</strong> <strong>R(s, a)</strong>. Instead, the agent learns <strong>directly from complete episodes of interaction</strong> with the environment.</p><p>The algorithm begins by initializing the action-value function <strong>Q(s, a)</strong> arbitrarily and maintaining an empty <strong>Returns</strong> list for every state-action pair. An <strong>ε-soft policy</strong> is then used to generate a complete episode, ensuring that every action has a non-zero probability of being selected for sufficient exploration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7zvG_NQeyD_OseC0SKmz_g.png" /><figcaption>Monte Carlo Control First Visit</figcaption></figure><p>After the episode terminates, the return is computed for the <strong>first occurrence</strong> of every visited state-action pair <strong>(s, a)</strong>:</p><p><strong>Gₜ = Rₜ₊₁ + γRₜ₊₂ + γ²Rₜ₊₃ + ⋯ + γⁿRₜ₊ₙ</strong></p><p>The computed return is appended to <strong>Returns(s, a)</strong>, and the action-value estimate is updated as</p><p><strong>Q(s, a) = Average(Returns(s, a))</strong></p><p>Finally, the policy is improved using the updated action-value estimates. For each state, the action with the highest estimated value is selected as the greedy action, while all other actions continue to receive a small exploration probability according to the <strong>ε-soft policy</strong>.</p><p>Let</p><p><strong>a* = arg maxₐ Q(s, a)</strong></p><p>Then the updated policy is</p><p><strong>π(a|s) = 1 − ε + ε/|A(s)|</strong>, if <strong>a = a*</strong></p><p><strong>π(a|s) = ε/|A(s)|</strong>, if <strong>a ≠ a*</strong></p><p>where <strong>|A(s)|</strong> denotes the number of available actions in state <strong>s</strong>.</p><p>By repeatedly generating episodes, updating <strong>Q(s, a)</strong> using <strong>first-visit returns</strong>, and improving the policy after each episode, the algorithm gradually converges toward the <strong>optimal action-value function</strong> <strong>Q*(s, a)</strong> and the corresponding <strong>optimal policy</strong> <strong>π*(a|s)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2B7l-wt1UuzOTcqdhqBZHg.png" /><figcaption>Monte carlo control</figcaption></figure><p>Here’s a concise, professional introduction suitable for a <strong>Medium article</strong>, <strong>blog</strong>, or <strong>lecture notes</strong>.</p><h3>First-Visit Monte Carlo vs Every-Visit Monte Carlo</h3><p>Monte Carlo methods estimate the <strong>state-value function</strong> <strong>V(s)</strong> by learning directly from <strong>complete episodes of experience</strong>, without requiring a model of the environment. Instead of using the Bellman equation or transition probabilities, these methods observe actual returns from interactions with the environment and estimate state values by averaging those returns over many episodes.</p><p>The two most common variants are <strong>First-Visit Monte Carlo</strong> and <strong>Every-Visit Monte Carlo</strong>. Both compute the return <strong>Gₜ</strong> from a visited state until the end of the episode and use these returns to estimate <strong>V(s)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LFKgRL3cuqsUPwtTjHuxLg.png" /><figcaption>First-Vist vs Every Visit</figcaption></figure><p>The key difference lies in <strong>which occurrences of a state are used</strong> during an episode.</p><ul><li><strong>First-Visit Monte Carlo</strong> updates the value of a state using <strong>only its first occurrence</strong> in an episode.</li><li><strong>Every-Visit Monte Carlo</strong> updates the value using <strong>every occurrence</strong> of the state within the same episode.</li></ul><p>Although both algorithms converge to the true value function with a sufficient number of episodes, they differ in how much information they extract from each episode, their computational cost, and their learning speed. The image above illustrates these differences using the same example episode, making it easy to understand how each method processes experience and updates state-value estimates.</p><h3>Incremental Implementation</h3><p>In <strong>First-Visit</strong> and <strong>Every-Visit Monte Carlo</strong>, we estimate the value of a state by averaging all the returns observed so far. But this raises an important question:</p><p><strong>Do we really need to store every return forever?</strong></p><p>Imagine training an agent for thousands of episodes. Storing every return for every state would require a lot of memory.</p><p>Fortunately, the answer is <strong>No</strong>. Whenever a new return is observed, we can simply update the current estimate without storing all previous returns. This approach is called <strong>Incremental Implementation</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2T-dBAXWFG6saYM1qEsRSg.png" /><figcaption>Incremental Implementation</figcaption></figure><p>The update rule is</p><p><strong>V(S) ← V(S) + (1 / N(S)) [ G − V(S) ]</strong></p><p>where <strong>N(S)</strong> is the number of times state <strong>S</strong> has been visited, and <strong>G</strong> is the newly observed return.</p><p>Sometimes, instead of using <strong>1 / N(S)</strong>, we use a fixed learning rate <strong>α</strong>.</p><p><strong>V(S) ← V(S) + α [ G − V(S) ]</strong></p><p>This gives the same intuition as averaging all returns but uses much less memory, making it more practical for large Reinforcement Learning problems.</p><h3>Off-Policy Monte Carlo Methods and Importance Sampling</h3><p>In <strong>On-Policy Monte Carlo</strong>, the same policy is used to <strong>generate episodes</strong> and <strong>learn the value function</strong>. However, in many real-world situations, this assumption is not practical.</p><p>We may already have data collected by one policy, but our objective is to evaluate or improve a <strong>different policy</strong>. This setting is known as <strong>Off-Policy Learning</strong>.</p><p>Off-Policy Monte Carlo methods estimate the value function of a <strong>target policy</strong> <strong>π</strong> using episodes generated by a different <strong>behavior policy</strong> <strong>μ</strong>. Since the collected episodes were not produced by the target policy, their returns cannot be used directly. Instead, they must be adjusted to account for the difference between the two policies.</p><p>This adjustment is performed using <strong>Importance Sampling</strong>, which assigns a weight to each trajectory based on how likely it is under the target policy compared with the behavior policy. The importance sampling ratio is given by</p><p><strong>ρ = π(Aₜ | Sₜ) / μ(Aₜ | Sₜ)</strong></p><p>For an entire trajectory from time <strong>t</strong> until the terminal state <strong>T</strong>, the cumulative importance ratio is</p><p><strong>ρₜ:ₜ₋₁ = ∏ₖ₌ₜᵀ⁻¹ π(Aₖ | Sₖ) / μ(Aₖ | Sₖ)</strong></p><p>If a trajectory is <strong>more likely</strong> under the target policy than the behavior policy (<strong>ρ &gt; 1</strong>), its return receives <strong>more weight</strong>. Conversely, if the trajectory is <strong>less likely</strong> under the target policy (<strong>ρ &lt; 1</strong>), its return is assigned <strong>less weight</strong>. By appropriately weighting returns, Off-Policy Monte Carlo methods can accurately estimate the value function of the target policy even though the data were collected using a different policy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qRgK-T91iEn3iQMlaZbS9Q.png" /><figcaption>off policy and Sampling Ratio</figcaption></figure><p>The following figure introduces the concepts of <strong>behavior policy</strong>, <strong>target policy</strong>, <strong>importance sampling ratio</strong>, <strong>ordinary and weighted importance sampling</strong>, and demonstrates how importance sampling enables Off-Policy Monte Carlo methods to learn from previously collected experience.</p><h3>Off-Policy Monte Carlo Control</h3><p>So far, we have used <strong>importance sampling</strong> to estimate the value of a <strong>target policy</strong> using episodes generated by a different <strong>behavior policy</strong>.</p><p>However, in Reinforcement Learning, our goal is not only to evaluate a policy but also to <strong>improve it</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bUn2tkuG-lYYkQ30Q0TRcg.png" /><figcaption>Off-Policy Monte Carlo Control</figcaption></figure><p>This leads to <strong>Off-Policy Monte Carlo Control</strong>, where the agent learns the <strong>optimal policy</strong> while collecting experience from a different behavior policy.</p><p>The idea is simple:</p><pre>Behavior Policy<br>        ↓<br>Generate Episodes<br>        ↓<br>Importance Sampling<br>        ↓<br>Update Q(s, a)<br>        ↓<br>Improve Target Policy<br>        ↓<br>Repeat</pre><p>The behavior policy continues to explore the environment, while the target policy is gradually improved using the estimated action values. This allows the agent to learn the optimal policy without requiring the data to be generated by that policy itself.</p><h3>Importance Sampling on Truncated Returns</h3><p>So far, we have used <strong>Importance Sampling</strong> to estimate and improve a target policy using episodes generated from a different behavior policy. However, there is one practical problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-SofZu5TmnwMhEQQpUh5IA.png" /><figcaption>Importance Sampling on Truncated Returns</figcaption></figure><p>Sometimes, the importance sampling ratio</p><p><strong>ρ = ∏ π(Aₖ|Sₖ) / b(Aₖ|Sₖ) </strong>can become very large.</p><p>When this happens, a single episode can receive an extremely high weight and dominate the learning process. As a result, the estimates become unstable and have <strong>high variance</strong>.</p><p>A simple solution is to <strong>truncate (clip)</strong> the importance sampling ratio by limiting its maximum value.</p><p>For example,</p><p><strong>ρ = 25 </strong>can be replaced with <strong>ρ = min(25, 5) = 5</strong></p><p>Although truncation introduces a small amount of bias, it greatly reduces variance and makes learning more stable in practice.</p><p>This trade-off between <strong>bias</strong> and <strong>variance</strong> is often worthwhile, especially in large reinforcement learning problems.</p><h3>From Monte Carlo to Temporal Difference Methods</h3><p>In the previous article, we explored <strong>Dynamic Programming (DP)</strong>, where the agent learns by using a complete model of the environment. In this article, we studied <strong>Monte Carlo (MC) Methods</strong>, which remove the need for a model and instead learn directly from experience by waiting until an entire episode is completed before updating the value function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5GpphtvbUk1xLCWPLqPW9g.png" /><figcaption>DP vs MC vs TD</figcaption></figure><p>Although Monte Carlo methods are simple and effective, they have an important limitation: <strong>learning can only begin after the episode ends</strong>. This delay makes learning slow, especially for long or continuing tasks where waiting until the terminal state is impractical.</p><p>This naturally leads to an important question:</p><blockquote><strong><em>Can an agent learn directly from experience without waiting for the end of an episode?</em></strong></blockquote><p>The answer is <strong>Temporal Difference (TD) Learning</strong>.</p><p>Temporal Difference methods combine the strengths of both <strong>Dynamic Programming</strong> and <strong>Monte Carlo Methods</strong>.</p><p>Like Monte Carlo, they are <strong>model-free</strong> and learn from sampled experience.</p><p>Like Dynamic Programming, they <strong>bootstrap</strong> by using the current estimate of the next state’s value to update the current state. As a result, TD methods can update their estimates <strong>after every step</strong>, making learning faster, more efficient, and suitable for online reinforcement learning.</p><p>In the next article, we will develop an intuitive understanding of <strong>Temporal Difference Methods</strong>, beginning with the core idea of <strong>bootstrapping</strong>, and then study the <strong>TD(0)</strong> algorithm in detail before moving on to more advanced control methods such as <strong>SARSA</strong> and <strong>Q-Learning</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*beQ-KSoSKbS29foeOKBO3A.png" /><figcaption>SARSA vs Q-Learning</figcaption></figure><h3>Conclusion</h3><p>In this article, we explored <strong>Monte Carlo Methods</strong>, a model-free approach that learns directly from experience by estimating value functions from complete episodes.</p><p>We studied <strong>Monte Carlo Prediction</strong>, <strong>Monte Carlo Control</strong>, <strong>ε-greedy exploration</strong>, <strong>First-Visit vs Every-Visit Monte Carlo</strong>, and <strong>Off-Policy Monte Carlo using Importance Sampling</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3UYpeeykcwqpVvABTZaK1g.png" /><figcaption>Numerical</figcaption></figure><p>To reinforce these concepts, we also solved a <strong>comprehensive numerical problem</strong> covering discounted returns, state-value and action-value estimation, incremental updates, ε-greedy action selection, importance sampling, and one iteration of Monte Carlo Control. This example demonstrates how the theory is applied step by step in practice.</p><p>Although Monte Carlo methods remove the need for an environment model, they must wait until an episode ends before updating their estimates.</p><p>This limitation naturally leads to <strong>Temporal Difference (TD) Learning</strong>, which combines the strengths of Dynamic Programming and Monte Carlo by learning <strong>online through bootstrapping</strong>.</p><p>In the next article, we will study <strong>Temporal Difference Methods</strong> and the <strong>TD(0)</strong> algorithm in detail.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b24243ae844" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reinforcement Learning from Scratch (Part 2): Understanding Markov Decision Processes (MDPs)]]></title>
            <link>https://medium.com/@sujangyawali177/reinforcement-learning-from-scratch-part-2-understanding-markov-decision-processes-mdps-a79d16983583?source=rss-23434c2aa8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a79d16983583</guid>
            <category><![CDATA[markov-decision-process]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sujangyawali]]></dc:creator>
            <pubDate>Thu, 02 Jul 2026 16:02:08 GMT</pubDate>
            <atom:updated>2026-07-02T16:02:08.566Z</atom:updated>
            <content:encoded><![CDATA[<p>In the previous article, we studied Multi-Armed Bandits, where an agent repeatedly chose among several actions with unknown rewards. While bandits introduced exploration and exploitation, they ignored one crucial aspect of real-world problems: state. In this article, we’ll extend those ideas by introducing Markov Decision Processes (MDPs), where every action changes the environment and influences future decisions.</p><h3>Introduction</h3><p>In the previous article, we studied <strong>Multi-Armed Bandits</strong>, where an agent learns to balance exploration and exploitation without considering the concept of state. However, most real-world decision-making problems involve environments that change over time, making the current state an essential part of the learning process.</p><p>To model such sequential decision-making problems, Reinforcement Learning uses <strong>Markov Decision Processes (MDPs)</strong>. In this article, we will build a step-by-step understanding of MDPs by covering four fundamental concepts:</p><ol><li><strong>Agent–Environment Interaction</strong></li><li><strong>Rewards and Expected Returns</strong></li><li><strong>Markov Property</strong></li><li><strong>Markov Decision Processes (MDPs)</strong></li></ol><p>These concepts form the mathematical foundation of Reinforcement Learning and prepare us for advanced algorithms such as Dynamic Programming, Monte Carlo Methods, and Temporal Difference Learning.</p><h3><strong>1. Agent–Environment Interaction</strong></h3><p>It explains how an <strong>agent</strong> interacts with an <strong>environment</strong> by observing the current state, selecting an action, receiving a reward, and transitioning to the next state.</p><p>The lecture also introduces key concepts such as <strong>states, actions, rewards, next states, and policies</strong>, illustrates them with practical examples, and highlights the differences between <strong>Multi-Armed Bandits</strong> and <strong>Markov Decision Processes (MDPs)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yLgEL5xnEWCyu8fMmXQ8Jw.png" /><figcaption>MDPs</figcaption></figure><p>Understanding these concepts provides the basis for advanced RL algorithms such as <strong>Q-Learning, SARSA, Deep Q-Networks (DQN), Policy Gradient, Actor-Critic, and PPO</strong>.</p><h4><strong>Agent–Environment Interaction Loop</strong></h4><p>The <strong>Agent–Environment Interaction</strong> is the fundamental interaction loop in Reinforcement Learning (RL).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1FWtAyLWvOmhXgwUsklUrw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KTDtOigIMbMwuUuG2QAOrQ.png" /><figcaption>Agent–Environment Interaction</figcaption></figure><p>At the beginning of each time step <strong>t</strong>, the environment provides the agent with the <strong>current state (Sₜ)</strong> and the <strong>current reward (Rₜ)</strong>, where <strong>Rₜ</strong> is the reward generated by the previous action <strong>Aₜ₋₁</strong>.</p><p>Based on the current state, the agent selects an action <strong>Aₜ</strong>. The environment then executes this action, transitions to the <strong>next state (Sₜ₊₁)</strong>, and generates the <strong>next reward (Rₜ₊₁)</strong>.</p><p>At the next time step (<strong>t + 1</strong>), the pair <strong>(Sₜ₊₁, Rₜ₊₁)</strong> becomes available to the agent. By repeating this interaction, the agent learns to select actions that maximize the cumulative reward over time.</p><h4>Practical Examples of Markov Decision Processes (MDPs)</h4><p>To better understand how <strong>Markov Decision Processes (MDPs)</strong> work in real-world applications, consider two practical examples: a <strong>Bioreactor</strong> and a <strong>Pick-and-Place Robot</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nOrU33xwilditFLoEfV_ZA.png" /><figcaption>Example in MDPs</figcaption></figure><p>In both cases, the agent observes the current <strong>state</strong>, selects an <strong>action</strong>, receives a <strong>reward</strong>, and transitions to the <strong>next state</strong>.</p><p>By repeating this interaction, the agent learns a policy that maximizes the long-term cumulative reward while performing the task efficiently.</p><h3><strong>2. Rewards and Expected Returns</strong></h3><p>One of the most fundamental ideas in Reinforcement Learning (RL) is that an agent learns by interacting with its environment and receiving <strong>rewards</strong> as feedback for its actions.</p><p>Rather than being explicitly programmed with the correct behavior, the agent discovers an optimal strategy by maximizing the rewards it receives over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*75hMOS98ZTkoCeXS4rn5IA.png" /><figcaption>Goal and Rewards in Reinforcement Learning</figcaption></figure><p>In this section, we will understand how <strong>reward signals</strong> define the goals of an RL problem, why agents aim to maximize the <strong>expected return</strong> instead of immediate rewards, and how the concept of <strong>discounting</strong> enables agents to balance short-term and long-term benefits.</p><p>We will also explore the difference between <strong>episodic</strong> and <strong>continuing</strong> tasks, understand the role of the <strong>discount factor (γ)</strong>, and learn how the <strong>discounted return</strong> forms the objective function that every reinforcement learning algorithm seeks to optimize.</p><h4>Example: Pole-Balancing Task as Episodic and Continuing Reinforcement Learning</h4><p>The <strong>pole-balancing (Cart-Pole) task</strong> is a classic example used in reinforcement learning to illustrate the difference between <strong>episodic</strong> and <strong>continuing</strong> tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L3lR4HWIB_dyDUfa7XGc4A.png" /><figcaption>Episodic vs Continuing Task</figcaption></figure><p>In this problem, the agent applies forces to a moving cart to keep the pole balanced for as long as possible.</p><p>The same task can be formulated in two ways: as an episodic task, where the episode ends when the pole falls, or as a continuing task, where the interaction is treated as ongoing using discounted future rewards.</p><p>Both formulations encourage the agent to maximize the duration for which the pole remains balanced, demonstrating how different reward designs can lead to the same learning objective.</p><h3>3. Markov Property</h3><p><strong>Markov Property</strong> says: <strong><em>The future should depend only on the current state, not on the entire history.</em></strong></p><p>A state is <strong>Markov</strong> if it contains <strong>all the information required to predict what happens next</strong>.</p><p>The examples in the slide show that whether a state is Markov depends entirely on <strong>what information is included in the state</strong>.</p><h4>Understanding the Markov Property: Why the Present Is Enough</h4><p>The <strong>Markov Property</strong> states that a state should contain all the relevant information needed to predict the future. If the current state completely summarizes the past, the agent does not need to remember the entire history to make the next decision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RIYWS246VkwpzX-BYNC8zA.png" /><figcaption>Markov Property</figcaption></figure><p>The slide explains this idea using three intuitive examples. In a <strong>conversation</strong>, the current context captures what has already been discussed.</p><p>In <strong>chess</strong>, the current board configuration is sufficient to determine the next legal moves.</p><p>Similarly, a <strong>cannonball’s</strong> current position and velocity are enough to predict its future motion. In each case, the present state retains all the information that matters.</p><h4>Understanding the Mathematical Notation of the Markov Property</h4><p>This slide explains the mathematical notation used to describe <strong>environment dynamics</strong> in Reinforcement Learning. It compares a <strong>general environment</strong>, where the next state and reward may depend on the entire interaction history, with a <strong>Markov environment</strong>, where they depend only on the <strong>current state and action</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BkORVa4dlM2QUtgwDIvYUQ.png" /><figcaption>Math Notation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WnLdrvkdZRZBRCV0_DJPWA.png" /><figcaption>Markov property</figcaption></figure><p>Without the Markov Property,</p><p><strong>P(Rₜ₊₁ = r, Sₜ₊₁ = s′ | S₀, A₀, R₁, …, Sₜ, Aₜ)</strong></p><p>the environment may require the complete history of states, actions, and rewards to predict the future.</p><p>With the Markov Property,</p><p><strong>P(Rₜ₊₁ = r, Sₜ₊₁ = s′ | Sₜ, Aₜ),</strong></p><p>the current state already summarizes all relevant past information, making the history unnecessary.</p><p>The slide also explains the notation: <strong>Sₜ</strong> represents the current state, <strong>Aₜ</strong> the current action, <strong>Rₜ₊₁</strong> the next reward, and <strong>Sₜ₊₁</strong> the next-state random variable.</p><p>In contrast, <strong>r</strong> and <strong>s′</strong> denote specific values of the reward and next state.</p><p>This simplification is the foundation of <strong>Markov Decision Processes (MDPs)</strong> and makes Reinforcement Learning algorithms computationally efficient.</p><h4>Practical Examples of the Markov Property</h4><p>This slide illustrates the <strong>Markov Property</strong> using two intuitive examples. In the <strong>pole-balancing task</strong>, the state consisting of the cart’s position, velocity, pole angle, and angular velocity satisfies the Markov Property in an ideal setting because it contains all the information needed to predict the future. However, in a real system, hidden factors such as pole bending, temperature, and wheel friction also affect the dynamics, making the state <strong>non-Markov</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qcCPsJcDwUUZiSwJLznNtg.png" /><figcaption>Example of Markov property</figcaption></figure><p>The second example is <strong>Draw Poker</strong>, where making the best decision may require remembering opponents’ betting patterns, previous actions, and bluffing behavior. Since players cannot retain all this information, the state they use is incomplete and therefore <strong>non-Markov</strong>. The key takeaway is that a state is <strong>Markov only if it includes all relevant information needed to predict the future</strong>.</p><h3>4. Markov Decision Processes</h3><p>A <strong>Markov Decision Process (MDP)</strong> is the mathematical framework used to model most Reinforcement Learning problems. It describes how an <strong>agent</strong> interacts with an <strong>environment</strong> by observing the current <strong>state</strong>, taking an <strong>action</strong>, receiving a <strong>reward</strong>, and transitioning to the <strong>next state</strong>.</p><p>An MDP assumes that the environment satisfies the <strong>Markov Property</strong>, meaning the future depends only on the <strong>current state and action</strong>, not on the entire history.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sNB4UhkZe3RPTfPpsXkIPw.png" /><figcaption>MDP</figcaption></figure><p>This slide introduces the core components of an MDP through a practical <strong>recycling robot</strong> example, illustrating states, actions, transition probabilities, rewards, and environment dynamics.</p><h4>Introduction to the Transition Diagram</h4><p>This transition diagram provides an intuitive view of how an MDP models the <strong>Search</strong> action for a recycling robot.</p><p>It shows how the robot’s <strong>current battery state</strong> and the <strong>chosen action</strong> determine the possible <strong>next states</strong>, along with their <strong>transition probabilities</strong> and <strong>rewards</strong>.</p><p>By following the arrows, we can understand how uncertainty is represented in an MDP and how the environment responds probabilistically to the agent’s actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kwC08wtwkDC8a9NpEvc5WA.png" /><figcaption>Transition Diagram</figcaption></figure><h4>Introduction to the Recycling Robot Transition Graph</h4><p>This transition graph visualizes the <strong>Markov Decision Process (MDP)</strong> for the recycling robot.</p><p>Each node represents a <strong>battery state</strong>, while each directed edge represents an <strong>action</strong> and its possible outcome.</p><p>The labels on the edges indicate the <strong>transition probability</strong> and the <strong>reward</strong> received after taking that action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ndoqg_kCIrMx1Xl8cj5rWQ.png" /></figure><p>This graph provides a clear picture of how the robot moves between states under uncertainty and helps us understand the environment’s dynamics in an MDP.</p><h3>Conclusion</h3><p>In this article, we introduced the fundamentals of <strong>Markov Decision Processes (MDPs)</strong>, including the Agent–Environment Interaction, rewards, expected returns, the Markov Property, and the core components of an MDP. These concepts provide the mathematical foundation for Reinforcement Learning.</p><p>In the next article, we will learn about <strong>Policies and Value Functions</strong>, which help an agent evaluate states and choose the best actions. These ideas form the basis of almost every Reinforcement Learning algorithm and bring us one step closer to understanding how RL is used in modern Large Language Models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a79d16983583" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>