<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bessie Delight Kekeli on Medium]]></title>
        <description><![CDATA[Stories by Bessie Delight Kekeli on Medium]]></description>
        <link>https://medium.com/@bessiedelight?source=rss-4a5744f8548f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vgw1zUTZNVcKBszAQqoDVw.png</url>
            <title>Stories by Bessie Delight Kekeli on Medium</title>
            <link>https://medium.com/@bessiedelight?source=rss-4a5744f8548f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:03:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bessiedelight/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Tokens | Context window | Caching| everything you got wrong…]]></title>
            <link>https://pub.towardsai.net/tokens-context-window-caching-everything-you-got-wrong-10b5a32f7456?source=rss-4a5744f8548f------2</link>
            <guid isPermaLink="false">https://medium.com/p/10b5a32f7456</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[token]]></category>
            <dc:creator><![CDATA[Bessie Delight Kekeli]]></dc:creator>
            <pubDate>Mon, 28 Sep 2026 14:31:01 GMT</pubDate>
            <atom:updated>2026-09-29T00:11:20.382Z</atom:updated>
            <content:encoded><![CDATA[<h4>The hidden mechanics behind token costs, context growth, latency, and prompt caching in production AI agents.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lbA6vRBwisdj2evcG9Za-A.png" /></figure><p>Most developers discover the problem in production. The agent that ran fine during testing suddenly costs three times what they budgeted. Response times creep up as conversations go on. Something that worked on a short demo fails when a user asks four questions in a row. And the logs show nothing useful, because nothing actually crashed.</p><p>The cause is almost always the same three things: a misunderstanding of what a token is, a misunderstanding of what the context window does, and zero thought given to caching. None of these are exotic failures. They are the default outcome when you build without understanding the machine underneath.</p><p>This article goes after all three, in order, from the surface down.</p><h3>Part One: Tokens</h3><h3>What people think a token is</h3><p>Most developers hear “token” and picture a word. One word, one token. That mental model is wrong enough to cause real problems.</p><p>A token is a fragment of text, usually a few characters, that a model’s tokenizer carved out of your input during the training process. The carving is not based on grammar or meaning. It is statistical. Common sequences of characters that appear together often in the training corpus get assigned their own token. Rarer sequences get split up.</p><p>So “the” is one token. “running” is one token. But “idempotency” might be three tokens: “idem,” “pot,” “ency.” Your API key? Every UUID looks different, so each character group has to be encoded individually. A single UUID can cost twelve to fourteen tokens. A list of ten UUIDs costs more than a short paragraph.</p><p>Here is the practical consequence. Nobody writes agents that process only clean English prose. Agents process JSON. They process code. They process database records with IDs and timestamps and field names. All of that costs more per character than English prose does — sometimes dramatically more.</p><p>The common advice is “approximately four characters per token.” That is an average over English prose. It does not hold for anything your agent actually works with. Run this through a tokenizer and count yourself:</p><pre>from anthropic import Anthropic<br>client = Anthropic()<br><br>result = client.messages.count_tokens(<br>    model=&quot;claude-opus-5&quot;,<br>    system=your_system_prompt,<br>    tools=your_tool_definitions,<br>    messages=your_conversation_history,<br>)<br>print(f&quot;Actual token count: {result.input_tokens}&quot;)</pre><p>That number will be larger than you expect. It always is.</p><h3>The tool definition problem</h3><p>Here is the piece that catches almost everyone. Your tool definitions ,the JSON schemas that describe what tools your agent can call, are sent to the model on every single request. Not once. Every request.</p><p>A dozen well-documented tools can run to four or five thousand tokens. You pay for those tokens on turn one, turn two, turn fifteen, turn forty. For the entire life of every conversation. If your agent handles ten thousand conversations a month and each conversation averages eight turns, you are paying for those tool schemas eighty thousand times.</p><p>Before you add your eighth tool with its detailed parameter descriptions, count what you are already sending:</p><pre>result = client.messages.count_tokens(<br>    model=&quot;claude-opus-5&quot;,<br>    tools=your_tool_definitions,<br>    messages=[],  # empty — just the tools<br>    system=&quot;&quot;,<br>)<br>print(f&quot;Tool definitions alone: {result.input_tokens} tokens per request, forever&quot;)</pre><p>That number does not go away. It just multiplies.</p><h3>Why output is so much more expensive than input</h3><p>Look at any provider’s pricing page and output tokens cost roughly five times what input tokens cost. This is not a pricing choice. It reflects what the hardware is doing.</p><p>When your prompt goes in the prefill phase, the GPU processes the entire sequence in one pass. All forty thousand tokens at once, in parallel. It is fast because parallelism is what GPUs are built for.</p><p>When the model generates a response, the decode phase produces one token at a time. Each token requires a full forward pass through the model. Each pass depends on the token before it, so nothing can be parallelized. The process is memory-bandwidth-bound, which is the most expensive constraint modern hardware faces.</p><pre>PREFILL                                DECODE<br>one pass over the entire prompt        one pass per output token<br><br>┌──────────────────────────────┐       ┌─┐ ┌─┐ ┌─┐ ┌─┐ ┌─┐ ┌─┐<br>│ ████████████████████████████ │  ───▶ │█│▶│█│▶│█│▶│█│▶│█│▶│█│ ...<br>└──────────────────────────────┘       └─┘ └─┘ └─┘ └─┘ └─┘ └─┘<br>        40,000 tokens                   sequential · unavoidable<br>        parallel · fast                  ~1 pass each</pre><p>Two things follow from this directly.</p><p>Latency is controlled by output length, not input length. An agent that feels slow is almost always generating too much text. Doubling the system prompt adds maybe a hundred milliseconds. Doubling the response length doubles the wait. If you want your agent to feel faster, the answer is almost never a shorter prompt. It is a shorter answer.</p><p>A verbose response costs twice. You pay output rates when the agent generates it. Then it goes into the conversation history, and you pay input rates again on every subsequent turn. A chatty response early in a long conversation gets billed a dozen times by the time the conversation ends.</p><h3>Part Two: The Context Window</h3><h3>The misconception that costs real money</h3><p>The context window looks like memory. You put the conversation history in. The agent refers back to earlier messages. Everything appears to have continuity.</p><p>It is not memory. The model stores nothing between calls. Every time you call the API, you reconstruct the entire history and send it again. The model reads it fresh. What looks like memory is you paying to reconstruct it on every turn.</p><p>This reframes the economics entirely. A ten-turn conversation is not ten requests. Each request includes every prior turn plus the new message plus every tool result plus the system prompt plus the tool definitions. The tenth request is not ten times more expensive than the first — it is closer to fifty times more expensive, because the total tokens sent grows with every turn.</p><pre>Turn 1 → sends 6,000 tokens<br>Turn 5 → sends 22,000 tokens<br>Turn 20 → sends 80,000 tokens<br>Turn 40 → sends 160,000 tokens (approaching window limits)</pre><p>Total spend across a forty-turn conversation does not grow linearly. It grows quadratically. This is the curve that breaks agent economics in production.</p><h3>What is actually inside the window</h3><p>Break down a real request from the middle of a conversation and you get something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ute7tNTOTlsclj5fSP4mwg.png" /></figure><p>The user typed 40 tokens. You are paying for roughly 46,000. The user’s actual question is under a tenth of a percent of the bill.</p><p>Everything else is architecture you chose. And since you chose it, you can measure it, change it, and cut it.</p><h3>Three distinct ways context kills agents</h3><p>They look similar from the outside but have different causes and require different fixes.</p><p>The first is overflow. The window fills up, the request fails or the provider silently compacts the history, and you lose something the agent needed. This is the failure everyone prepares for and the least dangerous of the three, because it announces itself. You either get an error or an obviously confused response. Fixable.</p><p>The second is dilution. Long contexts degrade attention unevenly. Information at the start and end of a long prompt is recalled reliably. Information in the middle is not. The effect grows with context length. A fact the agent quoted correctly at 8,000 tokens can be ignored at 80,000 — no error, no warning, a confident answer that is simply wrong. This is why “the model supports a million tokens” does not mean “you can put a million tokens in it and expect consistent behavior.” Capacity is not attention.</p><p>The third is poisoning. The agent writes something incorrect into its own history — a hallucinated value, a misread tool result, an incorrect intermediate conclusion. That error is now in the context for every subsequent turn, and the agent treats its own prior output as established fact. Poisoning compounds. An agent twenty turns into a poisoned conversation is confidently reasoning from a premise that was never true, and the transcript reads as perfectly coherent.</p><p>Overflow announces itself. Dilution is silent. Poisoning is a feedback loop that gets worse the longer it runs.</p><h3>Practical responses</h3><p>For long conversations, implement a rolling summary. When the conversation history exceeds a threshold — say, 20,000 tokens — summarize everything older than the last three turns into a single paragraph and replace the raw turns with it. This flattens the growth curve dramatically.</p><pre>def maybe_compress_history(messages: list, threshold: int = 20_000) -&gt; list:<br>    from anthropic import Anthropic<br>    client = Anthropic()<br><br>result = client.messages.count_tokens(<br>        model=&quot;claude-opus-5&quot;,<br>        messages=messages,<br>        system=&quot;&quot;,<br>    )<br>    if result.input_tokens &lt; threshold:<br>        return messages<br>    # Summarize all but the last three turns<br>    to_summarize = messages[:-6]   # -6 because each turn is 2 messages (user + assistant)<br>    recent = messages[-6:]<br>    summary_response = client.messages.create(<br>        model=&quot;claude-opus-5&quot;,<br>        max_tokens=512,<br>        messages=[<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: (<br>                &quot;Summarize the following conversation history in 3-5 sentences, &quot;<br>                &quot;preserving any specific facts, decisions, or values mentioned:\n\n&quot;<br>                + &quot;\n&quot;.join(m[&quot;content&quot;] for m in to_summarize if isinstance(m[&quot;content&quot;], str))<br>            )}<br>        ],<br>        system=&quot;You summarize conversation history accurately and briefly.&quot;,<br>    )<br>    summary_message = {<br>        &quot;role&quot;: &quot;assistant&quot;,<br>        &quot;content&quot;: f&quot;[Previous conversation summary]: {summary_response.content[0].text}&quot;<br>    }<br>    return [summary_message] + recent</pre><p>For tool output, trim aggressively. A database query that returns a hundred records when the agent only needs the first three is wasting 97 records worth of tokens on every subsequent turn. Return exactly what the agent needs and nothing more.</p><p>Keep verified facts in a typed object, not in the conversation transcript. If the agent confirmed an order ID, store it in state. Retrieve it directly when needed. Do not let the agent reconstruct it from what it said three turns ago.</p><h3>Part Three: Caching</h3><h3>What caching actually is</h3><p>Every provider that offers prompt caching is offering the same underlying mechanism: if the start of your prompt matches the start of a prompt you sent recently, they skip processing the matching portion and charge you a fraction of the normal rate for it.</p><p>The savings are real. Anthropic’s cache reads cost roughly 10% of normal input token rates. Cache writes cost about 125% of normal rates. Break-even on a short-lived cache entry is two requests.</p><p>The mechanism is a prefix match on exact bytes. Not semantic similarity. Not “close enough.” Exact bytes.</p><p>This one sentence explains why most caching setups fail silently.</p><h3>The render order you cannot change</h3><p>When you send a request to Anthropic, the content is rendered in a fixed order regardless of how you structured your API call:</p><pre>[tool definitions] → [system prompt] → [conversation history]</pre><p>Tool definitions go first, always. They sit at position zero in the token sequence. This has one consequence that most developers discover by accident: if your tool list changes between requests, the cache is invalidated for the entire request. System prompt, all of history, everything.</p><p>A tool list that changes per user. A tool list built dynamically from user permissions. A tool list where one tool’s schema gets regenerated on each request with a slightly different property order. Any of these mean you pay full price on every request, for every user, every turn, and nothing in the API response will tell you it happened.</p><h3>The invisible invalidators</h3><p>The silent failures are more common than the obvious ones.</p><p>A timestamp in the system prompt. If your system prompt contains datetime.now() to tell the agent the current time, the cache entry is different every minute. You never get a cache hit.</p><p>A UUID or request ID near the front. Same problem. Every request is unique by construction.</p><p>Unsorted JSON serialization. json.dumps(schema) without sort_keys=True produces key ordering that depends on dictionary insertion order, which Python does not guarantee across calls. Two calls with the same schema can produce different byte sequences and therefore miss the cache.</p><p>A user’s name or ID interpolated into the system prompt. Every user has a different prompt, so no cache is shared across users.</p><pre># This misses the cache for every request<br>system = f&quot;You are a helpful assistant. Today is {datetime.now()}. User: {user.name}.&quot;<br><br># This caches correctly<br>system = &quot;You are a helpful assistant.&quot;<br># Pass dynamic values in the first user message instead, after the cached prefix</pre><h3>Placing the cache marker</h3><p>You mark where the stable prefix ends using cache_control:</p><pre>from anthropic import Anthropic<br>client = Anthropic()<br><br>response = client.messages.create(<br>    model=&quot;claude-opus-5&quot;,<br>    max_tokens=1024,<br>    tools=TOOL_DEFINITIONS,          # stable - same for all users, sorted schemas<br>    system=[<br>        {<br>            &quot;type&quot;: &quot;text&quot;,<br>            &quot;text&quot;: SYSTEM_PROMPT,   # stable - no interpolation, no timestamps<br>            &quot;cache_control&quot;: {&quot;type&quot;: &quot;ephemeral&quot;},<br>        }<br>    ],<br>    messages=conversation_history,   # volatile - changes every turn<br>)</pre><p>That single marker on the last block of the system prompt caches the tool definitions and system prompt together. Everything after it — the conversation history — is volatile and does not need to be cached, because it changes on every turn anyway.</p><p>There is a minimum prefix length before caching kicks in. On Claude models it ranges from 512 to 4,096 tokens depending on the specific model. A system prompt shorter than the minimum simply does not cache, and nothing in the response will tell you. If you want to confirm caching is working:</p><pre>print(f&quot;Cache writes: {response.usage.cache_creation_input_tokens}&quot;)<br>print(f&quot;Cache reads:  {response.usage.cache_read_input_tokens}&quot;)<br>print(f&quot;Normal input: {response.usage.input_tokens}&quot;)</pre><p>If cache_read_input_tokens is zero across repeated requests that should share a prefix, an invalidator is running. Render two consecutive requests&#39; prompts as strings, diff them, and look for anything that changed. The cause is almost always in the list above.</p><h3>The agent-specific traps</h3><p>Two failure modes hit agents specifically and not ordinary chat applications.</p><p>Cache entries are found by searching backward through a bounded window of content blocks, roughly twenty blocks. An agent turn that fires six parallel tool calls appends twelve content blocks. Two turns like that and the cache entry from the start of the conversation is out of the lookback window. The entry exists — the match just cannot find it. Long tool-heavy turns need an intermediate cache breakpoint every ten or twelve blocks.</p><p>The second trap is fan-out. When an orchestrator fires five parallel requests with the same system prompt, none of them can read a cache entry that another is still writing. The entry only becomes readable once the first response starts streaming. Fire all five at once and all five pay full price. Fire one, wait for its first streaming token, then release the rest. The cache is warm by the time they arrive.</p><h3>What this adds up to</h3><p>None of these are obscure edge cases. They are the default behavior of every agent that was built without accounting for them.</p><p>The tool definitions sitting at position zero in every request, costing the same tokens every turn of every conversation across every user. The context growing quadratically as the conversation goes on, causing slowdowns that have nothing to do with the model’s capabilities. The cache that was supposed to cut costs but is silently invalidated by a timestamp in the system prompt or unsorted JSON keys or dynamic tool schemas.</p><p>These compound. An agent with verbose responses, no history compression, and a broken cache does not cost two or three times what it should. It costs an order of magnitude more, gets slower as conversations lengthen, and produces outputs that quietly degrade in accuracy as the context fills up. All without throwing a single error.</p><p>The fix is not exotic. Count your tokens before you assume. Measure the context at turn ten and at turn forty. Put datetime.now() in the user turn, not the system prompt. Sort your JSON serialization. Check cache_read_input_tokens and verify the cache is actually working.</p><p>Most of the failures described here take an afternoon to fix. Finding them takes longer only because nobody knows to look.</p><p><em>This is the first article in a series on building AI agents that work in production.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10b5a32f7456" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/tokens-context-window-caching-everything-you-got-wrong-10b5a32f7456">Tokens | Context window | Caching| everything you got wrong…</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Graph Engineering]]></title>
            <link>https://pub.towardsai.net/graph-engineering-e49337901a42?source=rss-4a5744f8548f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e49337901a42</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[graph-engineering]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Bessie Delight Kekeli]]></dc:creator>
            <pubDate>Mon, 07 Sep 2026 19:31:00 GMT</pubDate>
            <atom:updated>2026-09-07T19:31:00.935Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>We spent 2024 getting agents to do things. Now we need them to do the right things, in the right order, without going off the rails.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9qELVAysBqtZ0mqBpEUPgg.png" /></figure><h3>Let’s Start With What You Already Know</h3><p>You have probably built an agent. Maybe it calls a search tool, reasons over the results, formats a response, done. One loop. Works fine. You feel good about it.</p><p>Then a coworker drops a harder problem on your desk. Not a single-step retrieval task. Something with moving parts, multiple specialised checks, parallel processing, a human who needs to approve before anything irreversible happens. You try to cram it all into one agent loop. The system starts behaving strangely. It skips steps. It forgets what it was doing three turns back. You add more instructions to the prompt. Things get worse.</p><p>This is the moment graph engineering becomes relevant.</p><h3>What Graph Engineering Actually Is</h3><p>Graph engineering is the practice of designing the <em>structure</em> an AI agent runs inside, not the agent itself.</p><p>Think of it this way. An agent is like an employee: it has skills, memory, tools, and judgment. Graph engineering is the org chart that defines who that employee talks to, when they talk to them, what information gets passed along, and under what conditions work moves forward versus gets sent back. You are not just hiring smart people. You are designing how the organisation works.</p><p>The nodes are the workers. The edges are the handoffs. The state is the memory shared between them.</p><h3>Three Building Blocks You Need to Know</h3><p>Before the patterns, three terms. Get these right and the rest of the article clicks.</p><h3>The Harness</h3><p>Every agent sits inside what practitioners call a harness. It is the wrapper around the model itself , the memory it has access to, the tools it can call, and the guardrails that prevent it from doing something catastrophic. The harness is not the agent’s intelligence. It is the agent’s operating environment.</p><p>A well-built harness gives the agent exactly what it needs for its specific job and nothing extra. A poorly built harness dumps everything into the context window and hopes for the best.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xVr86K2iHit4EP-gfESZIA.png" /></figure><h3>The Loop</h3><p>A loop is the iterative cycle inside a single agent. The agent reads a problem, decides whether to call a tool or produce an answer, acts, observes the result, and goes again. This is the ReAct pattern. It is powerful for focused tasks.</p><p>The key word is <em>focused</em>. One agent. One goal. Loop until done.</p><p>Loops break down when the problem has distinct phases with different context requirements. Asking one agent to review code for security issues <em>and</em> performance issues <em>and</em> API design <em>and</em> test coverage in one loop is like asking one person to simultaneously be your security auditor, your performance engineer, your API designer, and your QA lead. They cannot fully inhabit all four mental models at once.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CjZpqKFMmJqJ-ubaBMXs-w.png" /></figure><h3>The Graph</h3><p>A graph is a structured workflow that connects multiple nodes — each node being an agent, a function, or a processing step — with edges that define how work moves between them.</p><p>The simplest graph is one node with an edge looping back to itself. That is just a loop with extra vocabulary. The interesting graphs are the ones with branching, parallelism, synchronisation points, and conditional routing. Those are the patterns below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bFWznGcHihAiFd-duvPM_w.png" /></figure><h3>The Four Control Flow Patterns</h3><p>These four patterns cover the overwhelming majority of what real production systems do. Everything complex you will ever build is some combination of these.</p><h3>Pattern 1: Sequential Chain (Prompt Chaining)</h3><p>The simplest structure. Node A completes, passes its output to Node B, which completes and passes to Node C. Each step processes the output of the previous one.</p><pre>[Node A] → [Node B] → [Node C] → Output</pre><p>When do you use this? When a task genuinely benefits from decomposition into sequential steps. A code review pipeline might: first parse the diff, then check it for security issues, then format the findings into a report. Each step has a different job and benefits from receiving a focused, clean input rather than raw everything.</p><p>The critical point here: the graph is not making the steps smarter. It is preventing them from interfering with each other.</p><p>A small code example shows the shape clearly:</p><pre># Each node does one focused job and returns a state update<br>def parse_diff(state: State) -&gt; dict:<br>    parsed = extract_changed_files(state[&quot;raw_diff&quot;])<br>    return {&quot;parsed_diff&quot;: parsed}<br>def check_security(state: State) -&gt; dict:<br>    findings = security_llm.invoke(state[&quot;parsed_diff&quot;])<br>    return {&quot;security_findings&quot;: findings}<br>def format_report(state: State) -&gt; dict:<br>    report = format_llm.invoke(state[&quot;security_findings&quot;])<br>    return {&quot;final_report&quot;: report}</pre><p>Each function receives the full state and returns only the fields it changed. The graph wires them together. No node needs to know what the others are doing.</p><h3>Pattern 2: Fan-Out and Join (Parallelisation)</h3><p>This is where graphs justify their existence.</p><p>Fan-out takes one task and splits it into multiple parallel sub-tasks that run simultaneously. Join is the synchronisation node that waits for all of them to finish, then merges the results.</p><pre>┌→ [Security Agent] ─┐<br>              ├→ [Perf Agent]    ──┤<br>Input → Router┤→ [API Agent]     ──┼→ [Join] → Output<br>              ├→ [Test Agent]    ──┤<br>              └→ [Quality Agent] ─┘</pre><p>The real-world example from the video is a pull request review system. You could send the same PR diff through a security reviewer, a performance reviewer, and a test coverage reviewer sequentially. Or you fan out to all three simultaneously, wait for all three to finish, and merge the findings. Total time is roughly equal to the slowest single reviewer, not the sum of all three.</p><p>The fan-out pattern in code uses the Send API to dispatch parallel branches:</p><pre>from langgraph.constants import Send<br>def fan_out_to_reviewers(state: State) -&gt; list[Send]:<br>    # Dispatch all reviewers simultaneously<br>    # Each runs independently, returns findings to the same state field<br>    return [<br>        Send(&quot;security_agent&quot;, state),<br>        Send(&quot;performance_agent&quot;, state),<br>        Send(&quot;api_agent&quot;, state),<br>        Send(&quot;test_agent&quot;, state),<br>    ]</pre><p>The join happens automatically when you use an accumulator on the findings field:</p><pre>from typing import Annotated<br>import operator<br>class State(TypedDict):<br>    # operator.add concatenates lists from parallel branches safely<br>    # Without this, the last agent to finish would overwrite all the others<br>    findings: Annotated[list, operator.add]</pre><p>That single operator.add annotation is doing serious work. When four agents all return findings simultaneously, LangGraph collects every write and concatenates them instead of letting the last writer win. This is not a coincidence. It is a deliberate design decision you make when you define the state schema.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mwjHO0FYLLeyYIs1gwRalw.png" /></figure><h3>Pattern 3: Router</h3><p>A router reads the current state and sends work to a different node depending on what it finds.</p><pre>┌→ [Fixer Agent] → back to start<br>Input → [Reviewer] ─┤<br>                    └→ [Human Review Gate] → Approved → Done</pre><p>From the video: after a code review agent evaluates a pull request, the router checks the outcome. Tests failed? Send it to a fixer agent. Tests passed? Route it to a human for final approval. The reviewer’s output determines where work goes next.</p><p>This is where the distinction between LLM routing and Python routing matters enormously. You could ask the LLM to decide where to send the work. Or you could write a Python function that reads a structured field from the state and returns a node name.</p><pre># The routing function is Python — not an LLM call<br>def route_after_review(state: State) -&gt; str:<br>    if state[&quot;test_result&quot;] == &quot;failed&quot;:<br>        return &quot;fixer_agent&quot;      # deterministic, testable, free<br>    elif state[&quot;risk_level&quot;] == &quot;critical&quot;:<br>        return &quot;human_review&quot;     # also deterministic<br>    return &quot;approve_and_merge&quot;</pre><p>&lt;cite index=”22–1&quot;&gt;The router’s output should be structured, recorded, and testable rather than buried in a free-form prompt.&lt;/cite&gt; LLM routing is non-deterministic and expensive. Python conditional routing is testable in two lines and costs nothing. Production systems move toward Python routing as fast as they can.</p><h3>Pattern 4: Evaluator-Optimizer (The Loop That Knows When to Stop)</h3><p>This pattern runs a generator and an evaluator in a loop until the output meets a quality threshold — or until a retry counter says enough.</p><pre>[Generator] → [Evaluator] → Good? Yes → Done<br>                          → No, here&#39;s why → back to [Generator]</pre><p>The generator produces something. The evaluator grades it with structured output — not a prose critique, an actual typed judgment with a pass/fail field. If it fails, the feedback flows back into the generator’s next call. If it passes, execution moves on.</p><pre>class EvalResult(BaseModel):<br>    passed: bool<br>    feedback: str   # only matters if passed is False</pre><pre>def route_after_eval(state: State) -&gt; str:<br>    if state[&quot;eval_result&quot;].passed:<br>        return &quot;done&quot;<br>    if state[&quot;retry_count&quot;] &gt;= 3:<br>        return &quot;escalate&quot;   # never loop forever<br>    return &quot;generator&quot;      # try again with feedback</pre><p>The retry counter is not optional. Without it, a generator that consistently fails the evaluator runs until you run out of money or patience.</p><h3>What Graph Engineering Is Not</h3><p>This trips people up, so worth being direct.</p><p><strong>It is not a knowledge graph.</strong> A knowledge graph is a data structure for representing facts and their relationships — think of a database where entities are nodes and relationships are edges. Neo4j, GraphRAG, graph neural networks, these are knowledge graph tools. Graph engineering is about <em>execution flow</em>, not data storage.</p><p><strong>It is not an agent swarm.</strong> An agent swarm is a collection of agents that interact dynamically, often without a predefined topology. Agents negotiate, bid for tasks, and self-organise. Swarms are flexible and adaptable. They are also harder to debug and less predictable.</p><p>Graph engineering sits at the other end of that spectrum. You pre-define the nodes, the edges, and the conditions under which work moves. The path is explicit. When something goes wrong, you can trace exactly which node produced the bad output and why.</p><p>Neither approach is universally better. Complex open-ended research tasks might benefit from swarm flexibility. A pull request review pipeline that needs to reliably block broken code from merging benefits from graph predictability. Know which problem you have.</p><h3>Shared State: The Nervous System</h3><p>The graph handles the structure. Shared state handles the information.</p><p>Every node in a graph reads from a shared state object and writes updates back to it. This is not just passing variables around. It is the mechanism that lets a fan-out of five agents all contribute their findings to one coherent result, lets a router make a decision based on what three previous nodes discovered, and lets the final report node see everything that happened without any node needing to talk directly to any other.</p><p>Google’s Agent Development Kit makes this explicit. Each node in a graph can read the shared state for context relevant to its job, write its results back, and the graph engine handles the merging, the ordering, and the persistence.</p><p>The state schema is the most important design decision in the whole system. Designed well, agents collaborate without interfering. Designed poorly — fields that overwrite instead of accumulate, missing fields that agents expect to find, no versioning strategy — and the graph produces subtly wrong outputs that are hard to trace.</p><p>Here is the minimal shape every state schema needs:</p><pre>from typing import TypedDict, Annotated<br>import operator<br>class GraphState(TypedDict):<br>    # Overwrite fields: last write wins, only one valid value at a time<br>    status: str<br>    current_task: str<br>    # Accumulator fields: parallel branches append safely<br>    findings: Annotated[list, operator.add]<br>    errors:   Annotated[list, operator.add]</pre><p>Overwrite fields for things where only one value is valid at a time. Accumulator fields for everything that multiple agents might contribute to simultaneously. The difference between these two is the difference between parallel agents working correctly and parallel agents silently eating each other’s output.</p><h3>A Real Example End to End: The PR Review Pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mwjHO0FYLLeyYIs1gwRalw.png" /></figure><p>The video uses this example. It is worth walking through completely because it uses all four patterns in one system.</p><p>A developer pushes a pull request. Here is what the graph does:</p><p><strong>Step 1 — Ingestion (Sequential).</strong> A single node parses the diff, extracts the changed files, and detects any function signature changes that might affect external callers. This is preprocessing. No LLM involved. Pure Python.</p><p><strong>Step 2 — Fan-Out (Parallel).</strong> The orchestrator fans out to four or five specialist reviewer agents simultaneously: security, performance, API design, test coverage, code quality. They all read the same parsed diff from state. They all write their findings back to the accumulator field. Total time: the slowest single reviewer, not the sum.</p><p><strong>Step 3 — Join.</strong> Once all reviewers finish, the join node has all findings in one list. It deduplicates, ranks by severity, computes a security score.</p><p><strong>Step 4 — Router.</strong> The router reads the aggregated findings. Critical security issue? Route to a human approval checkpoint using interrupt(). No critical issues, all tests passing? Route to auto-approve.</p><p><strong>Step 5 — Human Gate (Optional).</strong> If routing sent work here, the graph pauses. A human sees the findings. They approve, reject, or ask for changes. The graph resumes exactly where it left off. The PR either merges or goes back to the developer.</p><p>Every piece of that is a pattern you have seen above. Fan-out, join, router, and the human gate is just the interrupt-based HITL from the LangGraph series. The graph is the composition of these patterns, not a new invention.</p><h3>The Bigger Picture: Why This Is Happening Now</h3><p>From compiler control-flow graphs to database query plans, graph topology has always been the standard language for describing how work flows.</p><p>Graph engineering did not invent nodes and edges. It borrowed them from a tradition that goes back to Dijkstra’s work on compiler optimisation, to RACI matrices in organisational design, to Petri nets in concurrent systems theory. The insight is not structural novelty. It is recognising that the problems practitioners are hitting with AI agents — coordination failures, unpredictable routing, context getting lost between steps, are the same problems that led to those tools in the first place.</p><p>What changed is that AI agents introduced a new kind of node: one that is probabilistic, context-sensitive, and capable of producing structured outputs from natural language inputs. The old graph machinery needed new defaults. LangGraph, Google ADK, and Microsoft AutoGen are those defaults.</p><p>One speaker in the video raises an interesting question toward the end: are we just reinventing computer science? Stack engineering, hash engineering — will those be next? Maybe. But the more grounding observation is this: every generation of software complexity has produced a new organisational layer. Procedures replaced raw assembly. Objects replaced procedures. Microservices replaced monoliths. Graph engineering is the organisational layer for coordinated AI agents. That it resembles what came before is not a criticism. It is evidence that the problem is real and the solution is recognisable.</p><h3>The Practical Takeaway</h3><p>If you are building something with one agent doing one focused job in one loop, graph engineering is overhead you do not need. Keep it simple.</p><p>If your system has genuinely independent sub-tasks that could run in parallel, distinct specialists that each need their own context, conditional paths where different inputs need different handling, or irreversible actions that need a human in the loop, that is when a graph earns its place.</p><p>Start with the state schema. Draw the graph on paper before writing code. Identify which nodes accumulate and which overwrite. Then build, one pattern at a time.</p><p>The structure is the system. Get that right and the agents have somewhere sensible to work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e49337901a42" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/graph-engineering-e49337901a42">Graph Engineering</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Every LangGraph Pattern You’ll Actually Use : Explained Properly]]></title>
            <link>https://pub.towardsai.net/every-langgraph-pattern-youll-actually-use-explained-properly-a3fc2f947075?source=rss-4a5744f8548f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3fc2f947075</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Bessie Delight Kekeli]]></dc:creator>
            <pubDate>Wed, 22 Jul 2026 18:01:01 GMT</pubDate>
            <atom:updated>2026-07-22T18:01:01.611Z</atom:updated>
            <content:encoded><![CDATA[<h4>Prompt chains, routers, and full agents, walked through line by line with one reusable template</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SmhpVg0fbWYqWGdosdupIg.png" /></figure><p>If you’ve looked at LangGraph code before and felt lost in a pile of StateGraph, add_node, and add_edge calls, this guide is for you. We&#39;re going to slow down and actually explain what&#39;s happening at each step, not just show code and assume it&#39;s obvious.</p><p>By the end, you’ll understand six ways to structure an LLM-powered system, when to reach for each one, and how to read (or write) the code without guessing.</p><p>This article is adapted from the reference material: <a href="https://mirror-feeling-d80.notion.site/Workflow-And-Agents-17e808527b1780d792a0d934ce62bee6#180808527b1780639a64c45574fee836">Workflow And Agents</a></p><h3>Why this matters before we touch any code</h3><p>A single call to an LLM is just a question and an answer. You send a prompt, you get text back, and that’s it. The model has no memory of what it should do next, no way to check its own work, and no ability to call a search engine or a calculator on its own.</p><p>The moment you need more than one step (write a draft, then check it, then fix it), you need something to hold the pieces together: what happened so far, what should happen next, and how to move from one step to the other. That “something” is what this whole guide is about.</p><p>There are two broad ways to build that structure. Anthropic’s own engineering team wrote a good explainer on this, and it’s worth reading directly: <a href="https://www.anthropic.com/research/building-effective-agents">Building Effective Agents</a>.</p><h3>Workflows vs. agents: the actual difference</h3><p>People throw these two words around interchangeably, which causes a lot of confusion. Here’s the distinction that matters:</p><p><strong>A workflow is a fixed path you designed in advance.</strong> You, the developer, decide: step one happens, then step two, maybe with a branch depending on a condition, then step three. The LLM fills in the content at each step, but it doesn’t get to change the route. Think of it like a recipe: the steps are printed on the card, and the cook (the LLM) follows them in order.</p><p><strong>An agent doesn’t have a fixed path.</strong> Instead, it’s given a goal, a set of tools, and the freedom to decide what to do next based on what just happened. It might call a tool, look at the result, decide it needs another tool, call that, and keep going until it thinks the job is done. Nobody wrote down the exact sequence of steps ahead of time. The model works that out as it goes.</p><p>Neither one is “better.” Workflows are predictable, easier to test, and cheaper to run, because you know exactly what will happen. Agents are more flexible and can handle problems you didn’t fully anticipate, but that flexibility comes at the cost of predictability. An agent can wander, loop unnecessarily, or make a bad call that a fixed workflow would never have allowed in the first place.</p><p>A rough rule of thumb: if you can draw the flowchart before you write any code, build a workflow. If you can’t, because the right next step genuinely depends on things you can’t predict, you probably need an agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LAl3yjC6qTQiIdNNRLdyyg.png" /></figure><h3>Do you actually need LangGraph?</h3><p>Short answer: no, not technically. Every pattern in this guide could be written as plain Python: a few functions and some if/else logic. LangGraph doesn’t add new capabilities that Python doesn’t already have.</p><p>What it adds is infrastructure you’d otherwise have to build yourself, and would probably build badly the first few times:</p><ul><li><strong>Persistence</strong>: saving the state of a conversation or task so it survives a restart, and letting a human pause the process to approve or reject a step before it continues.</li><li><strong>Streaming</strong>: showing partial output as it’s generated, from any point in your workflow, instead of making the user stare at a blank screen until everything finishes.</li><li><strong>Deployment tooling</strong>: ways to test, inspect, and debug what actually happened inside a run, which matters a lot once something goes wrong in production and you need to know why.</li></ul><p>If you’re just experimenting with a single prompt, skip LangGraph entirely. Once you’re chaining multiple steps together and need to keep track of what happened, it starts pulling its weight.</p><h3>The template we’ll use for every example</h3><p>Every code example below follows the same seven-part layout. Learn this once, and you can read any of the patterns further down without re-learning how the code is organized each time.</p><ol><li><strong>Imports</strong>: what we’re pulling in and why</li><li><strong>State</strong>: the shared data structure every node reads from and writes to</li><li><strong>Tools</strong> <em>(only when the pattern uses them)</em>: functions the LLM can choose to call</li><li><strong>Nodes</strong>: the individual steps in the graph, each one usually a single Python function</li><li><strong>Edges</strong>: the wiring that decides which node runs after which</li><li><strong>Assembly</strong>: where we actually build the graph object and compile it</li><li><strong>Entrypoint</strong>: the code that runs the finished graph with real input</li></ol><p>Keeping this order consistent is the whole point. Once “State” always means the same thing and “Edges” always means the same thing, you stop having to reverse-engineer someone else’s code structure every time you open a new file.</p><h3>Building block: the “augmented” LLM</h3><p>Before any of the six patterns, you need one core idea: a plain LLM and an LLM that’s been given extra capabilities are not the same thing, and LangChain calls the upgraded version an “augmented” LLM.</p><p>On its own, an LLM just turns text into more text. Two upgrades change what it can do:</p><p><strong>Structured output</strong> forces the model to answer in a specific shape, a Python object with named fields, instead of free-form prose you’d have to parse yourself. You describe the shape you want using a Pydantic model (a class that defines field names, types, and short descriptions), and the LLM fills it in.</p><p><strong>Tool binding</strong> gives the model a menu of functions it’s allowed to call. The LLM doesn’t actually run the function. It decides <em>whether</em> a function is needed and <em>what arguments</em> to pass, and your code is the one that actually executes it and hands the result back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IGqpklcF8RwxuRMZo0UhAQ.png" /></figure><pre>from langchain_anthropic import ChatAnthropic<br><br># This is just a plain, unmodified LLM connection.<br>llm = ChatAnthropic(model=&quot;claude-3-5-sonnet-latest&quot;)</pre><p>Here’s structured output in practice:</p><pre>from pydantic import BaseModel, Field<br><br># This class describes the exact shape we want the LLM&#39;s answer to take.<br># Two fields: the search query itself, and a short justification for it.<br>class SearchQuery(BaseModel):<br>    search_query: str = Field(description=&quot;A web-search-optimized version of the question.&quot;)<br>    justification: str = Field(description=&quot;Why this query answers the user&#39;s request.&quot;)<br># .with_structured_output() wraps the LLM so it always replies in this shape.<br>structured_llm = llm.with_structured_output(SearchQuery)<br>output = structured_llm.invoke(&quot;How does a calcium CT score relate to cholesterol?&quot;)<br>print(output.search_query)     # a clean, searchable string<br>print(output.justification)    # the model&#39;s reasoning for picking that query</pre><p>Without with_structured_output, you&#39;d get back a paragraph of text and have to write regex or string-parsing logic to pull the query out of it. That&#39;s fragile: the model might phrase things slightly differently every time. Structured output sidesteps the problem entirely by making the shape non-negotiable.</p><p>And here’s tool binding:</p><pre>def multiply(a: int, b: int) -&gt; int:<br>    return a * b<br><br># bind_tools tells the LLM this function exists and describes what it does,<br># based on the function&#39;s name, type hints, and docstring.<br>llm_with_tools = llm.bind_tools([multiply])<br>msg = llm_with_tools.invoke(&quot;What is 2 times 3?&quot;)<br># The LLM doesn&#39;t compute 2*3 itself. It recognizes multiply() fits this task<br># and returns a *request* to call it, with the arguments filled in.<br>msg.tool_calls<br># -&gt; [{&#39;name&#39;: &#39;multiply&#39;, &#39;args&#39;: {&#39;a&#39;: 2, &#39;b&#39;: 3}, &#39;id&#39;: &#39;...&#39;}]</pre><p>Notice the model never actually multiplies anything. It just says “call multiply with a=2, b=3.&quot; Your code has to be the one that runs it. We&#39;ll see exactly how in the full agent example near the end.</p><p>With those two ideas in hand, we can move through the six patterns, from simplest to most autonomous.</p><h3>Pattern 1: Prompt Chaining</h3><p><strong>The idea:</strong> break one big task into several smaller LLM calls that run one after another, where each step builds on the output of the one before it. Use this when a task naturally breaks into stages, and getting each stage right on its own is easier than trying to get the whole thing right in a single shot.</p><p>A good comparison is editing an essay: you write a rough draft, then you tighten the language, then you do a final polish pass. Each pass has a narrow job, and narrow jobs are easier for an LLM (and for a human) to do well than “write a perfect essay in one attempt.”</p><p>Our example: write a joke, improve it with wordplay, then add a twist ending. There’s also a quality check in the middle: if the first draft doesn’t look like a real joke, we stop early instead of wasting two more LLM calls polishing something broken.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IdEp8B3jZSV9ArK2JM4XbA.png" /></figure><pre># ============================================================<br># MODULE 1: IMPORTS<br># ============================================================<br>from typing_extensions import TypedDict<br>from langgraph.graph import StateGraph, START, END<br><br># ============================================================<br># MODULE 2: STATE<br># ============================================================<br># State is just a dictionary with a known shape. Every node function<br># receives the current state and returns a partial update to it.<br># LangGraph merges that update back into the shared state automatically.<br>class State(TypedDict):<br>    topic: str<br>    joke: str<br>    improved_joke: str<br>    final_joke: str<br># ============================================================<br># MODULE 3: TOOLS<br># ============================================================<br># No tools in this pattern. Every step is a plain LLM call.<br># ============================================================<br># MODULE 4: NODES<br># ============================================================<br>def generate_joke(state: State):<br>    &quot;&quot;&quot;Step 1: write a first-draft joke about the topic.&quot;&quot;&quot;<br>    msg = llm.invoke(f&quot;Write a short joke about {state[&#39;topic&#39;]}&quot;)<br>    return {&quot;joke&quot;: msg.content}<br>def improve_joke(state: State):<br>    &quot;&quot;&quot;Step 2: take that draft and add wordplay to it.&quot;&quot;&quot;<br>    msg = llm.invoke(f&quot;Make this joke funnier by adding wordplay: {state[&#39;joke&#39;]}&quot;)<br>    return {&quot;improved_joke&quot;: msg.content}<br>def polish_joke(state: State):<br>    &quot;&quot;&quot;Step 3: give it a final twist ending.&quot;&quot;&quot;<br>    msg = llm.invoke(f&quot;Add a surprising twist to this joke: {state[&#39;improved_joke&#39;]}&quot;)<br>    return {&quot;final_joke&quot;: msg.content}<br>def check_punchline(state: State):<br>    &quot;&quot;&quot;<br>    This isn&#39;t a node that generates anything. It&#39;s a gate function<br>    used by a conditional edge. It looks at the draft joke and decides<br>    whether it&#39;s worth continuing, based on a rough heuristic: does it<br>    contain a &#39;?&#39; or &#39;!&#39;? If not, we treat it as a failed first draft<br>    and stop rather than polishing something broken.<br>    &quot;&quot;&quot;<br>    if &quot;?&quot; in state[&quot;joke&quot;] or &quot;!&quot; in state[&quot;joke&quot;]:<br>        return &quot;Pass&quot;<br>    return &quot;Fail&quot;<br># ============================================================<br># MODULE 5: EDGES<br># ============================================================<br>workflow = StateGraph(State)<br>workflow.add_node(&quot;generate_joke&quot;, generate_joke)<br>workflow.add_node(&quot;improve_joke&quot;, improve_joke)<br>workflow.add_node(&quot;polish_joke&quot;, polish_joke)<br>workflow.add_edge(START, &quot;generate_joke&quot;)<br># add_conditional_edges routes to a *different* next node depending on<br># what the gate function returns. Here: &quot;Pass&quot; continues the chain,<br># &quot;Fail&quot; jumps straight to the end.<br>workflow.add_conditional_edges(<br>    &quot;generate_joke&quot;,<br>    check_punchline,<br>    {&quot;Pass&quot;: &quot;improve_joke&quot;, &quot;Fail&quot;: END},<br>)<br>workflow.add_edge(&quot;improve_joke&quot;, &quot;polish_joke&quot;)<br>workflow.add_edge(&quot;polish_joke&quot;, END)<br># ============================================================<br># MODULE 6: ASSEMBLY<br># ============================================================<br>chain = workflow.compile()<br># ============================================================<br># MODULE 7: ENTRYPOINT<br># ============================================================<br>if __name__ == &quot;__main__&quot;:<br>    state = chain.invoke({&quot;topic&quot;: &quot;cats&quot;})<br>    print(&quot;Initial joke:&quot;, state[&quot;joke&quot;])<br>    if &quot;improved_joke&quot; in state:<br>        print(&quot;Improved joke:&quot;, state[&quot;improved_joke&quot;])<br>        print(&quot;Final joke:&quot;, state[&quot;final_joke&quot;])<br>    else:<br>        print(&quot;Joke failed the quality gate: no punchline detected.&quot;)</pre><p>A few things worth slowing down on:</p><ul><li><strong>Nodes only return what changed.</strong> generate_joke returns {&quot;joke&quot;: msg.content}, not the entire state object. LangGraph takes that small dictionary and merges it into the existing state for you.</li><li><strong>The gate function doesn’t touch state at all.</strong> It just reads it and returns a string (“Pass” or “Fail”), and that string is what add_conditional_edges uses to pick the next node.</li><li><strong>START and </strong><strong>END are LangGraph&#39;s built-in markers</strong> for &quot;the graph begins here&quot; and &quot;the graph is finished.&quot; Every graph needs both.</li></ul><h3>Pattern 2: Parallelization</h3><p><strong>The idea:</strong> when the sub-tasks don’t depend on each other, run them at the same time instead of one after another. If you’re generating a joke, a story, and a poem about the same topic, there’s no reason the poem has to wait for the joke to finish first. Nothing in the poem depends on the joke.</p><p>This is also useful when you want several independent “opinions” on the same input, for example, asking three different prompts to search for the same information from different angles, then combining what they each found.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*LtXE4Ri8y0Q0yOWCOfLUsw.png" /></figure><pre># ============================================================<br># MODULE 1: IMPORTS<br># ============================================================<br>from typing_extensions import TypedDict<br>from langgraph.graph import StateGraph, START, END<br><br># ============================================================<br># MODULE 2: STATE<br># ============================================================<br>class State(TypedDict):<br>    topic: str<br>    joke: str<br>    story: str<br>    poem: str<br>    combined_output: str<br># ============================================================<br># MODULE 3: TOOLS<br># ============================================================<br># None needed here either.<br># ============================================================<br># MODULE 4: NODES<br># ============================================================<br>def call_llm_1(state: State):<br>    &quot;&quot;&quot;Generate a joke about the topic.&quot;&quot;&quot;<br>    msg = llm.invoke(f&quot;Write a joke about {state[&#39;topic&#39;]}&quot;)<br>    return {&quot;joke&quot;: msg.content}<br>def call_llm_2(state: State):<br>    &quot;&quot;&quot;Generate a short story about the topic.&quot;&quot;&quot;<br>    msg = llm.invoke(f&quot;Write a story about {state[&#39;topic&#39;]}&quot;)<br>    return {&quot;story&quot;: msg.content}<br>def call_llm_3(state: State):<br>    &quot;&quot;&quot;Generate a poem about the topic.&quot;&quot;&quot;<br>    msg = llm.invoke(f&quot;Write a poem about {state[&#39;topic&#39;]}&quot;)<br>    return {&quot;poem&quot;: msg.content}<br>def aggregator(state: State):<br>    &quot;&quot;&quot;<br>    This node doesn&#39;t call the LLM at all. Its only job is to wait<br>    for all three branches to finish, then stitch their outputs<br>    together into one combined string.<br>    &quot;&quot;&quot;<br>    combined = f&quot;Here&#39;s a story, joke, and poem about {state[&#39;topic&#39;]}!\n\n&quot;<br>    combined += f&quot;STORY:\n{state[&#39;story&#39;]}\n\n&quot;<br>    combined += f&quot;JOKE:\n{state[&#39;joke&#39;]}\n\n&quot;<br>    combined += f&quot;POEM:\n{state[&#39;poem&#39;]}&quot;<br>    return {&quot;combined_output&quot;: combined}<br># ============================================================<br># MODULE 5: EDGES<br># ============================================================<br>parallel_builder = StateGraph(State)<br>parallel_builder.add_node(&quot;call_llm_1&quot;, call_llm_1)<br>parallel_builder.add_node(&quot;call_llm_2&quot;, call_llm_2)<br>parallel_builder.add_node(&quot;call_llm_3&quot;, call_llm_3)<br>parallel_builder.add_node(&quot;aggregator&quot;, aggregator)<br># All three branches start directly from START. That&#39;s what makes<br># them run in parallel instead of one after another. Then all three<br># feed into the same aggregator node, which waits until every branch<br># has completed before it runs.<br>parallel_builder.add_edge(START, &quot;call_llm_1&quot;)<br>parallel_builder.add_edge(START, &quot;call_llm_2&quot;)<br>parallel_builder.add_edge(START, &quot;call_llm_3&quot;)<br>parallel_builder.add_edge(&quot;call_llm_1&quot;, &quot;aggregator&quot;)<br>parallel_builder.add_edge(&quot;call_llm_2&quot;, &quot;aggregator&quot;)<br>parallel_builder.add_edge(&quot;call_llm_3&quot;, &quot;aggregator&quot;)<br>parallel_builder.add_edge(&quot;aggregator&quot;, END)<br># ============================================================<br># MODULE 6: ASSEMBLY<br># ============================================================<br>parallel_workflow = parallel_builder.compile()<br># ============================================================<br># MODULE 7: ENTRYPOINT<br># ============================================================<br>if __name__ == &quot;__main__&quot;:<br>    state = parallel_workflow.invoke({&quot;topic&quot;: &quot;cats&quot;})<br>    print(state[&quot;combined_output&quot;])</pre><p>The key structural difference from prompt chaining: three separate edges leave START, instead of one edge chaining into the next. LangGraph sees that call_llm_1, call_llm_2, and call_llm_3 don&#39;t depend on each other&#39;s output, so it can run them concurrently and simply wait for all three before moving on to aggregator.</p><h3>Pattern 3: Routing</h3><p><strong>The idea:</strong> look at the input first, classify what kind of request it is, and send it down a different path depending on the answer. This is useful whenever different types of input genuinely need different handling. A customer support message about billing shouldn’t go through the same logic as a technical bug report.</p><p>The trick is that the “router” itself is just another LLM call, one that uses structured output to guarantee it returns one of a fixed set of labels, rather than a free-text guess you’d have to interpret.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ULT_3wtvJlJdxO406XYH-Q.png" /></figure><pre># ============================================================<br># MODULE 1: IMPORTS<br># ============================================================<br>from typing_extensions import TypedDict, Literal<br>from pydantic import BaseModel, Field<br>from langchain_core.messages import HumanMessage, SystemMessage<br>from langgraph.graph import StateGraph, START, END<br><br># ============================================================<br># MODULE 2: STATE<br># ============================================================<br>class State(TypedDict):<br>    input: str<br>    decision: str<br>    output: str<br># ============================================================<br># MODULE 3: TOOLS<br># ============================================================<br># Not a callable tool, but the same idea: a schema that constrains<br># what the router is allowed to say. Literal[...] means the model<br># can ONLY answer with one of these three exact strings.<br>class Route(BaseModel):<br>    step: Literal[&quot;poem&quot;, &quot;story&quot;, &quot;joke&quot;] = Field(<br>        description=&quot;The next step in the routing process.&quot;<br>    )<br>router = llm.with_structured_output(Route)<br># ============================================================<br># MODULE 4: NODES<br># ============================================================<br>def llm_call_1(state: State):<br>    &quot;&quot;&quot;Handle requests classified as &#39;story&#39;.&quot;&quot;&quot;<br>    result = llm.invoke(state[&quot;input&quot;])<br>    return {&quot;output&quot;: result.content}<br>def llm_call_2(state: State):<br>    &quot;&quot;&quot;Handle requests classified as &#39;joke&#39;.&quot;&quot;&quot;<br>    result = llm.invoke(state[&quot;input&quot;])<br>    return {&quot;output&quot;: result.content}<br>def llm_call_3(state: State):<br>    &quot;&quot;&quot;Handle requests classified as &#39;poem&#39;.&quot;&quot;&quot;<br>    result = llm.invoke(state[&quot;input&quot;])<br>    return {&quot;output&quot;: result.content}<br>def llm_call_router(state: State):<br>    &quot;&quot;&quot;<br>    Classify the user&#39;s input into one of three buckets, using the<br>    Route schema above so the answer is guaranteed to be exactly<br>    &quot;story&quot;, &quot;joke&quot;, or &quot;poem&quot;, never something unexpected like<br>    &quot;Story!&quot; or &quot;I think it&#39;s a joke.&quot;<br>    &quot;&quot;&quot;<br>    decision = router.invoke([<br>        SystemMessage(content=&quot;Route the input to story, joke, or poem based on the user&#39;s request.&quot;),<br>        HumanMessage(content=state[&quot;input&quot;]),<br>    ])<br>    return {&quot;decision&quot;: decision.step}<br>def route_decision(state: State):<br>    &quot;&quot;&quot;<br>    Gate function for the conditional edge. Reads the classification<br>    made above and returns the name of the node that should handle it.<br>    &quot;&quot;&quot;<br>    if state[&quot;decision&quot;] == &quot;story&quot;:<br>        return &quot;llm_call_1&quot;<br>    elif state[&quot;decision&quot;] == &quot;joke&quot;:<br>        return &quot;llm_call_2&quot;<br>    elif state[&quot;decision&quot;] == &quot;poem&quot;:<br>        return &quot;llm_call_3&quot;<br># ============================================================<br># MODULE 5: EDGES<br># ============================================================<br>router_builder = StateGraph(State)<br>router_builder.add_node(&quot;llm_call_1&quot;, llm_call_1)<br>router_builder.add_node(&quot;llm_call_2&quot;, llm_call_2)<br>router_builder.add_node(&quot;llm_call_3&quot;, llm_call_3)<br>router_builder.add_node(&quot;llm_call_router&quot;, llm_call_router)<br>router_builder.add_edge(START, &quot;llm_call_router&quot;)<br>router_builder.add_conditional_edges(<br>    &quot;llm_call_router&quot;,<br>    route_decision,<br>    {<br>        &quot;llm_call_1&quot;: &quot;llm_call_1&quot;,<br>        &quot;llm_call_2&quot;: &quot;llm_call_2&quot;,<br>        &quot;llm_call_3&quot;: &quot;llm_call_3&quot;,<br>    },<br>)<br>router_builder.add_edge(&quot;llm_call_1&quot;, END)<br>router_builder.add_edge(&quot;llm_call_2&quot;, END)<br>router_builder.add_edge(&quot;llm_call_3&quot;, END)<br># ============================================================<br># MODULE 6: ASSEMBLY<br># ============================================================<br>router_workflow = router_builder.compile()<br># ============================================================<br># MODULE 7: ENTRYPOINT<br># ============================================================<br>if __name__ == &quot;__main__&quot;:<br>    state = router_workflow.invoke({&quot;input&quot;: &quot;Write me a joke about cats&quot;})<br>    print(state[&quot;output&quot;])</pre><p>Notice llm_call_1, llm_call_2, and llm_call_3 are nearly identical here. In a real project, they&#39;d usually differ by using different system prompts, different tools, or even entirely different models suited to each type of task. The point of routing isn&#39;t the destination nodes; it&#39;s that the router reliably sends each input to the right specialist.</p><h3>Pattern 4: Orchestrator-Worker</h3><p><strong>The idea:</strong> one LLM call plans out an unknown number of sub-tasks, then a worker handles each one (in parallel) and a final step stitches all the results together. The key difference from plain parallelization is that you don’t know ahead of time how many parallel branches you’ll need. In Pattern 2, we hardcoded exactly three branches (joke, story, poem). Here, the number of branches is decided at runtime by the orchestrator itself.</p><p>A good real-world case: writing a report. You don’t know in advance how many sections the report needs. That depends on the topic. So you let an LLM plan the sections first, and only then do you know how many workers to spin up.</p><p>This needs two new pieces of machinery:</p><ul><li><strong>Send</strong>: lets one node dynamically create multiple parallel branches at runtime, one per item in a list, instead of you hardcoding a fixed number of edges.</li><li><strong>operator.add</strong> as a reducer: when multiple workers all write to the same state key at the same time, LangGraph needs to know how to combine their results instead of one overwriting another. operator.add tells it: just concatenate the lists together.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wa2hDtjXsp3_R901b3_oCA.png" /></figure><pre># ============================================================<br># MODULE 1: IMPORTS<br># ============================================================<br>import operator<br>from typing import Annotated, List<br>from typing_extensions import TypedDict<br>from pydantic import BaseModel, Field<br>from langchain_core.messages import HumanMessage, SystemMessage<br>from langgraph.graph import StateGraph, START, END<br>from langgraph.constants import Send<br><br># ============================================================<br># MODULE 2: STATE<br># ============================================================<br>class Section(BaseModel):<br>    name: str = Field(description=&quot;Name for this section of the report.&quot;)<br>    description: str = Field(description=&quot;What this section should cover.&quot;)<br>class Sections(BaseModel):<br>    sections: List[Section] = Field(description=&quot;All sections of the report.&quot;)<br># The main graph state.<br>class State(TypedDict):<br>    topic: str<br>    sections: list[Section]<br>    # Annotated[list, operator.add] is the important part here: it tells<br>    # LangGraph that when multiple workers write to &quot;completed_sections&quot;<br>    # at the same time, it should APPEND their results together rather<br>    # than one worker&#39;s write erasing another&#39;s.<br>    completed_sections: Annotated[list, operator.add]<br>    final_report: str<br># A separate, smaller state just for what an individual worker needs.<br># It only has to know about the one section it&#39;s writing.<br>class WorkerState(TypedDict):<br>    section: Section<br>    completed_sections: Annotated[list, operator.add]<br># ============================================================<br># MODULE 3: TOOLS<br># ============================================================<br>planner = llm.with_structured_output(Sections)<br># ============================================================<br># MODULE 4: NODES<br># ============================================================<br>def orchestrator(state: State):<br>    &quot;&quot;&quot;Ask the LLM to plan out how many sections the report needs, and what each covers.&quot;&quot;&quot;<br>    report_sections = planner.invoke([<br>        SystemMessage(content=&quot;Generate a plan for the report.&quot;),<br>        HumanMessage(content=f&quot;Here is the report topic: {state[&#39;topic&#39;]}&quot;),<br>    ])<br>    return {&quot;sections&quot;: report_sections.sections}<br>def llm_call(state: WorkerState):<br>    &quot;&quot;&quot;A single worker: writes the content for exactly one section.&quot;&quot;&quot;<br>    section = llm.invoke([<br>        SystemMessage(content=&quot;Write a report section.&quot;),<br>        HumanMessage(content=f&quot;Section name: {state[&#39;section&#39;].name}\nDescription: {state[&#39;section&#39;].description}&quot;),<br>    ])<br>    # Wrapped in a list because operator.add expects to concatenate lists.<br>    return {&quot;completed_sections&quot;: [section.content]}<br>def synthesizer(state: State):<br>    &quot;&quot;&quot;Once every worker has finished, join all the sections into one document.&quot;&quot;&quot;<br>    completed_report_sections = &quot;\n\n---\n\n&quot;.join(state[&quot;completed_sections&quot;])<br>    return {&quot;final_report&quot;: completed_report_sections}<br>def assign_workers(state: State):<br>    &quot;&quot;&quot;<br>    This is what makes the dynamic fan-out possible. Send(node_name, payload)<br>    schedules one run of &quot;llm_call&quot; per section the orchestrator planned.<br>    If the plan has 3 sections, this creates 3 parallel workers; if it has<br>    7, it creates 7. We never had to know that number in advance.<br>    &quot;&quot;&quot;<br>    return [Send(&quot;llm_call&quot;, {&quot;section&quot;: s}) for s in state[&quot;sections&quot;]]<br># ============================================================<br># MODULE 5: EDGES<br># ============================================================<br>orchestrator_worker_builder = StateGraph(State)<br>orchestrator_worker_builder.add_node(&quot;orchestrator&quot;, orchestrator)<br>orchestrator_worker_builder.add_node(&quot;llm_call&quot;, llm_call)<br>orchestrator_worker_builder.add_node(&quot;synthesizer&quot;, synthesizer)<br>orchestrator_worker_builder.add_edge(START, &quot;orchestrator&quot;)<br># Instead of add_conditional_edges (which picks ONE next node),<br># we use add_conditional_edges with assign_workers, which can spawn<br># MANY next nodes at once via Send().<br>orchestrator_worker_builder.add_conditional_edges(<br>    &quot;orchestrator&quot;, assign_workers, [&quot;llm_call&quot;]<br>)<br>orchestrator_worker_builder.add_edge(&quot;llm_call&quot;, &quot;synthesizer&quot;)<br>orchestrator_worker_builder.add_edge(&quot;synthesizer&quot;, END)<br># ============================================================<br># MODULE 6: ASSEMBLY<br># ============================================================<br>orchestrator_worker = orchestrator_worker_builder.compile()<br># ============================================================<br># MODULE 7: ENTRYPOINT<br># ============================================================<br>if __name__ == &quot;__main__&quot;:<br>    state = orchestrator_worker.invoke({&quot;topic&quot;: &quot;Create a report on LLM scaling laws&quot;})<br>    print(state[&quot;final_report&quot;])</pre><p>If you only remember one thing from this pattern, make it this: Send is for when the <em>number</em> of parallel branches isn&#39;t known until the graph is already running. Everything else about it (nodes, edges, state) works the same way as the patterns before it.</p><h3>Pattern 5: Evaluator-Optimizer</h3><p><strong>The idea:</strong> one LLM generates something, a second LLM call grades it, and if it doesn’t pass, the feedback gets fed back into another generation attempt, looping until it’s good enough. This is essentially a built-in editor that won’t let a weak first draft through.</p><p>This is worth reaching for whenever “good” is something you can actually check for: grading a joke as funny or not, checking a generated SQL query for syntax errors, or verifying a RAG answer is actually backed by the retrieved documents rather than invented.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j1jL7yB3zhFyJD8fa3oAAg.png" /></figure><pre># ============================================================<br># MODULE 1: IMPORTS<br># ============================================================<br>from typing_extensions import TypedDict, Literal<br>from pydantic import BaseModel, Field<br>from langgraph.graph import StateGraph, START, END<br><br># ============================================================<br># MODULE 2: STATE<br># ============================================================<br>class State(TypedDict):<br>    joke: str<br>    topic: str<br>    feedback: str<br>    funny_or_not: str<br># ============================================================<br># MODULE 3: TOOLS<br># ============================================================<br>class Feedback(BaseModel):<br>    grade: Literal[&quot;funny&quot;, &quot;not funny&quot;] = Field(description=&quot;Whether the joke is funny.&quot;)<br>    feedback: str = Field(description=&quot;If not funny, concrete advice on how to fix it.&quot;)<br>evaluator = llm.with_structured_output(Feedback)<br># ============================================================<br># MODULE 4: NODES<br># ============================================================<br>def llm_call_generator(state: State):<br>    &quot;&quot;&quot;<br>    Write a joke. If this is a retry (there&#39;s feedback in state from a<br>    previous failed attempt), pass that feedback back to the model so<br>    it actually improves rather than generating a fresh random attempt.<br>    &quot;&quot;&quot;<br>    if state.get(&quot;feedback&quot;):<br>        msg = llm.invoke(<br>            f&quot;Write a joke about {state[&#39;topic&#39;]} but take into account this feedback: {state[&#39;feedback&#39;]}&quot;<br>        )<br>    else:<br>        msg = llm.invoke(f&quot;Write a joke about {state[&#39;topic&#39;]}&quot;)<br>    return {&quot;joke&quot;: msg.content}<br>def llm_call_evaluator(state: State):<br>    &quot;&quot;&quot;Grade the joke and, if it fails, explain why.&quot;&quot;&quot;<br>    grade = evaluator.invoke(f&quot;Grade the joke: {state[&#39;joke&#39;]}&quot;)<br>    return {&quot;funny_or_not&quot;: grade.grade, &quot;feedback&quot;: grade.feedback}<br>def route_joke(state: State):<br>    &quot;&quot;&quot;Gate function: accept and stop, or loop back with feedback.&quot;&quot;&quot;<br>    if state[&quot;funny_or_not&quot;] == &quot;funny&quot;:<br>        return &quot;Accepted&quot;<br>    return &quot;Rejected + Feedback&quot;<br># ============================================================<br># MODULE 5: EDGES<br># ============================================================<br>optimizer_builder = StateGraph(State)<br>optimizer_builder.add_node(&quot;llm_call_generator&quot;, llm_call_generator)<br>optimizer_builder.add_node(&quot;llm_call_evaluator&quot;, llm_call_evaluator)<br>optimizer_builder.add_edge(START, &quot;llm_call_generator&quot;)<br>optimizer_builder.add_edge(&quot;llm_call_generator&quot;, &quot;llm_call_evaluator&quot;)<br># This is the loop: if the evaluator says &quot;Rejected + Feedback&quot;, control<br># goes BACK to llm_call_generator instead of forward to END. That&#39;s what<br># turns this into an iterative refinement loop rather than a straight line.<br>optimizer_builder.add_conditional_edges(<br>    &quot;llm_call_evaluator&quot;,<br>    route_joke,<br>    {&quot;Accepted&quot;: END, &quot;Rejected + Feedback&quot;: &quot;llm_call_generator&quot;},<br>)<br># ============================================================<br># MODULE 6: ASSEMBLY<br># ============================================================<br>optimizer_workflow = optimizer_builder.compile()<br># ============================================================<br># MODULE 7: ENTRYPOINT<br># ============================================================<br>if __name__ == &quot;__main__&quot;:<br>    state = optimizer_workflow.invoke({&quot;topic&quot;: &quot;Cats&quot;})<br>    print(state[&quot;joke&quot;])</pre><p>One practical warning: nothing in this graph stops it from looping forever if the evaluator is a harsh grader and the generator can’t satisfy it. In real projects, you’d usually add a retry counter to State and force an exit after, say, three attempts. Otherwise, a stubborn evaluator can burn through your API budget without you noticing.</p><h3>Pattern 6: The Agent</h3><p><strong>The idea:</strong> this is where we stop pre-drawing the flowchart. Instead of you deciding the sequence of steps, the LLM decides (on every turn) whether it needs to call a tool, and it keeps looping between “think” and “act” until it decides the job is done.</p><p>This is genuinely different from every pattern above. In prompt chaining, routing, and the rest, <em>you</em> wrote down which node comes after which. Here, there’s really only one loop: the model thinks, optionally calls a tool, sees the result, and thinks again, and it can go around that loop as many times as it needs to.</p><p>Use this when the task is open-ended enough that you can’t lay out the steps in advance, for instance, a multi-step calculation where you don’t know how many operations will be needed until you’re partway through it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xL3QYNOYnuY4w0jmthKDEA.png" /></figure><pre># ============================================================<br># MODULE 1: IMPORTS<br># ============================================================<br>from typing_extensions import Literal<br>from langchain_core.tools import tool<br>from langchain_core.messages import SystemMessage, ToolMessage, HumanMessage<br>from langgraph.graph import StateGraph, START, END, MessagesState<br><br># ============================================================<br># MODULE 2: STATE<br># ============================================================<br># MessagesState is a ready-made state type from LangGraph that already<br># has one field, &quot;messages,&quot; designed to hold a running list of chat<br># messages. We don&#39;t need to define our own State class this time<br># because a conversation history is exactly what this pattern needs.<br># ============================================================<br># MODULE 3: TOOLS<br># ============================================================<br># The @tool decorator turns a normal Python function into something<br># the LLM can request to call. The docstring matters. The model reads<br># it to decide when this tool is the right one to use.<br>@tool<br>def multiply(a: int, b: int) -&gt; int:<br>    &quot;&quot;&quot;Multiply a and b.&quot;&quot;&quot;<br>    return a * b<br>@tool<br>def add(a: int, b: int) -&gt; int:<br>    &quot;&quot;&quot;Add a and b.&quot;&quot;&quot;<br>    return a + b<br>@tool<br>def divide(a: int, b: int) -&gt; float:<br>    &quot;&quot;&quot;Divide a and b.&quot;&quot;&quot;<br>    return a / b<br>tools = [add, multiply, divide]<br>tools_by_name = {t.name: t for t in tools}<br>llm_with_tools = llm.bind_tools(tools)<br># ============================================================<br># MODULE 4: NODES<br># ============================================================<br>def llm_call(state: MessagesState):<br>    &quot;&quot;&quot;<br>    The &quot;thinking&quot; step. The model looks at the full conversation so far<br>    and decides: answer directly, or request a tool call. Either way,<br>    its response gets appended to the messages list.<br>    &quot;&quot;&quot;<br>    return {<br>        &quot;messages&quot;: [<br>            llm_with_tools.invoke(<br>                [SystemMessage(content=&quot;You are a helpful assistant tasked with performing arithmetic on a set of inputs.&quot;)]<br>                + state[&quot;messages&quot;]<br>            )<br>        ]<br>    }<br>def tool_node(state: dict):<br>    &quot;&quot;&quot;<br>    The &quot;acting&quot; step. This is where a tool call the model requested<br>    actually gets executed. The LLM never runs code itself, it only<br>    asks for it. We look up the requested tool by name, run it with<br>    the arguments the model provided, and package the result as a<br>    ToolMessage so the model can read it on its next turn.<br>    &quot;&quot;&quot;<br>    result = []<br>    for tool_call in state[&quot;messages&quot;][-1].tool_calls:<br>        selected_tool = tools_by_name[tool_call[&quot;name&quot;]]<br>        observation = selected_tool.invoke(tool_call[&quot;args&quot;])<br>        result.append(ToolMessage(content=observation, tool_call_id=tool_call[&quot;id&quot;]))<br>    return {&quot;messages&quot;: result}<br>def should_continue(state: MessagesState) -&gt; Literal[&quot;environment&quot;, END]:<br>    &quot;&quot;&quot;<br>    This is the heart of the agent loop. After the model thinks, we check:<br>    did it ask for a tool? If yes, go run the tool (&quot;Action&quot;). If not,<br>    meaning it gave a final answer instead, the loop ends.<br>    &quot;&quot;&quot;<br>    last_message = state[&quot;messages&quot;][-1]<br>    if last_message.tool_calls:<br>        return &quot;Action&quot;<br>    return END<br># ============================================================<br># MODULE 5: EDGES<br># ============================================================<br>agent_builder = StateGraph(MessagesState)<br>agent_builder.add_node(&quot;llm_call&quot;, llm_call)<br>agent_builder.add_node(&quot;environment&quot;, tool_node)<br>agent_builder.add_edge(START, &quot;llm_call&quot;)<br>agent_builder.add_conditional_edges(<br>    &quot;llm_call&quot;,<br>    should_continue,<br>    {&quot;Action&quot;: &quot;environment&quot;, END: END},<br>)<br># This is the edge that makes it a loop rather than a straight line:<br># after a tool runs, control goes back to llm_call, not forward to END.<br># The model gets to see the tool&#39;s result and decide what to do next:<br># answer, or call another tool.<br>agent_builder.add_edge(&quot;environment&quot;, &quot;llm_call&quot;)<br># ============================================================<br># MODULE 6: ASSEMBLY<br># ============================================================<br>agent = agent_builder.compile()<br># ============================================================<br># MODULE 7: ENTRYPOINT<br># ============================================================<br>if __name__ == &quot;__main__&quot;:<br>    messages = [HumanMessage(content=&quot;Add 3 and 4. Then take the output and multiply it by 4.&quot;)]<br>    result = agent.invoke({&quot;messages&quot;: messages})<br>    for m in result[&quot;messages&quot;]:<br>        m.pretty_print()</pre><p>Walk through what actually happens on that input, step by step:</p><ol><li>llm_call runs. The model reads &quot;add 3 and 4, then multiply by 4&quot; and decides it needs the add tool first. It returns a tool call, not a final answer.</li><li>should_continue checks the last message, sees a tool call, and returns &quot;Action&quot;.</li><li>tool_node runs add(3, 4), gets 7, and appends that result as a ToolMessage.</li><li>Control loops back to llm_call. The model now sees &quot;3 + 4 = 7&quot; in its message history and decides the next step is multiply(7, 4).</li><li>should_continue sees another tool call, routes to &quot;Action&quot; again.</li><li>tool_node runs multiply(7, 4), gets 28.</li><li>Control loops back to llm_call one more time. Now the model has everything it needs, so it responds with a final answer instead of another tool call.</li><li>should_continue sees no tool calls this time and returns END. The loop stops.</li></ol><p>Nobody wrote “call add, then call multiply” anywhere in the code. The model figured that sequence out on its own, one decision at a time. That’s the entire difference between an agent and a workflow.</p><h3>Which pattern should you actually use?</h3><p>Start simple and add complexity only when you hit its limits:</p><ul><li>Just need to force a clean answer shape or let the model use one tool? You don’t need a graph at all. The augmented LLM alone is enough.</li><li>Task breaks cleanly into ordered stages, each one easier alone than combined? <strong>Prompt chaining.</strong></li><li>Independent sub-tasks that don’t need each other’s output? <strong>Parallelization.</strong></li><li>Different input types genuinely need different handling? <strong>Routing.</strong></li><li>Don’t know how many sub-tasks you’ll need until you see the input? <strong>Orchestrator-worker.</strong></li><li>“Good” output is something you can actually check programmatically or with a second LLM call? <strong>Evaluator-optimizer.</strong></li><li>The right sequence of steps can’t be known ahead of time at all? That’s the only case where you actually need a full <strong>agent</strong>, and it’s worth trying every other pattern first, because a workflow you can predict is a workflow you can debug.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3fc2f947075" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/every-langgraph-pattern-youll-actually-use-explained-properly-a3fc2f947075">Every LangGraph Pattern You’ll Actually Use : Explained Properly</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-Tune Your First LLM: A Guide with PyTorch and Hugging Face]]></title>
            <link>https://pub.towardsai.net/fine-tune-your-first-llm-a-guide-with-pytorch-and-hugging-face-bc4cdfb156c3?source=rss-4a5744f8548f------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc4cdfb156c3</guid>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Bessie Delight Kekeli]]></dc:creator>
            <pubDate>Sat, 27 Jun 2026 20:01:00 GMT</pubDate>
            <atom:updated>2026-06-27T20:01:00.910Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>A module-by-module guide to fine-tune a Pre-trained model with Pytorch and Hugging Face.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*21bg2FRI-k-RDPpBN40rKA.png" /></figure><blockquote>This article draws from the excellent full fine-tuning walkthrough at <a href="https://www.learnhuggingface.com/notebooks/hugging_face_llm_full_fine_tune_tutorial">learnhuggingface.com</a> as a primary study reference. What you are reading is my own version, restructured into a cleaner, more modular format built for beginners who want a straight line from setup to a working fine-tuned model.</blockquote><h3>What This Is and What You Will Build</h3><p>Pre-trained language models know a lot about language in general. What they do not know is your specific task, your specific output format, or your specific domain. Fine-tuning is the process of taking a pre-trained model and teaching it exactly that.</p><p>In this article, you will fine-tune <strong>Gemma 3 270M</strong>, a small but capable open-source language model from Google, to perform one focused task: extracting food and drink items from raw text and returning them in a clean, structured format.</p><p>The input will be plain text like this:</p><pre>British breakfast with baked beans, fried eggs, sausages, bacon, mushrooms, <br>a cup of tea, toast and fried tomatoes.</pre><p>And the output will be structured like this:</p><pre>food_or_drink: 1<br>tags: fi, di<br>foods: baked beans, fried eggs, sausages, bacon, mushrooms, toast, fried tomatoes<br>drinks: tea</pre><p>This specific task is called structured data extraction. It is one of the most common real-world use cases for fine-tuning: you have unstructured text, you want structured output, and a general-purpose model does not reliably give it to you in the format you need.</p><p>By the end of this article you will have a fine-tuned model saved locally and uploaded to Hugging Face, ready to use in any project.</p><h4>Before You Start</h4><p>You need a GPU for this. Training even a 270M parameter model on CPU is too slow to be practical.</p><p>The options in order of convenience: Google Colab with a T4 GPU (free tier), a local NVIDIA GPU with at least 8GB VRAM, or a cloud instance. If you are using Colab, go to Runtime, then Change Runtime Type, then select GPU before running anything.</p><p>Install the required libraries:</p><pre>pip install transformers trl datasets accelerate gradio</pre><p>And set one environment variable before anything else runs. This prevents tokenizer parallelism warnings that clutter your output without affecting anything:</p><pre>import os<br>os.environ[&quot;TOKENIZERS_PARALLELISM&quot;] = &quot;false&quot;</pre><h3>The Plan: Six Modules, One Straight Line</h3><p>Every step in this article maps to one of six modules. We move through them in order. No jumping ahead.</p><pre>Module 1: Setup and Hardware Check<br>Module 2: Load the Base Model<br>Module 3: Load and Prepare the Dataset<br>Module 4: Configure and Run Training<br>Module 5: Evaluate and Save the Model<br>Module 6: Upload to Hugging Face Hub</pre><p>That is the entire workflow. Let’s build it.</p><h3>Module 1: Setup and Hardware Check</h3><h4>What This Module Does</h4><p>Before touching a model or a dataset, confirm that your hardware is ready and your libraries are imported. A misconfigured runtime is the most common reason beginner fine-tuning runs fail before they start.</p><h4>The Code</h4><pre># ============================================================<br># MODULE 1: SETUP AND HARDWARE CHECK<br># ============================================================<br>import os<br>import torch<br>import transformers<br>import trl<br>import datasets<br><br>os.environ[&quot;TOKENIZERS_PARALLELISM&quot;] = &quot;false&quot;<br># Detect the best available compute backend.<br># CUDA = NVIDIA GPU. MPS = Apple Silicon. CPU = fallback.<br># You genuinely need CUDA or MPS for this to be practical.<br>if torch.cuda.is_available():<br>    DEVICE = &quot;cuda&quot;<br>elif torch.backends.mps.is_available():<br>    DEVICE = &quot;mps&quot;<br>else:<br>    DEVICE = &quot;cpu&quot;<br>print(f&quot;Device: {DEVICE}&quot;)<br># If you are on CUDA, print memory details so you know what you are working with.<br># 8GB minimum is recommended for this model at bfloat16 precision.<br>if DEVICE == &quot;cuda&quot;:<br>    gpu_name = torch.cuda.get_device_name(0)<br>    total_memory = torch.cuda.get_device_properties(0).total_memory / 1e9<br>    print(f&quot;GPU: {gpu_name}&quot;)<br>    print(f&quot;Total VRAM: {total_memory:.2f} GB&quot;)</pre><h4>What to Expect</h4><p>You should see your device printed clearly. If you see cpu, stop and fix your runtime before continuing. Training on CPU for this model will take hours instead of minutes.</p><h3>Module 2: Load the Base Model</h3><h4>What This Module Does</h4><p>A base model is the starting point. It already understands language in general — grammar, word relationships, how sentences are structured. Your fine-tuning run will take that general understanding and redirect it toward a specific task.</p><p>The model we are using is google/gemma-3-270m-it. The it stands for instruction-tuned, meaning it has already been aligned to follow user instructions. This makes it a better starting point than a raw base model for a task that involves following a specific output format.</p><h4>What a Tokenizer Is and Why It Matters</h4><p>The model does not read text. It reads numbers. A tokenizer is the translator between human-readable text and the integer sequences the model actually processes.</p><p>The tokenizer for this model converts:</p><pre>&quot;Hello my name is Daniel&quot;<br>→ [2, 9259, 1041, 1463, 563, 13108]</pre><p>Every model has its own tokenizer tied to its specific vocabulary. Always load the tokenizer that matches the model. Using a mismatched tokenizer will produce garbage output.</p><h4>The Code</h4><pre># ============================================================<br># MODULE 2: LOAD THE BASE MODEL<br># ============================================================<br>from transformers import AutoTokenizer, AutoModelForCausalLM<br><br>MODEL_NAME = &quot;google/gemma-3-270m-it&quot;<br># Load the model. dtype=&quot;auto&quot; lets the model choose the best precision<br># for your hardware (usually bfloat16 on modern GPUs).<br># device_map=&quot;auto&quot; places the model on your GPU automatically.<br>model = AutoModelForCausalLM.from_pretrained(<br>    MODEL_NAME,<br>    dtype=&quot;auto&quot;,<br>    device_map=&quot;auto&quot;,<br>    attn_implementation=&quot;eager&quot;<br>)<br># Load the tokenizer that matches this model exactly.<br>tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)<br>print(f&quot;Model loaded on: {model.device}&quot;)<br>print(f&quot;Model precision: {model.dtype}&quot;)<br># Count the total number of trainable parameters.<br># In full fine-tuning, ALL parameters are updated during training.<br># This is different from LoRA, where only a small adapter is trained.<br>total_params = sum(p.numel() for p in model.parameters())<br>trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)<br>print(f&quot;Total parameters: {total_params:,}&quot;)<br>print(f&quot;Trainable parameters: {trainable_params:,}&quot;)</pre><h4>Full Fine-Tuning vs. LoRA</h4><p>In <strong>full fine-tuning</strong>, every parameter in the model is updated during training. This gives the model the most flexibility to learn your task, but it requires more VRAM and takes longer. At 270M parameters, Gemma 3 270M is small enough to do this comfortably on a consumer GPU.</p><p><strong>LoRA</strong> (Low-Rank Adaptation) is an alternative where you freeze the original model weights and only train a small set of adapter weights. This uses far less memory and is the preferred approach for larger models. For a 270M model with a specific task and a GPU with at least 8GB VRAM, full fine-tuning is the right choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EmzhZoZCDmEIgGPq9dUIuQ.png" /></figure><h3>Module 3: Load and Prepare the Dataset</h3><h4>What This Module Does</h4><p>Training data is what separates a generic model from a task-specific one. The dataset needs three things to be ready for training: it needs to be loaded, it needs to be formatted into the input-output structure the model expects, and it needs to be split into training and test sets.</p><h4>The Dataset</h4><p>We are using mrdbourke/FoodExtract-1k, a dataset of 1,420 samples. Each sample contains a raw text string and a structured label showing what food and drink items should be extracted from it.</p><p>The labels were generated by a large teacher model (gpt-oss-120b) and condensed into a compact format that uses fewer tokens. This compact format is what we will train our model to produce. Fewer output tokens means faster inference at deployment time.</p><h4>What Supervised Fine-Tuning Means</h4><p>Supervised fine-tuning (SFT) means you give the model paired examples of input and ideal output. The model learns to produce outputs like the examples by minimizing the difference between what it generates and what the label says it should generate.</p><p>In our case: input is a raw text string. Output is the structured food extraction format. Given enough examples, the model learns the pattern.</p><h4>The Code</h4><pre># ============================================================<br># MODULE 3: LOAD AND PREPARE THE DATASET<br># ============================================================<br>from datasets import load_dataset<br><br># Load the dataset from Hugging Face Hub.<br>dataset = load_dataset(&quot;mrdbourke/FoodExtract-1k&quot;)<br>print(f&quot;Total samples: {len(dataset[&#39;train&#39;])}&quot;)<br>print(f&quot;Columns: {dataset[&#39;train&#39;].column_names}&quot;)<br># Inspect one sample so you understand the raw structure.<br># The two columns we care about are:<br>#   &#39;sequence&#39;                    → the raw text input<br>#   &#39;gpt-oss-120b-label-condensed&#39; → the structured output we want to produce<br>sample = dataset[&quot;train&quot;][0]<br>print(f&quot;\nInput:\n{sample[&#39;sequence&#39;]}&quot;)<br>print(f&quot;\nTarget output:\n{sample[&#39;gpt-oss-120b-label-condensed&#39;]}&quot;)</pre><h4>Formatting for the Model</h4><p>Language models trained for instruction following expect inputs in a conversational format with roles: user (the input) and assistant (the expected output). This is called a chat template. The model was pre-trained to recognize this structure, so matching it during fine-tuning produces better results than feeding raw text directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qHBF0p7IT-IwC8s0ht_lNw.png" /></figure><pre># Convert each raw sample into the prompt/completion format<br># the model was pre-trained to expect.<br>#<br># &#39;prompt&#39;     → what the user sends (the raw text to extract from)<br># &#39;completion&#39; → what the assistant should respond with (the structured output)<br>#<br># TRL&#39;s SFTTrainer knows how to handle this exact format automatically.<br><br>def format_sample(sample):<br>    return {<br>        &quot;prompt&quot;: [<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: sample[&quot;sequence&quot;]}<br>        ],<br>        &quot;completion&quot;: [<br>            {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: sample[&quot;gpt-oss-120b-label-condensed&quot;]}<br>        ]<br>    }<br>dataset = dataset.map(format_sample, batched=False)<br># Verify the formatting worked.<br>formatted_sample = dataset[&quot;train&quot;][0]<br>print(f&quot;Prompt role:     {formatted_sample[&#39;prompt&#39;][0][&#39;role&#39;]}&quot;)<br>print(f&quot;Completion role: {formatted_sample[&#39;completion&#39;][0][&#39;role&#39;]}&quot;)</pre><h4>Creating the Train and Test Split</h4><pre># Split into 80% training, 20% test.<br># The model trains on the training set and is evaluated on the test set.<br># The test set contains samples the model has never seen during training.<br># This is how we get an honest measure of how well fine-tuning worked.<br>dataset = dataset[&quot;train&quot;].train_test_split(<br>    test_size=0.2,<br>    shuffle=False,   # Keep order consistent for reproducibility<br>    seed=42<br>)<br><br>print(f&quot;Training samples: {len(dataset[&#39;train&#39;])}&quot;)<br>print(f&quot;Test samples:     {len(dataset[&#39;test&#39;])}&quot;)</pre><h3>Module 4: Configure and Run Training</h3><h4>What This Module Does</h4><p>This is where the actual learning happens. You configure the training settings (called hyperparameters), hand everything to a trainer, and let it run.</p><p>The two objects you need are SFTConfig (the settings) and SFTTrainer (the engine that runs training using those settings). SFT stands for Supervised Fine-Tuning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8uJppWdplTLcB_MAqskjhg.png" /></figure><h4>Understanding the Key Hyperparameters</h4><p><strong>num_train_epochs :</strong>how many times the trainer passes through the full training dataset. Three epochs is a reasonable default for a focused task like this. Too few and the model hasn&#39;t learned enough. Too many and it starts memorizing the training data instead of generalizing.</p><p><strong>per_device_train_batch_size</strong> :how many samples the model processes at once before updating its weights. Larger batches use more VRAM. If you hit an out-of-memory error, reduce this number first.</p><p><strong>learning_rate</strong> :how large each weight update step is. Too high and training becomes unstable. Too low and the model barely changes. 5e-5 is a safe starting point for fine-tuning a pre-trained model.</p><p><strong>completion_only_loss</strong> :this tells the trainer to only compute the training loss on the output tokens, not the input tokens. The model only needs to learn how to generate the structured output. Computing loss on the input would confuse this signal.</p><p><strong>max_length</strong> :the maximum number of tokens per training sample. Any sample longer than this is truncated. Set based on the length of your data and your available VRAM.</p><h4>The Code</h4><pre># ============================================================<br># MODULE 4: CONFIGURE AND RUN TRAINING<br># ============================================================<br>from trl import SFTConfig, SFTTrainer<br><br>CHECKPOINT_DIR = &quot;./fine_tuned_model&quot;<br>BATCH_SIZE     = 16   # Reduce to 8 or 4 if you run out of VRAM<br>LEARNING_RATE  = 5e-5<br>NUM_EPOCHS     = 3<br>sft_config = SFTConfig(<br>    # Where to save the model and training checkpoints<br>    output_dir = CHECKPOINT_DIR,<br>    # Training duration<br>    num_train_epochs = NUM_EPOCHS,<br>    # Batch size per GPU<br>    per_device_train_batch_size = BATCH_SIZE,<br>    per_device_eval_batch_size  = BATCH_SIZE,<br>    # Maximum token length per sample (longer samples are truncated)<br>    max_length = 512,<br>    packing    = False,<br>    # Only compute loss on the output (completion) tokens, not the input tokens.<br>    # This focuses the model on learning the output format.<br>    completion_only_loss = True,<br>    # Optimizer - adamw_torch_fused is faster than standard adamw on GPU<br>    optim = &quot;adamw_torch_fused&quot;,<br>    # Learning rate settings<br>    learning_rate     = LEARNING_RATE,<br>    lr_scheduler_type = &quot;constant&quot;,<br>    # Precision: set based on model dtype<br>    # bfloat16 is more stable than float16 for most modern GPUs<br>    bf16 = (model.dtype == torch.bfloat16),<br>    fp16 = (model.dtype == torch.float16),<br>    # Evaluation and checkpointing<br>    eval_strategy  = &quot;epoch&quot;,   # Evaluate after each full pass through the data<br>    save_strategy  = &quot;epoch&quot;,   # Save a checkpoint after each epoch<br>    # Load the best checkpoint at the end of training<br>    load_best_model_at_end   = True,<br>    metric_for_best_model    = &quot;mean_token_accuracy&quot;,<br>    greater_is_better        = True,<br>    # Logging<br>    logging_steps = 10,<br>    # Disable external reporting for simplicity<br>    report_to = &quot;none&quot;,<br>    push_to_hub = False,<br>)</pre><h4>Running the Trainer</h4><pre># Create the trainer object.<br># This is the engine that runs the actual fine-tuning loop.<br>trainer = SFTTrainer(<br>    model            = model,<br>    args             = sft_config,<br>    train_dataset    = dataset[&quot;train&quot;],<br>    eval_dataset     = dataset[&quot;test&quot;],<br>    processing_class = tokenizer,<br>)<br><br># Start training.<br># This is the line that actually trains the model.<br># Watch the training loss - it should trend downward across steps.<br>training_output = trainer.train()<br>print(f&quot;Training complete.&quot;)<br>print(f&quot;Final training loss: {training_output.training_loss:.4f}&quot;)</pre><h4>What to Watch During Training</h4><p>The trainer prints a log every logging_steps steps. The two numbers to focus on are the training loss and the validation loss. Training loss should decrease steadily. Validation loss should follow a similar trend.</p><p>If validation loss starts increasing while training loss keeps decreasing, the model is beginning to overfit to the training data. At that point, load_best_model_at_end=True ensures you end up with the best checkpoint, not the most recently trained one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iIMXZG_Cek-H1rNVHhXXsw.png" /></figure><h3>Module 5: Evaluate and Save the Model</h3><h4>What This Module Does</h4><p>Training is not finished when the loop ends. You need to confirm the model actually learned the task by evaluating it on samples it has never seen, then save it to disk in a reusable format.</p><h4>Evaluating with Metrics</h4><pre># ============================================================<br># MODULE 5: EVALUATE AND SAVE THE MODEL<br># ============================================================<br><br># Run the model across the full test dataset and collect metrics.<br>eval_results = trainer.evaluate()<br>print(f&quot;Evaluation results:&quot;)<br>print(f&quot;  Loss:                {eval_results[&#39;eval_loss&#39;]:.4f}&quot;)<br>print(f&quot;  Mean token accuracy: {eval_results[&#39;eval_mean_token_accuracy&#39;]*100:.2f}%&quot;)<br>print(f&quot;  Best metric:         {trainer.state.best_metric*100:.2f}%&quot;)</pre><p>Mean token accuracy measures what percentage of generated tokens exactly match the target label tokens. Because the output format is highly structured and repetitive (the same format keys appear in every sample), even a model that has partially learned the task will score relatively high here. The more diagnostic check is the manual inspection below.</p><h4>Manual Inspection on Test Samples</h4><p>Numbers do not tell the full story. Always look at actual model outputs on test samples and compare them to the ground truth labels. This is where you catch failure modes that metrics miss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O03CLb1fdAzqSLoMWVt18Q.png" /></figure><pre>from transformers import pipeline<br><br># Load the model into a text-generation pipeline for easy inference.<br>inference_pipeline = pipeline(<br>    &quot;text-generation&quot;,<br>    model     = model,<br>    tokenizer = tokenizer,<br>)<br>def predict(input_text: str, max_new_tokens: int = 256) -&gt; str:<br>    &quot;&quot;&quot;<br>    Run inference on a raw text string.<br>    Formats the input into chat template format, runs the model,<br>    and returns only the generated output - not the input prompt.<br>    &quot;&quot;&quot;<br>    formatted_input = [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: input_text}]<br>    prompt = inference_pipeline.tokenizer.apply_chat_template(<br>        conversation           = formatted_input,<br>        tokenize               = False,<br>        add_generation_prompt  = True,<br>    )<br>    output = inference_pipeline(<br>        text_inputs     = prompt,<br>        max_new_tokens  = max_new_tokens,<br>        disable_compile = True,<br>    )<br>    # Return only the newly generated text, not the input prompt<br>    return output[0][&quot;generated_text&quot;][len(prompt):]<br><br># Inspect 5 random test samples.<br># For each one: print the input, the model&#39;s output, and the ground truth label.<br>import random<br>random.seed(42)<br>test_indices = random.sample(range(len(dataset[&quot;test&quot;])), 5)<br>for i, idx in enumerate(test_indices):<br>    sample = dataset[&quot;test&quot;][idx]<br>    raw_input    = sample[&quot;sequence&quot;]<br>    ground_truth = sample[&quot;gpt-oss-120b-label-condensed&quot;]<br>    model_output = predict(raw_input)<br>    print(f&quot;\n{&#39;=&#39;*55}&quot;)<br>    print(f&quot;Sample {i+1}&quot;)<br>    print(f&quot;{&#39;=&#39;*55}&quot;)<br>    print(f&quot;Input:\n{raw_input[:200]}...&quot;)<br>    print(f&quot;\nModel output:\n{model_output}&quot;)<br>    print(f&quot;\nGround truth:\n{ground_truth}&quot;)</pre><h4>What Good Output Looks Like</h4><p>A well fine-tuned model should produce outputs that closely match the ground truth in structure and content. The format should be consistent across every sample: the same keys in the same order, the same field names, no extra text before or after.</p><p>If the model is occasionally missing a food item or tagging things slightly differently, that is normal for a model trained on 1,000 samples. The format consistency is what matters most. Content accuracy improves with more training data.</p><h4>Saving the Model</h4><pre># Save the best model to disk.<br># trainer.save_model() saves to the output_dir specified in SFTConfig.<br># This includes the model weights, tokenizer, and configuration files.<br>trainer.save_model()<br><br>print(f&quot;Model saved to: {CHECKPOINT_DIR}&quot;)</pre><p>After saving, your checkpoint directory will contain everything needed to reload and use the model: model.safetensors, config.json, tokenizer.json, and supporting files. You can reload it at any time with AutoModelForCausalLM.from_pretrained(CHECKPOINT_DIR).</p><h4>Reloading and Confirming</h4><p>Always reload the model from disk and run one inference call to confirm the saved files are valid before uploading or deploying.</p><pre>from transformers import AutoModelForCausalLM, AutoTokenizer<br><br># Reload from disk<br>reloaded_model = AutoModelForCausalLM.from_pretrained(<br>    CHECKPOINT_DIR,<br>    dtype      = &quot;auto&quot;,<br>    device_map = &quot;auto&quot;,<br>    attn_implementation = &quot;eager&quot;<br>)<br>reloaded_tokenizer = AutoTokenizer.from_pretrained(CHECKPOINT_DIR)<br># Quick sanity check<br>reloaded_pipeline = pipeline(<br>    &quot;text-generation&quot;,<br>    model     = reloaded_model,<br>    tokenizer = reloaded_tokenizer,<br>)<br>test_text   = &quot;A plate with scrambled eggs, avocado toast, and a glass of orange juice.&quot;<br>test_prompt = reloaded_pipeline.tokenizer.apply_chat_template(<br>    conversation          = [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: test_text}],<br>    tokenize              = False,<br>    add_generation_prompt = True,<br>)<br>test_output = reloaded_pipeline(test_prompt, max_new_tokens=128)<br>generated   = test_output[0][&quot;generated_text&quot;][len(test_prompt):]<br>print(f&quot;Input:  {test_text}&quot;)<br>print(f&quot;Output: {generated}&quot;)</pre><p>You should see a correctly formatted extraction output. If you do, the model is saved correctly.</p><h3>Module 6: Upload to Hugging Face Hub</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5tuwvfVw49GIuR2yfq540w.png" /></figure><h4>What This Module Does</h4><p>A model saved only to your local disk disappears when your Colab session ends or your machine resets. Uploading to Hugging Face Hub gives your model a permanent home, makes it reusable across machines, and optionally shareable with anyone.</p><h4>Before You Upload</h4><p>You need a Hugging Face account and an access token. Create one at <a href="https://huggingface.co/">huggingface.co</a>, then generate a token under Settings → Access Tokens. Give it write permissions.</p><p>In Colab, store it under Secrets (the key icon in the left panel) with the name HF_TOKEN. In a local environment, set it as an environment variable.</p><pre>from huggingface_hub import login<br><br># This will prompt for your token if it isn&#39;t already set.<br># In Colab, use userdata.get(&quot;HF_TOKEN&quot;) to read from Secrets.<br>login()</pre><h4>Writing a Model Card</h4><p>A model card is the README for your model. It tells anyone who finds it what the model does, how it was trained, and how to use it. This is not optional — an undocumented model is nearly useless to anyone, including your future self.</p><pre># ============================================================<br># MODULE 6: UPLOAD TO HUGGING FACE HUB<br># ============================================================<br><br>YOUR_HF_USERNAME = &quot;your_username&quot;   # Replace with your actual Hugging Face username<br>MODEL_REPO_ID    = f&quot;{YOUR_HF_USERNAME}/FoodExtract-gemma-3-270m-finetuned&quot;<br>model_card = f&quot;&quot;&quot;---<br>base_model: google/gemma-3-270m-it<br>library_name: transformers<br>tags:<br>  - sft<br>  - trl<br>  - fine-tuned<br>  - food-extraction<br>license: gemma<br>---<br># FoodExtract: Fine-Tuned Gemma 3 270M<br>A fine-tuned version of [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it)<br>trained to extract food and drink items from raw text.<br>## What It Does<br>Given any text input, this model outputs a structured extraction:</pre><p>food_or_drink: 1 tags: fi, di foods: eggs, bacon, toast drinks: orange juice</p><pre>If the input contains no food or drink items:</pre><p>food_or_drink: 0 tags: foods: drinks:</p><pre>## Tag Reference<br><br>| Tag | Meaning |<br>|-----|---------|<br>| np  | nutrition_panel |<br>| il  | ingredient_list |<br>| me  | menu |<br>| re  | recipe |<br>| fi  | food_items |<br>| di  | drink_items |<br>| fa  | food_advertisement |<br>| fp  | food_packaging |<br>## How to Use<br>```python<br>from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline<br>model_id = &quot;{MODEL_REPO_ID}&quot;<br>model     = AutoModelForCausalLM.from_pretrained(model_id, dtype=&quot;auto&quot;, device_map=&quot;auto&quot;)<br>tokenizer = AutoTokenizer.from_pretrained(model_id)<br>pipe      = pipeline(&quot;text-generation&quot;, model=model, tokenizer=tokenizer)<br>input_text = &quot;A bowl of ramen with soft-boiled eggs, nori, and a side of green tea.&quot;<br>formatted  = pipe.tokenizer.apply_chat_template(<br>    [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: input_text}],<br>    tokenize=False,<br>    add_generation_prompt=True<br>)<br>output = pipe(formatted, max_new_tokens=128)<br>print(output[0][&quot;generated_text&quot;][len(formatted):])</pre><h3>Training</h3><ul><li>Base model: google/gemma-3–270m-it</li><li>Dataset: mrdbourke/FoodExtract-1k (1,136 training samples)</li><li>Method: Full supervised fine-tuning (SFT) via Hugging Face TRL</li><li>Epochs: 3</li><li>Batch size: 16</li><li>Learning rate: 5e-5</li></ul><h3>Save the model card to the checkpoint directory.</h3><p>with open(f”{CHECKPOINT_DIR}/README.md”, “w”) as f: f.write(model_card)</p><p>print(“Model card written.”)</p><pre>### Creating the Repository and Uploading<br><br>```python<br>from huggingface_hub import HfApi, create_repo<br>api = HfApi()<br># Create the repository on Hugging Face Hub.<br># exist_ok=True means it will not error if the repo already exists.<br>create_repo(<br>    repo_id  = MODEL_REPO_ID,<br>    repo_type = &quot;model&quot;,<br>    private   = False,<br>    exist_ok  = True,<br>)<br>print(f&quot;Repository ready: {MODEL_REPO_ID}&quot;)<br># Upload the entire checkpoint directory.<br># This includes model weights, tokenizer files, config, and your README.<br>api.upload_folder(<br>    folder_path = CHECKPOINT_DIR,<br>    repo_id     = MODEL_REPO_ID,<br>    repo_type   = &quot;model&quot;,<br>)<br>print(f&quot;Upload complete: https://huggingface.co/{MODEL_REPO_ID}&quot;)</pre><p>Your model is now live at https://huggingface.co/{YOUR_HF_USERNAME}/FoodExtract-gemma-3-270m-finetuned. Anyone with the link can download and use it. You can also load it from any machine in the future with just the model ID.</p><h3>The Complete Flow at a Glance</h3><p>Here is the entire workflow condensed to its essentials — every module, every critical line, in sequence:</p><pre>Module 1: Detect hardware → confirm GPU is available<br>Module 2: Load model + tokenizer → google/gemma-3-270m-it<br>Module 3: Load dataset → format into prompt/completion → split 80/20<br>Module 4: Configure SFTConfig → build SFTTrainer → call trainer.train()<br>Module 5: Evaluate metrics → inspect outputs manually → save model to disk<br>Module 6: Write model card → create HF repo → upload_folder</pre><p>Each module has one clear input and one clear output. None of them depend on anything outside their own section except the objects passed from the previous one.</p><h3>What You Now Have</h3><p>You have a fine-tuned language model. Not a wrapper around an API. Not a prompt-engineered workaround. A model whose weights have been updated to reliably perform a specific task, saved to disk, and living on Hugging Face where you can reload it in one line from anywhere.</p><p>The task in this article was food extraction. The pattern is universal. The same six modules apply to any structured data extraction problem: replace the dataset, adjust the output format in your model card, change the task description in your evaluation function. Everything else stays the same.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc4cdfb156c3" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/fine-tune-your-first-llm-a-guide-with-pytorch-and-hugging-face-bc4cdfb156c3">Fine-Tune Your First LLM: A Guide with PyTorch and Hugging Face</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an LLM from Scratch with Pytorch]]></title>
            <link>https://pub.towardsai.net/building-an-llm-from-scratch-with-pytorch-b211a0d6793d?source=rss-4a5744f8548f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b211a0d6793d</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Bessie Delight Kekeli]]></dc:creator>
            <pubDate>Thu, 25 Jun 2026 12:31:00 GMT</pubDate>
            <atom:updated>2026-06-27T00:38:32.792Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>A module-by-module walkthrough of how language models actually work and built (with code)</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j4MebF1lHyCdJDbhpqoDcQ.png" /></figure><blockquote><strong><em>Who this is for:</em></strong><em> You have used PyTorch at least once, or you are comfortable reading it. You know python and have seen and read PyTorch code before. The rest you will pick up as we build. Lets go!</em></blockquote><p>Github repo : <a href="https://github.com/Bessiedelight/llm-pytorch">Pytorch LLM</a></p><h3>Why Build One Yourself</h3><p>There are two ways to understand how a language model works. The first is to read descriptions of it: attention mechanisms, transformer blocks, token embeddings. The second is to build one yourself, watch it fail, fix the failure, and watch it work. The second way produces a completely different quality of understanding.</p><p>This article takes the second approach. We are going to build a small but complete language model from the ground up using PyTorch. Not a wrapper around an existing model. Not a fine-tuning tutorial. The actual components, assembled in the actual order, with the actual reasons explained at each step.</p><p>By the end, you will have a working model that can generate text. More importantly, you will know why every piece is there, what it is doing, and what breaks if you remove it.</p><p>Before anything: make sure these three imports are at the top of your file. Every code block in this article assumes them.</p><pre>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F</pre><h4>The Big Picture: What a Language Model Actually Does</h4><p>Before writing a single class, you need the right mental model. A language model does one thing: given a sequence of words, it predicts what word comes next.</p><p>That is it. That is the entire job.</p><p>Read the sentence “The cat sat on the”. A language model looks at those five words and assigns a probability to every word in its vocabulary. “Mat” gets a high probability. “Quantum” gets a low one. “Floor” gets a medium one. The model picks the most likely next word, adds it to the sequence, and repeats.</p><p>Everything in this article : the tokenizer, the embeddings, the attention mechanism, the transformer blocks exists to make that one prediction as accurate as possible.</p><p>Here is the architecture we are building, in order:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*saR88ciSYNUEOQ9X_-4dFQ.png" /></figure><p>Each section below builds one row of this diagram. By the time we reach the bottom, we will assemble them into one complete model.</p><h3>Part 1: The Tokenizer</h3><h4>The Problem It Solves</h4><p>Neural networks only understand numbers. They cannot read the word “hello.” They can read the number 8730. A tokenizer is the bridge between human language and machine arithmetic , it converts raw text into a sequence of numbers, and converts sequences of numbers back into text.</p><p>Every language model has a tokenizer. The tokenizer decides what the smallest unit of language is. In this article, we use character-level tokenization: each individual character is one token. This keeps the implementation simple and clear without hiding anything important.</p><h4>How It Works</h4><p>The tokenizer builds a vocabulary: a dictionary that maps every character it has seen to a unique integer. The word “hello” becomes five integers, one per character. To convert back, you reverse the dictionary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cd5zjNUavlRe59h9SclI-w.png" /></figure><pre>class Tokenizer:<br><br>@staticmethod<br>    def build_vocab(text: str) -&gt; dict:<br>        &quot;&quot;&quot;<br>        Scan the text and assign a unique integer to every unique character.<br>        The result is the vocabulary: the complete set of characters this<br>        tokenizer knows how to handle.<br>        A special &lt;unk&gt; token is added for characters seen at inference<br>        time that were not in the training text.<br>        &quot;&quot;&quot;<br>        # sorted() makes the vocabulary deterministic across runs.<br>        # Without sorting, the same text could produce different integer<br>        # assignments each time, making saved models incompatible.<br>        vocab = {<br>            char: idx<br>            for idx, char in enumerate(sorted(set(text)))<br>        }<br>        vocab[&quot;&lt;unk&gt;&quot;] = len(vocab)<br>        return vocab<br>    def __init__(self, vocab: dict):<br>        self.encode_map = {str(k): int(v) for k, v in vocab.items()}<br>        # The decode map is just the encode map reversed.<br>        self.decode_map = {v: k for k, v in self.encode_map.items()}<br>    def encode(self, text: str) -&gt; list[int]:<br>        &quot;&quot;&quot;<br>        Convert a string into a list of integers.<br>        Unknown characters map to the &lt;unk&gt; token rather than crashing.<br>        &quot;&quot;&quot;<br>        return [<br>            self.encode_map.get(char, self.encode_map[&quot;&lt;unk&gt;&quot;])<br>            for char in text<br>        ]<br>    def decode(self, indices: list[int]) -&gt; str:<br>        &quot;&quot;&quot;<br>        Convert a list of integers back into a string.<br>        &quot;&quot;&quot;<br>        return &quot;&quot;.join([<br>            self.decode_map.get(idx, &quot;&lt;unk&gt;&quot;)<br>            for idx in indices<br>        ])</pre><h4>Why Character Level</h4><p>Word-level tokenization has a simpler vocabulary but breaks on any word it has never seen. Character-level tokenization has a tiny vocabulary (roughly 100 characters cover most English text) and handles any input without crashing. For learning purposes, character level is the right choice: the vocabulary fits in memory, the code is transparent, and nothing is hidden by an external library.</p><h4>Trying It</h4><pre>text = &quot;Hello, PyTorch.&quot;<br>vocab = Tokenizer.build_vocab(text)<br>tokenizer = Tokenizer(vocab)<br><br>encoded = tokenizer.encode(&quot;Hello&quot;)<br>print(encoded)   # Something like [3, 4, 8, 8, 11] depending on vocab order<br>decoded = tokenizer.decode(encoded)<br>print(decoded)   # Hello</pre><h3>Part 2: The Embedding Layer</h3><h4>The Problem It Solves</h4><p>After tokenization, each word is a single integer. The problem is that a single integer carries no information about meaning. The integer 42 is not “closer” to the integer 43 than it is to the integer 1. But the word “car” should be conceptually close to the word “vehicle.” Single integers cannot represent that relationship.</p><p>An embedding layer solves this by replacing each integer with a vector: a list of floating-point numbers, typically between 50 and 1024 values long. These vectors are learnable during training, the model adjusts them so that words with similar meanings end up with similar vectors. Similarity between vectors can be measured, and that measurement reflects real semantic similarity between words.</p><h4>What a Lookup Table Is</h4><p>The simplest mental model for an embedding layer is a spreadsheet. It has one row per token in the vocabulary, and one column per dimension of the embedding. When the model sees token number 42, it looks up row 42 of the spreadsheet and extracts that entire row as a vector. That vector is what travels through the rest of the model.</p><p>The values in that spreadsheet start random and get adjusted during training. That is the entire mechanism.</p><h4>Positional Encoding: The Second Problem</h4><p>Here is something subtle that trips up most people the first time: the attention mechanism you will see in the next section has no inherent sense of order. If you fed it “the cat sat” and “sat the cat,” it would see the same tokens and, without extra information, produce the same output. But word order clearly matters.</p><p>Positional encoding fixes this. For each position in the sequence (position 0, position 1, position 2…), we create a learnable vector that represents that position. We add this positional vector to the word embedding vector before anything enters the attention mechanism. Now the model sees both the meaning of the word and where it appears in the sequence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d0gETGfOCecPhcHv9VzO9Q.png" /></figure><pre>class Embedding(nn.Module):<br><br>def __init__(self, vocab_size: int, embedding_dim: int):<br>        &quot;&quot;&quot;<br>        Two embedding tables live here:<br>          self.token_embedding - one row per vocabulary token, learns word meaning<br>          self.position_embedding - one row per position, learns word order<br>        Both tables have the same number of columns (embedding_dim) so they<br>        can be added together.<br>        Args:<br>            vocab_size (int): Total number of unique tokens in the vocabulary.<br>            embedding_dim (int): Size of the vector representing each token.<br>                                 Larger = more expressive, but more expensive.<br>        &quot;&quot;&quot;<br>        super().__init__()<br>        self.token_embedding    = nn.Embedding(vocab_size, embedding_dim)<br>        self.position_embedding = nn.Embedding(vocab_size, embedding_dim)<br>    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:<br>        &quot;&quot;&quot;<br>        Args:<br>            x: Token indices of shape (batch_size, sequence_length)<br>        Returns:<br>            Combined embedding of shape (batch_size, sequence_length, embedding_dim)<br>        &quot;&quot;&quot;<br>        # Token meaning<br>        word_vectors     = self.token_embedding(x)<br>        # Positional order - x serves double duty as position indices<br>        position_vectors = self.position_embedding(x)<br>        # Adding them gives the model both dimensions of information<br>        return word_vectors + position_vectors</pre><h3>Part 3: The Attention Block</h3><h4>The Problem It Solves</h4><p>Reading a sentence is not just about knowing what each word means in isolation. It is about understanding which words relate to which other words. In the sentence “The bank by the river was steep,” the word “bank” means something entirely different than it does in “The bank approved my loan.” The model needs to look at the surrounding words to disambiguate.</p><p>The attention mechanism is how a language model looks at surrounding words. For each word in the sequence, it asks: “which other words in this sequence are most relevant to understanding this word right now?” It then produces a new representation of that word that blends in information from the relevant other words.</p><h4>Query, Key, and Value</h4><p>The attention mechanism uses three concepts with names borrowed from information retrieval.</p><p>The <strong>Query</strong> represents what a word is looking for. The <strong>Key</strong> represents what each word has to offer. The <strong>Value</strong> is the actual information that gets blended in once relevance is established.</p><p>The simplest analogy is a search engine. You type a search query. The engine compares your query against the keys (page titles, metadata) of every page it knows about. The pages with the most relevant keys get their values (actual content) returned to you. Attention does the same thing, but for every word in the sequence simultaneously.</p><p>In practice, Query, Key, and Value are all computed by passing the input through three separate linear layers. They start as the same data and get projected into three different “views” of it.</p><h4>The Mask</h4><p>There is one more piece: the mask. During training, the model sees the full sentence at once. But it is being trained to predict the next word, which means it should only be allowed to look at previous words, not future ones. If it could see future words, it would cheat — the answer would be right there in its context.</p><p>The mask enforces this by setting the attention score between a word and any future word to negative infinity. After the softmax step, negative infinity becomes zero, which means those positions contribute nothing to the output. Each word attends only to itself and to everything before it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jNQV398-JMl6iUKFIcaL6Q.png" /></figure><pre>class AttentionBlock(nn.Module):<br><br>def __init__(self, embedding_dim: int, head_dim: int, context_size: int):<br>        &quot;&quot;&quot;<br>        One attention head. Computes relevance between every pair of tokens<br>        in the sequence and uses that relevance to build enriched representations.<br>        Args:<br>            embedding_dim (int): Size of the input vectors coming in.<br>            head_dim (int): Size of the Query, Key, and Value projections.<br>                            In multi-head attention this is embedding_dim // num_heads.<br>            context_size (int): Maximum sequence length the model can handle.<br>                                 Determines the size of the causal mask.<br>        &quot;&quot;&quot;<br>        super().__init__()<br>        self.query = nn.Linear(embedding_dim, head_dim, bias=False)<br>        self.key   = nn.Linear(embedding_dim, head_dim, bias=False)<br>        self.value = nn.Linear(embedding_dim, head_dim, bias=False)<br>        # The mask is not a learned parameter - it is a fixed triangular matrix.<br>        # register_buffer makes it part of the module (moves to GPU with the model)<br>        # without making it a trainable weight.<br>        ones = torch.ones(context_size, context_size)<br>        self.register_buffer(&quot;mask&quot;, torch.tril(ones))<br>    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:<br>        &quot;&quot;&quot;<br>        Args:<br>            x: Input tensor of shape (batch_size, seq_len, embedding_dim)<br>        Returns:<br>            Enriched representation of shape (batch_size, seq_len, head_dim)<br>        &quot;&quot;&quot;<br>        B, T, C = x.shape<br>        Q = self.query(x)   # What each token is looking for<br>        K = self.key(x)     # What each token has to offer<br>        V = self.value(x)   # The actual content to blend in<br>        # Compute relevance scores between every pair of positions.<br>        # Scaling by head_dim**-0.5 prevents the scores from getting so large<br>        # that the softmax collapses to one-hot distributions (one dominant<br>        # token drowning out everything else).<br>        scores = Q @ K.transpose(-2, -1) * (C ** -0.5)<br>        # Apply the causal mask: future positions become -inf, then 0 after softmax.<br>        scores = scores.masked_fill(self.mask[:T, :T] == 0, float(&quot;-inf&quot;))<br>        # Softmax turns the scores into weights that sum to 1.<br>        weights = F.softmax(scores, dim=-1)<br>        # Blend the Value vectors according to those weights.<br>        return weights @ V</pre><h4>Why This Is the Heart of the Model</h4><p>Every other component in a transformer is supporting this one. The embeddings exist to give attention something meaningful to compare. The positional encoding exists to give attention a sense of order. The feed-forward layer that comes next exists to process what attention discovered. Attention is the mechanism that makes language models different from every other neural network that came before, and it is the piece worth spending extra time understanding before moving on.</p><h3>Part 4: Multi-Head Attention</h3><h4>The Problem It Solves</h4><p>A single attention head looks for one type of relationship between tokens. But language has many simultaneous relationship types. In “She gave him the book she wrote,” one relationship is between “she” and “wrote” (same person). Another relationship is between “him” and “gave” (recipient). A third is between “book” and “wrote” (what she wrote).</p><p>Multi-head attention runs several attention heads in parallel, each with its own Query, Key, and Value weights. Each head specialises in finding different patterns. Their outputs are then concatenated and projected back to the original dimension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hTXVC5AO5QLN6QU33l3HLw.png" /></figure><pre>class MultiHeadAttention(nn.Module):<br><br>def __init__(self, embedding_dim: int, num_heads: int, context_size: int):<br>        &quot;&quot;&quot;<br>        Runs num_heads attention heads in parallel and combines their outputs.<br>        Each head operates on a slice of the embedding dimension (head_dim),<br>        so the total computation stays proportional to embedding_dim regardless<br>        of how many heads are used.<br>        Args:<br>            embedding_dim (int): Size of the input and output vectors.<br>            num_heads (int): Number of parallel attention heads.<br>                             Must divide evenly into embedding_dim.<br>            context_size (int): Maximum sequence length.<br>        &quot;&quot;&quot;<br>        super().__init__()<br>        assert embedding_dim % num_heads == 0, (<br>            f&quot;embedding_dim ({embedding_dim}) must be divisible by &quot;<br>            f&quot;num_heads ({num_heads})&quot;<br>        )<br>        head_dim = embedding_dim // num_heads<br>        # Each head is a full AttentionBlock with its own learnable weights.<br>        # nn.ModuleList registers them properly so PyTorch tracks their parameters.<br>        self.heads = nn.ModuleList([<br>            AttentionBlock(embedding_dim, head_dim, context_size)<br>            for _ in range(num_heads)<br>        ])<br>        # After concatenation, project back to embedding_dim.<br>        # This lets the model mix information across heads.<br>        self.projection = nn.Linear(embedding_dim, embedding_dim)<br>    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:<br>        &quot;&quot;&quot;<br>        Run all heads in parallel, concatenate their outputs, project.<br>        Args:<br>            x: Input of shape (batch_size, seq_len, embedding_dim)<br>        Returns:<br>            Output of shape (batch_size, seq_len, embedding_dim)<br>        &quot;&quot;&quot;<br>        # Each head produces (batch_size, seq_len, head_dim).<br>        # Concatenating along the last dimension gives (batch_size, seq_len, embedding_dim).<br>        concatenated = torch.cat([head(x) for head in self.heads], dim=-1)<br>        return self.projection(concatenated)</pre><h3>Part 5: The Feed-Forward Layer</h3><h4>The Problem It Solves</h4><p>Attention is very good at moving information between positions in the sequence. What it does not do well is process that information once it has been gathered. The feed-forward layer is where that processing happens.</p><p>After attention has figured out “what is relevant to what,” the feed-forward layer applies a learnable non-linear transformation to each position independently. This is where the model learns to reason about the relationships that attention surfaced.</p><p>The structure is always the same: expand the dimension, apply a non-linearity, compress back down. The expansion (typically to four times the embedding dimension) gives the model a larger space to work in during the reasoning step.</p><pre>class FeedForward(nn.Module):<br><br>  def __init__(self, embedding_dim: int, ff_dim: int):<br>          &quot;&quot;&quot;<br>          A two-layer network applied to each position independently.<br>          The expansion to ff_dim (typically 4 * embedding_dim) creates room<br>          for the model to represent complex transformations before compressing<br>          back down. ReLU introduces the non-linearity that lets the network<br>          learn things a pure linear model cannot.<br>          Args:<br>              embedding_dim (int): Input and output dimension.<br>              ff_dim (int): Intermediate dimension. Typically 4 * embedding_dim.<br>          &quot;&quot;&quot;<br>          super().__init__()<br>          self.expand   = nn.Linear(embedding_dim, ff_dim)<br>          self.activate = nn.ReLU()<br>          self.compress = nn.Linear(ff_dim, embedding_dim)<br>      def forward(self, x: torch.Tensor) -&gt; torch.Tensor:<br>          x = self.expand(x)<br>          x = self.activate(x)<br>          x = self.compress(x)<br>          return x</pre><h3>Part 6: The Transformer Block</h3><h3>Assembling the Reasoning Unit</h3><p>A transformer block is the full reasoning unit of the model: multi-head attention, followed by a feed-forward layer, with two important additions: residual connections and layer normalisation.</p><p><strong>Residual connections</strong> (also called skip connections) add the block’s input back to its output. If the attention or feed-forward layer learns something useful, the residual carries it forward. If it learns something harmful, the original signal is preserved. This is what makes it possible to stack many transformer blocks without training becoming impossible.</p><p><strong>Layer normalisation</strong> stabilises the values flowing through the network. Without it, values can grow or shrink uncontrollably across many layers. Normalisation keeps them in a predictable range, which makes training faster and more stable. The standard practice is to normalise the input before passing it to each sub-layer, so the attention and feed-forward layers always receive well-scaled inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nKsqsJOyofJ0akddMXbxXg.png" /></figure><pre>class TransformerBlock(nn.Module):<br><br>  def __init__(<br>          self,<br>          embedding_dim: int,<br>          num_heads: int,<br>          context_size: int,<br>          ff_dim: int<br>      ):<br>          &quot;&quot;&quot;<br>          One complete transformer block: the repeating unit stacked N times<br>          to form the body of the language model.<br>          Processing order (Pre-LN variant, most common today):<br>            1. Normalise → Attention → Add residual<br>            2. Normalise → FeedForward → Add residual<br>          Normalising before the sub-layer (Pre-LN) rather than after<br>          produces more stable gradients during training.<br>          Args:<br>              embedding_dim (int): Model&#39;s core vector dimension.<br>              num_heads (int): Number of attention heads.<br>              context_size (int): Maximum sequence length.<br>              ff_dim (int): Feed-forward intermediate dimension.<br>          &quot;&quot;&quot;<br>          super().__init__()<br>          self.norm_1      = nn.LayerNorm(embedding_dim)<br>          self.attention   = MultiHeadAttention(embedding_dim, num_heads, context_size)<br>          self.norm_2      = nn.LayerNorm(embedding_dim)<br>          self.feed_forward = FeedForward(embedding_dim, ff_dim)<br>      def forward(self, x: torch.Tensor) -&gt; torch.Tensor:<br>          # Step 1: Attention with residual<br>          # Normalise first, run attention, add the result back to the original x.<br>          x = x + self.attention(self.norm_1(x))<br>          # Step 2: Feed-forward with residual<br>          # Normalise first, run feed-forward, add the result back.<br>          x = x + self.feed_forward(self.norm_2(x))<br>          return x</pre><h3>Part 7: The Complete Language Model</h3><h4>Assembling All the Parts</h4><p>Every piece is now built. The language model stacks them in order:</p><p>The embedding layer converts token indices into vectors with positional information. A sequence of transformer blocks processes those vectors, with each block refining the representation by attending to context and reasoning about it. A final layer normalisation stabilises the output. A linear projection (the “language model head”) maps from the embedding dimension to the vocabulary size, producing one score per token. The token with the highest score is the model’s prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jAZN2d56SQscWK_ptvy7CQ.png" /></figure><pre>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F<br>import requests<br><br>class LanguageModel(nn.Module):<br><br>    @staticmethod<br>    def load_shakespeare():<br>        &quot;&quot;&quot;<br>        Downloads Tiny Shakespeare (~1MB) and returns the raw text string.<br>        Cached in memory — call once and pass the result around.<br>        &quot;&quot;&quot;<br>        url = &quot;https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt&quot;<br>        response = requests.get(url)<br>        response.raise_for_status()<br>        print(f&quot;Dataset loaded: {len(response.text):,} characters&quot;)<br>        return response.text<br><br>    def __init__(<br>        self,<br>        vocab_size: int,<br>        embedding_dim: int,<br>        num_heads: int,<br>        num_layers: int,<br>        context_size: int,<br>        ff_dim: int<br>    ):<br>        &quot;&quot;&quot;<br>        A complete decoder-only language model.<br>        Args:<br>            vocab_size (int): Number of unique tokens. Determines embedding table<br>                              size and the output dimension of the LM head.<br>            embedding_dim (int): Core vector dimension throughout the model.<br>                                 All layers communicate using vectors of this size.<br>            num_heads (int): Attention heads per transformer block.<br>                             Must divide evenly into embedding_dim.<br>            num_layers (int): Number of stacked transformer blocks.<br>                              More layers = deeper reasoning, more computation.<br>            context_size (int): Maximum number of tokens the model processes at once.<br>                                The attention mask is sized by this value.<br>            ff_dim (int): Feed-forward intermediate dimension in each block.<br>                          Typically 4 * embedding_dim.<br>        &quot;&quot;&quot;<br>        super().__init__()<br>        self.embedding = Embedding(vocab_size, embedding_dim)<br>        self.blocks = nn.Sequential(*[<br>            TransformerBlock(embedding_dim, num_heads, context_size, ff_dim)<br>            for _ in range(num_layers)<br>        ])<br>        self.final_norm = nn.LayerNorm(embedding_dim)<br>        self.lm_head = nn.Linear(embedding_dim, vocab_size)<br><br>    def forward(<br>        self,<br>        x: torch.Tensor,<br>        targets: torch.Tensor = None<br>    ) -&gt; tuple[torch.Tensor, torch.Tensor | None]:<br>        &quot;&quot;&quot;<br>        Forward pass.<br>        Args:<br>            x: Token indices of shape (batch_size, seq_len)<br>            targets: Optional ground-truth next tokens for training.<br>                     If provided, computes and returns cross-entropy loss.<br>        Returns:<br>            logits: Raw scores of shape (batch_size, seq_len, vocab_size).<br>            loss:   Cross-entropy loss if targets provided, otherwise None.<br>        &quot;&quot;&quot;<br>        x = self.embedding(x)<br>        x = self.blocks(x)<br>        x = self.final_norm(x)<br>        logits = self.lm_head(x)<br><br>        loss = None<br>        if targets is not None:<br>            B, T, V = logits.shape<br>            loss = F.cross_entropy(<br>                logits.view(B * T, V),<br>                targets.view(B * T)<br>            )<br>        return logits, loss<br><br>    @torch.no_grad()<br>    def generate(<br>        self,<br>        prompt: torch.Tensor,<br>        max_new_tokens: int,<br>        context_size: int<br>    ) -&gt; torch.Tensor:<br>        &quot;&quot;&quot;<br>        Autoregressively generate new tokens from a prompt.<br>        Args:<br>            prompt: Starting token indices of shape (1, seq_len)<br>            max_new_tokens: How many tokens to generate<br>            context_size: The model&#39;s maximum context window<br>        Returns:<br>            Full sequence including prompt + generated tokens<br>        &quot;&quot;&quot;<br>        self.eval()<br>        for _ in range(max_new_tokens):<br>            context    = prompt[:, -context_size:]<br>            logits, _  = self(context)<br>            last_logits = logits[:, -1, :]<br>            probs       = F.softmax(last_logits, dim=-1)<br>            next_token  = torch.multinomial(probs, num_samples=1)<br>            prompt      = torch.cat([prompt, next_token], dim=1)<br>        return prompt</pre><h3>Part 8: Training the Model</h3><h4>Putting It All Together</h4><p>Here is a complete, runnable training loop. Each section is commented with what it is doing and why, not just what the code says.</p><pre>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F<br>import requests<br><br># ── Configuration ─────────────────────────────────────────────<br>EMBEDDING_DIM = 128    # Vector size throughout the model<br>NUM_HEADS     = 4      # Must divide evenly into EMBEDDING_DIM<br>NUM_LAYERS    = 4      # Number of stacked transformer blocks<br>CONTEXT_SIZE  = 64     # Maximum tokens the model sees at once<br>FF_DIM        = 512    # Feed-forward intermediate size (4 × EMBEDDING_DIM)<br>BATCH_SIZE    = 32     # Sequences per training step<br>LEARNING_RATE = 3e-4   # Standard starting point for transformers<br>MAX_STEPS     = 5000   # Total training steps<br><br># ── Data preparation ──────────────────────────────────────────<br>raw_text = LanguageModel.load_shakespeare()<br><br>vocab      = Tokenizer.build_vocab(raw_text)<br>tokenizer  = Tokenizer(vocab)<br>vocab_size = len(vocab)<br><br>data       = torch.tensor(tokenizer.encode(raw_text), dtype=torch.long)<br>split      = int(0.9 * len(data))<br>train_data = data[:split]<br>val_data   = data[split:]<br><br>def get_batch(split: str) -&gt; tuple[torch.Tensor, torch.Tensor]:<br>    &quot;&quot;&quot;<br>    Sample a random batch of (input, target) pairs from the dataset.<br>    For each sequence in the batch:<br>      input  = tokens at positions [i : i + CONTEXT_SIZE]<br>      target = tokens at positions [i+1 : i + CONTEXT_SIZE + 1]<br>    The target is the input shifted one position to the right.<br>    This is how next-token prediction is set up: for every token in the<br>    input sequence, the target is the very next token.<br>    &quot;&quot;&quot;<br>    source  = train_data if split == &quot;train&quot; else val_data<br>    indices = torch.randint(len(source) - CONTEXT_SIZE, (BATCH_SIZE,))<br>    x = torch.stack([source[i     : i + CONTEXT_SIZE    ] for i in indices])<br>    y = torch.stack([source[i + 1 : i + CONTEXT_SIZE + 1] for i in indices])<br>    return x, y<br><br># ── Model and optimiser ───────────────────────────────────────<br>model = LanguageModel(<br>    vocab_size    = vocab_size,<br>    embedding_dim = EMBEDDING_DIM,<br>    num_heads     = NUM_HEADS,<br>    num_layers    = NUM_LAYERS,<br>    context_size  = CONTEXT_SIZE,<br>    ff_dim        = FF_DIM,<br>)<br><br>optimiser = torch.optim.AdamW(model.parameters(), lr=LEARNING_RATE)<br>print(f&quot;Model parameters: {sum(p.numel() for p in model.parameters()):,}&quot;)<br><br># ── Training loop ─────────────────────────────────────────────<br>for step in range(MAX_STEPS):<br>    model.train()<br>    x_batch, y_batch = get_batch(&quot;train&quot;)<br>    logits, loss     = model(x_batch, targets=y_batch)<br><br>    optimiser.zero_grad()<br>    loss.backward()<br>    optimiser.step()<br><br>    if step % 500 == 0:<br>        model.eval()<br>        with torch.no_grad():<br>            _, val_loss = model(*get_batch(&quot;val&quot;))<br>        print(f&quot;Step {step:5d} | train loss: {loss.item():.4f} | val loss: {val_loss.item():.4f}&quot;)<br><br># ── Generation ────────────────────────────────────────────────<br>model.eval()<br>prompt_text   = &quot;T&quot;<br>prompt_tokens = torch.tensor(tokenizer.encode(prompt_text)).unsqueeze(0)<br><br>generated = model.generate(<br>    prompt         = prompt_tokens,<br>    max_new_tokens = 200,<br>    context_size   = CONTEXT_SIZE,<br>)<br><br>output_text = tokenizer.decode(generated[0].tolist())<br>print(&quot;\nGenerated text:&quot;)<br>print(output_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1nsHyEYllqdo5D8wzMwWEg.png" /></figure><h3>What You Just Built</h3><p>Let’s be explicit about every piece and what it contributes, now that the full picture is visible.</p><p><strong>The Tokenizer</strong> converts text to integers so the model can process it mathematically, and converts integers back to text so you can read the output. Without it, the model cannot handle language at all.</p><p><strong>The Embedding Layer</strong> converts those integers into vectors that can carry meaning. The token embedding learns what each word means. The positional embedding learns where in the sequence each word appears. Together, they give the model both content and context.</p><p><strong>The Attention Block</strong> figures out which tokens in the sequence are relevant to each other. The causal mask ensures the model only looks backward, not forward, so it cannot cheat during training by peeking at the answer.</p><p><strong>Multi-Head Attention</strong> runs several attention heads in parallel, each specialising in a different type of relationship. This gives the model the ability to track multiple linguistic patterns simultaneously.</p><p><strong>The Feed-Forward Layer</strong> processes the information that attention surfaced. Attention moves information between positions; the feed-forward layer reasons about that information at each position independently.</p><p><strong>The Transformer Block</strong> combines attention, feed-forward, residual connections, and normalisation into one repeating unit. Stacking these blocks is what gives the model its depth of reasoning.</p><p><strong>The Language Model Head</strong> maps the final representation back into vocabulary space, producing a score for every possible next token. The highest score wins.</p><p><strong>The Training Loop</strong> teaches the model to make better predictions by repeatedly sampling sequences, computing how wrong the model’s predictions were, and adjusting every parameter in the model slightly toward being more correct.</p><h3>Part 9: The Same Model Using nn.Transformer</h3><h3>Why This Version Exists</h3><p>You have now built every component by hand. You understand what a Query is, why the mask exists, what a residual connection protects against, and why layer normalisation belongs before the sub-layer rather than after. That knowledge is what makes the next version readable.</p><p>PyTorch ships with a built-in class called nn.Transformer. It packages the multi-head attention and feed-forward layers into a ready-made block. Using it, you can build the same language model in significantly fewer lines. The trade-off is that the internals are hidden behind the API — which is exactly why this version comes second, not first. Reading it without having built the components yourself produces understanding of the shape of the code, not understanding of the mechanism.</p><p>This version uses exactly the same tokenizer, the same embedding layer, the same training loop, and the same generation logic as the from-scratch version. The only thing that changes is the transformer block itself.</p><h3>What nn.Transformer Provides</h3><p>nn.Transformer gives you a complete encoder-decoder transformer. For a language model, which is decoder-only, you use only the decoder side. PyTorch also provides nn.TransformerDecoderLayer and nn.TransformerDecoder for exactly this purpose: a single decoder block and a stack of them, respectively.</p><p>Internally, nn.TransformerDecoderLayer contains the same pieces you built: multi-head attention, a feed-forward network, layer normalisation, and residual connections. The configuration parameters map directly to the ones you already know.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pdxb6tXma8SAcnZ_MeFBhQ.png" /></figure><h3>The Causal Mask</h3><p>The one thing nn.Transformer does not build for you is the causal mask. You hand it in explicitly at the forward pass. PyTorch provides a utility function to generate it: nn.Transformer.generate_square_subsequent_mask(size). This produces the same upper-triangular mask of negative infinity values that your AttentionBlock registered as a buffer — just generated on demand rather than stored on the module.</p><pre># ── The nn.Transformer Version ───────────────────────────────<br><br>class TransformerLanguageModel(nn.Module):<br>    def __init__(<br>        self,<br>        vocab_size: int,<br>        embedding_dim: int,<br>        num_heads: int,<br>        num_layers: int,<br>        context_size: int,<br>        ff_dim: int,<br>        dropout: float = 0.1<br>    ):<br>        &quot;&quot;&quot;<br>        A decoder-only language model built with PyTorch&#39;s nn.TransformerDecoder.<br>        Compared to the from-scratch version:<br>          - The Embedding layer is identical<br>          - nn.TransformerDecoderLayer replaces your TransformerBlock<br>          - nn.TransformerDecoder replaces the nn.Sequential stack of blocks<br>          - The LM head and generation logic are identical<br>          - The causal mask is generated at forward time, not stored as a buffer<br>        Args:<br>            vocab_size (int):    Total number of unique tokens.<br>            embedding_dim (int): Core vector dimension. Called d_model in PyTorch docs.<br>            num_heads (int):     Attention heads per layer. Must divide embedding_dim.<br>            num_layers (int):    Number of stacked decoder layers.<br>            context_size (int):  Maximum sequence length.<br>            ff_dim (int):        Feed-forward intermediate dimension (dim_feedforward).<br>            dropout (float):     Dropout rate applied inside each decoder layer.<br>                                 0.1 is the standard starting value.<br>        &quot;&quot;&quot;<br>        super().__init__()<br>        self.context_size = context_size<br>        # Identical to the from-scratch version<br>        self.embedding = Embedding(vocab_size, embedding_dim)<br>        # One decoder layer: the nn.Transformer equivalent of your TransformerBlock.<br>        # batch_first=True tells PyTorch our tensors are (batch, seq, features)<br>        # rather than the legacy (seq, batch, features) default.<br>        decoder_layer = nn.TransformerDecoderLayer(<br>            d_model         = embedding_dim,<br>            nhead           = num_heads,<br>            dim_feedforward = ff_dim,<br>            dropout         = dropout,<br>            batch_first     = True,<br>        )<br>        # Stack num_layers decoder layers.<br>        # nn.TransformerDecoder is the equivalent of your nn.Sequential block stack.<br>        self.transformer = nn.TransformerDecoder(<br>            decoder_layer = decoder_layer,<br>            num_layers    = num_layers,<br>        )<br>        self.final_norm = nn.LayerNorm(embedding_dim)<br>        self.lm_head    = nn.Linear(embedding_dim, vocab_size)<br>    def forward(<br>        self,<br>        x: torch.Tensor,<br>        targets: torch.Tensor = None<br>    ) -&gt; tuple[torch.Tensor, torch.Tensor | None]:<br>        &quot;&quot;&quot;<br>        Args:<br>            x:       Token indices of shape (batch_size, seq_len)<br>            targets: Optional ground-truth next tokens for loss computation<br>        Returns:<br>            logits: Shape (batch_size, seq_len, vocab_size)<br>            loss:   Cross-entropy loss if targets provided, else None<br>        &quot;&quot;&quot;<br>        B, T = x.shape<br>        x = self.embedding(x)   # (B, T, embedding_dim)<br>        # Generate the causal mask for the current sequence length.<br>        # This is the same triangular mask as in your AttentionBlock,<br>        # expressed as additive negative-infinity values rather than a<br>        # boolean fill. PyTorch&#39;s attention implementation adds this<br>        # directly to the attention scores before softmax.<br>        causal_mask = nn.Transformer.generate_square_subsequent_mask(<br>            T,<br>            device=x.device<br>        )<br>        # nn.TransformerDecoder expects both a target sequence and a memory<br>        # sequence (the encoder&#39;s output in a full encoder-decoder model).<br>        # For a decoder-only language model, we pass the same tensor for both.<br>        # The tgt_mask enforces causality on the self-attention side.<br>        x = self.transformer(<br>            tgt      = x,<br>            memory   = x,<br>            tgt_mask = causal_mask,<br>        )<br>        x      = self.final_norm(x)<br>        logits = self.lm_head(x)<br>        loss = None<br>        if targets is not None:<br>            B, T, V = logits.shape<br>            loss = F.cross_entropy(<br>                logits.view(B * T, V),<br>                targets.view(B * T)<br>            )<br>        return logits, loss<br>    @torch.no_grad()<br>    def generate(<br>        self,<br>        prompt: torch.Tensor,<br>        max_new_tokens: int,<br>        context_size: int<br>    ) -&gt; torch.Tensor:<br>        &quot;&quot;&quot;<br>        Identical generation logic to the from-scratch version.<br>        The interface is the same - swap the model, keep the loop.<br>        &quot;&quot;&quot;<br>        self.eval()<br>        for _ in range(max_new_tokens):<br>            context       = prompt[:, -context_size:]<br>            logits, _     = self(context)<br>            last_logits   = logits[:, -1, :]<br>            probs         = F.softmax(last_logits, dim=-1)<br>            next_token    = torch.multinomial(probs, num_samples=1)<br>            prompt        = torch.cat([prompt, next_token], dim=1)<br>        return prompt</pre><h3>Swapping It In</h3><p>The training loop and generation code from Part 8 require exactly one change: replace LanguageModel(...) with TransformerLanguageModel(...). Every other line stays identical, because the two classes share the same forward signature and the same generate interface.</p><pre># Replace this line in the training setup from Part 8:<br>model = LanguageModel(<br>    vocab_size    = vocab_size,<br>    embedding_dim = EMBEDDING_DIM,<br>    num_heads     = NUM_HEADS,<br>    num_layers    = NUM_LAYERS,<br>    context_size  = CONTEXT_SIZE,<br>    ff_dim        = FF_DIM,<br>)<br><br># With this:<br>model = TransformerLanguageModel(<br>    vocab_size    = vocab_size,<br>    embedding_dim = EMBEDDING_DIM,<br>    num_heads     = NUM_HEADS,<br>    num_layers    = NUM_LAYERS,<br>    context_size  = CONTEXT_SIZE,<br>    ff_dim        = FF_DIM,<br>    dropout       = 0.1,<br>)<br># Everything else in the training loop and generation block is unchanged.</pre><h3>What nn.Transformer Gives You and What It Doesn&#39;t</h3><p>The two versions produce the same architecture and will converge to similar loss values on the same data. The difference is entirely about control and transparency.</p><p>nn.TransformerDecoderLayer is a complete, tested, PyTorch-maintained implementation. It uses PyTorch&#39;s optimised F.scaled_dot_product_attention under the hood, which is meaningfully faster than the manual matrix multiplication in your AttentionBlock, especially on GPU. If you are training something real and do not need to modify the attention internals, the nn.Transformer version is the faster and more practical choice.</p><p>What it does not give you is the ability to change the attention mechanism itself. If you want to experiment with a different masking strategy, a modified scoring function, a different way of combining heads, or any of the architectural variations that appear in modern research, you need the from-scratch version. The internals of nn.TransformerDecoderLayer are not designed to be overridden piecemeal.</p><p>This is the honest trade-off: convenience and speed versus control and transparency. Most application developers want the former. Most researchers need the latter. Knowing how to build both means you can choose rather than being forced.</p><h3>Where to Go From Here</h3><p>This model is small by modern standards, and deliberately so. The architecture is real. The training loop is real. The generation logic is real. What is small is the configuration: the embedding dimension, the number of layers, the context size, the dataset.</p><p>Scale any of those up and you are on the same path that produced every large language model you have heard of. The architecture is not fundamentally different. The ingredients are identical. What changes is size, data, and compute.</p><p>The most productive next experiments, in order of effort:</p><p>Train on a larger and more interesting text file. Shakespeare, a Wikipedia dump, or a domain-specific dataset will produce noticeably different generations and teach you a lot about what the model actually learns.</p><p>Increase the number of layers and the embedding dimension, then observe how the validation loss changes and how long training takes. This gives you a direct intuition for the compute-versus-quality trade-off that governs all of modern AI.</p><p>Add a dropout layer inside the transformer block for regularisation. When the model has more parameters than training data to justify them, dropout prevents overfitting by randomly zeroing activations during training.</p><p>Replace the character-level tokenizer with a subword tokenizer from the tiktoken library. This dramatically increases the practical vocabulary coverage while keeping the vocabulary size manageable, and it is what every production language model uses.</p><p>Build it. Break it. Fix it. Extend it. That is the only path from reading an article to understanding a technology.</p><p>Github repo : <a href="https://github.com/Bessiedelight/llm-pytorch">Pytorch LLM</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b211a0d6793d" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/building-an-llm-from-scratch-with-pytorch-b211a0d6793d">Building an LLM from Scratch with Pytorch</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 5 RAG Architectures and Exactly When to Use Each One in Production]]></title>
            <link>https://pub.towardsai.net/the-5-rag-architectures-and-exactly-when-to-use-each-one-in-production-d73c9acedbf7?source=rss-4a5744f8548f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d73c9acedbf7</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[llamaindex]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Bessie Delight Kekeli]]></dc:creator>
            <pubDate>Thu, 25 Jun 2026 00:01:01 GMT</pubDate>
            <atom:updated>2026-06-25T00:01:01.912Z</atom:updated>
            <content:encoded><![CDATA[<h4>A deep, practical breakdown of every retrieval pattern worth knowing, with LangGraph and LlamaIndex.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o3xu_d-_pUI4feMjiWHpeQ.png" /></figure><blockquote><em>Part 6 of the LangGraph Mental Model series — an expansion of the RAG chapter, going broader and deeper across the retrieval landscape that production systems live in today.</em></blockquote><h3>Before We Begin: What This Article Is Really About</h3><p>Part 4 of this series introduced you to one specific RAG pattern: load documents, build a LlamaIndex VectorStoreIndex, wrap the QueryEngine as a @tool, and hand it to a LangGraph agent. That pattern works, and it works well for the problems it was designed to solve.</p><p>But the word “RAG” today covers a family of meaningfully different architectures, each built to solve a different class of problem. Using the wrong one is not just a performance issue. It is the difference between a system that works and one that quietly gives your users confident, wrong answers at scale.</p><p>This article maps the entire family. By the end of it, you will be able to look at any retrieval problem and know, without guessing, which architecture it calls for — and how to build it using LangGraph and LlamaIndex.</p><p>Here is what we will cover, in order from simplest to most complex:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ckLp77hjBFi3ZcUCvzlDsQ.png" /></figure><p>One more thing to carry with you through this entire article: these five architectures are not competitors. They are layers you add progressively as your problem demands more. Most production systems combine at least two of them.</p><h3>The Problem That All Five Architectures Are Solving</h3><p>Every RAG system exists to answer one fundamental question: <em>how do you give a language model access to knowledge it was never trained on, at the moment it needs it, in the right form for it to reason over?</em></p><p>Training data has a cutoff. It has no memory of your company’s internal documents, your product specifications, or anything written after the model was frozen. Fine-tuning on that data is expensive, slow, and produces a model that still cannot update when the documents change.</p><p>RAG sidesteps the entire problem. Rather than teaching the model new knowledge, you retrieve the relevant knowledge at query time and include it in the prompt as context. The model never needed to memorize it — it just needs to read it when it matters.</p><p>The five architectures in this article are five different answers to the question of <em>how to retrieve well</em>. Each answer is better suited to a different retrieval problem.</p><h3>Architecture 1: Naive RAG</h3><h3>The Concept</h3><p>Naive RAG is the baseline. It is the architecture that Part 4 of this series taught, and it is the right architecture for a large class of real problems — internal policy bots, FAQ assistants, documentation search tools, onboarding helpers. Do not let the word “naive” mislead you. This is a well-understood, well-tested production pattern used at scale today.</p><p>The pipeline has five sequential steps, and they map directly to what LlamaIndex gives you out of the box:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u9KurjQ8vwZejlDyyyYt3Q.png" /></figure><pre>  Stages 1–4: Indexing time (run once)<br>  Stage 5:    Query time (run on every user question)</pre><p>The single most important distinction in all of RAG is between indexing time and query time. You pay the embedding cost once, up front. At query time, you are only paying for one similarity search and one LLM call.</p><h3>Why It Works</h3><p>When a user asks a question, their question is embedded into the same vector space as your document chunks. The chunks whose vectors sit geometrically closest to the question vector are the ones returned. The assumption is that semantic similarity in vector space corresponds to relevance in meaning. For clean, factual document corpora, this assumption holds surprisingly well.</p><h3>The Code</h3><pre># ============================================================<br># NAIVE RAG — COMPLETE TEMPLATE<br># ============================================================<br># ── MODULE 1: IMPORTS &amp; CONFIGURATION ───────────────────────<br>import os<br>from typing import Literal<br>from langchain_openai import ChatOpenAI<br>from langchain_core.messages import HumanMessage, SystemMessage<br>from langchain_core.tools import tool<br>from langgraph.graph import StateGraph, MessagesState, START, END<br>from langgraph.prebuilt import ToolNode<br>from langgraph.checkpoint.memory import MemorySaver<br>from llama_index.core import (<br>    Settings, SimpleDirectoryReader, VectorStoreIndex,<br>    StorageContext, load_index_from_storage<br>)<br>from llama_index.llms.openai import OpenAI as LlamaOpenAI<br>from llama_index.embeddings.openai import OpenAIEmbedding<br># LangGraph&#39;s reasoning model<br>llm = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br># LlamaIndex&#39;s internal model - separate from the above<br># These two model configs do NOT share state or conversation history<br>Settings.llm = LlamaOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0.1)<br>Settings.embed_model = OpenAIEmbedding(model=&quot;text-embedding-3-small&quot;)<br>Settings.chunk_size = 512<br>Settings.chunk_overlap = 50<br># ── MODULE 2: STATE ──────────────────────────────────────────<br>class State(MessagesState):<br>    pass  # messages list inherited; extend if you need more<br># ── MODULE 3: KNOWLEDGE BASE (Build once, persist, reload) ───<br>PERSIST_DIR = &quot;./storage/naive&quot;<br>if os.path.exists(PERSIST_DIR):<br>    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)<br>    index = load_index_from_storage(storage_context)<br>else:<br>    documents = SimpleDirectoryReader(&quot;./data&quot;).load_data()<br>    index = VectorStoreIndex.from_documents(documents, show_progress=True)<br>    index.storage_context.persist(persist_dir=PERSIST_DIR)<br>query_engine = index.as_query_engine(similarity_top_k=3)<br># ── MODULE 3 (cont.): TOOL ───────────────────────────────────<br>@tool<br>def search_knowledge_base(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search internal company documents for policies, product specs,<br>    and procedures. Use this for any question requiring domain-specific<br>    knowledge rather than general world knowledge.&quot;&quot;&quot;<br>    response = query_engine.query(query)<br>    return str(response)<br>tools = [search_knowledge_base]<br>llm_with_tools = llm.bind_tools(tools)<br>tool_node = ToolNode(tools)<br># ── MODULE 4: NODES ──────────────────────────────────────────<br>def agent_node(state: State) -&gt; dict:<br>    system_prompt = SystemMessage(content=(<br>        &quot;You are a helpful assistant with access to an internal knowledge base. &quot;<br>        &quot;Use search_knowledge_base for company-specific questions. &quot;<br>        &quot;Answer general questions directly.&quot;<br>    ))<br>    response = llm_with_tools.invoke([system_prompt] + state[&quot;messages&quot;])<br>    return {&quot;messages&quot;: [response]}<br># ── MODULE 5: ROUTING ────────────────────────────────────────<br>def should_continue(state: State) -&gt; Literal[&quot;tools&quot;, &quot;__end__&quot;]:<br>    last = state[&quot;messages&quot;][-1]<br>    if hasattr(last, &quot;tool_calls&quot;) and last.tool_calls:<br>        return &quot;tools&quot;<br>    return &quot;__end__&quot;<br># ── MODULE 6: GRAPH ASSEMBLY ─────────────────────────────────<br>builder = StateGraph(State)<br>builder.add_node(&quot;agent&quot;, agent_node)<br>builder.add_node(&quot;tools&quot;, tool_node)<br>builder.add_edge(START, &quot;agent&quot;)<br>builder.add_conditional_edges(&quot;agent&quot;, should_continue,<br>    {&quot;tools&quot;: &quot;tools&quot;, &quot;__end__&quot;: END})<br>builder.add_edge(&quot;tools&quot;, &quot;agent&quot;)<br>graph = builder.compile(checkpointer=MemorySaver())</pre><h3>When Naive RAG Breaks</h3><p>Naive RAG makes one assumption that fails in two common situations.</p><p>The first is terminology mismatch. A user asks: <em>“What’s the SLA for tier-1 clients?”</em> The document says: <em>“Gold-tier customers are guaranteed a 99.9% uptime commitment.”</em> The words SLA, tier-1, and Gold-tier are semantically close but not identical. Vector similarity may not rank this chunk highly enough, and the answer gets missed.</p><p>The second is relational questions. A user asks: <em>“Which of our products are affected if Supplier X goes offline?”</em> Answering this requires traversing a chain of relationships across multiple documents. No single chunk answers it. Naive RAG returns chunks from different documents with no way to connect them.</p><p>These two failure modes are exactly what the next two architectures solve.</p><h3>Architecture 2: Hybrid RAG</h3><h3>The Concept</h3><p>Hybrid RAG acknowledges a truth that practitioners discovered in production: semantic similarity and exact keyword match are complementary, not competing, signals of relevance. Neither one alone is sufficient.</p><p>Dense retrieval (vector search) is excellent at finding semantic equivalents — it will find the “Gold-tier uptime commitment” document when you ask about “SLA.” But it struggles when the query contains proper nouns, product codes, medical terms, legal citations, or any highly specific terminology that carries precise meaning in its exact form.</p><p>Sparse retrieval (BM25/keyword search) is the opposite. It is brilliant at exact term matching — it will always find “SKU-4829” if you search for “SKU-4829.” But it has no concept of semantic equivalence. It will not find “uptime guarantee” when you search for “SLA.”</p><p>Hybrid RAG runs both searches in parallel, then uses a reranker to produce a single, unified ranked list from the merged results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LpLmmztb7RX8OyLcIU67Lw.png" /></figure><p>The reranker is the critical piece here. Unlike embedding models, which compare a query and a chunk independently, a cross-encoder reranker reads the query and each candidate chunk <em>together</em> and produces a relevance score that reflects their relationship directly. It is slower than similarity search, but it is applied only to the already-reduced candidate set, keeping latency manageable.</p><h3>The Code</h3><pre># ============================================================<br># HYBRID RAG — LLAMAINDEX IMPLEMENTATION<br># pip install llama-index-retrievers-bm25 rank-bm25<br># ============================================================<br><br>from llama_index.core import SimpleDirectoryReader, VectorStoreIndex<br>from llama_index.core.retrievers import VectorIndexRetriever<br>from llama_index.retrievers.bm25 import BM25Retriever<br>from llama_index.core.retrievers import QueryFusionRetriever<br>from llama_index.core.query_engine import RetrieverQueryEngine<br>from langchain_core.tools import tool<br># Build document store and index as before<br>documents = SimpleDirectoryReader(&quot;./data&quot;).load_data()<br>index = VectorStoreIndex.from_documents(documents)<br># ── DENSE RETRIEVER (vector similarity) ──────────────────────<br>vector_retriever = VectorIndexRetriever(<br>    index=index,<br>    similarity_top_k=5,  # Fetch more candidates pre-rerank<br>)<br># ── SPARSE RETRIEVER (BM25 keyword match) ────────────────────<br># BM25Retriever builds directly from the index&#39;s nodes<br>bm25_retriever = BM25Retriever.from_defaults(<br>    index=index,<br>    similarity_top_k=5,<br>)<br># ── FUSION: Merge both retrievers with Reciprocal Rank Fusion ─<br># QueryFusionRetriever is LlamaIndex&#39;s built-in hybrid combiner<br># mode=&quot;reciprocal_rerank&quot; implements the RRF algorithm:<br># it combines ranked lists without needing score calibration<br>hybrid_retriever = QueryFusionRetriever(<br>    retrievers=[vector_retriever, bm25_retriever],<br>    similarity_top_k=3,        # Final top-k after fusion<br>    num_queries=1,             # No query expansion in this mode<br>    mode=&quot;reciprocal_rerank&quot;,  # The fusion algorithm<br>    use_async=True,            # Run both retrievers in parallel<br>)<br># Build a QueryEngine on top of the hybrid retriever<br>hybrid_query_engine = RetrieverQueryEngine.from_args(<br>    retriever=hybrid_retriever,<br>)<br># ── LANGGRAPH TOOL ────────────────────────────────────────────<br>@tool<br>def search_hybrid(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search internal knowledge base using both semantic similarity<br>    and keyword matching. More precise than pure vector search,<br>    especially for technical terms, product codes, and exact names.&quot;&quot;&quot;<br>    response = hybrid_query_engine.query(query)<br>    return str(response)<br># The LangGraph graph structure is identical to Naive RAG.<br># You are only swapping the tool. Everything from Module 4 onward<br># stays exactly the same.</pre><h3>A Note on Chunk Size in Hybrid RAG</h3><p>With hybrid retrieval, you will often want to index with smaller chunks than in Naive RAG. Smaller chunks make BM25 matching more precise because a high-frequency term in a small chunk is a stronger signal of relevance than the same term in a large chunk. A setting of 256 tokens with 30-token overlap is a reasonable starting point when BM25 is in the mix.</p><h3>When to Use Hybrid RAG</h3><p>Use Hybrid RAG any time your documents contain a mix of free-form prose and structured terminology. This covers nearly every serious enterprise use case: legal document review (where citation forms must match exactly), medical records (where drug names, dosage codes, and ICD codes are precise), financial analysis (ticker symbols, contract clause identifiers), and technical documentation (error codes, API method names, version numbers).</p><h3>Architecture 3: Graph RAG</h3><h3>The Concept</h3><p>Graph RAG is a fundamentally different way of thinking about what a “document” is. In Naive and Hybrid RAG, a document is a blob of text, and retrieval finds the blobs whose text is most relevant to your query. In Graph RAG, a document is a set of <em>entities</em> and <em>relationships</em>, and retrieval follows a path through a network.</p><p>Consider this question: <em>“Which of our enterprise clients would be affected if we deprecated the legacy authentication module?”</em></p><p>A naive retrieval system would search for chunks that mention “enterprise clients” and “authentication module” together. It might find a few. But what you actually need is to traverse a chain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cyWl3Rqn-6QfD0SAB5trWw.png" /></figure><p>No single document chunk contains that answer. The answer <em>emerges from the structure</em> of the knowledge graph. This is what Graph RAG is built for: multi-hop reasoning, relationship tracing, and questions whose answers require connecting facts that live in different parts of your corpus.</p><h3>How Graph RAG Builds Its Index</h3><p>Instead of chunking documents and embedding the chunks, Graph RAG runs an entity extraction pass over all documents first. It identifies named entities (products, people, organizations, concepts) and the relationships between them (“depends on,” “is a client of,” “is authored by,” “was superseded by”). These become nodes and edges in a knowledge graph. The graph is then organized into communities of closely related entities using graph clustering algorithms, and each community gets a summary written by an LLM. At query time, the system searches community summaries and then traverses the graph to find relevant entities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vFUB5elTtiwYxgX70TfH9w.png" /></figure><h3>The Code</h3><pre># ============================================================<br># GRAPH RAG — LLAMAINDEX PROPERTY GRAPH INDEX<br># pip install llama-index-core<br># ============================================================<br>from llama_index.core import SimpleDirectoryReader, PropertyGraphIndex<br>from llama_index.core.indices.property_graph import (<br>    ImplicitPathExtractor,<br>    SimpleLLMPathExtractor,<br>)<br>from langchain_core.tools import tool<br># Load documents<br>documents = SimpleDirectoryReader(&quot;./data&quot;).load_data()<br># ── BUILD THE PROPERTY GRAPH INDEX ───────────────────────────<br># LlamaIndex&#39;s PropertyGraphIndex handles the full extraction pipeline.<br># SimpleLLMPathExtractor uses an LLM to extract (subject, relation, object)<br># triples from each chunk - these become graph edges.<br># ImplicitPathExtractor uses fast heuristics (cheaper, less precise).<br>index = PropertyGraphIndex.from_documents(<br>    documents,<br>    kg_extractors=[<br>        # LLM-based extraction: higher quality, higher cost<br>        SimpleLLMPathExtractor(<br>            llm=Settings.llm,<br>            max_paths_per_chunk=10,<br>        ),<br>        # Heuristic extraction: fast fallback<br>        ImplicitPathExtractor(),<br>    ],<br>    show_progress=True,<br>)<br># ── CREATE A GRAPH-AWARE RETRIEVER ────────────────────────────<br># This retriever traverses the graph rather than searching flat vectors.<br>kg_retriever = index.as_retriever(<br>    include_text=True,       # Include surrounding text with each entity<br>    retriever_mode=&quot;hybrid&quot;, # Combines keyword + embedding on the graph<br>    similarity_top_k=3,<br>)<br>from llama_index.core.query_engine import RetrieverQueryEngine<br>graph_query_engine = RetrieverQueryEngine.from_args(retriever=kg_retriever)<br># ── LANGGRAPH TOOL ────────────────────────────────────────────<br>@tool<br>def search_knowledge_graph(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search the knowledge graph for questions involving relationships<br>    between entities - dependencies, organizational hierarchies, impact<br>    analysis, and multi-hop reasoning across connected information.<br>    Use this when the answer requires tracing a chain of relationships<br>    rather than finding a single relevant document.&quot;&quot;&quot;<br>    response = graph_query_engine.query(query)<br>    return str(response)<br># ── COMBINING WITH NAIVE RAG: DUAL-TOOL AGENT ─────────────────<br># In practice, Graph RAG and Naive RAG are often combined.<br># The agent&#39;s LLM decides which tool fits the query.<br>@tool<br>def search_knowledge_base(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search for factual information in company documents.<br>    Best for direct questions with answers in a single document.&quot;&quot;&quot;<br>    response = query_engine.query(query)  # The naive query engine<br>    return str(response)<br># Give the LangGraph agent BOTH tools.<br># It will route to the right one based on the question type.<br>tools = [search_knowledge_base, search_knowledge_graph]<br>llm_with_tools = llm.bind_tools(tools)<br>tool_node = ToolNode(tools)<br># Graph assembly remains identical - only the tool list changes.</pre><h3>A Critical Cost Warning</h3><p>Graph RAG’s entity extraction phase runs an LLM call over every chunk in your corpus. For a large document set, this means thousands of LLM calls at indexing time. This is intentionally expensive and slow — you are paying a one-time indexing cost for a much richer data structure. Do not build a Graph RAG index on every startup. Always persist the graph and reload it, exactly as shown for Naive RAG in Part 4.</p><h3>When to Use Graph RAG</h3><p>Graph RAG is the right architecture when your questions require following chains of relationships: compliance and risk analysis (“which processes are affected by regulation X”), supply chain intelligence (“what products depend on this supplier”), organizational knowledge (“who owns what, and how do those ownership chains connect”), and software dependency mapping (“what breaks if we remove module Y”).</p><h3>Architecture 4: Advanced RAG</h3><h3>The Concept</h3><p>Advanced RAG is not a single new technique. It is a structured set of improvements that sit on top of whatever base retrieval mechanism you are already using. Where Naive RAG trusts its first retrieval pass, Advanced RAG questions it, refines it, and validates it.</p><p>There are three categories of improvement, and they slot into different parts of the pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*9a-ISX3lQwmQIKgSrzwciw.png" /></figure><h3>Pre-Retrieval: Query Rewriting and HyDE</h3><p>The query a user types is rarely the optimal search query. “What’s the deal with our returns policy for enterprise?” is a perfectly natural human question that will retrieve worse results than “enterprise customer return and refund policy procedures.” Query rewriting uses the LLM to transform the user’s natural language question into a better search query before hitting the index.</p><p>HyDE (Hypothetical Document Embedding) takes a different approach: instead of searching with the question, it asks the LLM to generate a hypothetical document that <em>would answer</em> the question, then embeds that document to search. The insight is that an answer-shaped text will sit closer in vector space to other answer-shaped texts than a question-shaped text will.</p><h3>Query Decomposition</h3><p>Multi-step questions fail Naive RAG because they require multiple retrievals. Advanced RAG decomposes them first.</p><p>“Compare our refund policy for enterprise and retail customers, and summarize the key differences” is not one question. It is three: retrieve enterprise policy, retrieve retail policy, compare them. Query decomposition breaks this into sub-queries, retrieves against each, and merges the results before synthesis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9cElpURJNE-ypBGKojRyiw.png" /></figure><h3>Post-Retrieval: Reranking and CRAG</h3><p>Reranking was covered in Hybrid RAG above. The same cross-encoder reranker applies here as a post-retrieval step over whatever chunks were retrieved, even if you only used dense retrieval.</p><p>CRAG (Corrective RAG) is the most sophisticated post-retrieval technique. After retrieval, it runs a lightweight evaluation: are the retrieved chunks actually relevant to the question? If the evaluator judges them insufficient, CRAG falls back to an alternative source (web search, a broader index) rather than forcing the LLM to answer from poor context.</p><h3>The Code</h3><pre># ============================================================<br># ADVANCED RAG — MULTI-TECHNIQUE IMPLEMENTATION<br># ============================================================<br><br>from llama_index.core import SimpleDirectoryReader, VectorStoreIndex<br>from llama_index.core.query_engine import (<br>    SubQuestionQueryEngine,    # Handles query decomposition<br>    RetrieverQueryEngine,<br>)<br>from llama_index.core.tools import QueryEngineTool, ToolMetadata<br>from llama_index.core.postprocessor import SentenceTransformerRerank<br>from langchain_core.tools import tool<br>documents = SimpleDirectoryReader(&quot;./data&quot;).load_data()<br>index = VectorStoreIndex.from_documents(documents)<br># ── RERANKER: Post-retrieval cross-encoder scoring ────────────<br># Fetch 8 candidates, rerank down to 3<br># This is the most impactful single improvement in Advanced RAG<br>reranker = SentenceTransformerRerank(<br>    model=&quot;cross-encoder/ms-marco-MiniLM-L-2-v2&quot;,<br>    top_n=3,<br>)<br># ── RETRIEVER WITH RERANKING ──────────────────────────────────<br>base_retriever = index.as_retriever(similarity_top_k=8)<br>reranked_engine = RetrieverQueryEngine.from_args(<br>    retriever=base_retriever,<br>    node_postprocessors=[reranker],  # Applied after retrieval<br>)<br># ── QUERY DECOMPOSITION with SubQuestionQueryEngine ───────────<br># Wrap the base engine as a &quot;tool&quot; that the decomposer can call<br>engine_tools = [<br>    QueryEngineTool(<br>        query_engine=reranked_engine,<br>        metadata=ToolMetadata(<br>            name=&quot;company_knowledge_base&quot;,<br>            description=(<br>                &quot;Searches company documents for policies, procedures, &quot;<br>                &quot;and product information.&quot;<br>            ),<br>        ),<br>    )<br>]<br># SubQuestionQueryEngine decomposes complex questions into<br># sub-questions, runs each against the available tools,<br># and synthesizes a final answer from all sub-answers<br>decomposed_engine = SubQuestionQueryEngine.from_defaults(<br>    query_engine_tools=engine_tools,<br>    use_async=True,  # Sub-queries run in parallel when possible<br>)<br># ── HYDE: Hypothetical Document Embedding ─────────────────────<br>from llama_index.core.indices.query.query_transform.base import (<br>    HyDEQueryTransform,<br>)<br>from llama_index.core.query_engine import TransformQueryEngine<br>hyde_transform = HyDEQueryTransform(include_original=True)<br>hyde_engine = TransformQueryEngine(<br>    query_engine=reranked_engine,<br>    query_transform=hyde_transform,<br>)<br># ── LANGGRAPH TOOLS: Different engines for different patterns ──<br>@tool<br>def search_with_decomposition(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search for answers to complex questions that may require<br>    combining information from multiple documents. Automatically<br>    breaks the question into sub-questions and merges the results.<br>    Best for comparison questions, multi-part questions, and anything<br>    requiring synthesis across different topics.&quot;&quot;&quot;<br>    response = decomposed_engine.query(query)<br>    return str(response)<br>@tool<br>def search_with_hyde(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search the knowledge base using hypothetical document embedding.<br>    More effective than standard search for abstract or exploratory<br>    questions where the exact terminology in the answer differs from<br>    the terminology in the question.&quot;&quot;&quot;<br>    response = hyde_engine.query(query)<br>    return str(response)<br># The LangGraph agent now has specialized retrieval tools<br># and its LLM decides which retrieval strategy each question needs.<br>tools = [search_with_decomposition, search_with_hyde]</pre><h3>The Golden Rule of Advanced RAG</h3><p>Do not implement all of these techniques at once. Start with the single intervention most likely to help your specific failure mode. Reranking is almost always the highest-value first addition. Query decomposition is second. HyDE is a good third step for conceptual or abstract corpora. Add complexity incrementally, and measure recall after each addition.</p><h3>Architecture 5: Agentic RAG</h3><h3>The Concept</h3><p>Agentic RAG is the architecture that this entire series has been building toward. It does not just improve on how you retrieve — it changes <em>who makes the retrieval decisions</em>.</p><p>In all four previous architectures, retrieval is a pipeline: a fixed, predetermined sequence of operations that runs the same way every time. In Agentic RAG, retrieval is a loop: an LLM agent that decides what to search for, evaluates what it found, decides whether to search again, and keeps going until it has enough to answer — or until it determines the answer is unanswerable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DgK03nlYi64JP7-ehQ0iCw.png" /></figure><p>This is exactly what LangGraph was designed to do. The agent node, the tool node, the conditional edge — the entire seven-module structure from Part 1 of this series is an Agentic RAG scaffold. What changes is the richness of the tool suite you give it and the sophistication of the routing logic you build around it.</p><h3>Agentic RAG with Multiple Retrieval Tools</h3><p>The real power of Agentic RAG in LangGraph is that you can give the agent access to every retrieval strategy discussed in this article simultaneously. The agent’s LLM decides which tool to use for each sub-question.</p><pre># ============================================================<br># AGENTIC RAG — COMPLETE MULTI-TOOL TEMPLATE<br># This is the full production scaffold: all five architectures<br># available to a single agent, which selects dynamically.<br># ============================================================<br><br># ── MODULE 1: IMPORTS &amp; CONFIGURATION ───────────────────────<br>import os<br>from typing import Literal, Annotated<br>from langchain_openai import ChatOpenAI<br>from langchain_core.messages import HumanMessage, SystemMessage, BaseMessage<br>from langchain_core.tools import tool<br>from langgraph.graph import StateGraph, MessagesState, START, END<br>from langgraph.prebuilt import ToolNode<br>from langgraph.checkpoint.memory import MemorySaver<br>from llama_index.core import (<br>    Settings, SimpleDirectoryReader, VectorStoreIndex,<br>    StorageContext, load_index_from_storage, PropertyGraphIndex<br>)<br>from llama_index.core.indices.property_graph import SimpleLLMPathExtractor<br>from llama_index.core.postprocessor import SentenceTransformerRerank<br>from llama_index.core.query_engine import RetrieverQueryEngine, SubQuestionQueryEngine<br>from llama_index.core.tools import QueryEngineTool, ToolMetadata<br>from llama_index.llms.openai import OpenAI as LlamaOpenAI<br>from llama_index.embeddings.openai import OpenAIEmbedding<br>from llama_index.retrievers.bm25 import BM25Retriever<br>from llama_index.core.retrievers import QueryFusionRetriever, VectorIndexRetriever<br># The agent&#39;s main reasoning model (LangGraph)<br>llm = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br># LlamaIndex internal configuration<br>Settings.llm = LlamaOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0.1)<br>Settings.embed_model = OpenAIEmbedding(model=&quot;text-embedding-3-small&quot;)<br>Settings.chunk_size = 512<br>Settings.chunk_overlap = 50<br># ── MODULE 2: STATE ──────────────────────────────────────────<br>class AgentState(MessagesState):<br>    # Extend with retrieval metadata if you need observability<br>    retrieval_count: int = 0  # Track how many retrieval calls were made<br># ── MODULE 3: KNOWLEDGE BASES (all built at startup) ──────────<br>VECTOR_DIR = &quot;./storage/vector&quot;<br>GRAPH_DIR  = &quot;./storage/graph&quot;<br>documents  = SimpleDirectoryReader(&quot;./data&quot;).load_data()<br># Vector index (for Naive, Hybrid, Advanced)<br>if os.path.exists(VECTOR_DIR):<br>    ctx = StorageContext.from_defaults(persist_dir=VECTOR_DIR)<br>    vector_index = load_index_from_storage(ctx)<br>else:<br>    vector_index = VectorStoreIndex.from_documents(documents, show_progress=True)<br>    vector_index.storage_context.persist(persist_dir=VECTOR_DIR)<br># Property Graph index (for Graph RAG)<br>if os.path.exists(GRAPH_DIR):<br>    ctx = StorageContext.from_defaults(persist_dir=GRAPH_DIR)<br>    graph_index = load_index_from_storage(ctx)<br>else:<br>    graph_index = PropertyGraphIndex.from_documents(<br>        documents,<br>        kg_extractors=[SimpleLLMPathExtractor(llm=Settings.llm)],<br>        show_progress=True,<br>    )<br>    graph_index.storage_context.persist(persist_dir=GRAPH_DIR)<br># ── Set up individual engines ─────────────────────────────────<br>reranker = SentenceTransformerRerank(<br>    model=&quot;cross-encoder/ms-marco-MiniLM-L-2-v2&quot;, top_n=3<br>)<br># Tool 1: Naive / vector search<br>vector_engine = vector_index.as_query_engine(<br>    similarity_top_k=3,<br>    node_postprocessors=[reranker],<br>)<br># Tool 2: Hybrid (vector + BM25)<br>hybrid_retriever = QueryFusionRetriever(<br>    retrievers=[<br>        VectorIndexRetriever(index=vector_index, similarity_top_k=5),<br>        BM25Retriever.from_defaults(index=vector_index, similarity_top_k=5),<br>    ],<br>    similarity_top_k=3,<br>    mode=&quot;reciprocal_rerank&quot;,<br>    use_async=True,<br>)<br>hybrid_engine = RetrieverQueryEngine.from_args(<br>    retriever=hybrid_retriever,<br>    node_postprocessors=[reranker],<br>)<br># Tool 3: Graph traversal<br>graph_engine = graph_index.as_query_engine(<br>    include_text=True,<br>    retriever_mode=&quot;hybrid&quot;,<br>    similarity_top_k=3,<br>)<br># Tool 4: Decomposed (for complex multi-part questions)<br>decomposed_engine = SubQuestionQueryEngine.from_defaults(<br>    query_engine_tools=[<br>        QueryEngineTool(<br>            query_engine=vector_engine,<br>            metadata=ToolMetadata(<br>                name=&quot;docs&quot;,<br>                description=&quot;Company documents and policies.&quot;<br>            )<br>        )<br>    ],<br>    use_async=True,<br>)<br># ── MODULE 3 (cont.): ALL TOOLS ───────────────────────────────<br>@tool<br>def search_documents(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search company documents by semantic meaning. Best for<br>    conceptual questions where the exact wording in the answer<br>    may differ from the question.&quot;&quot;&quot;<br>    return str(vector_engine.query(query))<br>@tool<br>def search_exact_terms(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search using both keyword and semantic matching. Best when<br>    the query contains specific terminology, product codes, names,<br>    or exact phrases that must appear in the result.&quot;&quot;&quot;<br>    return str(hybrid_engine.query(query))<br>@tool<br>def search_relationships(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search the knowledge graph for questions about how things<br>    connect: dependencies, impact chains, organizational links,<br>    and multi-hop reasoning. Use when the answer requires tracing<br>    a relationship across multiple entities.&quot;&quot;&quot;<br>    return str(graph_engine.query(query))<br>@tool<br>def search_complex_question(query: str) -&gt; str:<br>    &quot;&quot;&quot;For multi-part questions requiring synthesis across several<br>    topics. Automatically decomposes the question into sub-queries,<br>    retrieves each independently, and combines the results.&quot;&quot;&quot;<br>    return str(decomposed_engine.query(query))<br>tools = [<br>    search_documents,<br>    search_exact_terms,<br>    search_relationships,<br>    search_complex_question,<br>]<br>llm_with_tools = llm.bind_tools(tools)<br>tool_node = ToolNode(tools)<br># ── MODULE 4: NODES ──────────────────────────────────────────<br>def agent_node(state: AgentState) -&gt; dict:<br>    system_prompt = SystemMessage(content=(<br>        &quot;You are a precise research assistant with access to four retrieval tools:\n\n&quot;<br>        &quot;1. search_documents - semantic search over company documents\n&quot;<br>        &quot;2. search_exact_terms - hybrid semantic + keyword search\n&quot;<br>        &quot;3. search_relationships - graph traversal for relationship questions\n&quot;<br>        &quot;4. search_complex_question - decomposed retrieval for multi-part questions\n\n&quot;<br>        &quot;Think step by step. Use the tool that best fits the question type. &quot;<br>        &quot;You may call multiple tools if a question has multiple parts. &quot;<br>        &quot;Only answer when you have retrieved sufficient evidence.&quot;<br>    ))<br>    response = llm_with_tools.invoke([system_prompt] + state[&quot;messages&quot;])<br>    return {<br>        &quot;messages&quot;: [response],<br>        &quot;retrieval_count&quot;: state.get(&quot;retrieval_count&quot;, 0),<br>    }<br># ── MODULE 5: ROUTING ────────────────────────────────────────<br>def should_continue(state: AgentState) -&gt; Literal[&quot;tools&quot;, &quot;__end__&quot;]:<br>    last = state[&quot;messages&quot;][-1]<br>    if hasattr(last, &quot;tool_calls&quot;) and last.tool_calls:<br>        return &quot;tools&quot;<br>    return &quot;__end__&quot;<br># ── MODULE 6: GRAPH ASSEMBLY ─────────────────────────────────<br>builder = StateGraph(AgentState)<br>builder.add_node(&quot;agent&quot;, agent_node)<br>builder.add_node(&quot;tools&quot;, tool_node)<br>builder.add_edge(START, &quot;agent&quot;)<br>builder.add_conditional_edges(&quot;agent&quot;, should_continue,<br>    {&quot;tools&quot;: &quot;tools&quot;, &quot;__end__&quot;: END})<br>builder.add_edge(&quot;tools&quot;, &quot;agent&quot;)<br>graph = builder.compile(checkpointer=MemorySaver())<br># ── MODULE 7: ENTRYPOINT ──────────────────────────────────────<br>if __name__ == &quot;__main__&quot;:<br>    config = {&quot;configurable&quot;: {&quot;thread_id&quot;: &quot;agentic-session-001&quot;}}<br>    print(&quot;Agentic RAG ready. Ask anything.\n&quot;)<br>    while True:<br>        user_text = input(&quot;You: &quot;).strip()<br>        if not user_text or user_text.lower() in (&quot;exit&quot;, &quot;quit&quot;):<br>            break<br>        response = graph.invoke(<br>            {&quot;messages&quot;: [HumanMessage(content=user_text)]},<br>            config=config,<br>        )<br>        print(f&quot;\nAgent: {response[&#39;messages&#39;][-1].content}\n&quot;)</pre><h3>What Agentic RAG Solves That Pipelines Cannot</h3><p>The critical difference between Agentic RAG and every other architecture in this article is <em>self-correction</em>. A pipeline cannot realize it retrieved the wrong thing. An agent can.</p><p>If the first retrieval returns weak results, the agent recognizes this in its next reasoning step and issues a different query with different search terms. If a question has an unexpected dependency, the agent discovers this mid-answer and makes an additional retrieval call to resolve it. If the question was ambiguous, the agent can ask for clarification before searching at all.</p><p>This is the architecture to reach for when the cost of a wrong answer is high — compliance, legal, financial, medical — because you can add verification steps, confidence thresholds, and human-in-the-loop checkpoints from Part 2 of this series directly into the agent graph.</p><h3>The Latency Tradeoff</h3><p>Agentic RAG is genuinely slower. A single-tool pipeline runs in 200 to 500 milliseconds. An agent that makes three retrieval calls before answering may take 8 to 12 seconds. For real-time user-facing interfaces, this is often too slow for the primary interaction path. The two production patterns that resolve this are: streaming intermediate steps to the user so they see progress rather than silence, and running agentic retrieval asynchronously to pre-fetch answers for anticipated follow-up questions.</p><h3>Putting It All Together: The Decision Framework</h3><p>Every retrieval problem has a right answer among these five. Here is how to find it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UTJuSiuln8rxitH_nUM3cw.png" /></figure><h3>The Stacking Pattern: How These Architectures Combine</h3><p>One of the most important things to understand about this family is that the architectures are composable. You do not pick one and discard the others. The most common production pattern is a stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z7wwFwDiG_uyzSXRUhwSSg.png" /></figure><p>In LangGraph, this stacking pattern translates directly to the tool list. An Agentic RAG agent with access to a Naive tool, a Hybrid tool, a Graph tool, and a Decomposed tool is exactly the five-architecture stack — the agent (Layer 5) selects from and orchestrates the others (Layers 1 to 4) on every turn.</p><h3>The Keyword Reference Card</h3><p>An extension of the reference cards from Parts 1 through 5.</p><pre>NAIVE RAG<br>SimpleDirectoryReader           Load files into Document objects<br>VectorStoreIndex.from_documents Build the embed-and-store index<br>index.as_query_engine()         Full retrieve-and-answer pipeline<br>index.as_retriever()            Retrieve only (no answer generation)<br>Settings.chunk_size             Token size per Node<br>Settings.chunk_overlap          Token overlap between adjacent Nodes<br><br>HYBRID RAG<br>BM25Retriever                   Keyword-based sparse retriever<br>VectorIndexRetriever            Dense embedding retriever<br>QueryFusionRetriever            Merges multiple retrievers (RRF algorithm)<br>SentenceTransformerRerank       Cross-encoder reranker for post-retrieval<br><br>GRAPH RAG<br>PropertyGraphIndex              Builds a knowledge graph from documents<br>SimpleLLMPathExtractor          LLM-based entity and relation extraction<br>ImplicitPathExtractor           Heuristic-based entity extraction (fast)<br><br>ADVANCED RAG<br>SubQuestionQueryEngine          Decomposes complex queries into sub-queries<br>HyDEQueryTransform              Hypothetical Document Embedding transform<br>TransformQueryEngine            Wraps any engine with a query transform<br>node_postprocessors             Where rerankers and filters attach<br><br>AGENTIC RAG (LANGGRAPH LAYER)<br>@tool                           The bridge - every LlamaIndex engine becomes<br>                                a LangGraph tool through this decorator<br>ToolNode                        Executes whatever tool the agent selects<br>bind_tools()                    Gives the agent LLM its tool registry<br>MemorySaver / SqliteSaver       Thread-level memory across turns (Part 2)<br>interrupt()                     Human approval checkpoint before retrieval (Part 3)</pre><h3>Conclusion: Retrieval as Infrastructure</h3><p>The five architectures in this article are not five ways to do the same thing. They are five answers to five different retrieval problems, and they sit in a clean progression from simple to sophisticated.</p><p>Naive RAG is fast, cheap, and right for most document Q&amp;A problems. Hybrid RAG is the production default for anything with specialized terminology. Graph RAG is the answer when relationships matter more than individual documents. Advanced RAG is the pattern for when accuracy needs to go up and the problem is retrieval quality. Agentic RAG is the architecture for open-ended, high-stakes, autonomous reasoning tasks.</p><p>Combined, with LlamaIndex handling the data layer and LangGraph handling the orchestration layer, these five patterns cover the overwhelming majority of what a production AI application built on retrieval actually needs.</p><p>The seam between the two frameworks remains exactly what Part 4 taught: one @tool-decorated function. Everything else is a choice about what goes</p><p><em>Bessie Delight Kekeli — AI engineer. Writing about what actually works in production.</em> <em>Connect on LinkedIn: linkedin.com/in/delight-bessie</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d73c9acedbf7" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/the-5-rag-architectures-and-exactly-when-to-use-each-one-in-production-d73c9acedbf7">The 5 RAG Architectures and Exactly When to Use Each One in Production</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving Our LangGraph Agent for Real-World E-Commerce: Enterprise Validation, Business Logic…]]></title>
            <link>https://pub.towardsai.net/improving-our-langgraph-agent-for-real-world-e-commerce-enterprise-validation-business-logic-a63f2607808a?source=rss-4a5744f8548f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a63f2607808a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Bessie Delight Kekeli]]></dc:creator>
            <pubDate>Tue, 23 Jun 2026 14:31:01 GMT</pubDate>
            <atom:updated>2026-06-26T22:54:07.882Z</atom:updated>
            <content:encoded><![CDATA[<h3>Improving Our LangGraph Agent for Real-World E-Commerce: Enterprise Validation, Business Logic Guards, and a Multi-Agent Architecture</h3><h4>The patterns that separate a LangGraph demo from a system you can actually deploy.</h4><p><em>The article </em><a href="https://medium.com/@murat_geldyev"><strong><em>Murat Geldyev’s</em></strong><em> </em></a><em>comment made inevitable. Building on the original ShopBot, closing every gap, and turning a sandbox agent into something you could actually defend in a production review.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xCU1yWDDnDN9ihBr9nhwvg.png" /></figure><blockquote><strong><em>What this article assumes:</em></strong><em> You have read and understood the original ShopBot article (</em><a href="https://medium.com/towards-artificial-intelligence/your-first-real-langgraph-project-ac5eb00f923a"><em>LINK</em></a><em>) . You know the seven-module structure, how tools work, how the ReAct loop runs, and how interrupt() pauses a graph for human approval. If you haven’t built ShopBot yet, build it first. This article is a direct continuation, not a standalone.</em></blockquote><blockquote><strong><em>What we are building:</em></strong><em> The original ShopBot, extended with three new capabilities that the comment section identified as real gaps. Business Logic Gates that validate every LLM output before it touches an external system. A validation loop that automatically corrects hallucinations rather than crashing. A multi-agent architecture that splits ShopBot into three specialist sub-agents orchestrated by a Supervisor. By the end, the codebase looks like something you could walk into a production review with.</em></blockquote><h3>The Gap in the Original ShopBot</h3><p>The original ShopBot was honest about what it was: a complete implementation of everything from Parts 0 through 5of the series. Every concept worked. Every module was in the right place. But it had one silent assumption built into every node, every tool call, every routing decision.</p><p>It assumed the LLM was right.</p><p>Read through the original code and you will see this assumption everywhere. The agent calls check_refund_eligibility and trusts that the order ID it passes was correctly extracted from the conversation. The review_refund node surfaces the order ID to the human reviewer, but the human is approving or rejecting the action, not validating whether the order ID is even real. The process_refund tool gets called with whatever arguments the LLM decided to use. If the LLM hallucinates “ORD-99X” instead of “ORD-001”, the tool will return an error, the agent will see that error, and something reasonable might happen. But “something reasonable might happen” is not an acceptable description of a financial system’s error handling.</p><p>Murat Geldyev named this precisely: in enterprise workflows like automated procurement or compliance checking, you cannot afford probabilistic failures. The fix is not to trust the LLM more. The fix is to stop trusting it at all, and validate every structured output with deterministic Python code before it touches anything external.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u_l56EaxaqiAEGehw3kp4A.png" /></figure><p>That is what this article builds.</p><h3>What We Are Adding and Why</h3><p>Three new capabilities, each closing a specific gap:</p><p><strong>Business Logic Gates</strong> sit inside nodes and validate LLM output before it acts. They are pure Python. They have no LLM calls. They either pass the data through or they reject it with a specific, structured reason. This is deterministic validation in a probabilistic system.</p><p><strong>Validation Loops</strong> handle gate failures. When a gate rejects something, the graph does not crash and does not silently produce a wrong answer. It routes back to a correction node that feeds the failure reason to the LLM and asks for a corrected output. A retry counter in state prevents infinite loops. A hard maximum triggers an escalation to a human rather than looping forever.</p><p><strong>Multi-Agent Architecture</strong> splits the single agent_node into three specialist sub-agents: an Order Agent that handles lookups and status queries, a Refund Agent that handles eligibility checking and refund processing, and a Complaints Agent that handles frustrated customers and escalations. A Supervisor node reads each incoming message and routes to the right specialist. Each specialist is its own compiled subgraph.</p><p>These three capabilities are not independent. They compose. The Refund Agent has a Business Logic Gate. The gate can trigger a Validation Loop. The Supervisor reads the loop’s outcome and decides whether to retry, escalate, or abort. That composition is what makes the final system genuinely enterprise-grade.</p><h3>Before We Write Code: The New Plan</h3><p>The original article made the point that LangGraph rewards planning. It is even more true now that the graph is larger. Draw this before you touch a keyboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WznxoplxIXSINOX5jTtUbw.png" /></figure><p>That is the full system. Every box maps to exactly one node or one routing function. Every arrow maps to exactly one edge. Write this down. The code that follows is just translating this picture into Python, module by module, exactly as you have learned.</p><p>This tutorial, paired with a Jungle album, is all you need right now. Try it if you don’t believe me 😁✌️ Happy coding!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cf_hGK7hFAg8dzMB_LY-3Q.png" /></figure><h3>The New File Structure</h3><p>The original ShopBot lived in one file: shopbot.py. As the system grows, one file becomes unmanageable. We split into three files:</p><pre>shopbot_enterprise/<br>├── gates.py          ← Business Logic Gates (pure Python, no LangGraph imports)<br>├── agents.py         ← All three specialist subgraphs<br>└── shopbot.py        ← Supervisor graph + entrypoint (unchanged module structure)</pre><p>The gates.py file having no LangGraph imports is deliberate. It makes gates independently testable with plain Python unit tests. You do not need an LLM, a graph, or even a running process to test whether your validation logic is correct. You just call the gate function with sample data and assert the result. This is what “deterministic” means in practice.</p><h3>Module 2 Extended: The New State Schema</h3><p>Before anything else, the state needs new fields. The original had three custom fields: summary, customer_name, and ticket_category. We add four more.</p><pre># ============================================================<br># MODULE 2: STATE (Enterprise Version)<br># ============================================================<br>from typing import TypedDict, Annotated<br>from langgraph.graph import MessagesState<br>import operator<br><br>class SupportState(MessagesState):<br>    # From the original ShopBot - unchanged<br>    summary: str<br>    customer_name: str<br>    ticket_category: str<br>    # NEW: The structured data the LLM extracted from the conversation.<br>    # Gates validate this field. Tools use this field. The LLM never<br>    # writes to external systems directly - it writes here first.<br>    extracted_data: dict<br>    # NEW: The current status of the gate decision for this turn.<br>    # Values: &quot;pending&quot; | &quot;passed&quot; | &quot;failed&quot; | &quot;escalated&quot; | &quot;aborted&quot;<br>    # The supervisor reads this to decide what to do after a gate runs.<br>    gate_status: str<br>    # NEW: A log of every gate failure in this conversation.<br>    # Uses operator.add so entries accumulate rather than replace.<br>    # This is your audit trail. Every compliance system needs one.<br>    validation_errors: Annotated[list[str], operator.add]<br>    # NEW: How many correction attempts have fired in this turn.<br>    # The hard cap is 2. After 2 failed corrections, escalate to human.<br>    # Without this, a broken LLM output creates an infinite loop.<br>    retry_count: int</pre><p>The most important new field is extracted_data. In the original ShopBot, the LLM produced free-form text and also called tools directly. The agent trusted whatever arguments the LLM put in its tool call. Now, the LLM is asked to produce structured data first. That structured data goes into state[“extracted_data”]. The gate validates it. Only after the gate passes does the data become a tool argument. The LLM no longer has a direct path to external systems.</p><p>The validation_errors field being Annotated with operator.add means it accumulates across the entire conversation, not just one turn. If the LLM hallucinated a bad order ID in turn three and a bad refund reason in turn seven, both errors are in the audit log. This matters for compliance reviews.</p><h3>The New File: gates.py</h3><p>Business Logic Gates are pure Python validation functions. Each gate takes extracted data as input and returns a tuple: (True, cleaned_data) on success, (False, error_reason) on failure. That’s the entire interface. No LLM calls. No graph dependencies. Just Python.</p><pre># ============================================================<br># gates.py — Business Logic Gates<br># Pure Python. No LangGraph. No LLM calls.<br># Each gate returns (bool, data_or_error_string).<br># ============================================================<br>import re<br><br># The same fake database as the original ShopBot.<br># In production this would be a database query.<br>ORDERS_DB = {<br>    &quot;ORD-001&quot;: {&quot;customer&quot;: &quot;Alex&quot;, &quot;product&quot;: &quot;Wireless Headphones&quot;,<br>                &quot;status&quot;: &quot;Delivered&quot;, &quot;amount&quot;: 89.99,<br>                &quot;delivery_date&quot;: &quot;2025-06-10&quot;},<br>    &quot;ORD-002&quot;: {&quot;customer&quot;: &quot;Sam&quot;, &quot;product&quot;: &quot;Phone Case&quot;,<br>                &quot;status&quot;: &quot;In Transit&quot;, &quot;amount&quot;: 14.99,<br>                &quot;delivery_date&quot;: &quot;Expected 2025-06-18&quot;},<br>    &quot;ORD-003&quot;: {&quot;customer&quot;: &quot;Jordan&quot;, &quot;product&quot;: &quot;Laptop Stand&quot;,<br>                &quot;status&quot;: &quot;Processing&quot;, &quot;amount&quot;: 45.00,<br>                &quot;delivery_date&quot;: &quot;Expected 2025-06-20&quot;},<br>}<br>MAX_REFUND_AMOUNT = 500.00   # Business rule: auto-refunds above this need director sign-off<br>MIN_REASON_LENGTH = 10       # Business rule: refund reasons must be substantive<br><br>def validate_order_id(extracted_data: dict) -&gt; tuple[bool, dict | str]:<br>    &quot;&quot;&quot;<br>    Gate 1: Order ID Validation<br>    Checks that the order ID the LLM extracted actually exists in our system<br>    and follows the correct format. This is the most basic hallucination check<br>    in a support workflow. LLMs frequently confuse order IDs, especially when<br>    a customer has mentioned multiple orders or used an informal format like<br>    &quot;order one&quot; instead of &quot;ORD-001&quot;.<br>    Why this matters: if the agent calls a tool with a hallucinated order ID,<br>    the tool returns an error, the agent sees the error, and may hallucinate<br>    a recovery that looks plausible but is wrong. Catching it here, before<br>    the tool call, is the correct place to stop the failure.<br>    &quot;&quot;&quot;<br>    order_id = extracted_data.get(&quot;order_id&quot;, &quot;&quot;).strip().upper()<br>    # Check 1: Format validation - must match ORD-XXX pattern<br>    if not re.match(r&quot;^ORD-\d{3}$&quot;, order_id):<br>        return (<br>            False,<br>            f&quot;Order ID &#39;{order_id}&#39; does not match the required format ORD-XXX &quot;<br>            f&quot;(example: ORD-001). The LLM may have extracted the wrong value. &quot;<br>            f&quot;Ask the customer to confirm their exact order number.&quot;<br>        )<br>    # Check 2: Existence validation - must exist in our system<br>    if order_id not in ORDERS_DB:<br>        return (<br>            False,<br>            f&quot;Order ID &#39;{order_id}&#39; passed format validation but does not exist &quot;<br>            f&quot;in the orders database. This is likely a hallucination. &quot;<br>            f&quot;Ask the customer to provide the exact order ID from their confirmation email.&quot;<br>        )<br>    # Gate passed: return cleaned data with the normalised order ID<br>    cleaned = {**extracted_data, &quot;order_id&quot;: order_id}<br>    return True, cleaned<br><br>def validate_refund_request(extracted_data: dict) -&gt; tuple[bool, dict | str]:<br>    &quot;&quot;&quot;<br>    Gate 2: Refund Request Validation<br>    A compound gate that checks every business rule that must be true<br>    before a refund can be processed. Runs after validate_order_id passes,<br>    so we can assume order_id is valid at this point.<br>    Business rules checked:<br>      1. The order must exist and have Delivered status<br>      2. The refund amount must be below the auto-approval threshold<br>      3. The reason must be substantive (not empty or one word)<br>    Each failed rule produces a specific error message that gets fed back<br>    to the LLM as a correction instruction. Vague errors produce vague<br>    corrections. Specific errors produce specific corrections.<br>    &quot;&quot;&quot;<br>    order_id = extracted_data.get(&quot;order_id&quot;, &quot;&quot;).strip().upper()<br>    reason   = extracted_data.get(&quot;reason&quot;, &quot;&quot;).strip()<br>    order    = ORDERS_DB.get(order_id, {})<br>    errors = []<br>    # Rule 1: Order must be in Delivered status<br>    if order.get(&quot;status&quot;) != &quot;Delivered&quot;:<br>        errors.append(<br>            f&quot;Order {order_id} has status &#39;{order.get(&#39;status&#39;)}&#39;. &quot;<br>            f&quot;Refunds can only be processed for Delivered orders. &quot;<br>            f&quot;Inform the customer and do not attempt to process a refund.&quot;<br>        )<br>    # Rule 2: Amount must be within auto-approval limit<br>    amount = order.get(&quot;amount&quot;, 0)<br>    if amount &gt; MAX_REFUND_AMOUNT:<br>        errors.append(<br>            f&quot;Refund amount ${amount:.2f} exceeds the auto-approval limit &quot;<br>            f&quot;of ${MAX_REFUND_AMOUNT:.2f}. This requires director-level sign-off. &quot;<br>            f&quot;Escalate to a senior agent - do not process automatically.&quot;<br>        )<br>    # Rule 3: Reason must be substantive<br>    if len(reason) &lt; MIN_REASON_LENGTH:<br>        errors.append(<br>            f&quot;Refund reason &#39;{reason}&#39; is too short (minimum {MIN_REASON_LENGTH} characters). &quot;<br>            f&quot;Ask the customer to provide a more detailed reason for their return request.&quot;<br>        )<br>    if errors:<br>        # Return all errors joined - the correction node will feed this<br>        # exact string back to the LLM as a correction instruction<br>        return False, &quot; | &quot;.join(errors)<br>    return True, extracted_data<br><br>def validate_complaint(extracted_data: dict) -&gt; tuple[bool, dict | str]:<br>    &quot;&quot;&quot;<br>    Gate 3: Complaint Validation<br>    Ensures the complaints agent has enough structured data to work with<br>    before attempting any escalation or response. Complaints without<br>    a clear issue category cannot be routed correctly in a real ticketing system.<br>    &quot;&quot;&quot;<br>    issue_summary = extracted_data.get(&quot;issue_summary&quot;, &quot;&quot;).strip()<br>    sentiment     = extracted_data.get(&quot;sentiment&quot;, &quot;&quot;).strip().lower()<br>    valid_sentiments = {&quot;frustrated&quot;, &quot;angry&quot;, &quot;neutral&quot;, &quot;disappointed&quot;}<br>    if len(issue_summary) &lt; 20:<br>        return (<br>            False,<br>            f&quot;Complaint issue summary is too vague: &#39;{issue_summary}&#39;. &quot;<br>            f&quot;Ask the customer to describe their problem in more detail before escalating.&quot;<br>        )<br>    if sentiment not in valid_sentiments:<br>        return (<br>            False,<br>            f&quot;Sentiment &#39;{sentiment}&#39; is not a recognised value. &quot;<br>            f&quot;Valid values are: {&#39;, &#39;.join(valid_sentiments)}. &quot;<br>            f&quot;Re-classify the customer&#39;s sentiment and try again.&quot;<br>        )<br>    return True, extracted_data</pre><p>Notice what is not in this file: imports from LangGraph, from LangChain, from any AI library. This is pure Python that any developer on your team can read, test, and modify without understanding agent architecture. When a compliance auditor asks “what exactly validates a refund request before money moves?”, you show them this file. It is readable by non-engineers.</p><h3>Module 4 Extended: The Validation Loop Pattern</h3><p>Now the critical new node that makes gates useful: the correction node. When a gate fails, execution does not stop. It routes here. The correction node takes the specific failure reason from state, builds an instruction for the LLM explaining exactly what was wrong, and asks for a corrected structured output.</p><pre># ── agents.py (partial) ─────────────────────────────────────<br>from langchain_openai import ChatOpenAI<br>from langchain_core.messages import HumanMessage, SystemMessage, AIMessage<br>from pydantic import BaseModel, Field<br><br>llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br><br># ── Pydantic Models (Structured LLM Output) ──────────────────<br># These are the schemas that define what the LLM must return.<br># with_structured_output() enforces these - if the LLM returns<br># something that doesn&#39;t match, Pydantic raises a validation error<br># before the output even reaches a gate.<br>class OrderExtraction(BaseModel):<br>    order_id: str = Field(description=&quot;The order ID in the format ORD-XXX, e.g. ORD-001&quot;)<br>    query_type: str = Field(description=&quot;What the customer wants to know: &#39;status&#39;, &#39;delivery_date&#39;, &#39;product&#39;, or &#39;general&#39;&quot;)<br>class RefundExtraction(BaseModel):<br>    order_id: str = Field(description=&quot;The order ID in the format ORD-XXX&quot;)<br>    reason: str = Field(description=&quot;The customer&#39;s stated reason for wanting a refund, in their own words&quot;)<br>class ComplaintExtraction(BaseModel):<br>    issue_summary: str = Field(description=&quot;A one-sentence summary of the customer&#39;s complaint&quot;)<br>    sentiment: str = Field(description=&quot;The customer&#39;s emotional state: &#39;frustrated&#39;, &#39;angry&#39;, &#39;neutral&#39;, or &#39;disappointed&#39;&quot;)<br><br># ── The Correction Node ───────────────────────────────────────<br>def correction_node(state) -&gt; dict:<br>    &quot;&quot;&quot;<br>    The validation loop&#39;s core. When a Business Logic Gate fails, this<br>    node receives control. It reads the most recent gate failure from<br>    state, builds a precise correction instruction for the LLM, and<br>    asks for a new extraction attempt.<br>    This node is shared across all three specialist agents. The<br>    ticket_category in state tells it which Pydantic model to use<br>    for the re-extraction, so the same node handles order corrections,<br>    refund corrections, and complaint corrections without duplicating logic.<br>    Critical design decision: the correction instruction goes into the<br>    messages list as a HumanMessage, not a SystemMessage. This makes it<br>    part of the visible conversation history. If the LLM makes the same<br>    mistake twice, the history shows both the first failure and the second<br>    correction attempt, which helps diagnose systematic prompt problems.<br>    &quot;&quot;&quot;<br>    # Read the most recent validation error from the audit log<br>    errors = state.get(&quot;validation_errors&quot;, [])<br>    last_error = errors[-1] if errors else &quot;Unknown validation error.&quot;<br>    retry_count = state.get(&quot;retry_count&quot;, 0)<br>    # Hard stop: after 2 retries, do not attempt again - escalate<br>    if retry_count &gt;= 2:<br>        return {<br>            &quot;gate_status&quot;: &quot;escalated&quot;,<br>            &quot;messages&quot;: [<br>                AIMessage(<br>                    content=(<br>                        f&quot;I was unable to process your request correctly after {retry_count} attempts. &quot;<br>                        f&quot;I&#39;m escalating this to a senior support agent who will contact you directly. &quot;<br>                        f&quot;I apologise for the inconvenience.&quot;<br>                    )<br>                )<br>            ]<br>        }<br>    # Build the correction instruction - be maximally specific<br>    correction_instruction = (<br>        f&quot;Your previous response contained a validation error that must be corrected &quot;<br>        f&quot;before we can proceed. The exact error was:\n\n&quot;<br>        f&quot;{last_error}\n\n&quot;<br>        f&quot;Please re-read the conversation carefully and provide a corrected response &quot;<br>        f&quot;that addresses this specific error. Do not repeat the same mistake.&quot;<br>    )<br>    category = state.get(&quot;ticket_category&quot;, &quot;order_inquiry&quot;)<br>    # Choose the right structured output model based on what kind of<br>    # correction is needed<br>    if category == &quot;refund_request&quot;:<br>        structured_llm = llm.with_structured_output(RefundExtraction)<br>    elif category == &quot;complaint&quot;:<br>        structured_llm = llm.with_structured_output(ComplaintExtraction)<br>    else:<br>        structured_llm = llm.with_structured_output(OrderExtraction)<br>    messages = (<br>        [SystemMessage(content=&quot;You are a customer support assistant. Extract the requested information accurately.&quot;)]<br>        + state[&quot;messages&quot;]<br>        + [HumanMessage(content=correction_instruction)]<br>    )<br>    try:<br>        corrected = structured_llm.invoke(messages)<br>        return {<br>            &quot;extracted_data&quot;: corrected.model_dump(),<br>            &quot;gate_status&quot;: &quot;pending&quot;,<br>            &quot;retry_count&quot;: retry_count + 1,<br>            &quot;messages&quot;: [HumanMessage(content=correction_instruction)],<br>        }<br>    except Exception as e:<br>        # Structured output itself failed (Pydantic validation error)<br>        # Count this as a retry and surface the raw error<br>        return {<br>            &quot;gate_status&quot;: &quot;failed&quot;,<br>            &quot;retry_count&quot;: retry_count + 1,<br>            &quot;validation_errors&quot;: [f&quot;Structured output failed: {str(e)}&quot;],<br>        }</pre><p>The retry_count check at the top of this node is the most important safety mechanism in the entire system. Without it, a genuinely broken input (a customer who refuses to provide a valid order ID, or an LLM configuration that consistently hallucinates) creates an infinite loop between the correction node and the gate. Two retries is a reasonable default for a support agent: it gives the LLM two chances to self-correct before involving a human. Adjust this number based on your tolerance for latency versus accuracy.</p><h3>Module 4 Continued: The Three Specialist Subgraphs</h3><p>Each specialist is a compiled subgraph. Each has its own tools, its own extraction logic, its own gate, and its own internal routing. The parent Supervisor graph doesn’t know or care about any of that internal structure — it just sees a node that receives state and returns state.</p><h3>The Order Agent Subgraph</h3><pre># ── agents.py (continued) ────────────────────────────────────<br>from langchain_core.tools import tool<br>from langgraph.graph import StateGraph, START, END<br>from langgraph.prebuilt import ToolNode<br>from gates import validate_order_id, ORDERS_DB<br><br>@tool<br>def lookup_order(order_id: str) -&gt; str:<br>    &quot;&quot;&quot;Look up the details of a customer&#39;s order by order ID.<br>    Use this when the customer wants to know their order status,<br>    product name, or expected delivery date.<br>    &quot;&quot;&quot;<br>    order = ORDERS_DB.get(order_id.upper())<br>    if not order:<br>        return f&quot;No order found with ID &#39;{order_id}&#39;.&quot;<br>    return (<br>        f&quot;Order {order_id.upper()}: {order[&#39;product&#39;]} | &quot;<br>        f&quot;Status: {order[&#39;status&#39;]} | &quot;<br>        f&quot;Amount: ${order[&#39;amount&#39;]:.2f} | &quot;<br>        f&quot;Delivery: {order[&#39;delivery_date&#39;]}&quot;<br>    )<br>order_tools = [lookup_order]<br>order_tool_node = ToolNode(order_tools)<br>order_llm = llm.bind_tools(order_tools)<br><br>def order_extract_node(state) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Step 1 for the Order Agent: extract the order ID from the conversation<br>    using structured output. The result goes into state[&#39;extracted_data&#39;].<br>    The gate runs next. The tool call only happens if the gate passes.<br>    &quot;&quot;&quot;<br>    structured_llm = llm.with_structured_output(OrderExtraction)<br>    messages = (<br>        [SystemMessage(content=&quot;Extract the order ID and query type from the customer&#39;s message.&quot;)]<br>        + state[&quot;messages&quot;]<br>    )<br>    try:<br>        result = structured_llm.invoke(messages)<br>        return {<br>            &quot;extracted_data&quot;: result.model_dump(),<br>            &quot;gate_status&quot;: &quot;pending&quot;,<br>            &quot;ticket_category&quot;: &quot;order_inquiry&quot;,<br>        }<br>    except Exception as e:<br>        return {<br>            &quot;gate_status&quot;: &quot;failed&quot;,<br>            &quot;validation_errors&quot;: [f&quot;Extraction failed: {str(e)}&quot;],<br>            &quot;ticket_category&quot;: &quot;order_inquiry&quot;,<br>        }<br><br>def order_gate_node(state) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Step 2: Run the Business Logic Gate on the extracted data.<br>    Either passes gate_status to &#39;passed&#39; and lets the tool run,<br>    or sets it to &#39;failed&#39; and logs the reason to validation_errors.<br>    &quot;&quot;&quot;<br>    extracted = state.get(&quot;extracted_data&quot;, {})<br>    passed, result = validate_order_id(extracted)<br>    if passed:<br>        return {&quot;gate_status&quot;: &quot;passed&quot;, &quot;extracted_data&quot;: result}<br>    return {<br>        &quot;gate_status&quot;: &quot;failed&quot;,<br>        &quot;validation_errors&quot;: [result],   # operator.add accumulates this<br>    }<br><br>def order_agent_node(state) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Step 3 (only runs after gate passes): call the LLM with tools bound.<br>    Now the LLM uses the validated order_id from extracted_data rather<br>    than whatever it decided to extract ad hoc.<br>    &quot;&quot;&quot;<br>    extracted = state.get(&quot;extracted_data&quot;, {})<br>    order_id = extracted.get(&quot;order_id&quot;, &quot;&quot;)<br>    messages = (<br>        [SystemMessage(content=(<br>            &quot;You are the Order Agent for ShopBot. Your only job is to look up &quot;<br>            &quot;order information and report it clearly to the customer. &quot;<br>            f&quot;The validated order ID is {order_id}. Use lookup_order to get the details.&quot;<br>        ))]<br>        + state[&quot;messages&quot;]<br>    )<br>    response = order_llm.invoke(messages)<br>    return {&quot;messages&quot;: [response]}<br><br>def route_order_gate(state) -&gt; str:<br>    status = state.get(&quot;gate_status&quot;, &quot;pending&quot;)<br>    if status == &quot;passed&quot;:<br>        return &quot;order_agent_node&quot;<br>    if state.get(&quot;retry_count&quot;, 0) &gt;= 2:<br>        return &quot;order_agent_node&quot;   # correction_node already added the escalation message<br>    return &quot;correction_node&quot;<br><br>def route_order_agent(state) -&gt; str:<br>    last = state[&quot;messages&quot;][-1]<br>    if hasattr(last, &quot;tool_calls&quot;) and last.tool_calls:<br>        return &quot;order_tools&quot;<br>    return &quot;__end__&quot;<br><br># ── Order subgraph assembly ──────────────────────────────────<br>order_builder = StateGraph(state_schema=type(None))  # Uses parent state directly<br>order_builder = StateGraph(dict)<br>order_builder.add_node(&quot;order_extract_node&quot;, order_extract_node)<br>order_builder.add_node(&quot;order_gate_node&quot;,    order_gate_node)<br>order_builder.add_node(&quot;correction_node&quot;,    correction_node)<br>order_builder.add_node(&quot;order_agent_node&quot;,   order_agent_node)<br>order_builder.add_node(&quot;order_tools&quot;,        order_tool_node)<br>order_builder.add_edge(START,               &quot;order_extract_node&quot;)<br>order_builder.add_edge(&quot;order_extract_node&quot;, &quot;order_gate_node&quot;)<br>order_builder.add_conditional_edges(<br>    &quot;order_gate_node&quot;,<br>    route_order_gate,<br>    {&quot;order_agent_node&quot;: &quot;order_agent_node&quot;, &quot;correction_node&quot;: &quot;correction_node&quot;}<br>)<br>order_builder.add_edge(&quot;correction_node&quot;,   &quot;order_gate_node&quot;)<br>order_builder.add_conditional_edges(<br>    &quot;order_agent_node&quot;,<br>    route_order_agent,<br>    {&quot;order_tools&quot;: &quot;order_tools&quot;, &quot;__end__&quot;: END}<br>)<br>order_builder.add_edge(&quot;order_tools&quot;,       &quot;order_agent_node&quot;)<br>order_agent = order_builder.compile()</pre><p>Notice the loop: order_gate_node feeds into correction_node on failure, and correction_node feeds back into order_gate_node. This is the validation loop. It will run at most twice before the retry counter triggers the escalation message.</p><h3>The Refund Agent Subgraph</h3><p>The Refund Agent is the most consequential specialist because it handles money. It has two gates (order ID validation, then full refund validation), and the human approval interrupt from the original ShopBot is preserved inside this subgraph.</p><pre># ── agents.py (continued) ────────────────────────────────────<br>from gates import validate_order_id, validate_refund_request<br>from langgraph.types import interrupt, Command<br>from langchain_core.messages import ToolMessage<br>from datetime import datetime<br><br>@tool<br>def check_refund_eligibility(order_id: str) -&gt; str:<br>    &quot;&quot;&quot;Check whether an order is eligible for a refund.<br>    An order is eligible only if its status is Delivered.<br>    &quot;&quot;&quot;<br>    order = ORDERS_DB.get(order_id.upper())<br>    if not order:<br>        return f&quot;Cannot check refund: order &#39;{order_id}&#39; not found.&quot;<br>    if order[&quot;status&quot;] == &quot;Delivered&quot;:<br>        return (<br>            f&quot;Order {order_id.upper()} IS eligible for a refund. &quot;<br>            f&quot;Amount: ${order[&#39;amount&#39;]:.2f}.&quot;<br>        )<br>    return (<br>        f&quot;Order {order_id.upper()} is NOT eligible. &quot;<br>        f&quot;Status: {order[&#39;status&#39;]}.&quot;<br>    )<br><br>@tool<br>def process_refund(order_id: str, reason: str) -&gt; str:<br>    &quot;&quot;&quot;Process a refund for a delivered order.<br>    Only call this after human approval has been confirmed.<br>    &quot;&quot;&quot;<br>    order = ORDERS_DB.get(order_id.upper())<br>    if not order:<br>        return f&quot;Refund failed: order &#39;{order_id}&#39; not found.&quot;<br>    ref = f&quot;REF-{order_id.upper()}-{datetime.now().strftime(&#39;%H%M%S&#39;)}&quot;<br>    return (<br>        f&quot;Refund APPROVED and PROCESSED. Reference: {ref}. &quot;<br>        f&quot;${order[&#39;amount&#39;]:.2f} will be returned within 3 to 5 business days.&quot;<br>    )<br><br>refund_tools = [check_refund_eligibility, process_refund]<br>refund_tool_node = ToolNode(refund_tools)<br>refund_llm = llm.bind_tools(refund_tools)<br><br>def refund_extract_node(state) -&gt; dict:<br>    structured_llm = llm.with_structured_output(RefundExtraction)<br>    messages = (<br>        [SystemMessage(content=&quot;Extract the order ID and refund reason from the customer&#39;s message.&quot;)]<br>        + state[&quot;messages&quot;]<br>    )<br>    try:<br>        result = structured_llm.invoke(messages)<br>        return {<br>            &quot;extracted_data&quot;: result.model_dump(),<br>            &quot;gate_status&quot;: &quot;pending&quot;,<br>            &quot;ticket_category&quot;: &quot;refund_request&quot;,<br>        }<br>    except Exception as e:<br>        return {<br>            &quot;gate_status&quot;: &quot;failed&quot;,<br>            &quot;validation_errors&quot;: [f&quot;Refund extraction failed: {str(e)}&quot;],<br>            &quot;ticket_category&quot;: &quot;refund_request&quot;,<br>        }<br><br>def refund_gate_node(state) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Compound gate: runs both order ID validation and full refund validation.<br>    Both must pass for execution to continue.<br>    &quot;&quot;&quot;<br>    extracted = state.get(&quot;extracted_data&quot;, {})<br>    # Gate 1: Order ID<br>    id_passed, id_result = validate_order_id(extracted)<br>    if not id_passed:<br>        return {&quot;gate_status&quot;: &quot;failed&quot;, &quot;validation_errors&quot;: [id_result]}<br>    # Gate 2: Full refund rules (eligibility, amount, reason)<br>    ref_passed, ref_result = validate_refund_request(id_result)<br>    if not ref_passed:<br>        return {&quot;gate_status&quot;: &quot;failed&quot;, &quot;validation_errors&quot;: [ref_result]}<br>    return {&quot;gate_status&quot;: &quot;passed&quot;, &quot;extracted_data&quot;: ref_result}<br><br>def refund_review_node(state) -&gt; dict:<br>    &quot;&quot;&quot;<br>    The human approval gate, carried over from the original ShopBot<br>    and now living inside the Refund Agent subgraph.<br>    interrupt() pauses the entire graph here. The parent graph&#39;s<br>    checkpointer saves the state. A human reviews the details.<br>    Command(resume=...) restarts from this exact line.<br>    &quot;&quot;&quot;<br>    last_message = state[&quot;messages&quot;][-1]<br>    refund_tool_call = None<br>    if hasattr(last_message, &quot;tool_calls&quot;):<br>        for tc in last_message.tool_calls:<br>            if &quot;refund&quot; in tc[&quot;name&quot;].lower():<br>                refund_tool_call = tc<br>                break<br>    extracted = state.get(&quot;extracted_data&quot;, {})<br>    human_decision = interrupt({<br>        &quot;message&quot;: &quot;Refund approval required&quot;,<br>        &quot;order_id&quot;: extracted.get(&quot;order_id&quot;, &quot;Unknown&quot;),<br>        &quot;reason&quot;: extracted.get(&quot;reason&quot;, &quot;Not provided&quot;),<br>        &quot;tool_being_called&quot;: refund_tool_call[&quot;name&quot;] if refund_tool_call else &quot;process_refund&quot;,<br>        &quot;arguments&quot;: refund_tool_call[&quot;args&quot;] if refund_tool_call else extracted,<br>        &quot;summary&quot;: state.get(&quot;summary&quot;, &quot;No summary yet&quot;),<br>        &quot;options&quot;: [&quot;approve&quot;, &quot;reject&quot;, &quot;escalate&quot;],<br>    })<br>    if human_decision == &quot;approve&quot;:<br>        return {}<br>    if human_decision in (&quot;reject&quot;, &quot;escalate&quot;):<br>        label = &quot;declined&quot; if human_decision == &quot;reject&quot; else &quot;escalated to a senior agent&quot;<br>        return {<br>            &quot;messages&quot;: [<br>                ToolMessage(<br>                    content=f&quot;Refund request was {label} by the support team.&quot;,<br>                    tool_call_id=refund_tool_call[&quot;id&quot;] if refund_tool_call else &quot;unknown&quot;,<br>                )<br>            ]<br>        }<br>    return {}<br><br>def refund_agent_node(state) -&gt; dict:<br>    extracted = state.get(&quot;extracted_data&quot;, {})<br>    messages = (<br>        [SystemMessage(content=(<br>            &quot;You are the Refund Agent for ShopBot. Your job is to check refund &quot;<br>            &quot;eligibility and process approved refunds. &quot;<br>            f&quot;Validated order ID: {extracted.get(&#39;order_id&#39;, &#39;&#39;)}. &quot;<br>            f&quot;Customer reason: {extracted.get(&#39;reason&#39;, &#39;&#39;)}. &quot;<br>            &quot;Always check eligibility first, then process if eligible and approved.&quot;<br>        ))]<br>        + state[&quot;messages&quot;]<br>    )<br>    response = refund_llm.invoke(messages)<br>    return {&quot;messages&quot;: [response]}<br><br>def route_refund_gate(state) -&gt; str:<br>    status = state.get(&quot;gate_status&quot;, &quot;pending&quot;)<br>    if status == &quot;passed&quot;:<br>        return &quot;refund_agent_node&quot;<br>    if state.get(&quot;retry_count&quot;, 0) &gt;= 2:<br>        return &quot;refund_agent_node&quot;<br>    return &quot;correction_node&quot;<br><br>def route_refund_agent(state) -&gt; str:<br>    last = state[&quot;messages&quot;][-1]<br>    if hasattr(last, &quot;tool_calls&quot;) and last.tool_calls:<br>        tool_names = [tc[&quot;name&quot;] for tc in last.tool_calls]<br>        if &quot;process_refund&quot; in tool_names:<br>            return &quot;refund_review_node&quot;<br>        return &quot;refund_tools&quot;<br>    return &quot;__end__&quot;<br><br>def route_after_review(state) -&gt; str:<br>    last = state[&quot;messages&quot;][-1]<br>    if isinstance(last, ToolMessage):<br>        return &quot;refund_agent_node&quot;<br>    return &quot;refund_tools&quot;<br><br># ── Refund subgraph assembly ─────────────────────────────────<br>refund_builder = StateGraph(dict)<br>refund_builder.add_node(&quot;refund_extract_node&quot;, refund_extract_node)<br>refund_builder.add_node(&quot;refund_gate_node&quot;,    refund_gate_node)<br>refund_builder.add_node(&quot;correction_node&quot;,     correction_node)<br>refund_builder.add_node(&quot;refund_agent_node&quot;,   refund_agent_node)<br>refund_builder.add_node(&quot;refund_review_node&quot;,  refund_review_node)<br>refund_builder.add_node(&quot;refund_tools&quot;,        refund_tool_node)<br>refund_builder.add_edge(START,                &quot;refund_extract_node&quot;)<br>refund_builder.add_edge(&quot;refund_extract_node&quot;, &quot;refund_gate_node&quot;)<br>refund_builder.add_conditional_edges(<br>    &quot;refund_gate_node&quot;,<br>    route_refund_gate,<br>    {&quot;refund_agent_node&quot;: &quot;refund_agent_node&quot;, &quot;correction_node&quot;: &quot;correction_node&quot;}<br>)<br>refund_builder.add_edge(&quot;correction_node&quot;,    &quot;refund_gate_node&quot;)<br>refund_builder.add_conditional_edges(<br>    &quot;refund_agent_node&quot;,<br>    route_refund_agent,<br>    {<br>        &quot;refund_review_node&quot;: &quot;refund_review_node&quot;,<br>        &quot;refund_tools&quot;: &quot;refund_tools&quot;,<br>        &quot;__end__&quot;: END,<br>    }<br>)<br>refund_builder.add_conditional_edges(<br>    &quot;refund_review_node&quot;,<br>    route_after_review,<br>    {&quot;refund_tools&quot;: &quot;refund_tools&quot;, &quot;refund_agent_node&quot;: &quot;refund_agent_node&quot;}<br>)<br>refund_builder.add_edge(&quot;refund_tools&quot;,       &quot;refund_agent_node&quot;)<br>refund_agent = refund_builder.compile()</pre><h3>The Complaints Agent Subgraph</h3><p>The Complaints Agent is the simplest of the three. Its job is to acknowledge the customer’s frustration, validate that there is enough information to open a support ticket, and escalate to a human via interrupt() if the sentiment is strong enough.</p><pre># ── agents.py (continued) ────────────────────────────────────<br>from gates import validate_complaint<br><br>complaints_llm = llm   # Complaints agent doesn&#39;t need tool-calling<br><br>def complaints_extract_node(state) -&gt; dict:<br>    structured_llm = llm.with_structured_output(ComplaintExtraction)<br>    messages = (<br>        [SystemMessage(content=(<br>            &quot;Extract the complaint summary and the customer&#39;s emotional sentiment &quot;<br>            &quot;from the conversation. Sentiment must be one of: frustrated, angry, &quot;<br>            &quot;neutral, disappointed.&quot;<br>        ))]<br>        + state[&quot;messages&quot;]<br>    )<br>    try:<br>        result = structured_llm.invoke(messages)<br>        return {<br>            &quot;extracted_data&quot;: result.model_dump(),<br>            &quot;gate_status&quot;: &quot;pending&quot;,<br>            &quot;ticket_category&quot;: &quot;complaint&quot;,<br>        }<br>    except Exception as e:<br>        return {<br>            &quot;gate_status&quot;: &quot;failed&quot;,<br>            &quot;validation_errors&quot;: [f&quot;Complaint extraction failed: {str(e)}&quot;],<br>            &quot;ticket_category&quot;: &quot;complaint&quot;,<br>        }<br><br>def complaints_gate_node(state) -&gt; dict:<br>    extracted = state.get(&quot;extracted_data&quot;, {})<br>    passed, result = validate_complaint(extracted)<br>    if passed:<br>        return {&quot;gate_status&quot;: &quot;passed&quot;, &quot;extracted_data&quot;: result}<br>    return {&quot;gate_status&quot;: &quot;failed&quot;, &quot;validation_errors&quot;: [result]}<br><br>def complaints_respond_node(state) -&gt; dict:<br>    &quot;&quot;&quot;<br>    For high-sentiment complaints (frustrated, angry), trigger a human<br>    escalation via interrupt(). For lower-sentiment complaints, the LLM<br>    responds empathetically and the turn ends.<br>    This mirrors the Part 3 pattern but applied conditionally based on<br>    the validated sentiment field, not a keyword scan of free-form text.<br>    The gate already confirmed the sentiment is a valid value, so this<br>    routing decision is deterministic.<br>    &quot;&quot;&quot;<br>    extracted = state.get(&quot;extracted_data&quot;, {})<br>    sentiment = extracted.get(&quot;sentiment&quot;, &quot;neutral&quot;)<br>    if sentiment in (&quot;frustrated&quot;, &quot;angry&quot;):<br>        human_decision = interrupt({<br>            &quot;message&quot;: &quot;High-sentiment complaint requires human review&quot;,<br>            &quot;issue_summary&quot;: extracted.get(&quot;issue_summary&quot;, &quot;&quot;),<br>            &quot;sentiment&quot;: sentiment,<br>            &quot;options&quot;: [&quot;escalate_to_senior&quot;, &quot;provide_compensation&quot;, &quot;close_ticket&quot;],<br>        })<br>        if human_decision == &quot;escalate_to_senior&quot;:<br>            response_text = (<br>                &quot;I completely understand your frustration, and I want to make sure &quot;<br>                &quot;you receive the best possible support. I am escalating your case to &quot;<br>                &quot;a senior support agent who will contact you within 2 business hours. &quot;<br>                &quot;Your ticket has been flagged as priority.&quot;<br>            )<br>        elif human_decision == &quot;provide_compensation&quot;:<br>            response_text = (<br>                &quot;I sincerely apologise for the experience you have had. As a gesture &quot;<br>                &quot;of goodwill, our team will be in touch within 24 hours with a resolution &quot;<br>                &quot;that includes appropriate compensation for the inconvenience caused.&quot;<br>            )<br>        else:<br>            response_text = (<br>                &quot;Thank you for bringing this to our attention. Your feedback has been &quot;<br>                &quot;recorded and will help us improve our service.&quot;<br>            )<br>        from langchain_core.messages import AIMessage<br>        return {&quot;messages&quot;: [AIMessage(content=response_text)]}<br>    # Lower-sentiment complaints: LLM responds empathetically<br>    messages = (<br>        [SystemMessage(content=(<br>            &quot;You are the Complaints Agent for ShopBot. The customer has a complaint. &quot;<br>            &quot;Respond with genuine empathy, acknowledge their specific issue, and &quot;<br>            &quot;explain what steps will be taken. Be warm and professional.&quot;<br>        ))]<br>        + state[&quot;messages&quot;]<br>    )<br>    response = complaints_llm.invoke(messages)<br>    return {&quot;messages&quot;: [response]}<br><br>def route_complaints_gate(state) -&gt; str:<br>    status = state.get(&quot;gate_status&quot;, &quot;pending&quot;)<br>    if status == &quot;passed&quot;:<br>        return &quot;complaints_respond_node&quot;<br>    if state.get(&quot;retry_count&quot;, 0) &gt;= 2:<br>        return &quot;complaints_respond_node&quot;<br>    return &quot;correction_node&quot;<br><br># ── Complaints subgraph assembly ─────────────────────────────<br>complaints_builder = StateGraph(dict)<br>complaints_builder.add_node(&quot;complaints_extract_node&quot;, complaints_extract_node)<br>complaints_builder.add_node(&quot;complaints_gate_node&quot;,    complaints_gate_node)<br>complaints_builder.add_node(&quot;correction_node&quot;,         correction_node)<br>complaints_builder.add_node(&quot;complaints_respond_node&quot;, complaints_respond_node)<br>complaints_builder.add_edge(START,                    &quot;complaints_extract_node&quot;)<br>complaints_builder.add_edge(&quot;complaints_extract_node&quot;, &quot;complaints_gate_node&quot;)<br>complaints_builder.add_conditional_edges(<br>    &quot;complaints_gate_node&quot;,<br>    route_complaints_gate,<br>    {<br>        &quot;complaints_respond_node&quot;: &quot;complaints_respond_node&quot;,<br>        &quot;correction_node&quot;: &quot;correction_node&quot;,<br>    }<br>)<br>complaints_builder.add_edge(&quot;correction_node&quot;,        &quot;complaints_gate_node&quot;)<br>complaints_builder.add_edge(&quot;complaints_respond_node&quot;, END)<br>complaints_agent = complaints_builder.compile()</pre><h3>The Main shopbot.py: The Supervisor Graph</h3><p>With the three subgraphs compiled, the Supervisor graph is remarkably simple. Its only job is routing. It never does specialist work itself. This is the fundamental principle from Part 4’s multi-agent patterns: the supervisor routes, the specialists execute.</p><pre># ============================================================<br># shopbot.py — Supervisor Graph (Enterprise Version)<br># ============================================================<br># ── MODULE 1: IMPORTS &amp; CONFIGURATION ───────────────────────<br>import os<br>from typing import Literal<br>from dotenv import load_dotenv<br>from langchain_openai import ChatOpenAI<br>from langchain_core.messages import HumanMessage, SystemMessage, RemoveMessage, AIMessage<br>from langchain.chat_models import init_chat_model<br>from langgraph.graph import StateGraph, MessagesState, START, END<br>from langgraph.checkpoint.memory import MemorySaver<br>from langgraph.types import Command<br># Import the compiled subgraphs from agents.py<br>from agents import order_agent, refund_agent, complaints_agent<br>from state import SupportState   # The extended state from Module 2 above<br>load_dotenv()<br># The supervisor uses a plain LLM - no tools, just routing decisions<br>llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br><br># ── MODULE 4: SUPERVISOR NODE ────────────────────────────────<br>SUPERVISOR_SYSTEM = &quot;&quot;&quot;You are the ShopBot Supervisor. Your only job is to read<br>the customer&#39;s message and decide which specialist should handle it.<br>Available specialists:<br>  order_agent     - handles order status, delivery date, and product questions<br>  refund_agent    - handles refund requests and return eligibility<br>  complaints_agent - handles complaints, frustration, and negative feedback<br>  FINISH          - use this only when the conversation is fully resolved<br>Respond with exactly one word: the name of the specialist, or FINISH.&quot;&quot;&quot;<br><br>def supervisor_node(state: SupportState) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Reads the full conversation and routes to the correct specialist.<br>    Returns a next_agent string that the routing function reads.<br>    Uses plain LLM with no tools - the supervisor decides, it never acts.<br>    This separation keeps the routing logic deterministic and auditable.<br>    &quot;&quot;&quot;<br>    messages = [SystemMessage(content=SUPERVISOR_SYSTEM)] + state[&quot;messages&quot;]<br>    response = llm.invoke(messages)<br>    decision = response.content.strip().lower()<br>    # Normalise the response in case the LLM adds punctuation or casing<br>    if &quot;refund&quot; in decision:<br>        next_agent = &quot;refund_agent&quot;<br>    elif &quot;complaint&quot; in decision or &quot;frustrated&quot; in decision:<br>        next_agent = &quot;complaints_agent&quot;<br>    elif &quot;order&quot; in decision:<br>        next_agent = &quot;order_agent&quot;<br>    elif &quot;finish&quot; in decision:<br>        next_agent = &quot;FINISH&quot;<br>    else:<br>        next_agent = &quot;order_agent&quot;   # Safe default for ambiguous routing<br>    return {&quot;next_agent&quot;: next_agent}<br><br># ── MODULE 4 CONTINUED: SUMMARIZE NODE ───────────────────────<br># Carried over from the original ShopBot without changes.<br># The enterprise version still needs conversation compression.<br>def summarize_node(state: SupportState) -&gt; dict:<br>    existing_summary = state.get(&quot;summary&quot;, &quot;&quot;)<br>    instruction = (<br>        f&quot;Current summary:\n{existing_summary}\n\nExtend with new messages. &quot;<br>        &quot;Under 5 sentences. Include customer name, issue, orders mentioned, actions taken.&quot;<br>        if existing_summary<br>        else &quot;Summarise this support conversation in under 5 sentences.&quot;<br>    )<br>    messages = state[&quot;messages&quot;] + [HumanMessage(content=instruction)]<br>    response = llm.invoke(messages)<br>    to_delete = [RemoveMessage(id=m.id) for m in state[&quot;messages&quot;][:-2]]<br>    return {&quot;summary&quot;: response.content, &quot;messages&quot;: to_delete}<br><br># ── MODULE 5: ROUTING ────────────────────────────────────────<br>def route_supervisor(state: SupportState) -&gt; Literal[<br>    &quot;order_agent&quot;, &quot;refund_agent&quot;, &quot;complaints_agent&quot;, &quot;summarize_node&quot;, &quot;__end__&quot;<br>]:<br>    &quot;&quot;&quot;<br>    Reads next_agent from state and routes accordingly.<br>    Also checks whether the conversation is long enough to need summarisation.<br>    &quot;&quot;&quot;<br>    next_agent = state.get(&quot;next_agent&quot;, &quot;order_agent&quot;)<br>    if next_agent == &quot;FINISH&quot;:<br>        if len(state[&quot;messages&quot;]) &gt; 6:<br>            return &quot;summarize_node&quot;<br>        return &quot;__end__&quot;<br>    return next_agent<br><br># ── MODULE 6: GRAPH ASSEMBLY ─────────────────────────────────<br>graph_builder = StateGraph(SupportState)<br>graph_builder.add_node(&quot;supervisor&quot;,        supervisor_node)<br>graph_builder.add_node(&quot;order_agent&quot;,       order_agent)       # compiled subgraph<br>graph_builder.add_node(&quot;refund_agent&quot;,      refund_agent)      # compiled subgraph<br>graph_builder.add_node(&quot;complaints_agent&quot;,  complaints_agent)  # compiled subgraph<br>graph_builder.add_node(&quot;summarize_node&quot;,    summarize_node)<br>graph_builder.add_edge(START, &quot;supervisor&quot;)<br>graph_builder.add_conditional_edges(<br>    &quot;supervisor&quot;,<br>    route_supervisor,<br>    {<br>        &quot;order_agent&quot;:       &quot;order_agent&quot;,<br>        &quot;refund_agent&quot;:      &quot;refund_agent&quot;,<br>        &quot;complaints_agent&quot;:  &quot;complaints_agent&quot;,<br>        &quot;summarize_node&quot;:    &quot;summarize_node&quot;,<br>        &quot;__end__&quot;:           END,<br>    }<br>)<br># After each specialist finishes, always return to supervisor.<br># The supervisor re-evaluates and decides: another specialist, or FINISH.<br>graph_builder.add_edge(&quot;order_agent&quot;,      &quot;supervisor&quot;)<br>graph_builder.add_edge(&quot;refund_agent&quot;,     &quot;supervisor&quot;)<br>graph_builder.add_edge(&quot;complaints_agent&quot;, &quot;supervisor&quot;)<br>graph_builder.add_edge(&quot;summarize_node&quot;,   END)<br>memory = MemorySaver()<br>shopbot = graph_builder.compile(checkpointer=memory)<br><br># ── MODULE 7: ENTRYPOINT ──────────────────────────────────────<br>def run_shopbot():<br>    print(&quot;=&quot; * 60)<br>    print(&quot;  ShopBot Enterprise - Multi-Agent Customer Support&quot;)<br>    print(&quot;=&quot; * 60)<br>    print(&quot;Type your message. Type &#39;exit&#39; to quit. Type &#39;audit&#39; to&quot;)<br>    print(&quot;see the validation error log for this session.\n&quot;)<br>    config = {&quot;configurable&quot;: {&quot;thread_id&quot;: &quot;enterprise-session-001&quot;}}<br>    while True:<br>        user_input = input(&quot;You: &quot;).strip()<br>        if not user_input:<br>            continue<br>        if user_input.lower() == &quot;exit&quot;:<br>            print(&quot;ShopBot: Thank you for contacting ShopBot. Have a great day!&quot;)<br>            break<br>        if user_input.lower() == &quot;audit&quot;:<br>            snapshot = shopbot.get_state(config)<br>            errors = snapshot.values.get(&quot;validation_errors&quot;, [])<br>            print(f&quot;\n[AUDIT LOG] {len(errors)} validation event(s) this session:&quot;)<br>            for i, err in enumerate(errors, 1):<br>                print(f&quot;  {i}. {err}&quot;)<br>            print()<br>            continue<br>        result = shopbot.invoke(<br>            {&quot;messages&quot;: [HumanMessage(content=user_input)]},<br>            config=config,<br>        )<br>        while &quot;__interrupt__&quot; in result:<br>            interrupt_data = result[&quot;__interrupt__&quot;][0].value<br>            print(&quot;\n&quot; + &quot;=&quot; * 60)<br>            print(&quot;  HUMAN REVIEW REQUIRED&quot;)<br>            print(&quot;=&quot; * 60)<br>            for key, value in interrupt_data.items():<br>                if key != &quot;options&quot;:<br>                    print(f&quot;  {key.replace(&#39;_&#39;, &#39; &#39;).title()}: {value}&quot;)<br>            options = interrupt_data.get(&quot;options&quot;, [])<br>            print(f&quot;\n  Options: {&#39; | &#39;.join(f&#39;[{o[0]}] {o}&#39; for o in options)}&quot;)<br>            raw = input(&quot;  Your decision: &quot;).strip().lower()<br>            chosen = next((o for o in options if o.startswith(raw)), options[-1])<br>            result = shopbot.invoke(Command(resume=chosen), config=config)<br>        last = result[&quot;messages&quot;][-1]<br>        print(f&quot;\nShopBot: {last.content}\n&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    run_shopbot()</pre><h3>What Is Different Now and Why It Matters</h3><p>The original ShopBot and the enterprise ShopBot are both correct implementations of the series patterns. But they make different assumptions, and that difference matters when you move from a sandbox to a system with real consequences.</p><p>The original agent assumed the LLM was right. If the LLM extracted “ORD-0O1” (a letter O instead of zero), the tool would error, the agent would probably recover gracefully, and the customer might get a slightly confusing response. Probably fine.</p><p>The enterprise agent assumes the LLM will sometimes be wrong, and builds a recovery path before that wrongness reaches anything external. The gate catches “ORD-0O1” immediately, produces a specific error message that names exactly what was wrong and why, and the correction node feeds that exact message back to the LLM. The LLM gets a second chance with a precise instruction. If it fails again, a human takes over. No tool was called with bad data. No external system saw the hallucination.</p><p>The validation_errors field in state means that when a senior engineer reviews a session that produced a wrong answer, they have a complete log of every gate failure, every correction attempt, and every retry count. They can reproduce the failure, understand what went wrong, and either fix the prompt or tighten the gate. Without that log, debugging a probabilistic system is guesswork.</p><p>The three-file structure means a compliance auditor asking “what validates a refund before money moves?” gets shown gates.py. One file. Pure Python. No agent architecture to explain. Just clear, auditable business rules in a language any developer can read.</p><h3>The Keyword Summary</h3><p>Everything new in this enterprise version, in one place:</p><p>Business Logic Gate — a pure Python function that takes extracted data and returns (True, cleaned_data) or (False, error_reason). Lives in its own file with no LangGraph imports. Runs inside a node before any tool call.</p><p>with_structured_output(PydanticModel) — forces the LLM to return data matching a Pydantic schema instead of free-form text. Pydantic validates the schema before the gate even runs. The combination of structured output and a gate gives you two layers of validation.</p><p>extracted_data: dict in state — the field where structured LLM output lands before gates validate it. Tools read from here, not directly from the LLM&#39;s raw output.</p><p>validation_errors: Annotated[list[str], operator.add] — the audit trail. Every gate failure appends to this list. It accumulates across the whole conversation, not just one turn.</p><p>retry_count: int in state — the infinite loop prevention. The correction node increments it. The gate routing function checks it. If it exceeds the maximum, escalate rather than retry.</p><p>correction_node — the loop recovery node. Reads the most recent validation error, builds a specific correction instruction, re-invokes the LLM with structured output, and routes back to the gate. Shared across all three specialist agents.</p><p>gate_status: str — the communication channel between gate nodes and routing functions. Values: &quot;pending&quot;, &quot;passed&quot;, &quot;failed&quot;, &quot;escalated&quot;, &quot;aborted&quot;. Routing functions read this instead of parsing LLM output directly.</p><p>supervisor_node — reads conversation state, returns a specialist name. Never calls tools, never acts. Routes only.</p><p>Compiled subgraph as a node — graph_builder.add_node(&quot;name&quot;, compiled_subgraph). The parent graph sees a black box. The subgraph handles its own internal routing, gate logic, and tool calls without the parent knowing the details.</p><h3>Conclusion: The Gap Is Closed</h3><p>Read Murat’s comment again. The challenge isn’t the initial graph layout. It’s deterministic error handling and state validation at scale. You cannot afford probabilistic failures. The orchestration layer must be independent of any single LLM vendor.</p><p>Every one of those points is addressed in the code above. The gates are deterministic. The validation loop handles failures without crashing. The audit trail makes every decision inspectable. And the orchestration layer — the gates, the routing functions, the supervisor logic, the correction node — contains not a single LLM-vendor-specific import. Swap ChatOpenAI for ChatAnthropic or any other init_chat_model provider in the configuration, and every gate, every routing function, every business rule runs identically. The LLM is a pluggable module. The architecture is not.</p><p>The original ShopBot was the map. This is the territory.</p><p><em>Check out the project on Github : </em><a href="https://github.com/Bessiedelight/customer-service-agent">https://github.com/Bessiedelight/customer-service-agent</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a63f2607808a" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/improving-our-langgraph-agent-for-real-world-e-commerce-enterprise-validation-business-logic-a63f2607808a">Improving Our LangGraph Agent for Real-World E-Commerce: Enterprise Validation, Business Logic…</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MCP for LangGraph Developers: From Basics to Production]]></title>
            <link>https://pub.towardsai.net/mcp-for-langgraph-developers-from-basics-to-production-12ff52df3d3c?source=rss-4a5744f8548f------2</link>
            <guid isPermaLink="false">https://medium.com/p/12ff52df3d3c</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Bessie Delight Kekeli]]></dc:creator>
            <pubDate>Sun, 21 Jun 2026 20:31:00 GMT</pubDate>
            <atom:updated>2026-06-22T22:24:07.461Z</atom:updated>
            <content:encoded><![CDATA[<h4>The seven-level staircase</h4><p><em>Part 6 of the LangGraph Mental Model series, a ground-up introduction to the Model Context Protocol, building toward full integration with everything from Parts 1–6</em></p><p><em>For other parts of the series : </em><a href="https://medium.com/@bessiedelight/the-building-blocks-of-langgraph-1975904edac8?source=user_profile_page---------4-------------4a5744f8548f----------------------"><em>Part 0</em></a><em> , </em><a href="https://medium.com/towards-artificial-intelligence/the-langgraph-mental-model-a-standardized-architecture-guide-for-every-agent-youll-ever-build-d02265f3bebf?source=user_profile_page---------2-------------4a5744f8548f----------------------"><em>Part 1</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-memory-the-complete-practical-guide-to-managing-what-your-agent-remembers-d865d505a59e?source=user_profile_page---------1-------------4a5744f8548f----------------------"><em>Part 2</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-human-in-the-loop-pausing-reviewing-and-rewinding-your-agent-4028bd05b049?source=user_profile_page---------0-------------4a5744f8548f----------------------"><em>Part 3</em></a><em> </em>, <a href="https://pub.towardsai.net/your-first-real-langgraph-project-ac5eb00f923a"><em>Part 4</em></a><em> , </em><a href="https://medium.com/towards-artificial-intelligence/langgraph-multi-agent-systems-from-one-brain-to-many-4c1773055693"><em>Part 5</em></a>,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gMkSI-Ibe79lZGYc7xwoRw.png" /></figure><blockquote><strong><em>What this article assumes:</em></strong><em> You’re comfortable with the seven-module LangGraph structure and the idea of a </em><em>@tool-decorated function (Part 1, Module 3). That&#39;s the only prerequisite. MCP is a new piece of infrastructure, not a new way of thinking about graphs, so we start from zero on MCP itself, and only reconnect to LangGraph once the concept is solid.</em></blockquote><h3>Why This Article Exists</h3><p>In Part 1, you learned to write tools like this:</p><pre>@tool<br>def search_web(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search the web for current information.&quot;&quot;&quot;<br>    return f&quot;Search results for: {query}&quot;</pre><p>This works great, until it doesn’t scale. Imagine you’re building five different agents across five different projects, and every single one needs a “search the company database” tool. With the pattern above, you’d write that tool five separate times, in five separate codebases, in whatever language each project happens to use. If the database schema changes, you update five places. If you want to share that tool with a teammate building a different kind of AI app, maybe not even using LangGraph, they can’t use your @tool function. It&#39;s tied to LangChain&#39;s Python ecosystem.</p><p>This is the <strong>N×M integration problem</strong>: N different tools (databases, APIs, file systems) need to connect to M different AI applications (your LangGraph agent, a teammate’s custom agent, Claude Desktop, an IDE assistant). Without a shared standard, you end up writing N×M custom integrations.</p><p><strong>The Model Context Protocol (MCP)</strong> solves this by being a universal, open standard for connecting AI applications to tools and data. Build a tool once, as an MCP server, and <em>any</em> MCP-compatible application, our LangGraph agent, Claude Desktop, Cursor, a teammate’s custom agent, can use it immediately, with zero custom integration code. Think of it as a USB-C port for AI applications: one standard connector, any compatible device on either end.</p><p>This article takes you from “what even is MCP” to building and connecting a production-grade MCP server to a real LangGraph agent, in five levels, each adding exactly one new idea.</p><h3>Level 1: The Core Concept — What MCP Actually Is</h3><h3>The Mental Model</h3><p>Before any code, build the picture. Imagine a <strong>hotel concierge desk</strong>. The concierge (your AI application) doesn’t personally know how to book a restaurant, hail a cab, or arrange theater tickets. Instead, the concierge has a list of <strong>trusted local services</strong>, partners who specialize in each task. When a guest asks for a dinner reservation, the concierge picks up a dedicated phone line to the restaurant booking service, makes the request in a standard format both sides understand, and relays the answer back to the guest.</p><p>MCP is that standard phone line and that standard request format. The <strong>AI application</strong> is the concierge. The <strong>MCP server</strong> is a specialized service (a restaurant booker, a cab company, a ticket office). And <strong>MCP</strong> itself is the shared language and connection protocol that lets any concierge talk to any service, without the concierge needing to learn each service’s internal phone system from scratch.</p><h3>The Three Roles</h3><p>Every MCP system has exactly three participants. Get comfortable with these three words, they are the foundation of everything else in this article.</p><p><strong>Host</strong> :the AI application the end user actually interacts with. In our series so far, this is your LangGraph agent. (Other real-world examples: Claude Desktop, Cursor, VS Code with an AI copilot.) The host manages the conversation, decides when a tool is needed, and shows results to the user.</p><p><strong>Client</strong> :lives <em>inside</em> the host. Each client maintains a dedicated, one-to-one connection to exactly one server. If your host connects to three different MCP servers, it spins up three separate clients internally — one per server. This 1:1 mapping is a deliberate security boundary: a client for the file system server can’t accidentally leak data to the database server.</p><p><strong>Server</strong> :a lightweight, focused program that exposes specific capabilities. A server might wrap a database, a file system, a web search API, or your company’s internal CRM. Critically: an MCP server doesn’t know or care which host is calling it. The same server works for your LangGraph agent today and a completely different AI application tomorrow, unmodified.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wtklbug7zHe5KZPXZ9pEbg.png" /></figure><h3>What MCP Communicates With: JSON-RPC 2.0</h3><p>Every message between a client and a server is a JSON-RPC 2.0 message — a simple, well-established format with exactly three message types:</p><p><strong>Request</strong> :“please do this and tell me the result.” Always includes a unique id so the response can be matched back to it.</p><pre>{&quot;jsonrpc&quot;: &quot;2.0&quot;, &quot;id&quot;: 1, &quot;method&quot;: &quot;tools/call&quot;, &quot;params&quot;: {&quot;name&quot;: &quot;search_db&quot;, &quot;arguments&quot;: {&quot;query&quot;: &quot;Q3 revenue&quot;}}}</pre><p><strong>Response</strong> — the answer to a request. Contains either a result (success) or an error (failure), and the same id as the original request.</p><pre>{&quot;jsonrpc&quot;: &quot;2.0&quot;, &quot;id&quot;: 1, &quot;result&quot;: {&quot;content&quot;: [{&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: &quot;Q3 revenue was $4.2M&quot;}]}}</pre><p><strong>Notification</strong> :a one-way message that expects no response. Used for things like “my list of available tools just changed.”</p><p>You will almost never write raw JSON-RPC by hand, the SDKs handle this entirely. But understanding that this is the wire format underneath everything demystifies a lot of what follows. It’s the same reason understanding HTTP underneath a REST API makes you a better API developer, even if you never write raw HTTP by hand.</p><h3>Level 2: The Three Primitives : What a Server Can Offer</h3><p>An MCP server exposes its capabilities through exactly three types of primitives. This is the single most important conceptual table in this article, almost every design decision when building a server comes down to picking the right one of these three.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xpqR24dZhE_EUsQBhd8HSQ.png" /></figure><h3>Tools : Actions the Model Can Take</h3><p>A tool is a function the LLM can decide to call during reasoning, exactly like the @tool-decorated functions from Part 1. The difference is <em>where</em> that function lives: instead of being defined inside your LangGraph codebase, it&#39;s defined inside an MCP server, and your LangGraph agent discovers and calls it over the protocol.</p><p>Use a tool for anything that <strong>does something</strong> , runs a calculation, queries a database, sends an email, modifies a file.</p><h3>Resources : Read-Only Context</h3><p>A resource is data the application can pull in to give the model context, without the model needing to “decide” to call it the way it does with a tool. Resources are addressed by a URI, similar to how a web page is addressed by a URL.</p><p>Use a resource for anything that’s <strong>read-only context</strong> ,a configuration file, a document, a list of available data sources the model might want to know about before deciding what to search for.</p><h3>Prompts : Reusable Templates</h3><p>A prompt is a pre-written template that structures how a user (or the host application) kicks off a particular workflow. Unlike tools, prompts are typically <strong>user-invoked</strong>, not model-invoked , think of them as the MCP equivalent of slash commands.</p><p>Use a prompt for <strong>common, repeatable workflows</strong> ,“summarize this document,” “review this code for security issues” ,where you want consistent, well-engineered instructions every time, rather than relying on the user to phrase the request well.</p><h3>The Decision Rule</h3><p>When you’re not sure which primitive to use for something you’re building, ask: <em>Does this change anything, or just provide information?</em> If it changes something (creates, updates, deletes, sends, executes) → <strong>Tool</strong>. If it’s read-only data the model might want as background → <strong>Resource</strong>. If it’s a reusable instruction template a user explicitly kicks off → <strong>Prompt</strong>.</p><h3>Level 3: Building Your First MCP Server with FastMCP</h3><p>Time to write real code. We’ll use <strong>FastMCP</strong>, the dominant Python framework for building MCP servers, it turns the protocol’s considerable complexity (JSON-RPC handling, schema generation, lifecycle management) into a handful of decorators, in exactly the same spirit as how @tool simplified LangChain tool creation in Part 1.</p><h3>Installation</h3><pre>pip install fastmcp</pre><h3>Your First Server: One Tool</h3><pre># ── math_server.py ───────────────────────────────────────────<br>from fastmcp import FastMCP<br><br># Create the server - this object is the container for everything below<br>mcp = FastMCP(name=&quot;Math Server&quot;)<br>@mcp.tool<br>def add(a: int, b: int) -&gt; int:<br>    &quot;&quot;&quot;Add two numbers together.&quot;&quot;&quot;<br>    return a + b<br>@mcp.tool<br>def multiply(a: int, b: int) -&gt; int:<br>    &quot;&quot;&quot;Multiply two numbers together.&quot;&quot;&quot;<br>    return a * b<br>if __name__ == &quot;__main__&quot;:<br>    mcp.run()  # Defaults to stdio transport</pre><p>This is a complete, valid, runnable MCP server. Notice the parallels to Part 1’s @tool pattern: FastMCP reads your function&#39;s <strong>type hints</strong> to generate the input schema, and reads your <strong>docstring</strong> to generate the description the LLM sees. Exactly the same rule from Part 1 applies here with even higher stakes: a vague docstring means the model won&#39;t know when to call this tool, and now that vagueness affects every application that ever connects to this server, not just one LangGraph agent.</p><h3>Adding a Resource</h3><pre># ── Adding to math_server.py ─────────────────────────────────<br><br>@mcp.resource(&quot;config://server-info&quot;)<br>def server_info() -&gt; dict:<br>    &quot;&quot;&quot;Provides metadata about this server&#39;s capabilities.&quot;&quot;&quot;<br>    return {&quot;version&quot;: &quot;1.0&quot;, &quot;operations&quot;: [&quot;add&quot;, &quot;multiply&quot;]}</pre><p>Resources are addressed by a URI you define (config://server-info here). When a client requests that URI, FastMCP calls your function and returns the result. The function only runs when requested — this is lazy evaluation, so you&#39;re not computing resource data the model never ends up using.</p><h3>A Resource Template (Parameterized Resources)</h3><p>Sometimes you want a resource whose content depends on an argument — like looking up one specific record rather than a fixed configuration blob.</p><pre>@mcp.resource(&quot;history://{operation_id}&quot;)<br>def get_calculation(operation_id: str) -&gt; dict:<br>    &quot;&quot;&quot;Retrieve a past calculation by its ID.&quot;&quot;&quot;<br>    # In a real server, this would look up a database record<br>    return {&quot;id&quot;: operation_id, &quot;result&quot;: &quot;...&quot;}</pre><p>The {operation_id} placeholder in the URI makes this a <strong>resource template</strong>. When a client requests history://abc123, FastMCP extracts abc123 and passes it as the function argument automatically.</p><h3>Adding a Prompt</h3><pre>@mcp.prompt<br>def explain_calculation(operation: str, result: str) -&gt; str:<br>    &quot;&quot;&quot;Generate a request to explain a calculation in plain English.&quot;&quot;&quot;<br>    return f&quot;Explain how the operation &#39;{operation}&#39; produced the result &#39;{result}&#39;, in simple terms a beginner could follow.&quot;</pre><h3>Running and Testing Your Server</h3><pre>python math_server.py</pre><p>Before connecting this to any real AI application, you should verify it works using the <strong>MCP Inspector</strong> — a browser-based debugging tool that ships with the ecosystem and lets you list tools, call them manually, and inspect resources, all without needing an LLM in the loop.</p><pre>fastmcp dev math_server.py</pre><p>This opens a local web interface (typically http://127.0.0.1:6274) where you can click through your tools, fill in test arguments, and see exactly what JSON-RPC messages flow back and forth. <strong>Always test here first.</strong> Debugging a broken tool through an LLM&#39;s tool-calling behavior is much harder than debugging it directly in the Inspector.</p><h3>Level 4: Transports: How Clients and Servers Actually Connect</h3><p>So far, our server has run with the default transport. Now we need to understand the choice, because it directly determines how your LangGraph agent will connect to it.</p><h3>The Mental Model</h3><p>Think of transport as the <strong>physical wire</strong>, separate from the <strong>language spoken over it</strong>. Two people can speak English over a telephone line or over a radio , same language, different wire. Similarly, MCP servers speak the same JSON-RPC 2.0 “language” regardless of transport , but <em>how</em> the bytes travel differs.</p><h3>stdio : For Local, Same-Machine Tools</h3><pre>if __name__ == &quot;__main__&quot;:<br>    mcp.run()  # stdio is the default</pre><p>With <strong>stdio</strong> (standard input/output), the host application launches your server as a <strong>subprocess</strong> and communicates by writing to its stdin and reading from its stdout. There’s no network involved at all , it’s process-to-process communication on the same machine.</p><p>Use stdio when: your server runs locally alongside the host (a file system tool, a local script runner, a personal math utility). This is the simplest transport to set up and the most common for development and personal-use tools.</p><p><strong>Important constraint:</strong> when using stdio, your server’s stdout must be reserved <em>exclusively</em> for MCP protocol messages. If you print() debug output to stdout, you will corrupt the protocol stream. Always log to stderr instead.</p><h3>Streamable HTTP — For Remote, Networked Tools</h3><pre>if __name__ == &quot;__main__&quot;:<br>    mcp.run(transport=&quot;streamable-http&quot;, host=&quot;0.0.0.0&quot;, port=8000)</pre><p>With <strong>Streamable HTTP</strong>, your server runs as a standalone web service. Clients connect over the network using HTTP POST requests, with optional Server-Sent Events for streaming responses back. This is what makes a server genuinely <strong>remote, </strong>it can run on a different machine, in the cloud, behind a load balancer, serving many different host applications simultaneously.</p><p>Use Streamable HTTP when: your server needs to be shared across a team, deployed centrally, or accessed by host applications running on different machines (including a LangGraph agent running on a server, talking to a database-access MCP server running elsewhere).</p><h3>The Decision Rule</h3><p><strong>stdio</strong>: local tool, single user, simple setup, development and personal use. <strong>Streamable HTTP</strong>: shared tool, multiple users or applications, production deployment, needs authentication.</p><p>This exact choice is what you’ll configure when connecting from LangGraph in the next level — every server in your config will declare itself as one or the other.</p><h3>Level 5: Connecting MCP to LangGraph</h3><p>This is where everything from this article meets everything from Parts 1–5. The bridge is a package called <strong>langchain-mcp-adapters</strong>, maintained by the LangChain team specifically to make MCP tools usable inside LangGraph agents.</p><h3>The Core Idea</h3><p>langchain-mcp-adapters does for MCP what the @tool decorator did for plain Python functions in Part 1: it converts something external (an MCP tool) into a BaseTool object that slots directly into a LangGraph ToolNode, with zero special-casing in your graph logic. Once converted, your agent doesn&#39;t know or care that a tool came from an MCP server rather than being written in-process , it&#39;s just a tool.</p><h3>Installation</h3><pre>pip install langchain-mcp-adapters langgraph &quot;langchain[openai]&quot;</pre><h3>Step 1: Configure Your MCP Servers</h3><p>The central object is MultiServerMCPClient — it can connect to multiple MCP servers at once, mixing transports freely.</p><pre># ── MODULE 1 (extended): MCP CLIENT CONFIGURATION ───────────<br>from langchain_mcp_adapters.client import MultiServerMCPClient<br><br>client = MultiServerMCPClient({<br>    # A local server, launched as a subprocess via stdio<br>    &quot;math&quot;: {<br>        &quot;command&quot;: &quot;python&quot;,<br>        &quot;args&quot;: [&quot;/absolute/path/to/math_server.py&quot;],<br>        &quot;transport&quot;: &quot;stdio&quot;,<br>    },<br>    # A remote server, already running, connected over HTTP<br>    &quot;company_data&quot;: {<br>        &quot;url&quot;: &quot;http://localhost:8000/mcp&quot;,<br>        &quot;transport&quot;: &quot;http&quot;,<br>    },<br>})</pre><p>Each entry in this dictionary is one server. The key (&quot;math&quot;, &quot;company_data&quot;) is just a label you choose — it&#39;s how langchain-mcp-adapters keeps tools from different servers organized internally. Notice this directly mirrors the transport decision from Level 4: stdio servers get a command + args, HTTP servers get a url.</p><h3>Step 2: Load the Tools</h3><pre># This single call discovers every tool, on every configured server,<br># and converts each one into a LangChain-compatible BaseTool<br>tools = await client.get_tools()<br><br>print([t.name for t in tools])<br># [&#39;add&#39;, &#39;multiply&#39;, &#39;search_company_db&#39;, ...]</pre><p>This is the entire integration. client.get_tools() performs the MCP tools/list request against every server in your config, takes the returned schemas, and wraps each one into a tool object indistinguishable , from your graph&#39;s perspective — from a hand-written @tool function.</p><h3>Step 3: The Familiar LangGraph Wiring</h3><p>From here, every module looks exactly like Part 1. This is the entire point of the integration: <strong>MCP changes where your tools come from, not how your graph is built.</strong></p><pre># ============================================================<br># LANGGRAPH + MCP AGENT — COMPLETE TEMPLATE<br># Extends: Part 1 (core structure)<br># ============================================================<br><br># ── MODULE 1: IMPORTS &amp; CONFIGURATION ───────────────────────<br>import asyncio<br>from langchain_mcp_adapters.client import MultiServerMCPClient<br>from langchain.chat_models import init_chat_model<br>from langchain_core.messages import HumanMessage<br>from langgraph.graph import StateGraph, MessagesState, START, END<br>from langgraph.prebuilt import ToolNode, tools_condition<br>from langgraph.checkpoint.memory import MemorySaver<br>llm = init_chat_model(&quot;openai:gpt-4o&quot;)<br>client = MultiServerMCPClient({<br>    &quot;math&quot;: {<br>        &quot;command&quot;: &quot;python&quot;,<br>        &quot;args&quot;: [&quot;./math_server.py&quot;],<br>        &quot;transport&quot;: &quot;stdio&quot;,<br>    },<br>    &quot;company_data&quot;: {<br>        &quot;url&quot;: &quot;http://localhost:8000/mcp&quot;,<br>        &quot;transport&quot;: &quot;http&quot;,<br>    },<br>})<br><br># ── MODULE 2: STATE ──────────────────────────────────────────<br>class State(MessagesState):<br>    pass<br><br># ── MODULE 4: NODES ──────────────────────────────────────────<br># (Module 3, Tools, is now sourced from MCP rather than defined here)<br>def make_agent_node(llm_with_tools):<br>    &quot;&quot;&quot;Factory pattern: tools are loaded asynchronously, so the node<br>    closure is built after the async tool-loading step completes.&quot;&quot;&quot;<br>    def agent_node(state: State) -&gt; dict:<br>        response = llm_with_tools.invoke(state[&quot;messages&quot;])<br>        return {&quot;messages&quot;: [response]}<br>    return agent_node<br><br># ── MODULE 6: GRAPH ASSEMBLY (inside an async function) ──────<br>async def build_graph():<br>    tools = await client.get_tools()<br>    llm_with_tools = llm.bind_tools(tools)<br>    <br>    graph_builder = StateGraph(State)<br>    graph_builder.add_node(&quot;agent&quot;, make_agent_node(llm_with_tools))<br>    graph_builder.add_node(&quot;tools&quot;, ToolNode(tools))<br>    <br>    graph_builder.add_edge(START, &quot;agent&quot;)<br>    graph_builder.add_conditional_edges(&quot;agent&quot;, tools_condition)<br>    graph_builder.add_edge(&quot;tools&quot;, &quot;agent&quot;)<br>    <br>    return graph_builder.compile(checkpointer=MemorySaver())<br><br># ── MODULE 7: ENTRYPOINT ──────────────────────────────────────<br>async def main():<br>    graph = await build_graph()<br>    config = {&quot;configurable&quot;: {&quot;thread_id&quot;: &quot;session-001&quot;}}<br>    <br>    response = await graph.ainvoke(<br>        {&quot;messages&quot;: [HumanMessage(content=&quot;What&#39;s (3 + 5) times 12?&quot;)]},<br>        config=config<br>    )<br>    print(response[&quot;messages&quot;][-1].content)<br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><h3>Why Everything Is Async Now</h3><p>This is the one genuine adjustment MCP introduces to the patterns from earlier parts: <strong>MCP communication is inherently asynchronous</strong>, because every tool call is, under the hood, a network or subprocess round-trip , even for the stdio transport. This means client.get_tools() must be awaited, your graph-building step typically lives inside an async def, and you invoke the compiled graph with ainvoke instead of invoke. Everything else, StateGraph, MessagesState, tools_condition, MemorySaver, is identical to every prior article in this series.</p><h3>tools_condition: The Pre-Built Router Returns</h3><p>Notice Module 5 (Routing) effectively disappears from this template — we use tools_condition, the pre-built router from Part 1, instead of writing a custom one. This isn&#39;t a new concept; it&#39;s the same tools_condition from the very first canonical template, doing exactly the same job: check if the last message has tool calls, route to &quot;tools&quot; if so, otherwise END. MCP tools and hand-written tools are indistinguishable to this router, because they&#39;re both just BaseTool objects by the time your graph sees them.</p><h3>Level 6: Production Considerations</h3><p>A few things separate a working demo from a system you’d actually deploy. These are the most commonly hit issues in real MCP + LangGraph systems.</p><h3>Keep the Client Connection Alive for the Process Lifetime</h3><p>MultiServerMCPClient is stateless by default , every tool call opens a fresh connection, executes, and tears down. For a single request that&#39;s fine, but if your application serves many requests, opening a subprocess or HTTP connection on every single tool call adds real latency. In production, open the client once when your process starts and reuse it for the lifetime of the application, rather than recreating it per-request.</p><pre># Hold the connection open across the application&#39;s lifetime,<br># rather than opening/closing per request<br>async with MultiServerMCPClient({...}) as client:<br>    tools = await client.get_tools()<br>    graph = await build_graph_with(tools)<br>    # ... serve many requests using this same graph ...</pre><h3>Don’t Use stdio in a Web Server Context</h3><p>The stdio transport was designed for local, single-user, same-machine scenarios , like a desktop app launching a helper subprocess. If you’re deploying your LangGraph agent as a web service handling requests from many users, stdio servers become a liability: you’d be spawning subprocesses per request, with no natural way to share state or scale horizontally. For any server-side deployment, prefer Streamable HTTP, and genuinely ask whether you need a separate MCP server process at all — sometimes a plain @tool function calling an internal library directly is simpler and faster than going through the protocol.</p><h3>Combine with Human-in-the-Loop for Sensitive Tools</h3><p>MCP tools are still just tools by the time they reach your graph , which means every pattern from Part 3 applies unchanged. If an MCP server exposes something sensitive (sending emails, modifying a database, spending money), wrap it with the same interrupt() review pattern from Part 3&#39;s review_tool_call node. MCP gives you discovery and standardization; it does not give you safety by default. That&#39;s still your job, using the patterns you already know.</p><h3>MCP Servers Can’t See Your LangGraph State</h3><p>This is a structural fact worth internalizing: MCP servers run as <strong>separate processes</strong>. They cannot reach into your LangGraph state, your checkpointer, or any in-memory Python objects in your graph. If an MCP tool call needs to be personalized , using a stored user preference, for example , that information has to be passed explicitly as a tool argument, or handled through the more advanced <strong>interceptor</strong> pattern (a middleware hook in langchain-mcp-adapters that lets you inspect and modify a tool call before it&#39;s sent, using your graph&#39;s runtime context). For most agents, simply passing the needed values as arguments is sufficient and far simpler.</p><h3>Level 7: MCP in a Multi-Agent System (Tying Back to Part 4)</h3><p>To show this genuinely composes with everything earlier in the series, here’s how MCP slots into the supervisor pattern from Part 4 : multiple specialist agents, each backed by its own dedicated MCP server.</p><pre># ── Specialist agents, each pulling tools from a DIFFERENT MCP server ──<br><br>async def build_multi_agent_system():<br>    client = MultiServerMCPClient({<br>        &quot;research&quot;: {&quot;url&quot;: &quot;http://localhost:8001/mcp&quot;, &quot;transport&quot;: &quot;http&quot;},<br>        &quot;code&quot;:     {&quot;url&quot;: &quot;http://localhost:8002/mcp&quot;, &quot;transport&quot;: &quot;http&quot;},<br>    })<br>    <br>    research_tools = await client.get_tools(server_name=&quot;research&quot;)<br>    code_tools     = await client.get_tools(server_name=&quot;code&quot;)<br>    <br>    researcher_llm = llm.bind_tools(research_tools)<br>    coder_llm      = llm.bind_tools(code_tools)<br>    <br>    # From here, wire researcher_node, coder_node, and a supervisor_node<br>    # exactly as shown in Part 4 - the rest of the supervisor pattern<br>    # is completely unchanged.<br>    ...</pre><p>The takeaway: MCP doesn’t introduce a competing architecture to multi-agent systems. A specialist agent backed by an MCP server is structurally identical to a specialist agent backed by hand-written tools , the supervisor still routes the same way, the specialists still report back the same way. MCP simply changes <em>where the tool definitions live and how many applications can reuse them.</em></p><h3>The Updated Keyword Reference Card</h3><p>This extends the keyword cards from Parts 1–5 with MCP-specific terms.</p><p><strong>MCP Architecture Keywords</strong> Host — the AI application the user interacts with (your LangGraph agent). Client — lives inside the host, one per connected server, manages a 1:1 session. Server — the standalone program exposing tools, resources, and prompts. Doesn&#39;t know which host is connecting. JSON-RPC 2.0 — the message format underneath every MCP exchange: Requests, Responses, Notifications.</p><p><strong>The Three Primitives</strong> Tool — an executable action, model decides when to call it. Built with @mcp.tool. Resource — read-only data, addressed by URI, application/user-controlled. Built with @mcp.resource(&quot;uri://...&quot;). Prompt — a reusable instruction template, user-invoked. Built with @mcp.prompt.</p><p><strong>FastMCP Server Keywords</strong> FastMCP(name=&quot;...&quot;) — creates the server instance, the container for all tools/resources/prompts. @mcp.tool — decorator that exposes a Python function as a callable tool; reads type hints for schema, docstring for description. @mcp.resource(&quot;scheme://{param}&quot;) — decorator for read-only data; {param} placeholders create a resource template. @mcp.prompt — decorator for a reusable prompt template. mcp.run() — starts the server. Defaults to stdio; pass transport=&quot;streamable-http&quot; for networked deployment. fastmcp dev server.py — launches the MCP Inspector for manual testing without an LLM.</p><p><strong>Transport Keywords</strong> stdio — subprocess-based, local, same-machine. The host launches and manages the server process. Never write debug output to stdout. streamable-http — networked, remote-capable, supports many concurrent host connections. The production default for shared servers.</p><p><strong>LangGraph Integration Keywords</strong> MultiServerMCPClient({...}) — from langchain_mcp_adapters.client. Connects to one or more MCP servers, mixing transports freely in one config dict. await client.get_tools() — discovers and converts every tool from every configured server into LangChain-compatible BaseTool objects. tools_condition — (from Part 1) the pre-built LangGraph router; works identically whether tools are hand-written or MCP-sourced. <strong>Everything is async</strong> — client.get_tools(), graph building, and graph.ainvoke() all require await, because MCP calls are network/subprocess round-trips even over stdio.</p><h3>The Decision Guide: When Do You Actually Need MCP?</h3><p><strong>You’re building a single tool for a single agent in a single codebase</strong> → You don’t need MCP. A plain @tool function from Part 1 is simpler and has zero protocol overhead.</p><p><strong>You’re building a tool that should be reusable across multiple agents, multiple projects, or shared with teammates using different frameworks</strong> → Build it as an MCP server. This is the core value proposition.</p><p><strong>You want to use one of the hundreds of pre-built community MCP servers</strong> (GitHub, Slack, Postgres, Google Drive, Stripe, and more) <strong>instead of writing integration code yourself</strong> → Connect via MultiServerMCPClient and skip writing that integration entirely. This is often the single biggest time-saver MCP offers.</p><p><strong>Your tool needs to run on a different machine from your agent, or be shared by multiple host applications at once</strong> → Use Streamable HTTP transport, deployed as a standalone service.</p><p><strong>Your tool only ever runs locally, alongside one agent, for development or personal use</strong> → stdio transport is sufficient, and simpler to set up.</p><h3>Conclusion: A Standard, Not a Replacement</h3><p>The instinct when learning a new protocol is to wonder whether it replaces what you already know. It doesn’t. MCP doesn’t replace LangGraph’s tools, nodes, or graphs — it replaces <em>where those tools come from</em> and <em>how many places can reuse them</em>. Once an MCP tool reaches your ToolNode, it behaves exactly like the hand-written tools from Part 1, gets approved by the same interrupt() patterns from Part 3, and slots into the same supervisor architectures from Part 4.</p><p>The progression in this article followed the same staircase as every other part of this series: concept and roles (Level 1), the three primitives (Level 2), a working server (Level 3), the transport choice (Level 4), the LangGraph bridge (Level 5), production hardening (Level 6), and composition with multi-agent systems (Level 7). Each level added exactly one idea on top of a stable foundation.</p><p>You now have six articles’ worth of production scaffold: canonical structure (Part 1), memory management (Part 2), human-in-the-loop safety (Part 3), multi-agent orchestration (Part 4), real-world knowledge via RAG (Part 5), and standardized, shareable tooling via MCP (Part 6). Between them, these cover the overwhelming majority of what a serious, production-grade LangGraph application needs , and MCP is what lets the tooling half of that scale beyond any single codebase.</p><p><em>For other parts of the series : </em><a href="https://medium.com/@bessiedelight/the-building-blocks-of-langgraph-1975904edac8?source=user_profile_page---------4-------------4a5744f8548f----------------------"><em>Part 0</em></a><em> , </em><a href="https://medium.com/towards-artificial-intelligence/the-langgraph-mental-model-a-standardized-architecture-guide-for-every-agent-youll-ever-build-d02265f3bebf?source=user_profile_page---------2-------------4a5744f8548f----------------------"><em>Part 1</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-memory-the-complete-practical-guide-to-managing-what-your-agent-remembers-d865d505a59e?source=user_profile_page---------1-------------4a5744f8548f----------------------"><em>Part 2</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-human-in-the-loop-pausing-reviewing-and-rewinding-your-agent-4028bd05b049?source=user_profile_page---------0-------------4a5744f8548f----------------------"><em>Part 3</em></a><em> </em>, <a href="https://pub.towardsai.net/your-first-real-langgraph-project-ac5eb00f923a"><em>Part 4</em></a><em> , </em><a href="https://medium.com/towards-artificial-intelligence/langgraph-multi-agent-systems-from-one-brain-to-many-4c1773055693"><em>Part 5</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12ff52df3d3c" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/mcp-for-langgraph-developers-from-basics-to-production-12ff52df3d3c">MCP for LangGraph Developers: From Basics to Production</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG Without the Guesswork: A Standardized LangGraph + LlamaIndex Pattern.]]></title>
            <link>https://pub.towardsai.net/rag-without-the-guesswork-a-standardized-langgraph-llamaindex-pattern-bcaf14f9c811?source=rss-4a5744f8548f------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcaf14f9c811</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[llamaindex]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Bessie Delight Kekeli]]></dc:creator>
            <pubDate>Sun, 21 Jun 2026 16:01:01 GMT</pubDate>
            <atom:updated>2026-06-22T09:47:57.711Z</atom:updated>
            <content:encoded><![CDATA[<h4>LangGraph + LlamaIndex: Giving Your Agent Real Knowledge with RAG</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1EA_oPnBdeTITlcJHNEEZA.png" /></figure><p><em>Part 6 of the LangGraph Mental Model series , a focused detour into Retrieval-Augmented Generation, connected back to the canonical structure from Parts 1–4</em></p><p><em>For other parts of the series : </em><a href="https://medium.com/@bessiedelight/the-building-blocks-of-langgraph-1975904edac8?source=user_profile_page---------4-------------4a5744f8548f----------------------"><em>Part 0</em></a><em> , </em><a href="https://medium.com/towards-artificial-intelligence/the-langgraph-mental-model-a-standardized-architecture-guide-for-every-agent-youll-ever-build-d02265f3bebf?source=user_profile_page---------2-------------4a5744f8548f----------------------"><em>Part 1</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-memory-the-complete-practical-guide-to-managing-what-your-agent-remembers-d865d505a59e?source=user_profile_page---------1-------------4a5744f8548f----------------------"><em>Part 2</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-human-in-the-loop-pausing-reviewing-and-rewinding-your-agent-4028bd05b049?source=user_profile_page---------0-------------4a5744f8548f----------------------"><em>Part 3</em></a><em> </em>, <a href="https://pub.towardsai.net/your-first-real-langgraph-project-ac5eb00f923a"><em>Part 4</em></a><em> , </em><a href="https://medium.com/towards-artificial-intelligence/langgraph-multi-agent-systems-from-one-brain-to-many-4c1773055693"><em>Part 5</em></a></p><blockquote><strong><em>What this article assumes:</em></strong><em> You’re comfortable with the seven-module structure and know how to give a LangGraph agent a tool (Part 1, Module 3). That’s genuinely all you need. If not, you can still follow along very easily. We are deliberately rolling complexity back down to a single agent here, no supervisors, no subgraphs, no </em><em>Send. RAG is a </em>capability<em> you add to an agent, not a new orchestration pattern, and this article treats it that way.</em></blockquote><h3>Why This Article Exists</h3><p>Every agent you’ve built so far in this series has one blind spot: it only knows what the LLM was trained on. Ask it about your company’s internal pricing policy, last quarter’s financial report, or a product manual you wrote last week, and it will either say “I don’t know” or — worse — confidently make something up.</p><p><strong>Retrieval-Augmented Generation (RAG)</strong> fixes this. Instead of relying purely on what the model memorized during training, RAG retrieves relevant snippets from <em>your own documents</em> at query time and feeds them to the LLM as context before it answers. The LLM is no longer guessing ,it’s reading.</p><p>You could build a RAG pipeline by hand inside LangGraph: chunk documents yourself, generate embeddings yourself, build a vector index yourself, write the retrieval logic yourself. People do this. It’s a lot of plumbing for something that’s become a solved problem.</p><p><strong>LlamaIndex</strong> is a framework built specifically to solve that plumbing problem. It is, by a wide margin, the most mature and widely used tool for the “data” half of an AI application ,loading, chunking, embedding, indexing, and retrieving. LangGraph, as you already know, is built for the “orchestration” half — state, control flow, multi-step reasoning.</p><p>This article shows you how to use each tool for what it’s best at: <strong>LlamaIndex builds and serves the knowledge base. LangGraph wraps it as a tool and decides when to use it.</strong> This is the dominant pattern in real production RAG systems today, and it’s the pattern we’ll build, step by step.</p><h3>Part A: Understanding LlamaIndex (The Concept First)</h3><p>Before writing code, you need the same kind of mental model for LlamaIndex that you built for LangGraph in Part 1. Let’s build it the same way: concept, then keywords, then template.</p><h3>What LlamaIndex Actually Does</h3><p>Think of LlamaIndex as a <strong>librarian for your LLM</strong>. Imagine you hired a brilliant research assistant (the LLM) who has read most of the internet but has never seen your company’s files. You can’t just hand them a 10,000-page binder and say “remember all of this” — they don’t have room to hold it all in their head at once.</p><p>So instead, you hire a librarian (LlamaIndex). The librarian’s job has four parts: <strong>read</strong> all your documents, <strong>organize</strong> them into small, searchable index cards (chunks), <strong>file</strong> those cards in a system that allows fast lookup by topic (a vector index), and when your research assistant has a question, <strong>fetch</strong> only the 3–5 most relevant cards and hand them over.</p><p>The research assistant never has to memorize the binder. They just ask the librarian, get a tiny handful of relevant snippets, and reason over those.</p><h3>The Five-Stage RAG Pipeline</h3><p>Every RAG system, regardless of framework, follows this same five-stage pipeline. LlamaIndex gives you a clean abstraction for each stage:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hnzOPXm85PEkJzcblU8_zw.png" /></figure><pre>1. LOAD       → Read raw files (PDF, Word, web pages, databases) into Documents<br>2. CHUNK      → Split Documents into small, retrievable Nodes<br>3. EMBED      → Convert each Node&#39;s text into a vector (a list of numbers<br>                representing meaning)<br>4. STORE      → Save those vectors in a Vector Index for fast lookup<br>5. RETRIEVE   → At query time, embed the user&#39;s question, find the most<br>                  similar Nodes, and return them as context</pre><p>Stages 1–4 happen once, when you build your knowledge base (this is called <strong>indexing</strong>). Stage 5 happens every time a user asks a question (this is called <strong>querying</strong>). This distinction — indexing time vs. query time — is the single most important mental model in all of RAG. Keep it in your head as you read everything that follows.</p><h3>The Core Keywords You Need to Know</h3><p><strong>Document</strong> — a raw piece of source content (one PDF, one web page, one database row) plus its metadata. This is the unprocessed, full-length unit of data.</p><p><strong>Node</strong> — a chunk of a Document. LlamaIndex splits each Document into multiple smaller Node objects because LLMs and embedding models work better with focused, bite-sized text rather than entire documents at once. Nodes carry metadata back to their parent document (so you can always trace a chunk back to its source).</p><p><strong>SimpleDirectoryReader</strong> — the most common loader. Points at a folder and automatically reads PDFs, text files, Word docs, and more, turning each file into one or more Document objects.</p><p><strong>VectorStoreIndex</strong> — the core indexing class. Takes a list of Document (or Node) objects, embeds them, and builds a searchable vector index in memory (or in an external vector database, if you configure one).</p><p><strong>Settings</strong> — a global configuration object. Instead of passing your LLM and embedding model into every function call, you set them once on Settings and every LlamaIndex component uses them automatically.</p><p><strong>QueryEngine</strong> — the object you actually &quot;ask questions&quot; to. Created by calling .as_query_engine() on an index. Internally it does embedding the query, similarity search, and feeding retrieved context to the LLM to generate a final answer — all in one call.</p><p><strong>Retriever</strong> — a lower-level object created by calling .as_retriever() on an index. Unlike a QueryEngine, a Retriever only does the <em>search</em> step — it returns the raw matching Node objects without generating an LLM answer. You&#39;ll use this when you want retrieval and generation to be separate, controllable steps (which is exactly what we want once we bring in LangGraph).</p><h3>Part B: Building a Standalone LlamaIndex Knowledge Base</h3><p>Let’s build a working LlamaIndex pipeline on its own first — no LangGraph yet. This mirrors how Part 1 taught LangGraph’s modules in isolation before assembling them.</p><h3>Step 1: Installation</h3><pre># Core package + OpenAI LLM and embedding integrations (the common starting setup)<br>pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai<br><br># Readers for common file types (PDF, Word, etc.)<br>pip install llama-index-readers-file pypdf</pre><p>LlamaIndex is intentionally split into many small packages. llama-index-core has the framework logic. Everything else : which LLM, which embedding model, which vector database, which file reader is its own package you add only when you need it. This keeps your installs lean.</p><h3>Step 2: Global Configuration with Settings</h3><pre># ── llamaindex_config.py ─────────────────────────────────────<br>import os<br>from llama_index.core import Settings<br>from llama_index.llms.openai import OpenAI<br>from llama_index.embeddings.openai import OpenAIEmbedding<br><br># Settings is global - configure once, used everywhere in LlamaIndex<br>Settings.llm = OpenAI(<br>    model=&quot;gpt-4o-mini&quot;,       # Used for generating final answers from retrieved context<br>    temperature=0.1,            # Low temperature: factual, not creative<br>)<br>Settings.embed_model = OpenAIEmbedding(<br>    model=&quot;text-embedding-3-small&quot;,   # Used to convert text into vectors<br>)<br># Controls how documents are split into Nodes (chunks)<br>Settings.chunk_size = 512        # Max tokens per chunk<br>Settings.chunk_overlap = 50      # Overlap between consecutive chunks, to preserve context across boundaries</pre><p>A quick but important note on chunk_size: too large, and each chunk contains so much text that retrieval becomes imprecise (you fetch a chunk that&#39;s only 10% relevant). Too small, and you lose surrounding context that the LLM needs to understand a fact. 512 tokens with a 50-token overlap is a reasonable, widely-used starting point ,tune it based on your documents.</p><h3>Step 3: Load → Index → Query (The Standalone Pipeline)</h3><pre># ── build_knowledge_base.py ──────────────────────────────────<br>from llama_index.core import SimpleDirectoryReader, VectorStoreIndex<br><br># ── LOAD: Read all files in a folder into Document objects ──<br>documents = SimpleDirectoryReader(&quot;./data&quot;).load_data()<br>print(f&quot;Loaded {len(documents)} documents&quot;)<br># ── CHUNK + EMBED + STORE: all three happen inside this one call ──<br># VectorStoreIndex automatically:<br>#   1. Splits each Document into Nodes (using Settings.chunk_size)<br>#   2. Embeds each Node (using Settings.embed_model)<br>#   3. Stores the vectors in an in-memory index<br>index = VectorStoreIndex.from_documents(documents, show_progress=True)<br># ── RETRIEVE + GENERATE: ask a question ──────────────────────<br>query_engine = index.as_query_engine(<br>    similarity_top_k=3,   # Retrieve the 3 most relevant chunks for each query<br>)<br>response = query_engine.query(&quot;What is our refund policy for enterprise customers?&quot;)<br>print(response)</pre><p>That’s a complete, working RAG system in about ten lines. This is genuinely the “hello world” of LlamaIndex, and it’s a fair demonstration of why the framework exists — compare this to manually writing a chunker, an embedding loop, a similarity search function, and a prompt template yourself.</p><h3>Step 4: Persisting the Index (Don’t Re-Embed Every Time)</h3><p>Building the index calls the embedding model for every chunk — which costs money and time. You don’t want to rebuild it every time your app starts. Persist it to disk.</p><pre># ── Save the index after building it ─────────────────────────<br>index.storage_context.persist(persist_dir=&quot;./storage&quot;)<br><br># ── Load it back later without re-embedding anything ──────────<br>from llama_index.core import StorageContext, load_index_from_storage<br>storage_context = StorageContext.from_defaults(persist_dir=&quot;./storage&quot;)<br>index = load_index_from_storage(storage_context)</pre><p>storage_context.persist() writes the vector data, the node metadata, and the document store to a local folder. load_index_from_storage() reconstructs the exact same index from those files — no re-embedding, no API calls. This is the standalone equivalent of the checkpointer pattern you learned in Part 2: build once, persist, reload cheaply.</p><h3>Step 5: Using an External Vector Database (Chroma)</h3><p>The in-memory index above is fine for prototypes, but for production you typically want a dedicated vector database — something built to handle millions of vectors, concurrent reads, and metadata filtering at scale. Vector DB: <a href="https://www.trychroma.com/">Chroma</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jdhtSiWmLhNsZnWRcGz71A.png" /></figure><pre># ── Using Chroma as a persistent, production-grade vector store ──<br># pip install llama-index-vector-stores-chroma chromadb<br><br>import chromadb<br>from llama_index.vector_stores.chroma import ChromaVectorStore<br>from llama_index.core import StorageContext, VectorStoreIndex<br>chroma_client = chromadb.PersistentClient(path=&quot;./chroma_db&quot;)<br>chroma_collection = chroma_client.get_or_create_collection(&quot;my_knowledge_base&quot;)<br>vector_store = ChromaVectorStore(chroma_collection=chroma_collection)<br>storage_context = StorageContext.from_defaults(vector_store=vector_store)<br># Build the index directly into Chroma<br>index = VectorStoreIndex.from_documents(<br>    documents,<br>    storage_context=storage_context<br>)<br># Later, in a different process, reconnect without re-indexing:<br>index = VectorStoreIndex.from_vector_store(vector_store=vector_store)</pre><p>The pattern is identical to what you learned with checkpointers in Part 2: swap one object (MemorySaver → SqliteSaver, or here, in-memory → ChromaVectorStore) and everything downstream,your query engine, your retrieval logic — stays exactly the same. The same swap pattern works for Pinecone, Qdrant, Weaviate, and most other vector databases LlamaIndex supports.</p><h3>Part C: Connecting LlamaIndex to LangGraph</h3><p>Now for the part this whole article has been building toward. We need to bridge two frameworks that speak different languages: LlamaIndex’s QueryEngine is not a LangGraph tool. LangGraph&#39;s agent expects tools built with LangChain&#39;s @tool decorator.</p><h3>The Bridge: Wrap the Query Engine as a Tool</h3><p>The integration pattern is refreshingly simple once you see it: <strong>write a regular Python function, decorated with </strong><strong>@tool, whose body calls </strong><strong>query_engine.query(...).</strong> That&#39;s the entire bridge. LangGraph doesn&#39;t need to know anything about LlamaIndex internals — it just sees a tool that takes a string and returns a string, exactly like every other tool from Part 1.</p><pre># ── MODULE 3: TOOLS (LlamaIndex-backed) ─────────────────────<br>from langchain_core.tools import tool<br><br># The query_engine built in Part B - created once, at startup<br># (In a real app, you&#39;d load this from persisted storage, not rebuild it every time)<br>@tool<br>def search_knowledge_base(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search the internal knowledge base for company policies, product <br>    documentation, and internal procedures. Use this whenever the user asks <br>    a question that might be answered by internal company documents rather <br>    than general knowledge.<br>    <br>    Args:<br>        query: A natural-language question to search for.<br>    <br>    Returns:<br>        A synthesized answer based on the most relevant retrieved documents.<br>    &quot;&quot;&quot;<br>    response = query_engine.query(query)<br>    return str(response)</pre><p>That’s it. The docstring matters exactly as much as it did in Part 1 — it’s how the LLM in your LangGraph agent decides <em>when</em> to call this tool versus answering from its own knowledge or calling a different tool.</p><h3>The Complete Integration: Modules 1 Through 7</h3><p>Here is the full canonical agent template from Part 1, with a LlamaIndex-backed RAG tool slotted directly into Module 3. Nothing else changes.</p><pre># ============================================================<br># LANGGRAPH + LLAMAINDEX RAG AGENT — COMPLETE TEMPLATE<br># Extends: Part 1 (core structure)<br># ============================================================<br><br># ── MODULE 1: IMPORTS &amp; CONFIGURATION ───────────────────────<br>import os<br>from typing import Literal<br># LangChain / LangGraph imports (the orchestration layer)<br>from langchain_openai import ChatOpenAI<br>from langchain_core.messages import HumanMessage, SystemMessage, BaseMessage<br>from langchain_core.tools import tool<br>from langgraph.graph import StateGraph, MessagesState, START, END<br>from langgraph.prebuilt import ToolNode<br>from langgraph.checkpoint.memory import MemorySaver<br># LlamaIndex imports (the retrieval / data layer)<br>from llama_index.core import (1<br>    Settings, SimpleDirectoryReader, VectorStoreIndex,<br>    StorageContext, load_index_from_storage<br>)<br>from llama_index.llms.openai import OpenAI as LlamaOpenAI<br>from llama_index.embeddings.openai import OpenAIEmbedding<br># LangGraph&#39;s chat model - used by the agent&#39;s reasoning<br>llm = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br># LlamaIndex&#39;s model config - used internally by the query engine<br># Note: these are SEPARATE from the LangGraph llm above. Each framework<br># manages its own model instances; they don&#39;t share state.<br>Settings.llm = LlamaOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0.1)<br>Settings.embed_model = OpenAIEmbedding(model=&quot;text-embedding-3-small&quot;)<br>Settings.chunk_size = 512<br><br># ── MODULE 2: STATE ──────────────────────────────────────────<br>class State(MessagesState):<br>    pass  # messages field inherited; extend if your agent needs more<br><br># ── MODULE 3: TOOLS (RAG-backed) ─────────────────────────────<br># Build or load the LlamaIndex knowledge base ONCE, at startup<br>PERSIST_DIR = &quot;./storage&quot;<br>if os.path.exists(PERSIST_DIR):<br>    # Reload existing index - no re-embedding, fast startup<br>    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)<br>    index = load_index_from_storage(storage_context)<br>else:<br>    # First run - build the index and persist it<br>    documents = SimpleDirectoryReader(&quot;./data&quot;).load_data()<br>    index = VectorStoreIndex.from_documents(documents, show_progress=True)<br>    index.storage_context.persist(persist_dir=PERSIST_DIR)<br>query_engine = index.as_query_engine(similarity_top_k=3)<br><br>@tool<br>def search_knowledge_base(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search internal company documents for policies, product specs, <br>    procedures, and other domain-specific information. Use this for any <br>    question that requires knowledge specific to this organization rather <br>    than general world knowledge.&quot;&quot;&quot;<br>    response = query_engine.query(query)<br>    return str(response)<br><br>tools = [search_knowledge_base]<br>llm_with_tools = llm.bind_tools(tools)<br>tool_node = ToolNode(tools)<br><br># ── MODULE 4: NODES ──────────────────────────────────────────<br>def agent_node(state: State) -&gt; dict:<br>    &quot;&quot;&quot;The reasoning node. Decides whether to answer directly or <br>    search the knowledge base first.&quot;&quot;&quot;<br>    system_prompt = SystemMessage(content=(<br>        &quot;You are a helpful assistant with access to an internal knowledge base. &quot;<br>        &quot;Use the search_knowledge_base tool when the user asks about company-specific &quot;<br>        &quot;information. For general questions, answer directly.&quot;<br>    ))<br>    messages = [system_prompt] + state[&quot;messages&quot;]<br>    response = llm_with_tools.invoke(messages)<br>    return {&quot;messages&quot;: [response]}<br><br># ── MODULE 5: ROUTING ────────────────────────────────────────<br>def should_continue(state: State) -&gt; Literal[&quot;tools&quot;, &quot;__end__&quot;]:<br>    last_message = state[&quot;messages&quot;][-1]<br>    if hasattr(last_message, &quot;tool_calls&quot;) and last_message.tool_calls:<br>        return &quot;tools&quot;<br>    return &quot;__end__&quot;<br><br># ── MODULE 6: GRAPH ASSEMBLY ─────────────────────────────────<br>graph_builder = StateGraph(State)<br>graph_builder.add_node(&quot;agent&quot;, agent_node)<br>graph_builder.add_node(&quot;tools&quot;, tool_node)<br>graph_builder.add_edge(START, &quot;agent&quot;)<br>graph_builder.add_conditional_edges(<br>    &quot;agent&quot;, should_continue,<br>    {&quot;tools&quot;: &quot;tools&quot;, &quot;__end__&quot;: END}<br>)<br>graph_builder.add_edge(&quot;tools&quot;, &quot;agent&quot;)<br>graph = graph_builder.compile(checkpointer=MemorySaver())<br><br># ── MODULE 7: ENTRYPOINT ──────────────────────────────────────<br>if __name__ == &quot;__main__&quot;:<br>    config = {&quot;configurable&quot;: {&quot;thread_id&quot;: &quot;session-001&quot;}}<br>    <br>    print(&quot;RAG agent ready. Ask about your documents, or anything else.\n&quot;)<br>    <br>    while True:<br>        user_text = input(&quot;You: &quot;).strip()<br>        if not user_text or user_text.lower() == &quot;exit&quot;:<br>            break<br>        <br>        response = graph.invoke(<br>            {&quot;messages&quot;: [HumanMessage(content=user_text)]},<br>            config=config<br>        )<br>        print(f&quot;Agent: {response[&#39;messages&#39;][-1].content}\n&quot;)</pre><h3>What Actually Happens When You Run This</h3><p>It’s worth tracing through a single query so the two frameworks’ division of labor is completely clear:</p><pre>User: &quot;What&#39;s our policy on remote work?&quot;<br>        ↓<br>[agent_node] — LangGraph&#39;s LLM reads the message, recognizes this needs<br>               internal info, decides to call search_knowledge_base<br>        ↓<br>[tools] — ToolNode executes search_knowledge_base(&quot;What&#39;s our policy on remote work?&quot;)<br>        ↓<br>        Inside the tool: query_engine.query(...) runs —<br>        this is 100% LlamaIndex, invisible to LangGraph:<br>          1. Embeds the query<br>          2. Searches the vector index for the 3 closest chunks<br>          3. Feeds those chunks + the question to Settings.llm<br>          4. Returns a synthesized answer string<br>        ↓<br>[agent_node] — LangGraph&#39;s LLM receives the tool&#39;s string result,<br>               and crafts the final response shown to the user<br>        ↓<br>Response to user</pre><p>Notice the clean separation: LangGraph never sees a Node, an embedding, or a vector. It just sees a tool that returns a string — exactly like the search_web or calculate tools from Part 1. LlamaIndex never sees messages, state, or a thread_id — it just receives a query string and returns an answer. Each framework does exactly what it&#39;s good at, and the seam between them is one function call.</p><h3>Part D: One Level Deeper — Retriever-Only Mode (More Control)</h3><p>The query_engine.query() pattern above is the simplest integration, and it&#39;s right for most cases. But sometimes you want more control — for example, you want <em>your</em> LangGraph agent&#39;s LLM to be the one synthesizing the final answer (using your existing system prompt and conversation context), rather than letting LlamaIndex&#39;s internal Settings.llm generate a separate, disconnected answer.</p><p>For this, use a <strong>Retriever</strong> instead of a QueryEngine. The retriever does <em>only</em> the search step — it returns raw chunks, and you decide what to do with them.</p><pre># ── Retriever-only tool: returns raw chunks, not a synthesized answer ──<br>retriever = index.as_retriever(similarity_top_k=3)<br>@tool<br>def retrieve_documents(query: str) -&gt; str:<br>    &quot;&quot;&quot;Retrieve relevant document excerpts from the internal knowledge base.<br>    Returns raw excerpts for you to read and reason over yourself - <br>    use this when you need to cite specific sources or combine information <br>    from multiple documents.&quot;&quot;&quot;<br>    <br>    nodes = retriever.retrieve(query)<br>    <br>    # Format each retrieved chunk with its source for transparency<br>    formatted_chunks = []<br>    for i, node in enumerate(nodes):<br>        source = node.metadata.get(&quot;file_name&quot;, &quot;unknown source&quot;)<br>        formatted_chunks.append(f&quot;[Excerpt {i+1} from {source}]\n{node.text}&quot;)<br>    <br>    return &quot;\n\n---\n\n&quot;.join(formatted_chunks)</pre><p>With this pattern, your LangGraph agent_node&#39;s own LLM call (the one using llm_with_tools, with your full conversation history and system prompt) is the one that reads these raw excerpts and writes the final answer. This gives you a few real advantages: the agent can cite which document a fact came from, it can combine the retrieved excerpts with earlier conversation context, and you have one consistent &quot;voice&quot; for the agent rather than two different LLMs answering in two different styles.</p><h3>When to Use QueryEngine vs. Retriever</h3><p><strong>Use </strong><strong>QueryEngine</strong> (the Part C pattern) when you want the simplest possible integration and don&#39;t mind LlamaIndex&#39;s LLM generating the retrieval-based answer as a self-contained string. Good for most chatbots and assistants.</p><p><strong>Use </strong><strong>Retriever</strong> (this pattern) when you need source citations, want the agent&#39;s main LLM to maintain a consistent voice and reasoning style across both retrieved and non-retrieved answers, or want to combine retrieved chunks with other tool results before generating a final answer.</p><h3>The Updated Keyword Reference Card</h3><p>This extends the keyword cards from Parts 1–4 with LlamaIndex-specific terms.</p><p><strong>LlamaIndex Core Keywords</strong> Document — a raw loaded source (one file, one page). The unprocessed unit of data. Node — a chunk of a Document. The retrievable unit. Carries metadata back to its source. SimpleDirectoryReader(&quot;./path&quot;).load_data() — loads all supported files in a folder into Document objects. Settings — global configuration singleton. Set Settings.llm, Settings.embed_model, Settings.chunk_size once; every LlamaIndex component uses them. VectorStoreIndex.from_documents(documents) — builds a complete index: chunks, embeds, and stores in one call.</p><p><strong>Querying Keywords</strong> index.as_query_engine(similarity_top_k=3) — creates a QueryEngine. Retrieves top-k chunks AND generates a synthesized answer in one call. index.as_retriever(similarity_top_k=3) — creates a Retriever. Retrieves top-k chunks only — no answer generation. Use when you want your own LLM to synthesize. query_engine.query(&quot;...&quot;) — runs the full retrieve-then-generate pipeline. Returns a Response object (use str(response) to get text). retriever.retrieve(&quot;...&quot;) — runs only the retrieval step. Returns a list of Node objects with .text and .metadata.</p><p><strong>Persistence Keywords</strong> index.storage_context.persist(persist_dir=&quot;./storage&quot;) — saves the index to disk. Avoids re-embedding on every restart. StorageContext.from_defaults(persist_dir=&quot;./storage&quot;) + load_index_from_storage(...) — reloads a previously persisted index. ChromaVectorStore / PineconeVectorStore / QdrantVectorStore — external vector database backends. Swap-in replacements for the default in-memory store, following the same pattern as LangGraph&#39;s checkpointers from Part 2.</p><p><strong>The Bridge (LlamaIndex → LangGraph)</strong> @tool + a function body calling query_engine.query(...) — the entire integration pattern. LangGraph sees a normal tool; LlamaIndex internals stay fully encapsulated. <strong>Two separate LLM configs</strong> — Settings.llm (LlamaIndex&#39;s internal model, used only inside the query engine) and your LangGraph llm (the agent&#39;s main reasoning model) are independent. They do not share conversation history or state.</p><h3>The Decision Guide: When Do You Actually Need This?</h3><p><strong>Your agent only needs general knowledge, math, or live web search</strong> → You don’t need LlamaIndex. Stick with the Part 1 template and standard tools.</p><p><strong>Your agent needs to answer questions from a fixed set of documents (policies, manuals, reports) that don’t change often</strong> → This article’s Part C pattern (QueryEngine wrapped as a tool) is exactly right.</p><p><strong>Your agent needs to cite sources, or combine retrieved facts with multi-turn conversation reasoning</strong> → Use the Retriever pattern from Part D instead.</p><p><strong>Your knowledge base is large (100,000+ documents), needs frequent updates, or needs to scale to many concurrent users</strong> → Use an external vector database (Chroma, Pinecone, Qdrant) instead of the in-memory index, exactly as shown in Part B, Step 5.</p><p><strong>You need the retrieval step itself to be a reviewable, pausable action</strong> (e.g., compliance requires a human to approve what gets searched) → Combine this article with Part 3’s interrupt() pattern. Wrap the retrieval call inside a node that pauses for approval before querying, just like the tool-approval pattern for any other sensitive tool.</p><h3>Conclusion: Two Frameworks, One Seam</h3><p>The lesson underneath this entire article is one you’ve now seen twice: in Part 4, you learned that a complex multi-agent system is really just several single-agent graphs composed together. Here, you’ve learned that adding real-world knowledge to an agent isn’t a LangGraph problem at all — it’s a <em>data</em> problem, and the right tool for a data problem is a data framework.</p><p>LlamaIndex does the indexing and retrieval. LangGraph does the reasoning and orchestration. The entire integration between them is a single @tool-decorated function. That&#39;s not a limitation — it&#39;s the right amount of coupling. Each framework stays fully in its lane, and you can upgrade either side independently: swap your vector database without touching your graph, or add a supervisor pattern from Part 4 without touching your retrieval logic.</p><p>You now have the complete production scaffold across five articles: canonical structure (Part 1), memory management (Part 2), human-in-the-loop safety (Part 3), multi-agent orchestration (Part 4), and real-world knowledge via RAG (Part 5). Combined, these five patterns cover the overwhelming majority of what a serious, production-grade LangGraph application actually needs.</p><p><em>For other parts of the series : </em><a href="https://medium.com/@bessiedelight/the-building-blocks-of-langgraph-1975904edac8?source=user_profile_page---------4-------------4a5744f8548f----------------------"><em>Part 0</em></a><em> , </em><a href="https://medium.com/towards-artificial-intelligence/the-langgraph-mental-model-a-standardized-architecture-guide-for-every-agent-youll-ever-build-d02265f3bebf?source=user_profile_page---------2-------------4a5744f8548f----------------------"><em>Part 1</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-memory-the-complete-practical-guide-to-managing-what-your-agent-remembers-d865d505a59e?source=user_profile_page---------1-------------4a5744f8548f----------------------"><em>Part 2</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-human-in-the-loop-pausing-reviewing-and-rewinding-your-agent-4028bd05b049?source=user_profile_page---------0-------------4a5744f8548f----------------------"><em>Part 3</em></a><em> </em>, <a href="https://pub.towardsai.net/your-first-real-langgraph-project-ac5eb00f923a"><em>Part 4</em></a><em> , Part 5,</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcaf14f9c811" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/rag-without-the-guesswork-a-standardized-langgraph-llamaindex-pattern-bcaf14f9c811">RAG Without the Guesswork: A Standardized LangGraph + LlamaIndex Pattern.</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LangGraph Multi-Agent Systems: From One Brain to Many]]></title>
            <link>https://pub.towardsai.net/langgraph-multi-agent-systems-from-one-brain-to-many-4c1773055693?source=rss-4a5744f8548f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c1773055693</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Bessie Delight Kekeli]]></dc:creator>
            <pubDate>Fri, 19 Jun 2026 12:31:01 GMT</pubDate>
            <atom:updated>2026-06-20T17:07:24.321Z</atom:updated>
            <content:encoded><![CDATA[<h4>The jump from a single agent to a multi-agent system with langGraph.</h4><p><em>Part 5 of the LangGraph Mental Model series.<br>For other parts of the series : </em><a href="https://medium.com/@bessiedelight/the-building-blocks-of-langgraph-1975904edac8?source=user_profile_page---------4-------------4a5744f8548f----------------------"><em>Part 0</em></a><em> , </em><a href="https://medium.com/towards-artificial-intelligence/the-langgraph-mental-model-a-standardized-architecture-guide-for-every-agent-youll-ever-build-d02265f3bebf?source=user_profile_page---------2-------------4a5744f8548f----------------------"><em>Part 1</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-memory-the-complete-practical-guide-to-managing-what-your-agent-remembers-d865d505a59e?source=user_profile_page---------1-------------4a5744f8548f----------------------"><em>Part 2</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-human-in-the-loop-pausing-reviewing-and-rewinding-your-agent-4028bd05b049?source=user_profile_page---------0-------------4a5744f8548f----------------------"><em>Part 3</em></a><em> </em>, <a href="https://pub.towardsai.net/your-first-real-langgraph-project-ac5eb00f923a"><em>Part 4</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lMWSN3Xo1h2z-l9Md4vNQg.png" /></figure><blockquote><strong><em>What this article assumes:</em></strong><em> You understand the seven-module structure, can write a single-agent graph with tools and memory, and know how to pause and resume execution with </em><em>interrupt(). Everything here builds on that foundation. We go from &quot;why would I even use multiple agents?&quot; all the way to a full multi-agent research assistant.</em></blockquote><h3>Why Multiple Agents? The Problem That Forces the Architecture</h3><p>Imagine you’ve built the single-agent assistant from Part 1. It works. Then your users start asking for more: “Can it also check emails, research topics, write reports, and manage tasks , all in one conversation?” So you add tools. More tools. More instructions in the system prompt. Suddenly your agent has 15 tools and a 2,000-word system prompt, and its performance quietly gets worse. The LLM gets confused about which tool to use when. It sometimes uses the email tool for research tasks and the research tool for email tasks.</p><p>This is the <strong>cognitive overload problem</strong>, and it’s the primary reason multi-agent systems exist. The solution is the same one software engineers have used for decades: split the responsibility.</p><p>Multi-agent systems in LangGraph solve three specific problems:</p><p><strong>Cognitive overload</strong> — one LLM performing too many unrelated tasks at once. Split into specialized agents, each excellent at one thing.</p><p><strong>Sequential bottlenecks</strong> — tasks that could run at the same time but are forced to run one after another. Parallel agents fix latency.</p><p><strong>Complexity management</strong> — a 40-node graph is impossible to reason about. Breaking it into smaller subgraphs makes each piece understandable and testable independently.</p><p>Everything in this article addresses one of these three problems.</p><h3>Agent Architecture Levels</h3><p>Here’s the simple way to think about <strong>Level 1 to Level 4</strong> in the article’s agent architecture framing.</p><h4>Level 1</h4><p>A <strong>single agent</strong> handles the task and can use tools, but it is still one brain making the choices. The article describes this as a <strong>supervisor pattern</strong> with one supervisor coordinating work, so it is already more than a plain tool-using chatbot. In practice, this level is for straightforward delegation of work to focused workers, not a giant internal decision tree.</p><h4>Level 2</h4><p>This adds <strong>parallelization</strong>: independent tasks are split and run at the same time, then their outputs are combined. The important idea is <strong>fan-out / fan-in</strong>, where multiple branches execute concurrently and then feed into one later step. This is useful when the work is the same kind of task repeated for multiple sources or subtasks.</p><h4>Level 3</h4><p>This is where <strong>subgraphs</strong> come in. Instead of keeping everything in one huge workflow, each branch can be its own smaller graph with private internal steps and its own output schema. That makes the system easier to build, test, and reason about when each part is complex.</p><h4>Level 4</h4><p>This introduces <strong>dynamic parallelism</strong> with the Send pattern / map-reduce style execution. The system does not need to know the number of branches ahead of time; it can create as many workers as needed at runtime. So Level 4 is more flexible than fixed parallel branches because it can spawn one worker per item, per topic, or per analyst on demand.</p><p>Breaking it down:</p><ul><li><strong>Runtime-generated</strong> → the agents are <strong>not predefined</strong>. The system decides during execution how many agents are needed and creates them dynamically.</li><li><strong>Parallel</strong> → all those agents work simultaneously instead of one after another.</li><li><strong>Agents</strong> → independent workers, each responsible for a specific task.</li></ul><h4>Mental model</h4><p>A good shortcut is:</p><p><strong>Level 1</strong>: one supervisor routes work.</p><p><strong>Level 2</strong>: fixed parallel workers run together.</p><p><strong>Level 3</strong>: each branch can be a full subgraph.</p><p><strong>Level 4</strong>: dynamic workers are spawned at runtime.</p><p>So if you are building with LangGraph, Level 1 is not the same as a multi-agent team, Level 2 is parallel fan-out, Level 3 is modular subgraphs, and Level 4 is runtime-generated parallel agents.</p><h3>Level 1: The Supervisor Pattern (One Brain, Multiple Workers)</h3><p>This is the simplest multi-agent pattern and the right starting point. One <strong>supervisor</strong> agent receives the user’s request and decides which <strong>specialist</strong> agent should handle it. The specialist does the work and returns results to the supervisor, which then decides what to do next.</p><h3>The Mental Model</h3><p>Think of a <strong>law firm</strong>: a senior partner (supervisor) talks to the client, understands the need, and assigns work to junior associates (specialists) one for contracts, one for litigation, one for compliance. The junior associate does the deep work. The senior partner reviews and responds to the client.</p><h3>The Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z6jGRbKihbcsK_eqYZSRGQ.png" /></figure><pre>User Input<br>    ↓<br>[Supervisor Node]  ← decides who handles what<br>    ↓         ↓<br>[Specialist A] [Specialist B]   ← do the actual work<br>    ↓         ↓<br>[Supervisor Node]  ← reviews results, responds or routes again<br>    ↓<br>Response</pre><h3>Module 2: State for Supervisor Pattern</h3><p>The key design decision here is next_agent — a field the supervisor writes to, which the router reads.</p><pre># ── MODULE 2: STATE ─────────────────────────────────────────<br>from typing import TypedDict, Annotated, Literal<br>from langgraph.graph import MessagesState<br><br>class SupervisorState(MessagesState):<br>    # The supervisor writes this field to indicate who should act next.<br>    # &quot;researcher&quot;, &quot;writer&quot;, &quot;FINISH&quot; are the possible values.<br>    next_agent: str</pre><h3>Module 3: Specialist Tools</h3><p>Each specialist has its own, focused set of tools — no overlap.</p><pre># ── MODULE 3: TOOLS ─────────────────────────────────────────<br>from langchain_core.tools import tool<br><br># Researcher&#39;s tools<br>@tool<br>def search_web(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search the web for factual information, news, or research.&quot;&quot;&quot;<br>    return f&quot;[Web results for: {query}]&quot;<br>@tool<br>def search_academic(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search academic papers and journals for scholarly sources.&quot;&quot;&quot;<br>    return f&quot;[Academic results for: {query}]&quot;<br># Writer&#39;s tools<br>@tool<br>def format_as_report(content: str, title: str) -&gt; str:<br>    &quot;&quot;&quot;Format raw content into a structured report with headings.&quot;&quot;&quot;<br>    return f&quot;# {title}\n\n{content}&quot;<br>@tool<br>def check_grammar(text: str) -&gt; str:<br>    &quot;&quot;&quot;Check and correct grammar in the provided text.&quot;&quot;&quot;<br>    return f&quot;[Grammar-checked]: {text}&quot;<br># Each specialist gets only its own tools<br>researcher_tools = [search_web, search_academic]<br>writer_tools = [format_as_report, check_grammar]<br>from langchain_openai import ChatOpenAI<br>llm = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br>researcher_llm = llm.bind_tools(researcher_tools)<br>writer_llm     = llm.bind_tools(writer_tools)</pre><h3>Module 4: Nodes — Supervisor and Specialists</h3><pre># ── MODULE 4: NODES ─────────────────────────────────────────<br>from langchain_core.messages import SystemMessage, HumanMessage<br>from langgraph.prebuilt import ToolNode<br><br># ── Supervisor ────────────────────────────────────────────<br>SUPERVISOR_PROMPT = &quot;&quot;&quot;You are a supervisor coordinating a team of specialists.<br>Given the conversation, decide who should act next.<br>Your team:<br>- researcher: finds information, searches web and academic sources<br>- writer: formats content, writes reports, checks grammar<br>- FINISH: use this when the task is complete and you have a final answer<br>Respond with ONLY the name of the next agent: researcher, writer, or FINISH.&quot;&quot;&quot;<br>def supervisor_node(state: SupervisorState) -&gt; dict:<br>    &quot;&quot;&quot;Reads the conversation history and routes to the right specialist.&quot;&quot;&quot;<br>    messages = [SystemMessage(content=SUPERVISOR_PROMPT)] + state[&quot;messages&quot;]<br>    response = llm.invoke(messages)<br>    # The supervisor&#39;s plain text response IS the routing decision<br>    return {&quot;next_agent&quot;: response.content.strip()}<br><br># ── Researcher Specialist ──────────────────────────────────<br>def researcher_node(state: SupervisorState) -&gt; dict:<br>    &quot;&quot;&quot;Specialist that handles all research and information-gathering tasks.&quot;&quot;&quot;<br>    messages = [<br>        SystemMessage(content=&quot;You are a research specialist. Use your search tools to find accurate, thorough information.&quot;)<br>    ] + state[&quot;messages&quot;]<br>    response = researcher_llm.invoke(messages)<br>    return {&quot;messages&quot;: [response]}<br><br># ── Writer Specialist ──────────────────────────────────────<br>def writer_node(state: SupervisorState) -&gt; dict:<br>    &quot;&quot;&quot;Specialist that handles all writing, formatting, and editing tasks.&quot;&quot;&quot;<br>    messages = [<br>        SystemMessage(content=&quot;You are a writing specialist. Use your tools to produce well-structured, polished output.&quot;)<br>    ] + state[&quot;messages&quot;]<br>    response = writer_llm.invoke(messages)<br>    return {&quot;messages&quot;: [response]}<br><br># Tool execution nodes for each specialist<br>researcher_tool_node = ToolNode(researcher_tools)<br>writer_tool_node     = ToolNode(writer_tools)</pre><h3>Module 5: Routing</h3><pre># ── MODULE 5: ROUTING ────────────────────────────────────────<br>def route_after_supervisor(state: SupervisorState) -&gt; Literal[&quot;researcher&quot;, &quot;writer&quot;, &quot;__end__&quot;]:<br>    &quot;&quot;&quot;Read the supervisor&#39;s decision and route accordingly.&quot;&quot;&quot;<br>    decision = state[&quot;next_agent&quot;]<br>    if decision == &quot;FINISH&quot;:<br>        return &quot;__end__&quot;<br>    return decision  # &quot;researcher&quot; or &quot;writer&quot;<br><br>def route_researcher(state: SupervisorState) -&gt; Literal[&quot;researcher_tools&quot;, &quot;supervisor&quot;]:<br>    &quot;&quot;&quot;After researcher acts: did it call a tool, or produce a final answer?&quot;&quot;&quot;<br>    last = state[&quot;messages&quot;][-1]<br>    if hasattr(last, &quot;tool_calls&quot;) and last.tool_calls:<br>        return &quot;researcher_tools&quot;<br>    return &quot;supervisor&quot;  # Research done - report back to supervisor<br>def route_writer(state: SupervisorState) -&gt; Literal[&quot;writer_tools&quot;, &quot;supervisor&quot;]:<br>    last = state[&quot;messages&quot;][-1]<br>    if hasattr(last, &quot;tool_calls&quot;) and last.tool_calls:<br>        return &quot;writer_tools&quot;<br>    return &quot;supervisor&quot;</pre><h3>Module 6: Graph Assembly</h3><pre># ── MODULE 6: GRAPH ASSEMBLY ────────────────────────────────<br>from langgraph.graph import StateGraph, START, END<br>from langgraph.checkpoint.memory import MemorySaver<br><br>graph_builder = StateGraph(SupervisorState)<br># Register all nodes<br>graph_builder.add_node(&quot;supervisor&quot;,        supervisor_node)<br>graph_builder.add_node(&quot;researcher&quot;,        researcher_node)<br>graph_builder.add_node(&quot;researcher_tools&quot;,  researcher_tool_node)<br>graph_builder.add_node(&quot;writer&quot;,            writer_node)<br>graph_builder.add_node(&quot;writer_tools&quot;,      writer_tool_node)<br># Entry: always start at supervisor<br>graph_builder.add_edge(START, &quot;supervisor&quot;)<br># Supervisor routes to specialist or ends<br>graph_builder.add_conditional_edges(<br>    &quot;supervisor&quot;,<br>    route_after_supervisor,<br>    {&quot;researcher&quot;: &quot;researcher&quot;, &quot;writer&quot;: &quot;writer&quot;, &quot;__end__&quot;: END}<br>)<br># Researcher loop: tools → researcher → supervisor<br>graph_builder.add_conditional_edges(<br>    &quot;researcher&quot;, route_researcher,<br>    {&quot;researcher_tools&quot;: &quot;researcher_tools&quot;, &quot;supervisor&quot;: &quot;supervisor&quot;}<br>)<br>graph_builder.add_edge(&quot;researcher_tools&quot;, &quot;researcher&quot;)<br># Writer loop: tools → writer → supervisor<br>graph_builder.add_conditional_edges(<br>    &quot;writer&quot;, route_writer,<br>    {&quot;writer_tools&quot;: &quot;writer_tools&quot;, &quot;supervisor&quot;: &quot;supervisor&quot;}<br>)<br>graph_builder.add_edge(&quot;writer_tools&quot;, &quot;writer&quot;)<br>graph = graph_builder.compile(checkpointer=MemorySaver())</pre><p>The key structure here is the <strong>return-to-supervisor</strong> edge. After every specialist completes its work, it reports back to the supervisor. The supervisor then evaluates the full conversation and decides: assign to another specialist, loop the same specialist, or finish. This is the orchestration pattern at its most fundamental.</p><h3>Level 2: Parallelization (Fan-Out / Fan-In)</h3><p>The supervisor pattern is sequential — one specialist works at a time. But many real tasks have <strong>independent sub-tasks</strong> that can run simultaneously. Parallelization fixes the latency problem.</p><h3>The Mental Model</h3><p>Think of <strong>cooking a meal</strong>: you don’t finish the salad, then start the pasta, then start the sauce. You prep the salad, put the pasta water on, and start the sauce — all at the same time. The meal finishes faster because independent tasks run in parallel.</p><h3>The Core Mechanics: Fan-Out and Fan-In</h3><p>In LangGraph, parallelism is created by a single node having <strong>multiple outgoing edges</strong> to different nodes. LangGraph detects this “fan-out” pattern and runs all the destination nodes concurrently in what it calls a <strong>superstep</strong>. The results are collected before any subsequent node runs (“fan-in”).</p><h3>The Critical Rule: Parallel Nodes Need Reducers</h3><p>When multiple nodes run in the same superstep and try to write to the same state field, LangGraph needs to know how to combine those writes. Without a reducer, it throws an error. With operator.add, it safely concatenates the results.</p><pre># ── MODULE 2: STATE FOR PARALLEL PATTERN ────────────────────<br>import operator<br>from typing import Annotated<br><br>class ResearchState(TypedDict):<br>    query: str<br>    # Without Annotated + operator.add, parallel writes to &#39;context&#39; would CRASH.<br>    # With it, LangGraph concatenates results from all parallel nodes.<br>    context: Annotated[list[str], operator.add]<br>    final_answer: str</pre><h3>Module 4 &amp; 6: Parallel Research from Two Sources</h3><pre># ── MODULE 4: PARALLEL WORKER NODES ─────────────────────────<br><br>def web_search_node(state: ResearchState) -&gt; dict:<br>    &quot;&quot;&quot;Runs in parallel with academic_search_node.&quot;&quot;&quot;<br>    result = f&quot;[WEB] Results for: {state[&#39;query&#39;]}&quot;<br>    return {&quot;context&quot;: [result]}   # Returns a LIST - operator.add will concatenate<br>def academic_search_node(state: ResearchState) -&gt; dict:<br>    &quot;&quot;&quot;Runs in parallel with web_search_node.&quot;&quot;&quot;<br>    result = f&quot;[ACADEMIC] Papers about: {state[&#39;query&#39;]}&quot;<br>    return {&quot;context&quot;: [result]}   # Also a LIST<br>def synthesize_node(state: ResearchState) -&gt; dict:<br>    &quot;&quot;&quot;Runs AFTER both parallel nodes complete (fan-in point).<br>    state[&#39;context&#39;] already contains results from BOTH parallel nodes.&quot;&quot;&quot;<br>    combined = &quot;\n&quot;.join(state[&quot;context&quot;])<br>    prompt = [<br>        SystemMessage(content=&quot;Synthesize the following research into a clear answer.&quot;),<br>        HumanMessage(content=f&quot;Research:\n{combined}\n\nQuery: {state[&#39;query&#39;]}&quot;)<br>    ]<br>    response = llm.invoke(prompt)<br>    return {&quot;final_answer&quot;: response.content}<br><br># ── MODULE 6: PARALLEL GRAPH ASSEMBLY ───────────────────────<br>from langgraph.graph import StateGraph, START, END<br>graph_builder = StateGraph(ResearchState)<br>graph_builder.add_node(&quot;web_search&quot;,      web_search_node)<br>graph_builder.add_node(&quot;academic_search&quot;, academic_search_node)<br>graph_builder.add_node(&quot;synthesize&quot;,      synthesize_node)<br># Fan-out: START branches to BOTH search nodes simultaneously<br>graph_builder.add_edge(START,       &quot;web_search&quot;)<br>graph_builder.add_edge(START,       &quot;academic_search&quot;)<br># Fan-in: BOTH search nodes must complete before synthesize runs<br>graph_builder.add_edge(&quot;web_search&quot;,      &quot;synthesize&quot;)<br>graph_builder.add_edge(&quot;academic_search&quot;, &quot;synthesize&quot;)<br>graph_builder.add_edge(&quot;synthesize&quot;, END)<br>parallel_graph = graph_builder.compile()</pre><p>The fan-out is just two edges from the same source (START). LangGraph automatically detects this and runs both destinations concurrently. The fan-in happens naturally — synthesize can only run once <em>both</em> of its incoming edges are satisfied (i.e., both parallel nodes have completed).</p><h3>Level 3: Subgraphs (Encapsulation and Modularity)</h3><p>Parallelization with flat nodes works well for simple cases. But when each parallel “branch” is itself complex — with multiple steps, its own tools, its own logic — you need <strong>subgraphs</strong>. A subgraph is a fully compiled StateGraph that runs as a single node inside a parent graph.</p><h3>The Mental Model</h3><p>Think of <strong>departments in a company</strong>. The CEO (parent graph) delegates work to the Engineering department and the Marketing department. Each department has its own internal processes, meetings, and workflows. The CEO doesn’t care about the internals — they just hand off work and receive deliverables.</p><h3>The Key Design Rule: Communicate via Overlapping State Keys</h3><p>Subgraphs communicate with their parent graph through shared state keys. Any key that appears in <em>both</em> the parent’s state and the subgraph’s state is automatically passed in (when the subgraph starts) and passed back out (when the subgraph finishes).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BVnaYleOJPsjOZz4Uu3uFA.png" /></figure><pre>Parent State: {topic, cleaned_data, report_A, report_B, processed_ids}<br>                         ↓<br>Subgraph A State: {cleaned_data, internal_step_1, report_A, processed_ids}<br>                  ← reads &#39;cleaned_data&#39; from parent<br>                  → writes &#39;report_A&#39; and &#39;processed_ids&#39; back to parent<br><br>Subgraph B State: {cleaned_data, internal_step_2, report_B, processed_ids}<br>                  ← reads &#39;cleaned_data&#39; from parent<br>                  → writes &#39;report_B&#39; and &#39;processed_ids&#39; back to parent</pre><p>Keys that are only in the subgraph’s state (like internal_step_1) are private — the parent never sees them.</p><h3>Complete Subgraph Implementation</h3><pre># ── MODULE 2: STATE SCHEMAS (Three Levels) ──────────────────<br><br># Parent state - what the orchestrating graph tracks<br>class ParentState(TypedDict):<br>    raw_input: str<br>    cleaned_data: str                               # passed INTO both subgraphs<br>    failure_summary: str                            # written back by subgraph A<br>    performance_report: str                         # written back by subgraph B<br>    processed_ids: Annotated[list[int], operator.add]  # BOTH subgraphs write this<br><br># Subgraph A&#39;s internal state - includes its private fields + shared keys<br>class FailureAnalysisState(TypedDict):<br>    cleaned_data: str        # shared input from parent<br>    failures: list[str]      # private intermediate state - parent never sees this<br>    failure_summary: str     # shared output to parent<br>    processed_ids: Annotated[list[int], operator.add]  # shared output (accumulated)<br><br># Subgraph A&#39;s output schema - filters which keys return to parent<br># Without this, ALL keys return, which can cause conflicts<br>class FailureAnalysisOutput(TypedDict):<br>    failure_summary: str     # only these two keys go back to parent<br>    processed_ids: Annotated[list[int], operator.add]<br><br># Subgraph B&#39;s internal state<br>class PerformanceAnalysisState(TypedDict):<br>    cleaned_data: str<br>    metrics: list[str]          # private<br>    performance_report: str     # shared output<br>    processed_ids: Annotated[list[int], operator.add]<br><br>class PerformanceAnalysisOutput(TypedDict):<br>    performance_report: str<br>    processed_ids: Annotated[list[int], operator.add]<br><br># ── SUBGRAPH A: Failure Analysis ─────────────────────────────<br>def extract_failures(state: FailureAnalysisState) -&gt; dict:<br>    failures = [f&quot;Error in {state[&#39;cleaned_data&#39;]}: timeout&quot;]<br>    return {&quot;failures&quot;: failures, &quot;processed_ids&quot;: [1]}<br>def summarize_failures(state: FailureAnalysisState) -&gt; dict:<br>    summary = f&quot;Found {len(state[&#39;failures&#39;])} failure(s): {state[&#39;failures&#39;]}&quot;<br>    return {&quot;failure_summary&quot;: summary}<br>failure_builder = StateGraph(FailureAnalysisState, output=FailureAnalysisOutput)<br>failure_builder.add_node(&quot;extract_failures&quot;,  extract_failures)<br>failure_builder.add_node(&quot;summarize_failures&quot;, summarize_failures)<br>failure_builder.add_edge(START, &quot;extract_failures&quot;)<br>failure_builder.add_edge(&quot;extract_failures&quot;, &quot;summarize_failures&quot;)<br>failure_builder.add_edge(&quot;summarize_failures&quot;, END)<br>failure_subgraph = failure_builder.compile()  # compiled → can be used as a node<br><br># ── SUBGRAPH B: Performance Analysis ────────────────────────<br>def collect_metrics(state: PerformanceAnalysisState) -&gt; dict:<br>    metrics = [f&quot;Latency: 120ms&quot;, &quot;Uptime: 99.8%&quot;]<br>    return {&quot;metrics&quot;: metrics, &quot;processed_ids&quot;: [2]}<br>def write_report(state: PerformanceAnalysisState) -&gt; dict:<br>    report = f&quot;Performance Report:\n&quot; + &quot;\n&quot;.join(state[&quot;metrics&quot;])<br>    return {&quot;performance_report&quot;: report}<br>performance_builder = StateGraph(PerformanceAnalysisState, output=PerformanceAnalysisOutput)<br>performance_builder.add_node(&quot;collect_metrics&quot;, collect_metrics)<br>performance_builder.add_node(&quot;write_report&quot;,    write_report)<br>performance_builder.add_edge(START, &quot;collect_metrics&quot;)<br>performance_builder.add_edge(&quot;collect_metrics&quot;, &quot;write_report&quot;)<br>performance_builder.add_edge(&quot;write_report&quot;, END)<br>performance_subgraph = performance_builder.compile()<br><br># ── PARENT GRAPH ─────────────────────────────────────────────<br>def clean_data_node(state: ParentState) -&gt; dict:<br>    return {&quot;cleaned_data&quot;: state[&quot;raw_input&quot;].strip().lower()}<br>def combine_reports_node(state: ParentState) -&gt; dict:<br>    combined = f&quot;{state[&#39;failure_summary&#39;]}\n\n{state[&#39;performance_report&#39;]}&quot;<br>    print(f&quot;Final combined report:\n{combined}&quot;)<br>    return {}<br>parent_builder = StateGraph(ParentState)<br>parent_builder.add_node(&quot;clean_data&quot;,           clean_data_node)<br>parent_builder.add_node(&quot;failure_analysis&quot;,     failure_subgraph)    # subgraph as node<br>parent_builder.add_node(&quot;performance_analysis&quot;, performance_subgraph) # subgraph as node<br>parent_builder.add_node(&quot;combine_reports&quot;,      combine_reports_node)<br>parent_builder.add_edge(START, &quot;clean_data&quot;)<br># Fan-out to both subgraphs (parallel execution)<br>parent_builder.add_edge(&quot;clean_data&quot;,           &quot;failure_analysis&quot;)<br>parent_builder.add_edge(&quot;clean_data&quot;,           &quot;performance_analysis&quot;)<br># Fan-in: combine only after both subgraphs complete<br>parent_builder.add_edge(&quot;failure_analysis&quot;,     &quot;combine_reports&quot;)<br>parent_builder.add_edge(&quot;performance_analysis&quot;, &quot;combine_reports&quot;)<br>parent_builder.add_edge(&quot;combine_reports&quot;, END)<br>parent_graph = parent_builder.compile()</pre><p>The output schema (FailureAnalysisOutput, PerformanceAnalysisOutput) is the piece most tutorials skip — and then they wonder why they get state key conflicts. The output schema acts as a filter, controlling exactly which fields the subgraph exposes to the parent. Any field not in the output schema is treated as private internal state.</p><h3>Level 4: The Send API and Map-Reduce (Dynamic Parallelism)</h3><p>Parallelization and subgraphs cover the case where you know <em>at design time</em> how many parallel branches you’ll have. But what if you don’t? What if the user asks to research 5 topics today and 20 topics tomorrow? You can’t hard-code 20 parallel branches.</p><p>This is the <strong>map-reduce</strong> pattern, powered by LangGraph’s Send API.</p><h3>The Mental Model</h3><p>Think of <strong>a book publisher assigning chapters</strong>: one editor is given the manuscript. They split it into chapters and assign each chapter to a different copy editor simultaneously. However many chapters there are — five, twenty, two — that’s how many copy editors get hired. When they’re all done, the results are collected and assembled into the final book. The number of parallel workers is determined at <em>runtime</em>, not at design time.</p><h3>The Keywords You Need to Know</h3><p>Send(node_name, state) — from langgraph.constants. Instead of routing to a fixed next node, Send creates a new <em>instance</em> of a node with a specific state payload. Return a <strong>list</strong> of Send objects from a routing function, and LangGraph launches all of them in parallel, each with its own independent state.</p><p>operator.add on the collecting field — the &quot;reduce&quot; step. As each parallel worker finishes and returns its result, operator.add accumulates them into a growing list in the parent state.</p><p><strong>Worker state</strong> — each Send can include a custom state dict that doesn&#39;t have to match the parent graph&#39;s state. The worker node uses its own small local state — just the data it needs for its specific piece of work.</p><h3>Complete Map-Reduce Implementation</h3><pre># ── MODULE 2: MAP-REDUCE STATE ──────────────────────────────<br>from langgraph.constants import Send<br><br>class OverallState(TypedDict):<br>    topic: str<br>    subjects: list[str]                         # populated by the map step<br>    jokes: Annotated[list[str], operator.add]   # accumulated by the reduce step<br>    best_joke: str<br>class JokeState(TypedDict):<br>    &quot;&quot;&quot;The private state each parallel worker gets.<br>    Notice: this does NOT need to match OverallState.&quot;&quot;&quot;<br>    subject: str<br><br># ── MODULE 4: MAP-REDUCE NODES ──────────────────────────────<br>def generate_subjects(state: OverallState) -&gt; dict:<br>    &quot;&quot;&quot;MAP step 1: Expands the topic into a list of subjects to process.&quot;&quot;&quot;<br>    prompt = [<br>        SystemMessage(content=&quot;Generate a list of 3 sub-topics related to the given topic. Return as comma-separated values only.&quot;),<br>        HumanMessage(content=state[&quot;topic&quot;])<br>    ]<br>    response = llm.invoke(prompt)<br>    subjects = [s.strip() for s in response.content.split(&quot;,&quot;)]<br>    return {&quot;subjects&quot;: subjects}<br><br>def generate_joke(state: JokeState) -&gt; dict:<br>    &quot;&quot;&quot;MAP step 2: Worker node. Each instance handles exactly ONE subject.<br>    Receives its own isolated JokeState - not the full OverallState.&quot;&quot;&quot;<br>    prompt = [HumanMessage(content=f&quot;Write a short, funny joke about: {state[&#39;subject&#39;]}&quot;)]<br>    response = llm.invoke(prompt)<br>    # Returns a LIST - operator.add in OverallState will accumulate these<br>    return {&quot;jokes&quot;: [response.content]}<br><br>def pick_best_joke(state: OverallState) -&gt; dict:<br>    &quot;&quot;&quot;REDUCE step: All jokes are now in state[&#39;jokes&#39;]. Pick the winner.&quot;&quot;&quot;<br>    jokes_text = &quot;\n&quot;.join([f&quot;{i}. {j}&quot; for i, j in enumerate(state[&quot;jokes&quot;])])<br>    prompt = [<br>        SystemMessage(content=&quot;You are a comedy judge. Pick the funniest joke from the list below. Reply with ONLY the number.&quot;),<br>        HumanMessage(content=jokes_text)<br>    ]<br>    response = llm.invoke(prompt)<br>    winning_index = int(response.content.strip())<br>    return {&quot;best_joke&quot;: state[&quot;jokes&quot;][winning_index]}<br><br># ── MODULE 5: THE SEND ROUTER ────────────────────────────────<br>def continue_to_jokes(state: OverallState):<br>    &quot;&quot;&quot;The routing function that LAUNCHES parallel workers.<br>    <br>    Returning a LIST of Send objects (not a string) is what triggers map-reduce.<br>    Each Send creates an independent, parallel execution of &#39;generate_joke&#39;<br>    with its own state payload. The number of parallel workers is determined<br>    at runtime by how many subjects exist in state.<br>    &quot;&quot;&quot;<br>    return [<br>        Send(&quot;generate_joke&quot;, {&quot;subject&quot;: subject})<br>        for subject in state[&quot;subjects&quot;]<br>    ]<br><br># ── MODULE 6: MAP-REDUCE GRAPH ASSEMBLY ─────────────────────<br>graph_builder = StateGraph(OverallState)<br>graph_builder.add_node(&quot;generate_subjects&quot;, generate_subjects)<br>graph_builder.add_node(&quot;generate_joke&quot;,     generate_joke)<br>graph_builder.add_node(&quot;pick_best_joke&quot;,    pick_best_joke)<br>graph_builder.add_edge(START, &quot;generate_subjects&quot;)<br># This conditional edge uses the Send router - it fans out dynamically<br>graph_builder.add_conditional_edges(<br>    &quot;generate_subjects&quot;,<br>    continue_to_jokes,   # returns a LIST of Send objects<br>    [&quot;generate_joke&quot;]    # list of possible destination nodes (for graph validation)<br>)<br># Fan-in: pick_best_joke runs after ALL parallel generate_joke instances complete<br>graph_builder.add_edge(&quot;generate_joke&quot;, &quot;pick_best_joke&quot;)<br>graph_builder.add_edge(&quot;pick_best_joke&quot;, END)<br>map_reduce_graph = graph_builder.compile()</pre><p>The continue_to_jokes function is where the magic happens. Notice that it returns a <strong>list</strong> of Send objects, not a string. When LangGraph sees a list of Send objects from a routing function, it launches all of them in parallel immediately. If state[&quot;subjects&quot;] has 3 items, 3 parallel generate_joke nodes launch. If it has 20, 20 launch. The graph scales dynamically to whatever runtime produces.</p><h3>Level 5: Full Real-World System — The Multi-Agent Research Assistant</h3><p>Now we combine everything: supervisor orchestration, HITL approval (Part 3), parallel subgraphs, and the Send API — into one complete production-grade system. This is the LangGraph Academy&#39;s capstone project, annotated and explained.</p><p><strong>The system:</strong> A user provides a research topic. The system generates a team of AI analyst personas (with human approval to refine them), then runs each analyst as a parallel interview sub-agent. Each analyst interviews an AI “expert” using web search, producing a report section. Finally, a writer node compiles all sections into a final polished report.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hUatT-s_mF29vZ38XZbX9A.png" /></figure><pre>User provides topic<br>        ↓<br>[create_analysts] → LLM generates N analyst personas<br>        ↓<br>[human_feedback]  ← INTERRUPT: human approves or sends feedback<br>        ↓         ↑<br>[should_continue] → if feedback, loop back to create_analysts<br>        ↓<br>[initiate_research] → Send API: spawns N parallel interview subgraphs<br>        ↓↓↓ (N parallel workers running simultaneously)<br>[interview_subgraph × N] → each analyst interviews an &quot;expert&quot; via web search<br>        ↓<br>[write_report]    → LLM compiles all sections into a final report</pre><h3>State Schemas (Three Levels)</h3><pre>from pydantic import BaseModel, Field<br><br># ── Analyst persona - a Pydantic model for structured LLM output ──<br>class Analyst(BaseModel):<br>    name: str = Field(description=&quot;The analyst&#39;s full name&quot;)<br>    role: str = Field(description=&quot;Their professional role (e.g. &#39;Financial Analyst&#39;)&quot;)<br>    focus: str = Field(description=&quot;Their specific analytical focus area&quot;)<br>    <br>    @property<br>    def persona(self) -&gt; str:<br>        return f&quot;Name: {self.name}\nRole: {self.role}\nFocus: {self.focus}&quot;<br>class Perspectives(BaseModel):<br>    analysts: list[Analyst]<br><br># ── Outer graph state ─────────────────────────────────────────<br>class ResearchGraphState(TypedDict):<br>    topic: str<br>    max_analysts: int<br>    human_analyst_feedback: str      # human writes here during HITL pause<br>    analysts: list[Analyst]<br>    sections: Annotated[list, operator.add]  # accumulated from all parallel interviews<br>    final_report: str<br><br># ── Interview subgraph state ──────────────────────────────────<br>class InterviewState(MessagesState):<br>    # inherits: messages: Annotated[list[BaseMessage], add_messages]<br>    max_num_turns: int<br>    context: Annotated[list, operator.add]  # search results accumulate here<br>    analyst: Analyst<br>    interview: str       # formatted transcript<br>    sections: list       # completed section (sent back to outer graph)</pre><h3>The Analyst Generation Phase (with HITL)</h3><pre># ── ANALYST GENERATION NODE ───────────────────────────────────<br>ANALYST_PROMPT = &quot;&quot;&quot;You are generating a team of {max_analysts} research analysts <br>for the topic: {topic}.<br>{feedback_section}<br>Each analyst should have a unique perspective and area of focus.<br>Generate analysts with diverse viewpoints - financial, technical, social, etc.&quot;&quot;&quot;<br>def create_analysts(state: ResearchGraphState) -&gt; dict:<br>    &quot;&quot;&quot;Generates analyst personas using structured LLM output.&quot;&quot;&quot;<br>    feedback = state.get(&quot;human_analyst_feedback&quot;, &quot;&quot;)<br>    feedback_section = (<br>        f&quot;User feedback to incorporate:\n{feedback}&quot; if feedback<br>        else &quot;No feedback yet - generate your best initial set.&quot;<br>    )<br>    <br>    prompt = ANALYST_PROMPT.format(<br>        max_analysts=state[&quot;max_analysts&quot;],<br>        topic=state[&quot;topic&quot;],<br>        feedback_section=feedback_section<br>    )<br>    <br>    # with_structured_output: forces LLM to return a Pydantic model<br>    # instead of free-form text - ensures clean, usable data<br>    structured_llm = llm.with_structured_output(Perspectives)<br>    result = structured_llm.invoke([SystemMessage(content=prompt)])<br>    return {&quot;analysts&quot;: result.analysts}<br><br>def human_feedback_node(state: ResearchGraphState) -&gt; dict:<br>    &quot;&quot;&quot;No-op node. Its only purpose is to be a pause point.<br>    The graph is compiled with interrupt_before=[&#39;human_feedback&#39;].<br>    See Part 3 for the full HITL pattern.&quot;&quot;&quot;<br>    pass  # Does nothing - the pause happens BEFORE this node via interrupt_before<br><br>def should_continue_to_research(state: ResearchGraphState) -&gt; Literal[&quot;create_analysts&quot;, &quot;initiate_research&quot;]:<br>    &quot;&quot;&quot;After human feedback: loop back to refine, or proceed to interviews.&quot;&quot;&quot;<br>    if state.get(&quot;human_analyst_feedback&quot;):<br>        return &quot;create_analysts&quot;<br>    return &quot;initiate_research&quot;<br></pre><h3>The Interview Subgraph</h3><p>Each analyst runs as an independent subgraph. The subgraph is a mini ReAct agent that interviews an “expert” (another LLM call that plays the expert role) and searches the web.</p><pre>from langchain_core.tools import tool<br>from langgraph.prebuilt import ToolNode<br><br>@tool<br>def web_search_tool(query: str) -&gt; str:<br>    &quot;&quot;&quot;Search the web for information on a topic.&quot;&quot;&quot;<br>    return f&quot;[Search results for &#39;{query}&#39;]&quot;  # Replace with real search API<br>interview_tools = [web_search_tool]<br>interview_llm = llm.bind_tools(interview_tools)<br><br>def generate_question(state: InterviewState) -&gt; dict:<br>    &quot;&quot;&quot;The analyst node: generates the next interview question.&quot;&quot;&quot;<br>    analyst = state[&quot;analyst&quot;]<br>    messages = [<br>        SystemMessage(content=(<br>            f&quot;You are {analyst.name}, a {analyst.role} focused on {analyst.focus}. &quot;<br>            f&quot;You are interviewing an expert about the research topic. &quot;<br>            f&quot;Ask insightful questions that align with your analytical focus. &quot;<br>            f&quot;When you have enough information, say &#39;Thank you, that is all I needed.&#39;&quot;<br>        ))<br>    ] + state[&quot;messages&quot;]<br>    response = interview_llm.invoke(messages)<br>    return {&quot;messages&quot;: [response]}<br><br>def generate_answer(state: InterviewState) -&gt; dict:<br>    &quot;&quot;&quot;The expert node: answers the analyst&#39;s question, using web search.&quot;&quot;&quot;<br>    messages = [<br>        SystemMessage(content=(<br>            &quot;You are an expert being interviewed. Answer thoroughly and factually. &quot;<br>            &quot;Use the web search tool when you need current data.&quot;<br>        ))<br>    ] + state[&quot;messages&quot;]<br>    response = interview_llm.invoke(messages)<br>    return {&quot;messages&quot;: [response]}<br><br>def save_interview(state: InterviewState) -&gt; dict:<br>    &quot;&quot;&quot;Formats the full Q&amp;A exchange into a clean transcript string.&quot;&quot;&quot;<br>    transcript = []<br>    for msg in state[&quot;messages&quot;]:<br>        if hasattr(msg, &quot;content&quot;):<br>            role = &quot;Analyst&quot; if type(msg).__name__ == &quot;HumanMessage&quot; else &quot;Expert&quot;<br>            transcript.append(f&quot;{role}: {msg.content}&quot;)<br>    return {&quot;interview&quot;: &quot;\n\n&quot;.join(transcript)}<br><br>def write_section(state: InterviewState) -&gt; dict:<br>    &quot;&quot;&quot;The final node: uses the interview transcript to write a report section.&quot;&quot;&quot;<br>    analyst = state[&quot;analyst&quot;]<br>    prompt = [<br>        SystemMessage(content=(<br>            f&quot;Based on the following interview conducted by {analyst.name} ({analyst.role}), &quot;<br>            f&quot;write a structured report section focused on {analyst.focus}. &quot;<br>            f&quot;Be concise, factual, and cite specific points from the interview.&quot;<br>        )),<br>        HumanMessage(content=state[&quot;interview&quot;])<br>    ]<br>    response = llm.invoke(prompt)<br>    # sections is the key that Send passes back to the outer graph<br>    return {&quot;sections&quot;: [response.content]}<br><br>def route_interview(state: InterviewState) -&gt; Literal[&quot;generate_answer&quot;, &quot;save_interview&quot;]:<br>    &quot;&quot;&quot;Continue interviewing OR wrap up when analyst is satisfied or max turns hit.&quot;&quot;&quot;<br>    last_message = state[&quot;messages&quot;][-1]<br>    if (<br>        &quot;thank you, that is all&quot; in last_message.content.lower()<br>        or len(state[&quot;messages&quot;]) &gt;= state.get(&quot;max_num_turns&quot;, 6) * 2<br>    ):<br>        return &quot;save_interview&quot;<br>    # Check if analyst made a tool call (web search)<br>    if hasattr(last_message, &quot;tool_calls&quot;) and last_message.tool_calls:<br>        return &quot;generate_answer&quot;<br>    return &quot;generate_answer&quot;<br><br># ── INTERVIEW SUBGRAPH ASSEMBLY ───────────────────────────────<br>interview_builder = StateGraph(InterviewState)<br>interview_tool_node = ToolNode(interview_tools)<br>interview_builder.add_node(&quot;generate_question&quot;, generate_question)<br>interview_builder.add_node(&quot;generate_answer&quot;,   generate_answer)<br>interview_builder.add_node(&quot;interview_tools&quot;,   interview_tool_node)<br>interview_builder.add_node(&quot;save_interview&quot;,    save_interview)<br>interview_builder.add_node(&quot;write_section&quot;,     write_section)<br>interview_builder.add_edge(START, &quot;generate_question&quot;)<br>interview_builder.add_conditional_edges(<br>    &quot;generate_question&quot;,<br>    route_interview,<br>    {&quot;generate_answer&quot;: &quot;generate_answer&quot;, &quot;save_interview&quot;: &quot;save_interview&quot;}<br>)<br>interview_builder.add_conditional_edges(<br>    &quot;generate_answer&quot;,<br>    lambda s: &quot;interview_tools&quot; if (hasattr(s[&quot;messages&quot;][-1], &quot;tool_calls&quot;) and s[&quot;messages&quot;][-1].tool_calls) else &quot;generate_question&quot;,<br>    {&quot;interview_tools&quot;: &quot;interview_tools&quot;, &quot;generate_question&quot;: &quot;generate_question&quot;}<br>)<br>interview_builder.add_edge(&quot;interview_tools&quot;, &quot;generate_answer&quot;)<br>interview_builder.add_edge(&quot;save_interview&quot;,  &quot;write_section&quot;)<br>interview_builder.add_edge(&quot;write_section&quot;,   END)<br>interview_graph = interview_builder.compile()</pre><h3>The Outer Orchestration Graph</h3><pre># ── SEND ROUTER: Spawn one interview per analyst ─────────────<br>def initiate_research(state: ResearchGraphState):<br>    &quot;&quot;&quot;Launches N parallel interview subgraphs via Send — one per analyst.<br>    This is the map-reduce Send pattern from Level 4, applied to subgraphs.&quot;&quot;&quot;<br>    return [<br>        Send(&quot;interview&quot;, {<br>            &quot;analyst&quot;: analyst,<br>            &quot;messages&quot;: [HumanMessage(content=f&quot;Research topic: {state[&#39;topic&#39;]}&quot;)],<br>            &quot;context&quot;: [],<br>            &quot;max_num_turns&quot;: 3,<br>            &quot;interview&quot;: &quot;&quot;,<br>            &quot;sections&quot;: []<br>        })<br>        for analyst in state[&quot;analysts&quot;]<br>    ]<br># ── REPORT COMPILATION NODE ───────────────────────────────────<br>def write_report(state: ResearchGraphState) -&gt; dict:<br>    &quot;&quot;&quot;Fan-in point: all sections have arrived from parallel interviews.<br>    The LLM compiles them into a final report.&quot;&quot;&quot;<br>    sections_text = &quot;\n\n---\n\n&quot;.join(state[&quot;sections&quot;])<br>    prompt = [<br>        SystemMessage(content=(<br>            &quot;You are a senior editor. Compile the following report sections written by &quot;<br>            &quot;different analysts into one cohesive, well-structured final report. &quot;<br>            &quot;Add a proper introduction and conclusion. Do not repeat content.&quot;<br>        )),<br>        HumanMessage(content=f&quot;Topic: {state[&#39;topic&#39;]}\n\nSections:\n{sections_text}&quot;)<br>    ]<br>    response = llm.invoke(prompt)<br>    return {&quot;final_report&quot;: response.content}<br><br># ── OUTER GRAPH ASSEMBLY ─────────────────────────────────────<br>outer_builder = StateGraph(ResearchGraphState)<br>outer_builder.add_node(&quot;create_analysts&quot;,   create_analysts)<br>outer_builder.add_node(&quot;human_feedback&quot;,    human_feedback_node)<br>outer_builder.add_node(&quot;interview&quot;,         interview_graph)   # subgraph as node<br>outer_builder.add_node(&quot;write_report&quot;,      write_report)<br>outer_builder.add_edge(START, &quot;create_analysts&quot;)<br>outer_builder.add_edge(&quot;create_analysts&quot;, &quot;human_feedback&quot;)<br>outer_builder.add_conditional_edges(<br>    &quot;human_feedback&quot;,<br>    should_continue_to_research,<br>    {&quot;create_analysts&quot;: &quot;create_analysts&quot;, &quot;initiate_research&quot;: &quot;initiate_research&quot;}<br>)<br># initiate_research is a ROUTING FUNCTION (returns Send objects), not a node<br>outer_builder.add_conditional_edges(<br>    &quot;human_feedback&quot;,<br>    initiate_research,   # returns list of Send objects<br>    [&quot;interview&quot;]        # validation: lists possible destinations<br>)<br>outer_builder.add_edge(&quot;interview&quot;,    &quot;write_report&quot;)<br>outer_builder.add_edge(&quot;write_report&quot;, END)<br>research_graph = outer_builder.compile(<br>    checkpointer=MemorySaver(),<br>    interrupt_before=[&quot;human_feedback&quot;]  # HITL pause for analyst approval<br>)</pre><h3>Module 7: Running the Full System</h3><pre>if __name__ == &quot;__main__&quot;:<br>    config = {&quot;configurable&quot;: {&quot;thread_id&quot;: &quot;research-001&quot;}}<br>    <br>    # Step 1: Generate analysts<br>    result = research_graph.invoke(<br>        {<br>            &quot;topic&quot;: &quot;The impact of AI on the future of software engineering&quot;,<br>            &quot;max_analysts&quot;: 3,<br>            &quot;human_analyst_feedback&quot;: &quot;&quot;,<br>            &quot;analysts&quot;: [],<br>            &quot;sections&quot;: [],<br>            &quot;final_report&quot;: &quot;&quot;<br>        },<br>        config=config<br>    )<br>    <br>    # Step 2: HITL — graph paused at human_feedback node<br>    snapshot = research_graph.get_state(config)<br>    analysts = snapshot.values[&quot;analysts&quot;]<br>    <br>    print(&quot;Generated analysts:&quot;)<br>    for a in analysts:<br>        print(f&quot;  - {a.name} ({a.role}): {a.focus}&quot;)<br>    <br>    feedback = input(&quot;\nProvide feedback to refine analysts (or press Enter to approve): &quot;)<br>    <br>    if feedback.strip():<br>        # Human has feedback — inject it and let supervisor regenerate<br>        research_graph.update_state(<br>            config,<br>            {&quot;human_analyst_feedback&quot;: feedback},<br>            as_node=&quot;human_feedback&quot;<br>        )<br>    else:<br>        # Approved — clear feedback and proceed to interviews<br>        research_graph.update_state(<br>            config,<br>            {&quot;human_analyst_feedback&quot;: None},<br>            as_node=&quot;human_feedback&quot;<br>        )<br>    <br>    # Step 3: Resume — N parallel interviews launch via Send, then report is compiled<br>    print(&quot;\nRunning parallel interviews...&quot;)<br>    final_result = research_graph.invoke(None, config=config)<br>    <br>    print(&quot;\n&quot; + &quot;=&quot;*60)<br>    print(&quot;FINAL RESEARCH REPORT&quot;)<br>    print(&quot;=&quot;*60)<br>    print(final_result[&quot;final_report&quot;])</pre><h3>The Multi-Agent Architecture Decision Guide</h3><p>Use this to decide which pattern fits your situation:</p><p><strong>Single agent getting confused between tasks</strong> → Supervisor Pattern. Split work into specialized sub-agents. The supervisor routes; specialists execute.</p><p><strong>Tasks that are independent and slow</strong> → Parallelization (Fan-out / Fan-in). Run them simultaneously. Add operator.add reducers for shared fields.</p><p><strong>Parallel branches that are themselves complex</strong> → Subgraphs. Compile each complex branch as its own graph and embed it as a node. Use output schemas to control what returns to the parent.</p><p><strong>Number of parallel workers determined at runtime</strong> → Send API (Map-Reduce). Generate a list of Send objects from a router. Each Send spawns an independent worker with its own state payload.</p><p><strong>A real production system</strong> → All of the above, combined. The research assistant at Level 5 uses every pattern together because a real task requires all of them.</p><h3>The Updated Keyword Reference Card</h3><p>This extends the keyword cards from Parts 1–3.</p><p><strong>Multi-Agent Structure Keywords</strong> supervisor_node — the orchestrating node. Reads state, writes next_agent, never does specialist work itself. specialist_node — does one focused task with its own tools and system prompt. Always routes back to supervisor when done. next_agent: str — the standard state field for supervisor routing. The supervisor writes a name; the router reads it.</p><p><strong>Parallelization Keywords</strong> Fan-out — multiple add_edge calls from the same source node. LangGraph detects this and runs targets concurrently. Fan-in — multiple add_edge calls pointing to the same destination. Destination waits for all sources to complete. Annotated[list, operator.add] — mandatory on any state field that multiple parallel nodes write to. Without this, parallel writes crash. operator.add — the most common reducer for parallel patterns. Concatenates lists from concurrent nodes.</p><p><strong>Subgraph Keywords</strong> StateGraph(InternalState, output=OutputSchema) — the two-argument form of StateGraph. The second argument filters which keys are returned to the parent. Output schema — a TypedDict with only the keys the subgraph should expose to the parent. Keys absent from this schema are private to the subgraph. Overlapping keys — the communication channel between parent and subgraph. Any key in both state schemas is automatically shared. subgraph.compile() — seals the subgraph into a callable. After this, pass it to parent.add_node(&quot;name&quot;, compiled_subgraph). xray=1 — argument to graph.get_graph(xray=1).draw_mermaid() — visualizes internal subgraph structure in the parent graph diagram.</p><p><strong>Map-Reduce / Send Keywords</strong> Send(node_name, state_dict) — from langgraph.constants. Represents a single parallel worker invocation. Does not need to match parent graph state. [Send(&quot;node&quot;, {...}), Send(&quot;node&quot;, {...}), ...] — returning a list of Send objects from a routing function launches all of them in parallel. [&quot;node_name&quot;] — the third argument to add_conditional_edges when using Send. A list (not a dict) of valid destination node names, for graph validation only. with_structured_output(PydanticModel) — chains after an LLM to force structured JSON output that validates against a Pydantic schema. The standard pattern for supervisor decisions and analyst generation.</p><h3>Conclusion: Composition Is the Skill</h3><p>The jump from a single agent to a multi-agent system isn’t really a jump at all — it’s the same seven modules, applied multiple times and wired together. Every “agent” in a multi-agent system is just a graph, or a node, or a subgraph. The primitives don’t change. The skill is knowing how to <strong>compose</strong> them.</p><p>The progression in this article followed a deliberate staircase: one supervisor, then parallel flat nodes, then complex parallel subgraphs, then dynamic parallelism with Send, then the full combination. Each step added exactly one new concept. If any step felt comfortable, that&#39;s the design working — each level builds cleanly on the one before.</p><p>The research assistant at Level 5 is genuinely close to what you’d find in a production multi-agent codebase. Human approval loops, parallel specialist agents, structured LLM output, dynamic worker spawning, and a final synthesis step. You now have a mental model and a working template for all of it.</p><p><em>With all four parts complete, you have the full production scaffold: canonical structure (Part 0) + memory management (Part 1) + human-in-the-loop safety (Part 2) + multi-agent orchestration (Part 3). These four articles together cover the architecture behind the vast majority of real-world LangGraph applications.</em></p><p><em>For other parts of the series : </em><a href="https://medium.com/@bessiedelight/the-building-blocks-of-langgraph-1975904edac8?source=user_profile_page---------4-------------4a5744f8548f----------------------"><em>Part 0</em></a><em> , </em><a href="https://medium.com/towards-artificial-intelligence/the-langgraph-mental-model-a-standardized-architecture-guide-for-every-agent-youll-ever-build-d02265f3bebf?source=user_profile_page---------2-------------4a5744f8548f----------------------"><em>Part 1</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-memory-the-complete-practical-guide-to-managing-what-your-agent-remembers-d865d505a59e?source=user_profile_page---------1-------------4a5744f8548f----------------------"><em>Part 2</em></a><em> , </em><a href="https://medium.com/@bessiedelight/langgraph-human-in-the-loop-pausing-reviewing-and-rewinding-your-agent-4028bd05b049?source=user_profile_page---------0-------------4a5744f8548f----------------------"><em>Part 3</em></a><em> </em>, <a href="https://pub.towardsai.net/your-first-real-langgraph-project-ac5eb00f923a"><em>Part 4</em></a><em> .</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c1773055693" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/langgraph-multi-agent-systems-from-one-brain-to-many-4c1773055693">LangGraph Multi-Agent Systems: From One Brain to Many</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>