<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tanishk Soni on Medium]]></title>
        <description><![CDATA[Stories by Tanishk Soni on Medium]]></description>
        <link>https://medium.com/@sonitanishk2003?source=rss-df11a530c327------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Fvq63guiKrr6Hu7-AhJq_g.jpeg</url>
            <title>Stories by Tanishk Soni on Medium</title>
            <link>https://medium.com/@sonitanishk2003?source=rss-df11a530c327------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 19:26:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sonitanishk2003/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How AI Agent Memory Actually Works - And How to Build It]]></title>
            <link>https://pub.towardsai.net/how-ai-agent-memory-actually-works-and-how-to-build-it-0d0874e913bc?source=rss-df11a530c327------2</link>
            <guid isPermaLink="false">https://medium.com/p/0d0874e913bc</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Tanishk Soni]]></dc:creator>
            <pubDate>Sun, 12 Jul 2026 00:01:01 GMT</pubDate>
            <atom:updated>2026-07-12T00:01:01.348Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4MQPuB_Vf-9PnaN3PFFwhA.png" /></figure><p>Every call you make to a large language model is stateless. The model reads the text you send, generates a reply, and forgets everything the moment it finishes. That is fine for a one-off question and a disaster for an agent - a system that takes multi-step actions, learns from feedback, and picks up work across sessions. <strong>AI agent memory architecture</strong> is the layer you build on top of that stateless model to give it continuity: a way to remember what happened, what’s true, and how to act.</p><p>I wrote a full guide on <a href="https://medium.com/@sonitanishk2003/the-ultimate-guide-to-llm-memory-from-context-windows-to-advanced-agent-memory-systems-3ec106d2a345">The Ultimate Guide to LLM Memory</a> that covered the context window itself. This piece is the sequel - moving from “how a model holds a conversation” to “how you architect memory that survives across sessions.” By the end you’ll have runnable, framework-agnostic Python for each memory type and a decision framework for which ones you actually need. No vendor lock-in, no hand-waving.</p><h3>What is AI agent memory architecture?</h3><blockquote><em>An AI agent memory architecture is the two-layer system - working memory held in the context window plus long-term memory stored externally - that lets a stateless model persist and recall information across turns and sessions. Working memory holds the active task; long-term memory holds durable facts, past events, and learned behaviors that outlive any single conversation.</em></blockquote><p>A language model has no internal store that updates between calls. The <a href="https://arxiv.org/abs/2309.02427">Cognitive Architectures for Language Agents (CoALA) framework</a>, a peer-reviewed taxonomy from Sumers, Yao, Narasimhan, and Griffiths, formalizes the split most systems now use: information storage divided into <strong>working memory</strong> (the immediate context) and <strong>long-term memory</strong>(persistent, external stores the agent reads from and writes to).</p><p>The practical consequence is that memory is not one thing. It is a pipeline that moves information between a small, fast, in-context layer and larger, slower, external layers - a design <a href="https://arxiv.org/abs/2310.08560">MemGPT explicitly borrowed from operating systems</a>, treating the context window like RAM and external storage like disk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVWpt2CBPwhfzSl0uMFumQ.png" /></figure><h3>Why every LLM call is stateless by default</h3><p>The model receives a block of text, processes it, returns a response, and retains nothing. There is no hidden state carried forward. When ChatGPT “remembers” what you said three turns ago, it is because your application resent those turns inside the prompt - not because the model stored them.</p><h3>The two layers: working vs long-term</h3><p>Working memory lives inside the context window and is rebuilt on every call. Long-term memory lives outside the model in a database, vector store, or file, and is queried when relevant. The entire craft of agent memory is deciding what to promote from working into long-term, what to retrieve back, and what to drop.</p><h3>The four types of agent memory</h3><blockquote><em>Agent memory is usually split into four types: working memory (the active context window), episodic memory (records of past interactions), semantic memory (durable facts and preferences), and procedural memory (learned skills and workflows). Working memory is short-term; the other three are long-term stores that persist across sessions.</em></blockquote><p>This four-part split isn’t a marketing invention. The episodic-versus-semantic distinction comes from psychologist Endel Tulving’s 1972 work on human memory, and CoALA adapts it directly for language agents. <a href="https://machinelearningmastery.com/beyond-short-term-memory-the-3-types-of-long-term-memory-ai-agents-need/">MachineLearningMastery’s breakdown</a> and <a href="https://www.ibm.com/think/topics/ai-agent-memory">IBM’s overview</a> both land on the same categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qqL92exqnJSicVHdgItquQ.png" /></figure><h3>Working memory (the scratchpad)</h3><p>The context window itself - recent turns, the current task state, whatever the user just clarified. Think of it as RAM: fast, limited, and wiped when the session ends. Everything else in this article exists to feed the right information into this layer at the right time.</p><h3>Episodic memory (what happened)</h3><p>Records of specific past interactions, tied to a time and context: “On July 2, the user booked a trip to London and asked for city-center hotels.” Episodic memory is what lets an agent say “last week you told me…” It is typically stored as a timestamped log or as indexed conversation chunks.</p><h3>Semantic memory (what’s true)</h3><p>Durable facts, preferences, and constraints that hold across time: “prefers Python over JavaScript,” “budget cap is $50K,” “allergic to peanuts.” The key operation here is <em>updating</em> rather than appending. When “budget cap $50K” becomes “$75K,” you overwrite the entry - you do not store two contradictory facts and hope retrieval picks the right one.</p><h3>Procedural memory (how to act)</h3><p>Learned skills, tool-use patterns, and workflows - the agent’s repertoire of effective actions. A coding assistant that has learned your project’s test-then-refactor loop is using procedural memory. In practice it often lives as prompt templates, tool policies, or code rather than as retrieved text.</p><h3>Agent memory vs RAG vs a big context window</h3><blockquote><em>Retrieval-augmented generation (RAG) is read-only retrieval from a shared, static corpus; every user draws from the same documents and nothing changes based on the interaction. Agent memory reads </em>and<em> writes: it is stateful, user-scoped, and updates as the agent learns. A larger context window delays the memory problem but does not solve it.</em></blockquote><p>This is the single most confused distinction in the space, so it’s worth being precise. RAG, introduced by Lewis et al. in 2020, fetches relevant chunks from a document index and injects them into the prompt. It is stateless and query-matched. Agent memory is a two-phase system: a <strong>write</strong> phase that persists information about the user or task over time, and a <strong>read</strong>phase that retrieves it later. Mem0’s <a href="https://mem0.ai/blog/rag-vs-ai-memory">own writeup on the difference</a> frames it well - RAG reads the library, memory writes the autobiography.</p><p>And no, a bigger context window is not a substitute. Stuffing an entire history into a 1M-token window increases latency and cost, and models still struggle to use information buried deep in long contexts. A larger window filled with contradictory or outdated entries makes reasoning worse, not better. The two are complementary: the strongest architectures use RAG for shared domain knowledge and memory for user-specific state.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8088d118cb839f6e917eecb8ec626bd8/href">https://medium.com/media/8088d118cb839f6e917eecb8ec626bd8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1k1tGzsn_GoMqPL820qnJQ.png" /></figure><h3>The memory lifecycle: write, consolidate, retrieve, forget</h3><blockquote><em>Agent memory works as a four-stage loop: </em><strong><em>write</em></strong><em> extracts what matters from a turn, </em><strong><em>consolidate</em></strong><em> deduplicates and resolves contradictions, </em><strong><em>retrieve</em></strong><em> pulls relevant memories back into context, and </em><strong><em>forget</em></strong><em> decays or evicts stale entries. Skipping the consolidate and forget stages is why naive memory systems degrade over time.</em></blockquote><p>Most tutorials stop at “store embeddings, retrieve top-K.” That produces a memory that only grows and never cleans itself. A production-grade architecture treats memory as a lifecycle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJTcA33hAtrcXNQ2nMp2qw.png" /></figure><h3>Write</h3><p>After each turn, decide what is worth persisting. Not every message contains a durable fact - “thanks!” does not need to be remembered. A common pattern is an extraction step: an LLM call that pulls candidate facts out of the exchange before anything is stored.</p><h3>Consolidate</h3><p>This is the stage naive systems skip. New information must be integrated without contradicting what’s already stored. When a new fact overlaps an existing one, you update in place rather than duplicate. Mem0’s <a href="https://mem0.ai/blog/long-term-memory-ai-agents">production pipeline</a> formalizes this as an extract-then-consolidate step, and the <a href="https://arxiv.org/abs/2304.03442">Generative Agents paper</a> popularized “reflection” - periodically synthesizing raw episodic memories into higher-level semantic insights.</p><h3>Retrieve</h3><p>At query time, pull the memories relevant to the current message. Three strategies dominate: vector similarity (semantic match), recency (most recent events first), and graph traversal (following relationships between entities). Many systems combine them.</p><h3>Forget</h3><p>Memory that only grows becomes memory that retrieves noise. Decay and eviction - dropping low-value or stale entries - keep retrieval sharp. <a href="https://redis.io/blog/ai-agent-memory-stateful-systems/">Redis’s engineering writeup</a> makes the point that context pollution, where irrelevant information degrades reasoning, is a first-order problem, not an edge case.</p><h3>Build a minimal agent memory system in Python</h3><blockquote><em>You can implement working, semantic, and episodic memory in a few dozen lines of framework-agnostic Python before reaching for any library. The code below uses stand-in </em><em>llm() and </em><em>embed() functions so you can drop in whichever model and embedding provider you use.</em></blockquote><p>Here’s a working skeleton. It’s deliberately dependency-light - the goal is to understand the moving parts, not to hide them behind an SDK.</p><h3>Working memory: rolling buffer with summarization</h3><pre>class WorkingMemory:<br>    &quot;&quot;&quot;Keeps the last N turns verbatim; older turns roll into a running summary.&quot;&quot;&quot;<br>    def __init__(self, max_turns=10):<br>        self.max_turns = max_turns<br>        self.buffer = []      # recent turns: {&quot;role&quot;: ..., &quot;content&quot;: ...}<br>        self.summary = &quot;&quot;     # compressed history of evicted turns<br><br>    def add(self, role, content):<br>        self.buffer.append({&quot;role&quot;: role, &quot;content&quot;: content})<br>        if len(self.buffer) &gt; self.max_turns:<br>            # Evict the oldest turns into the running summary instead of dropping them<br>            cut = len(self.buffer) - self.max_turns<br>            overflow, self.buffer = self.buffer[:cut], self.buffer[cut:]<br>            self.summary = self._summarize(self.summary, overflow)<br><br>    def _summarize(self, prior, turns):<br>        text = &quot;\n&quot;.join(f&quot;{t[&#39;role&#39;]}: {t[&#39;content&#39;]}&quot; for t in turns)<br>        return llm(f&quot;Prior summary:\n{prior}\n\nNew turns:\n{text}\n\n&quot;<br>                   &quot;Update the summary in 3-4 sentences.&quot;)<br><br>    def render(self):<br>        head = f&quot;[Summary so far]\n{self.summary}\n\n&quot; if self.summary else &quot;&quot;<br>        recent = &quot;\n&quot;.join(f&quot;{t[&#39;role&#39;]}: {t[&#39;content&#39;]}&quot; for t in self.buffer)<br>        return head + recent</pre><h3>Semantic memory: embed, store, retrieve — and update instead of duplicate</h3><pre>import numpy as np<br><br>def cosine(a, b):<br>    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-9))<br><br>class SemanticMemory:<br>    &quot;&quot;&quot;Durable facts. Near-duplicate facts overwrite the old entry (consolidation).&quot;&quot;&quot;<br>    def __init__(self, dedupe_threshold=0.92):<br>        self.facts = []       # list[str]<br>        self.vectors = []     # list[np.ndarray]<br>        self.threshold = dedupe_threshold<br><br>    def write(self, fact):<br>        vec = embed(fact)     # your embedding model -&gt; np.ndarray<br>        for i, v in enumerate(self.vectors):<br>            if cosine(vec, v) &gt; self.threshold:<br>                # Same subject -&gt; update in place rather than store a contradiction<br>                self.facts[i], self.vectors[i] = fact, vec<br>                return<br>        self.facts.append(fact)<br>        self.vectors.append(vec)<br><br>    def retrieve(self, query, k=3):<br>        q = embed(query)<br>        ranked = sorted(zip(self.facts, self.vectors),<br>                        key=lambda fv: cosine(q, fv[1]), reverse=True)<br>        return [fact for fact, _ in ranked[:k]]</pre><p><em>(In production you’d swap the in-memory lists for a real vector store - the trade-offs there, like index choice and hybrid search, are a topic on their own that I’ll cover next in this series. The logic above stays identical.)</em></p><h3>Episodic memory: a timestamped event log</h3><pre>import time, json<br><br>class EpisodicMemory:<br>    &quot;&quot;&quot;Append-only log of what happened, with timestamps for recency retrieval.&quot;&quot;&quot;<br>    def __init__(self, path=&quot;episodes.jsonl&quot;):<br>        self.path = path<br><br>    def log(self, event, meta=None):<br>        record = {&quot;ts&quot;: time.time(), &quot;event&quot;: event, &quot;meta&quot;: meta or {}}<br>        with open(self.path, &quot;a&quot;) as f:<br>            f.write(json.dumps(record) + &quot;\n&quot;)<br><br>    def recent(self, n=3):<br>        with open(self.path) as f:<br>            lines = f.readlines()<br>        return [json.loads(line) for line in lines[-n:]]</pre><h3>Assembling context for one agent turn</h3><pre>def build_context(user_msg, working, semantic, episodic):<br>    facts = semantic.retrieve(user_msg, k=3)<br>    events = episodic.recent(n=3)<br>    return &quot;\n&quot;.join([<br>        &quot;## What I know about you:&quot;,<br>        *[f&quot;- {f}&quot; for f in facts],<br>        &quot;\n## Recent events:&quot;,<br>        *[f&quot;- {e[&#39;event&#39;]}&quot; for e in events],<br>        &quot;\n## Conversation:&quot;,<br>        working.render(),<br>        f&quot;\n## Current message:\n{user_msg}&quot;,<br>    ])<br><br>def agent_turn(user_msg, working, semantic, episodic):<br>    working.add(&quot;user&quot;, user_msg)<br>    reply = llm(build_context(user_msg, working, semantic, episodic))<br>    working.add(&quot;assistant&quot;, reply)<br>    # Write phase: persist durable facts + log the interaction<br>    for fact in extract_facts(user_msg, reply):   # an LLM extraction step<br>        semantic.write(fact)<br>    episodic.log(f&quot;user said: {user_msg}&quot;, {&quot;reply&quot;: reply})<br>    return reply</pre><p>That’s the whole architecture in one screen: working memory for coherence, semantic memory for facts, episodic memory for history, and a turn loop that reads before it answers and writes after. Procedural memory slots in the same way — as retrievable workflow templates or tool policies you inject into build_context.</p><h3>When to use which memory type</h3><blockquote><em>Not every agent needs every memory type. Match the architecture to the use case and add complexity only when operational value justifies it: episodic for interaction history, semantic for stable facts and personalization, procedural for repeatable skills. Start with working memory plus one long-term type, then expand.</em></blockquote><p>The most common mistake is building all four types on day one. <a href="https://redis.io/blog/build-smarter-ai-agents-manage-short-term-and-long-term-memory-with-redis/">Redis’s guidance</a> is blunt about it: use the simplest memory that reliably supports the task.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0be68ae91b7923ecb803c342ea56e7bb/href">https://medium.com/media/0be68ae91b7923ecb803c342ea56e7bb/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V596QCf15leGw_JmLGh4Rw.png" /></figure><h3>What breaks in production</h3><blockquote><em>The four failure modes that show up at scale are context pollution (irrelevant memories degrading reasoning), temporal conflicts (new facts contradicting old ones), retrieval latency and cost, and multi-session inconsistency. Each maps to a specific stage of the memory lifecycle, and each has a standard mitigation.</em></blockquote><p><strong>Context pollution.</strong> Retrieval returns too much, or the wrong things, and the model’s reasoning degrades. Mitigation: aggressive relevance filtering, smaller top-K, and the forget stage.</p><p><strong>Temporal conflicts.</strong> Old and new facts disagree. Mitigation: consolidate on write - overwrite rather than append, and timestamp facts so the current state is unambiguous.</p><p><strong>Latency and cost.</strong> Every retrieval and every LLM-managed memory operation adds tokens and time. MemGPT’s own design notes that letting the model page memory in and out adds overhead on every interaction where it decides to do so. Mitigation: cache, batch, and only retrieve when the query warrants it.</p><p><strong>Do you even need a framework?</strong> Rolling your own, as above, is the right call for learning and for simple cases. Once you need scale, temporal knowledge graphs, or multi-agent consistency, the mature options — Mem0, Letta (the production evolution of MemGPT), Zep, and LangMem — each take a different architectural stance. Mem0 is a framework-agnostic memory layer you bolt on; Letta is a full agent runtime built around the OS-style memory model. Pick based on whether you want a <em>library</em> or a <em>runtime</em>. The code above is enough to evaluate them with your eyes open instead of picking on vibes.</p><h3>FAQ</h3><p><strong>How does AI agent memory work?</strong> An external system captures information from each turn, stores it in working, episodic, semantic, or procedural memory, and retrieves the relevant pieces back into the context window on later turns - giving a stateless model the appearance of continuity.</p><p><strong>What are the types of AI agent memory?</strong> Four: working memory (the active context window), episodic memory (timestamped records of past interactions), semantic memory (durable facts and preferences), and procedural memory (learned skills and workflows). Working memory is short-term; the rest are long-term.</p><p><strong>What’s the difference between agent memory and RAG?</strong> RAG is read-only retrieval from a shared, static corpus. Agent memory reads and writes, is scoped to a specific user or session, and updates as the agent learns. They’re complementary - RAG for shared knowledge, memory for personal state.</p><p><strong>How do AI agents remember across sessions?</strong> By persisting memories in an external store (a database, vector store, or file) outside the model, then querying that store at the start of each new session and injecting the results into the prompt.</p><p><strong>Do I need a framework like Mem0, or can I build my own?</strong> For learning and simple agents, a few dozen lines of Python (like the code above) is enough. Reach for Mem0, Letta, Zep, or LangMem when you need scale, temporal knowledge graphs, or multi-agent consistency.</p><p><strong>Episodic vs semantic vs procedural — what’s the difference?</strong> Episodic stores <em>what happened</em> (events), semantic stores <em>what’s true</em> (facts), and procedural stores <em>how to act</em> (skills). The distinction traces back to Tulving’s 1972 memory research and is adapted for agents in the CoALA framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WsvyaJntnObf3p8z2ZcVRw.png" /></figure><p>If this helped, I’m writing a whole series on the internals of production LLM systems - RAG, vector databases, agents, and context engineering, all interlinked. Follow along for the rest, and if you haven’t yet, read <a href="https://medium.com/@sonitanishk2003/the-ultimate-guide-to-llm-memory-from-context-windows-to-advanced-agent-memory-systems-3ec106d2a345">[INTERNAL LINK: The Ultimate Guide to LLM Memory]</a> - the post this one builds on. RAG and vector databases are up next.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0d0874e913bc" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/how-ai-agent-memory-actually-works-and-how-to-build-it-0d0874e913bc">How AI Agent Memory Actually Works - And How to Build It</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced RAG with LangChain: The Phase 2 Capstone]]></title>
            <link>https://medium.com/@sonitanishk2003/advanced-rag-with-langchain-the-phase-2-capstone-d0010ebd62be?source=rss-df11a530c327------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0010ebd62be</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[Tanishk Soni]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 19:19:18 GMT</pubDate>
            <atom:updated>2026-03-31T19:19:18.133Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W3aiY_prEV9Hpp6WppcAMQ.png" /></figure><blockquote><strong><em>Series:</em></strong><em> LangChain Mastery | </em><strong><em>Post:</em></strong><em> #18 of 34 </em><strong><em>Phase:</em></strong><em> 2 — LangChain Core Concepts | </em><strong><em>Section:</em></strong><em> 2.8 (Capstone) </em><strong><em>Level:</em></strong><em> Advanced | </em><strong><em>Reading Time:</em></strong><em> ~15 min</em></blockquote><p>Eight posts into Phase 2. You know LCEL, parallel execution, memory, prompt templates, output parsers, and observability. Now we put all of it together into the most requested LLM application pattern in existence: <strong>Retrieval-Augmented Generation (RAG)</strong>.</p><p>RAG is the answer to two fundamental problems with LLMs:</p><ol><li>They don’t know about your private data</li><li>Their knowledge has a cutoff date</li></ol><p>The solution: before answering a question, retrieve relevant documents from your own knowledge base and include them in the prompt. The model answers based on your data, not just its training.</p><p>By the end of this post you’ll have a complete, production-grade RAG pipeline — document ingestion, chunking, vector storage, retrieval, a conversational interface with memory, and LangSmith tracing throughout.</p><h3>What We’re Building</h3><p>A conversational document Q&amp;A system with:</p><ul><li><strong>Document ingestion</strong> — load and chunk any text document</li><li><strong>Vector storage</strong> — embed chunks and store in a vector database</li><li><strong>Retrieval chain</strong> — find relevant chunks for any question</li><li><strong>Conversational interface</strong> — multi-turn chat with memory</li><li><strong>Source citations</strong> — show which documents answered the question</li><li><strong>LangSmith tracing</strong> — full observability on every query</li></ul><pre>project/<br>├── ingest.py      # Document loading, chunking, embedding<br>├── retriever.py   # Vector store setup and retrieval chain<br>├── chain.py       # Conversational RAG chain with memory<br>└── main.py        # CLI interface</pre><h3>How RAG Works — The Mental Model</h3><p>Before code, the mental model:</p><pre>Query: &quot;What is LangChain&#39;s LCEL?&quot;<br>         ↓<br>  [1. Embed query]<br>  Query → embedding vector [0.23, -0.87, ...]<br>         ↓<br>  [2. Similarity search]<br>  Find top-K chunks whose vectors are closest to query vector<br>         ↓<br>  [3. Build prompt]<br>  &quot;Answer the question using these documents:<br>   [Chunk 1: LCEL is LangChain&#39;s expression language...]<br>   [Chunk 2: The pipe operator connects Runnables...]<br>   Question: What is LangChain&#39;s LCEL?&quot;<br>         ↓<br>  [4. Generate answer]<br>  Model answers from retrieved context, not from training alone</pre><p>The model never sees your full document library — only the most relevant chunks for each question. This keeps prompts focused and costs manageable.</p><h3>Step 1: ingest.py — Document Loading &amp; Chunking</h3><pre># ingest.py<br>from langchain_community.document_loaders import (<br>    TextLoader,<br>    PyPDFLoader,<br>    WebBaseLoader,<br>    DirectoryLoader,<br>)<br>from langchain_text_splitters import RecursiveCharacterTextSplitter<br>from langchain_openai import OpenAIEmbeddings<br>from langchain_community.vectorstores import Chroma<br>from langchain_core.documents import Document<br>from typing import List<br>import os<br><br># ── Document Loading ────────────────────────────────────────<br><br>def load_text(file_path: str) -&gt; List[Document]:<br>    &quot;&quot;&quot;Load a plain text file.&quot;&quot;&quot;<br>    loader = TextLoader(file_path, encoding=&quot;utf-8&quot;)<br>    return loader.load()<br><br>def load_pdf(file_path: str) -&gt; List[Document]:<br>    &quot;&quot;&quot;Load a PDF — each page becomes a Document.&quot;&quot;&quot;<br>    loader = PyPDFLoader(file_path)<br>    return loader.load()<br><br>def load_url(/service/url: str) -&gt; List[Document]:<br>    &quot;&quot;&quot;Load a webpage.&quot;&quot;&quot;<br>    loader = WebBaseLoader(url)<br>    return loader.load()<br><br>def load_directory(dir_path: str, glob: str = &quot;**/*.txt&quot;) -&gt; List[Document]:<br>    &quot;&quot;&quot;Load all matching files from a directory.&quot;&quot;&quot;<br>    loader = DirectoryLoader(dir_path, glob=glob, loader_cls=TextLoader)<br>    return loader.load()<br><br><br># ── Chunking ────────────────────────────────────────────────<br><br>def chunk_documents(<br>    documents: List[Document],<br>    chunk_size: int = 1000,<br>    chunk_overlap: int = 200,<br>) -&gt; List[Document]:<br>    &quot;&quot;&quot;<br>    Split documents into overlapping chunks.<br><br>    chunk_size: target size in characters<br>    chunk_overlap: overlap between consecutive chunks<br>                   (prevents splitting a concept across chunk boundaries)<br>    &quot;&quot;&quot;<br>    splitter = RecursiveCharacterTextSplitter(<br>        chunk_size=chunk_size,<br>        chunk_overlap=chunk_overlap,<br>        # Try to split on these boundaries in order — paragraph &gt; sentence &gt; word<br>        separators=[&quot;\n\n&quot;, &quot;\n&quot;, &quot;. &quot;, &quot; &quot;, &quot;&quot;],<br>        length_function=len,<br>    )<br><br>    chunks = splitter.split_documents(documents)<br><br>    print(f&quot;  Split {len(documents)} documents → {len(chunks)} chunks&quot;)<br>    print(f&quot;  Avg chunk size: {sum(len(c.page_content) for c in chunks) // len(chunks)} chars&quot;)<br><br>    return chunks<br><br><br># ── Embedding &amp; Vector Storage ──────────────────────────────<br><br>def build_vectorstore(<br>    chunks: List[Document],<br>    persist_directory: str = &quot;./chroma_db&quot;,<br>    collection_name: str = &quot;rag_docs&quot;,<br>) -&gt; Chroma:<br>    &quot;&quot;&quot;<br>    Embed chunks and store in ChromaDB.<br>    Persists to disk — won&#39;t re-embed on restart.<br>    &quot;&quot;&quot;<br>    embeddings = OpenAIEmbeddings(model=&quot;text-embedding-3-small&quot;)<br><br>    vectorstore = Chroma.from_documents(<br>        documents=chunks,<br>        embedding=embeddings,<br>        persist_directory=persist_directory,<br>        collection_name=collection_name,<br>    )<br><br>    print(f&quot;  Embedded {len(chunks)} chunks → stored in {persist_directory}&quot;)<br>    return vectorstore<br><br><br>def load_vectorstore(<br>    persist_directory: str = &quot;./chroma_db&quot;,<br>    collection_name: str = &quot;rag_docs&quot;,<br>) -&gt; Chroma:<br>    &quot;&quot;&quot;Load an existing vector store from disk.&quot;&quot;&quot;<br>    embeddings = OpenAIEmbeddings(model=&quot;text-embedding-3-small&quot;)<br><br>    return Chroma(<br>        persist_directory=persist_directory,<br>        embedding_function=embeddings,<br>        collection_name=collection_name,<br>    )<br><br><br># ── Main ingestion pipeline ──────────────────────────────────<br><br>def ingest(source: str, source_type: str = &quot;text&quot;) -&gt; Chroma:<br>    &quot;&quot;&quot;<br>    Full ingestion pipeline: load → chunk → embed → store.<br>    Returns a ready-to-query vector store.<br>    &quot;&quot;&quot;<br>    print(f&quot;\n📥 Ingesting: {source}&quot;)<br><br>    # Load<br>    loaders = {<br>        &quot;text&quot;: load_text,<br>        &quot;pdf&quot;:  load_pdf,<br>        &quot;url&quot;:  load_url,<br>        &quot;dir&quot;:  load_directory,<br>    }<br>    documents = loaders[source_type](source)<br>    print(f&quot;  Loaded {len(documents)} document(s)&quot;)<br><br>    # Chunk<br>    chunks = chunk_documents(documents)<br><br>    # Embed &amp; store<br>    vectorstore = build_vectorstore(chunks)<br><br>    print(f&quot;✅ Ingestion complete\n&quot;)<br>    return vectorstore</pre><h4>Why chunk_overlap matters</h4><pre>Chunk 1: &quot;...LangChain uses the pipe operator to connect&quot;<br>Chunk 2: &quot;Runnables together into a sequence called...&quot;</pre><p>Without overlap, a question about “pipe operator connecting Runnables” might not retrieve either chunk cleanly. With 200-character overlap, chunk 2 repeats the end of chunk 1 — the relevant context stays together.</p><h3>Step 2: retriever.py — Retrieval Chain</h3><pre># retriever.py<br>from langchain_community.vectorstores import Chroma<br>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.runnables import RunnablePassthrough, RunnableLambda<br>from langchain_core.documents import Document<br>from typing import List<br><br><br>def format_docs(docs: List[Document]) -&gt; str:<br>    &quot;&quot;&quot;Format retrieved documents into a single context string.&quot;&quot;&quot;<br>    formatted = []<br>    for i, doc in enumerate(docs, 1):<br>        source = doc.metadata.get(&quot;source&quot;, &quot;unknown&quot;)<br>        page = doc.metadata.get(&quot;page&quot;, &quot;&quot;)<br>        page_str = f&quot;, page {page}&quot; if page else &quot;&quot;<br>        formatted.append(<br>            f&quot;[Document {i} — {source}{page_str}]\n{doc.page_content}&quot;<br>        )<br>    return &quot;\n\n---\n\n&quot;.join(formatted)<br><br><br>def build_rag_chain(vectorstore: Chroma, k: int = 4):<br>    &quot;&quot;&quot;<br>    Build a RAG chain that retrieves relevant docs and answers questions.<br><br>    k: number of chunks to retrieve per query<br>    &quot;&quot;&quot;<br>    retriever = vectorstore.as_retriever(<br>        search_type=&quot;mmr&quot;,    # Maximum Marginal Relevance — diversity + relevance<br>        search_kwargs={<br>            &quot;k&quot;: k,           # Number of chunks to return<br>            &quot;fetch_k&quot;: 20,    # Fetch 20, re-rank to k=4 for diversity<br>        }<br>    )<br><br>    prompt = ChatPromptTemplate.from_messages([<br>        (&quot;system&quot;,<br>         &quot;You are a helpful assistant that answers questions based on &quot;<br>         &quot;the provided documents. Be accurate and cite which document &quot;<br>         &quot;supports each claim.\n\n&quot;<br>         &quot;If the answer is not in the documents, say: &quot;<br>         &quot;&#39;I don&#39;t have information about that in the provided documents.&#39;\n\n&quot;<br>         &quot;Documents:\n{context}&quot;),<br>        (&quot;human&quot;, &quot;{question}&quot;),<br>    ])<br><br>    model = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br>    parser = StrOutputParser()<br><br>    # Chain: retrieve → format → prompt → model → parse<br>    rag_chain = (<br>        {<br>            &quot;context&quot;:  retriever | RunnableLambda(format_docs),<br>            &quot;question&quot;: RunnablePassthrough()<br>        }<br>        | prompt<br>        | model<br>        | parser<br>    )<br><br>    return rag_chain, retriever<br><br><br>def build_rag_chain_with_sources(vectorstore: Chroma, k: int = 4):<br>    &quot;&quot;&quot;<br>    RAG chain that returns both the answer AND the source documents.<br>    Useful for showing citations in your UI.<br>    &quot;&quot;&quot;<br>    retriever = vectorstore.as_retriever(<br>        search_type=&quot;mmr&quot;,<br>        search_kwargs={&quot;k&quot;: k, &quot;fetch_k&quot;: 20}<br>    )<br><br>    prompt = ChatPromptTemplate.from_messages([<br>        (&quot;system&quot;,<br>         &quot;Answer based on the documents below. &quot;<br>         &quot;If the answer isn&#39;t in the documents, say so.\n\n&quot;<br>         &quot;Documents:\n{context}&quot;),<br>        (&quot;human&quot;, &quot;{question}&quot;),<br>    ])<br><br>    model = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br>    parser = StrOutputParser()<br><br>    # Retrieve docs and keep them alongside the answer<br>    retrieve_chain = RunnablePassthrough.assign(<br>        docs=retriever<br>    )<br><br>    answer_chain = (<br>        RunnablePassthrough.assign(<br>            context=lambda x: format_docs(x[&quot;docs&quot;])<br>        )<br>        | RunnablePassthrough.assign(<br>            answer=prompt | model | parser<br>        )<br>    )<br><br>    full_chain = retrieve_chain | answer_chain<br><br>    return full_chain</pre><h4>MMR vs similarity search</h4><p>Standard similarity search returns the k most similar chunks. The problem: they&#39;re often near-duplicates of each other — the same information repeated from slightly different angles.</p><p><strong>Maximum Marginal Relevance (MMR)</strong> balances relevance with diversity. It fetches a larger candidate set (fetch_k=20) then selects k=4 that are both relevant AND different from each other. You get broader coverage of the topic with less redundancy.</p><h3>Step 3: chain.py — Conversational RAG with Memory</h3><p>A plain RAG chain can’t handle follow-up questions. If the user asks “Who wrote it?” after “What is LCEL?”, the chain has no idea what “it” refers to. We need memory.</p><p>The pattern: before retrieval, use the conversation history to rewrite the follow-up question into a standalone question:</p><pre>History: User asked &quot;What is LCEL?&quot;<br>Follow-up: &quot;Who created it?&quot;<br>Standalone: &quot;Who created LangChain&#39;s LCEL (LangChain Expression Language)?&quot;</pre><p>Then retrieve using the standalone question.</p><pre># chain.py<br>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.runnables import RunnablePassthrough, RunnableLambda<br>from langchain_core.runnables.history import RunnableWithMessageHistory<br>from langchain_community.chat_message_histories import ChatMessageHistory<br>from langchain_community.vectorstores import Chroma<br>from typing import List<br>from retriever import format_docs, build_rag_chain_with_sources<br><br><br># ── Question rewriting ───────────────────────────────────────<br><br>contextualize_prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;,<br>     &quot;Given the conversation history and a follow-up question, &quot;<br>     &quot;rewrite the follow-up as a standalone question that contains &quot;<br>     &quot;all necessary context. If it&#39;s already standalone, return it unchanged. &quot;<br>     &quot;Return ONLY the rewritten question — no explanation.&quot;),<br>    MessagesPlaceholder(variable_name=&quot;chat_history&quot;),<br>    (&quot;human&quot;, &quot;{input}&quot;),<br>])<br><br>contextualize_chain = (<br>    contextualize_prompt<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br>    | StrOutputParser()<br>)<br><br><br># ── Answer generation ────────────────────────────────────────<br><br>qa_prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;,<br>     &quot;You are a helpful assistant that answers questions based on &quot;<br>     &quot;retrieved documents. Be accurate, cite sources by document number, &quot;<br>     &quot;and acknowledge when information isn&#39;t available in the documents.\n\n&quot;<br>     &quot;Retrieved Documents:\n{context}&quot;),<br>    MessagesPlaceholder(variable_name=&quot;chat_history&quot;),<br>    (&quot;human&quot;, &quot;{input}&quot;),<br>])<br><br><br>def build_conversational_rag(vectorstore: Chroma):<br>    &quot;&quot;&quot;<br>    Build a full conversational RAG chain with:<br>    - Question contextualization (handles follow-up questions)<br>    - Document retrieval with MMR<br>    - Answer generation with source citations<br>    - Conversation memory<br>    &quot;&quot;&quot;<br>    retriever = vectorstore.as_retriever(<br>        search_type=&quot;mmr&quot;,<br>        search_kwargs={&quot;k&quot;: 4, &quot;fetch_k&quot;: 20}<br>    )<br><br>    model = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br>    parser = StrOutputParser()<br><br>    # Full chain:<br>    # 1. Rewrite question using history<br>    # 2. Retrieve relevant docs using rewritten question<br>    # 3. Answer using docs + history + rewritten question<br>    def get_context(inputs: dict) -&gt; str:<br>        &quot;&quot;&quot;Retrieve and format docs for the (possibly rewritten) question.&quot;&quot;&quot;<br>        question = inputs.get(&quot;standalone_question&quot;, inputs[&quot;input&quot;])<br>        docs = retriever.invoke(question)<br>        return format_docs(docs)<br><br>    rag_chain = (<br>        RunnablePassthrough.assign(<br>            standalone_question=contextualize_chain<br>        )<br>        | RunnablePassthrough.assign(<br>            context=RunnableLambda(get_context)<br>        )<br>        | RunnablePassthrough.assign(<br>            answer=qa_prompt | model | parser<br>        )<br>    )<br><br>    # Session management<br>    session_store = {}<br><br>    def get_session_history(session_id: str) -&gt; ChatMessageHistory:<br>        if session_id not in session_store:<br>            session_store[session_id] = ChatMessageHistory()<br>        return session_store[session_id]<br><br>    # Wrap with memory<br>    conversational_chain = RunnableWithMessageHistory(<br>        rag_chain,<br>        get_session_history,<br>        input_messages_key=&quot;input&quot;,<br>        history_messages_key=&quot;chat_history&quot;,<br>        output_messages_key=&quot;answer&quot;,<br>    )<br><br>    return conversational_chain, session_store</pre><h3>Step 4: main.py — The CLI Interface</h3><pre># main.py<br>import os<br>import sys<br>from langchain_core.runnables import RunnableConfig<br>from ingest import ingest, load_vectorstore<br>from chain import build_conversational_rag<br><br># LangSmith tracing — set these in your environment<br>os.environ.setdefault(&quot;LANGCHAIN_TRACING_V2&quot;, &quot;true&quot;)<br>os.environ.setdefault(&quot;LANGCHAIN_PROJECT&quot;, &quot;rag-capstone&quot;)<br><br><br>def run_rag_cli(source: str = None, source_type: str = &quot;text&quot;):<br>    # ── Build or load vector store ──────────────────────────<br>    chroma_dir = &quot;./chroma_db&quot;<br><br>    if source:<br>        # Fresh ingestion<br>        vectorstore = ingest(source, source_type)<br>    elif os.path.exists(chroma_dir):<br>        # Load existing<br>        print(&quot;📂 Loading existing vector store...&quot;)<br>        vectorstore = load_vectorstore(chroma_dir)<br>    else:<br>        print(&quot;❌ No source provided and no existing vector store found.&quot;)<br>        print(&quot;   Usage: python main.py &lt;path_to_file&gt;&quot;)<br>        sys.exit(1)<br><br>    # ── Build conversational chain ──────────────────────────<br>    chain, session_store = build_conversational_rag(vectorstore)<br>    session_id = &quot;default_session&quot;<br>    config = RunnableConfig(<br>        configurable={&quot;session_id&quot;: session_id},<br>        tags=[&quot;rag-capstone&quot;],<br>        metadata={&quot;session_id&quot;: session_id}<br>    )<br><br>    print(&quot;\n&quot; + &quot;═&quot; * 55)<br>    print(&quot;  🦜 RAG Q&amp;A — LangChain Mastery Capstone&quot;)<br>    print(&quot;═&quot; * 55)<br>    print(&quot;  Ask questions about your documents.&quot;)<br>    print(&quot;  Commands: /sources, /clear, /quit&quot;)<br>    print(&quot;═&quot; * 55 + &quot;\n&quot;)<br><br>    # ── Chat loop ───────────────────────────────────────────<br>    while True:<br>        try:<br>            user_input = input(&quot;  You: &quot;).strip()<br>        except (KeyboardInterrupt, EOFError):<br>            print(&quot;\n\n  👋 Goodbye!&quot;)<br>            break<br><br>        if not user_input:<br>            continue<br><br>        if user_input == &quot;/quit&quot;:<br>            break<br><br>        if user_input == &quot;/clear&quot;:<br>            if session_id in session_store:<br>                session_store[session_id].clear()<br>            print(&quot;  ✅ History cleared.&quot;)<br>            continue<br><br>        if user_input == &quot;/sources&quot;:<br>            history = session_store.get(session_id)<br>            if history and history.messages:<br>                print(f&quot;  Session has {len(history.messages)} messages&quot;)<br>            else:<br>                print(&quot;  No conversation history yet.&quot;)<br>            continue<br><br>        try:<br>            result = chain.invoke({&quot;input&quot;: user_input}, config=config)<br><br>            answer = result.get(&quot;answer&quot;, &quot;No answer generated.&quot;)<br>            standalone = result.get(&quot;standalone_question&quot;, user_input)<br>            docs = result.get(&quot;docs&quot;, [])<br><br>            print(f&quot;\n  🤖 Assistant: {answer}&quot;)<br><br>            # Show rewritten question if it changed<br>            if standalone.lower() != user_input.lower():<br>                print(f&quot;\n  🔍 [Searched for: \&quot;{standalone}\&quot;]&quot;)<br><br>            # Show sources<br>            if docs:<br>                sources = list(set(<br>                    d.metadata.get(&quot;source&quot;, &quot;unknown&quot;) for d in docs<br>                ))<br>                print(f&quot;  📄 Sources: {&#39;, &#39;.join(sources)}&quot;)<br><br>        except Exception as e:<br>            print(f&quot;\n  ❌ Error: {e}&quot;)<br><br>        print()<br><br>    print(&quot;\n  👋 Goodbye!\n&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    source = sys.argv[1] if len(sys.argv) &gt; 1 else None<br>    source_type = sys.argv[2] if len(sys.argv) &gt; 2 else &quot;text&quot;<br>    run_rag_cli(source, source_type)</pre><h3>Running It</h3><pre>pip install langchain langchain-openai langchain-community chromadb tiktoken pypdf<br><br># Ingest a document and start chatting<br>python main.py my_document.pdf pdf<br><br># Or ingest a URL<br>python main.py https://docs.langchain.com text url<br><br># Reload existing vector store (no re-ingestion)<br>python main.py</pre><h3>Demo Walkthrough</h3><pre>📥 Ingesting: langchain_docs.pdf<br>  Loaded 42 document(s)<br>  Split 42 documents → 187 chunks<br>  Avg chunk size: 847 chars<br>  Embedded 187 chunks → stored in ./chroma_db<br>✅ Ingestion complete<br><br>═══════════════════════════════════════════════════════<br>  🦜 RAG Q&amp;A — LangChain Mastery Capstone<br>═══════════════════════════════════════════════════════<br>  Ask questions about your documents.<br>  Commands: /sources, /clear, /quit<br>═══════════════════════════════════════════════════════<br><br>  You: What is LCEL?<br><br>  🤖 Assistant: LCEL (LangChain Expression Language) is a declarative way<br>to compose chains in LangChain. It uses the pipe operator (|) to connect<br>Runnable components — prompts, models, parsers, and custom functions —<br>into a sequence. Every component shares the same interface: .invoke(),<br>.stream(), .batch(), and their async counterparts. [Document 1]<br><br>  📄 Sources: langchain_docs.pdf<br><br>  You: Who designed it?<br><br>  🔍 [Searched for: &quot;Who designed LangChain&#39;s LCEL?&quot;]<br><br>  🤖 Assistant: Based on the provided documents, LCEL was designed by the<br>LangChain team, led by Harrison Chase. The documents don&#39;t provide<br>specific attribution for the LCEL design beyond the core team. [Document 3]<br><br>  📄 Sources: langchain_docs.pdf<br><br>  You: What are the execution modes?<br><br>  🔍 [Searched for: &quot;What are the LCEL execution modes?&quot;]<br><br>  🤖 Assistant: LCEL chains support six execution modes: invoke() for<br>single synchronous calls, stream() for token-by-token output, batch()<br>for parallel processing of multiple inputs, and async versions of each:<br>ainvoke(), astream(), and abatch(). [Document 2, Document 4]<br><br>  📄 Sources: langchain_docs.pdf</pre><p>Notice the question rewriting in action: “Who designed it?” becomes “Who designed LangChain’s LCEL?” — pulling in context from the previous turn. “What are the execution modes?” becomes “What are the LCEL execution modes?” — same pattern.</p><p>Without this step, the retriever would search for “Who designed it?” and return irrelevant chunks about product design in general.</p><h3>Production Improvements to Consider</h3><p>This capstone is complete and functional. For production, add:</p><p><strong>Better chunking:</strong> SemanticChunker splits on meaning rather than character count — keeps related sentences together regardless of length.</p><pre>from langchain_experimental.text_splitter import SemanticChunker<br>from langchain_openai import OpenAIEmbeddings<br><br>splitter = SemanticChunker(<br>    OpenAIEmbeddings(),<br>    breakpoint_threshold_type=&quot;percentile&quot;<br>)</pre><p><strong>Hybrid search:</strong> Combine vector similarity with BM25 keyword search — catches exact terms that embeddings sometimes miss.</p><pre>from langchain_community.retrievers import BM25Retriever<br>from langchain.retrievers import EnsembleRetriever<br><br>bm25 = BM25Retriever.from_documents(chunks, k=4)<br>vector = vectorstore.as_retriever(search_kwargs={&quot;k&quot;: 4})<br><br>hybrid = EnsembleRetriever(<br>    retrievers=[bm25, vector],<br>    weights=[0.4, 0.6]  # 40% keyword, 60% semantic<br>)</pre><p><strong>Re-ranking:</strong> Use a cross-encoder to re-rank retrieved chunks by true relevance before passing to the model.</p><p><strong>Streaming answers:</strong> Replace .invoke() with .stream() in the CLI loop — tokens appear as they&#39;re generated rather than waiting for the full answer.</p><h3>Phase 2 Complete</h3><p>Across eight posts, Phase 2 has covered the entire LangChain core:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d8eb107df68d00f9719e0a12425fcb52/href">https://medium.com/media/d8eb107df68d00f9719e0a12425fcb52/href</a></iframe><p><strong>Phase 3 starts next</strong> — production patterns. Cost optimization, latency tuning, observability, and everything that keeps a LangChain app healthy under real traffic.</p><p><em>Next up: </em><strong><em>Post #19 — Memory Management &amp; Cost Optimization: Production-Proofing Your Chain</em></strong></p><p><em>Follow the series on Medium and find the complete curriculum and Revision Notes on the </em><a href="https://bitphonix.github.io/Langchain-Mastery/"><em>LangChain Mastery website</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0010ebd62be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Metadata, Debugging & LangSmith: Seeing Inside Your Chain]]></title>
            <link>https://medium.com/@sonitanishk2003/metadata-debugging-langsmith-seeing-inside-your-chain-7d1cf2e1c190?source=rss-df11a530c327------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d1cf2e1c190</guid>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[debugging]]></category>
            <dc:creator><![CDATA[Tanishk Soni]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 19:07:15 GMT</pubDate>
            <atom:updated>2026-03-31T19:07:15.794Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pVLNNXS5VR5prfKZPcaI2Q.png" /></figure><blockquote><strong><em>Series:</em></strong><em> LangChain Mastery | </em><strong><em>Post:</em></strong><em> #17 of 34 </em><strong><em>Phase:</em></strong><em> 2 — LangChain Core Concepts | </em><strong><em>Sections:</em></strong><em> 2.7 + 2.8 (Combined) </em><strong><em>Level:</em></strong><em> Intermediate | </em><strong><em>Reading Time:</em></strong><em> ~9 min</em></blockquote><p>Here’s the debugging experience most developers have with LangChain:</p><p>Chain fails. You add print() statements. You can&#39;t tell which step failed. You add more prints. The output is a wall of AIMessage objects and dict dumps. You still don&#39;t know what the prompt looked like when it hit the model. You give up and hardcode something.</p><p>There’s a better way. LangChain has a built-in metadata and event system that gives you full visibility into every step of every chain — inputs, outputs, timing, token counts, errors. And LangSmith, LangChain’s observability platform, turns that event stream into a visual trace you can inspect, share, and query.</p><p>This post covers both: how to instrument your chains with metadata, and how to use LangSmith to debug, monitor, and improve them.</p><h3>What Metadata Is in LCEL</h3><p>Every LCEL chain execution produces a stream of events — on_chain_start, on_llm_start, on_llm_stream, on_chain_end, and so on. You can attach metadata to these events via RunnableConfig, which then shows up in your logs and traces.</p><pre>from langchain_core.runnables import RunnableConfig<br>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br><br>chain = (<br>    ChatPromptTemplate.from_template(&quot;Explain {concept} briefly.&quot;)<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>    | StrOutputParser()<br>)<br><br># Attach metadata to this invocation<br>result = chain.invoke(<br>    {&quot;concept&quot;: &quot;caching&quot;},<br>    config=RunnableConfig(<br>        run_name=&quot;explain-concept&quot;,               # Human-readable name in traces<br>        tags=[&quot;production&quot;, &quot;explain-feature&quot;],   # Filter by these in LangSmith<br>        metadata={                                 # Arbitrary key-value pairs<br>            &quot;user_id&quot;: &quot;u_789&quot;,<br>            &quot;session_id&quot;: &quot;sess_abc&quot;,<br>            &quot;feature&quot;: &quot;concept-explainer&quot;,<br>            &quot;model_version&quot;: &quot;v2&quot;<br>        }<br>    )<br>)</pre><p>tags and metadata appear on every child run inside the chain — the prompt, the model call, the parser — so you can filter and group traces by feature, user, or version without instrumenting each step individually.</p><h3>Callbacks — Hooking Into Chain Events</h3><p>Before LangSmith, callbacks were how you observed chain execution. They’re still useful for lightweight logging, custom metrics, or environments where LangSmith isn’t available.</p><pre>from langchain_core.callbacks import BaseCallbackHandler<br>from langchain_core.outputs import LLMResult<br>from typing import Any<br>import time<br><br>class TimingCallbackHandler(BaseCallbackHandler):<br>    &quot;&quot;&quot;Logs timing and token usage for every LLM call.&quot;&quot;&quot;<br><br>    def __init__(self):<br>        self.start_times = {}<br><br>    def on_llm_start(self, serialized: dict, prompts: list, **kwargs):<br>        run_id = str(kwargs.get(&quot;run_id&quot;, &quot;&quot;))<br>        self.start_times[run_id] = time.perf_counter()<br>        print(f&quot;\n⏱  LLM call started&quot;)<br>        print(f&quot;   Model: {serialized.get(&#39;name&#39;, &#39;unknown&#39;)}&quot;)<br>        print(f&quot;   Prompt tokens (est.): {sum(len(p.split()) for p in prompts)}&quot;)<br><br>    def on_llm_end(self, response: LLMResult, **kwargs):<br>        run_id = str(kwargs.get(&quot;run_id&quot;, &quot;&quot;))<br>        elapsed = time.perf_counter() - self.start_times.pop(run_id, time.perf_counter())<br>        usage = response.llm_output.get(&quot;token_usage&quot;, {}) if response.llm_output else {}<br>        print(f&quot;✅ LLM call completed in {elapsed:.2f}s&quot;)<br>        if usage:<br>            print(f&quot;   Tokens — in: {usage.get(&#39;prompt_tokens&#39;)} | &quot;<br>                  f&quot;out: {usage.get(&#39;completion_tokens&#39;)}&quot;)<br><br>    def on_llm_error(self, error: Exception, **kwargs):<br>        print(f&quot;❌ LLM error: {type(error).__name__}: {error}&quot;)<br><br>    def on_chain_start(self, serialized: dict, inputs: dict, **kwargs):<br>        name = serialized.get(&quot;name&quot;, &quot;Chain&quot;)<br>        print(f&quot;\n▶  {name} started | inputs: {list(inputs.keys())}&quot;)<br><br>    def on_chain_end(self, outputs: dict, **kwargs):<br>        print(f&quot;■  Chain ended | outputs: {list(outputs.keys())}&quot;)<br><br><br># Attach callback to invocation<br>timing_handler = TimingCallbackHandler()<br><br>result = chain.invoke(<br>    {&quot;concept&quot;: &quot;binary search&quot;},<br>    config=RunnableConfig(callbacks=[timing_handler])<br>)</pre><p>Output:</p><pre>▶  RunnableSequence started | inputs: [&#39;concept&#39;]<br>▶  ChatPromptTemplate started | inputs: [&#39;concept&#39;]<br>■  ChatPromptTemplate ended | outputs: [&#39;output&#39;]<br>▶  ChatOpenAI started | inputs: [&#39;messages&#39;]<br>⏱  LLM call started<br>   Model: ChatOpenAI<br>   Prompt tokens (est.): 8<br>✅ LLM call completed in 1.43s<br>   Tokens — in: 18 | out: 52<br>■  ChatOpenAI ended | outputs: [&#39;generations&#39;]<br>▶  StrOutputParser started | inputs: [&#39;input&#39;]<br>■  StrOutputParser ended | outputs: [&#39;output&#39;]<br>■  RunnableSequence ended | outputs: [&#39;output&#39;]</pre><p>Every step, timed, with token counts. No print statements inside your chain code — all observability is injected at the call site via config.</p><h3>Setting Up LangSmith</h3><p>LangSmith is where callbacks become genuinely powerful. It captures every event from every chain run and presents them as visual traces — searchable, filterable, and shareable.</p><h4>Setup</h4><pre>pip install langsmith</pre><p>Set environment variables:</p><pre>export LANGCHAIN_TRACING_V2=true<br>export LANGCHAIN_API_KEY=&quot;ls__your_key_here&quot;         # From smith.langchain.com<br>export LANGCHAIN_PROJECT=&quot;langchain-mastery-dev&quot;      # Groups traces by project</pre><p>That’s it. Once these are set, <strong>every chain invocation is automatically traced</strong> — no code changes needed. The same chain from before:</p><pre># No changes to chain code — LangSmith captures automatically<br>result = chain.invoke({&quot;concept&quot;: &quot;caching&quot;})<br># → Trace appears at smith.langchain.com under &quot;langchain-mastery-dev&quot;</pre><h4>What a trace looks like</h4><p>A LangSmith trace for a simple chain shows:</p><pre>RunnableSequence                    [1.82s total]<br>├── ChatPromptTemplate              [0.001s]<br>│   ├── Input:  {&quot;concept&quot;: &quot;caching&quot;}<br>│   └── Output: ChatPromptValue(messages=[...])<br>├── ChatOpenAI                      [1.73s]<br>│   ├── Input:  [SystemMessage, HumanMessage]<br>│   ├── Output: AIMessage(content=&quot;Caching is...&quot;)<br>│   └── Tokens: in=23, out=61, total=84<br>└── StrOutputParser                 [0.001s]<br>    ├── Input:  AIMessage(content=&quot;Caching is...&quot;)<br>    └── Output: &quot;Caching is the process of...&quot;</pre><p>You can see exactly what the prompt looked like when it hit the model, what the model returned, how long each step took, and how many tokens were used — all without adding a single line of observability code to your chain.</p><h3>Tracing Complex Chains &amp; Agents</h3><p>LangSmith gets more valuable as chains get more complex. For a chain with parallel steps, branches, and multiple model calls:</p><pre>from langchain_core.runnables import RunnableParallel, RunnablePassthrough<br><br>complex_chain = (<br>    RunnablePassthrough.assign(<br>        summary=summary_chain,<br>        sentiment=sentiment_chain,<br>        topics=topics_chain<br>    )<br>    | combine_prompt<br>    | ChatOpenAI(model=&quot;gpt-4o&quot;)<br>    | StrOutputParser()<br>)<br><br>result = complex_chain.invoke(<br>    {&quot;text&quot;: long_article},<br>    config=RunnableConfig(<br>        run_name=&quot;content-analysis&quot;,<br>        tags=[&quot;content-pipeline&quot;, &quot;v3&quot;],<br>        metadata={&quot;article_id&quot;: &quot;art_123&quot;, &quot;source&quot;: &quot;rss_feed&quot;}<br>    )<br>)</pre><p>In LangSmith, this traces as a tree:</p><pre>RunnableSequence [content-analysis]       [4.21s]<br>├── RunnablePassthrough<br>│   ├── summary_chain                     [3.82s] ← Slowest step<br>│   │   ├── ChatPromptTemplate<br>│   │   └── ChatOpenAI (in:312, out:89)<br>│   ├── sentiment_chain                   [1.23s]<br>│   │   ├── ChatPromptTemplate<br>│   │   └── ChatOpenAI (in:312, out:12)<br>│   └── topics_chain                      [1.45s]<br>│       ├── ChatPromptTemplate<br>│       └── ChatOpenAI (in:312, out:23)<br>├── ChatPromptTemplate<br>├── ChatOpenAI (in:156, out:201)          [2.11s]<br>└── StrOutputParser</pre><p>At a glance: summary_chain is the bottleneck at 3.82s. All three parallel branches share the same 312 input tokens — worth caching. The final model call used 201 output tokens — worth checking if that&#39;s more than needed.</p><p>This kind of insight is invisible without tracing.</p><h3>LangSmith for Debugging Failures</h3><p>When a chain fails, LangSmith shows you exactly where and why — including the full input that caused the failure.</p><pre>from langchain_core.exceptions import OutputParserException<br><br># Intentionally bad setup to trigger a parse failure<br>from langchain_core.output_parsers import PydanticOutputParser<br>from pydantic import BaseModel<br><br>class StrictOutput(BaseModel):<br>    score: int<br>    label: str<br><br>parser = PydanticOutputParser(pydantic_object=StrictOutput)<br><br>bad_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Rate {item} from 1-10.\n{format_instructions}&quot;<br>        # Missing &quot;Return ONLY JSON&quot; instruction — model will add explanation<br>    )<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0.7)<br>    | parser<br>)<br><br>try:<br>    result = bad_chain.invoke({<br>        &quot;item&quot;: &quot;Python&quot;,<br>        &quot;format_instructions&quot;: parser.get_format_instructions()<br>    })<br>except OutputParserException as e:<br>    print(f&quot;Parse failed: {e}&quot;)<br>    # LangSmith trace shows:<br>    # - Exact prompt sent to model<br>    # - Exact model response (probably had extra explanation text)<br>    # - Which parser step failed and why</pre><p>In LangSmith you’d see the exact model output that failed parsing — “Here is the JSON: json\n{...}&quot; — and immediately understand the fix: add &quot;Return ONLY the JSON object&quot; to the prompt.</p><p>Without tracing, you’d see OutputParserException: Failed to parse and guess.</p><h3>Custom Tracing with @traceable</h3><p>For code outside an LCEL chain — preprocessing functions, postprocessing steps, custom retrieval logic — use the @traceable decorator to include them in your traces:</p><pre>from langsmith import traceable<br><br>@traceable(name=&quot;preprocess-user-query&quot;, tags=[&quot;preprocessing&quot;])<br>def preprocess_query(raw_query: str) -&gt; str:<br>    &quot;&quot;&quot;Clean and normalize user input before sending to chain.&quot;&quot;&quot;<br>    cleaned = raw_query.strip().lower()<br>    # Remove common filler words<br>    fillers = [&quot;please&quot;, &quot;can you&quot;, &quot;could you&quot;, &quot;i want to know&quot;]<br>    for filler in fillers:<br>        cleaned = cleaned.replace(filler, &quot;&quot;).strip()<br>    return cleaned<br><br>@traceable(name=&quot;postprocess-response&quot;, tags=[&quot;postprocessing&quot;])<br>def postprocess_response(response: str, user_id: str) -&gt; dict:<br>    &quot;&quot;&quot;Add metadata to response before returning to user.&quot;&quot;&quot;<br>    return {<br>        &quot;content&quot;: response,<br>        &quot;user_id&quot;: user_id,<br>        &quot;word_count&quot;: len(response.split()),<br>        &quot;timestamp&quot;: time.time()<br>    }<br><br># These functions now appear in LangSmith traces alongside chain steps<br>query = preprocess_query(&quot;  Can you please explain what an API is?  &quot;)<br>response = chain.invoke({&quot;concept&quot;: query})<br>final = postprocess_response(response, user_id=&quot;u_789&quot;)</pre><p>@traceable makes your entire request lifecycle visible — not just the LangChain parts.</p><h3>LangSmith for Evaluation &amp; Testing</h3><p>Beyond debugging, LangSmith supports systematic evaluation — running your chain against a dataset of test cases and scoring the outputs.</p><pre>from langsmith import Client<br>from langsmith.evaluation import evaluate<br><br>client = Client()<br><br># Create a dataset of test cases<br>dataset = client.create_dataset(&quot;concept-explainer-tests&quot;)<br><br>client.create_examples(<br>    inputs=[<br>        {&quot;concept&quot;: &quot;recursion&quot;},<br>        {&quot;concept&quot;: &quot;async/await&quot;},<br>        {&quot;concept&quot;: &quot;REST API&quot;},<br>    ],<br>    outputs=[<br>        {&quot;expected_keywords&quot;: [&quot;function&quot;, &quot;itself&quot;, &quot;base case&quot;]},<br>        {&quot;expected_keywords&quot;: [&quot;non-blocking&quot;, &quot;event loop&quot;, &quot;await&quot;]},<br>        {&quot;expected_keywords&quot;: [&quot;HTTP&quot;, &quot;endpoint&quot;, &quot;stateless&quot;]},<br>    ],<br>    dataset_id=dataset.id<br>)<br><br># Define an evaluator<br>def keyword_coverage(run, example) -&gt; dict:<br>    &quot;&quot;&quot;Check what fraction of expected keywords appear in the response.&quot;&quot;&quot;<br>    response = run.outputs.get(&quot;output&quot;, &quot;&quot;).lower()<br>    keywords = example.outputs.get(&quot;expected_keywords&quot;, [])<br>    covered = sum(1 for kw in keywords if kw in response)<br>    score = covered / len(keywords) if keywords else 0<br>    return {&quot;score&quot;: score, &quot;comment&quot;: f&quot;{covered}/{len(keywords)} keywords found&quot;}<br><br># Run evaluation<br>results = evaluate(<br>    lambda inputs: chain.invoke(inputs),<br>    data=dataset.name,<br>    evaluators=[keyword_coverage],<br>    experiment_prefix=&quot;concept-explainer-v2&quot;<br>)<br><br>print(results.to_pandas())</pre><p>LangSmith shows evaluation results as a table — every test case, every score, side by side across experiments. When you change your prompt or model, you can run the evaluation again and see exactly what improved and what regressed.</p><h3>The Debugging Workflow</h3><p>Here’s the workflow that replaces print() debugging:</p><pre>1. Set LANGCHAIN_TRACING_V2=true in dev environment<br>2. Run your chain with a failing input<br>3. Open smith.langchain.com → find the failed run<br>4. Click through the trace tree:<br>   - Check prompt template output → did variables fill correctly?<br>   - Check model input → is the final prompt what you expected?<br>   - Check model output → what did the model actually return?<br>   - Check parser input → is it what the parser expected?<br>5. Fix the issue (usually: prompt wording, format instructions, or schema)<br>6. Re-run → verify trace shows correct behavior<br>7. Add to your evaluation dataset → prevent regression</pre><p>This loop — trace → inspect → fix → verify → add to eval set — is what separates teams that ship reliable LLM features from teams that are always firefighting.</p><h3>What’s Next</h3><p><strong>Post #18</strong> is the Phase 2 capstone: we build a complete Advanced RAG pipeline — document ingestion, vector storage, retrieval chain, and a full conversational interface with memory. Everything from Phase 2 in one working application.</p><p><em>Next up: </em><strong><em>Post #18 — Advanced RAG with LangChain: The Phase 2 Capstone</em></strong></p><p><em>Follow the series on Medium and find the complete curriculum and Revision Notes on the </em><a href="https://bitphonix.github.io/Langchain-Mastery/"><em>LangChain Mastery website</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d1cf2e1c190" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Output Parsers: Getting Structured Data from LLMs]]></title>
            <link>https://medium.com/@sonitanishk2003/output-parsers-getting-structured-data-from-llms-5398ee6ee0fb?source=rss-df11a530c327------2</link>
            <guid isPermaLink="false">https://medium.com/p/5398ee6ee0fb</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[generative-ai]]></category>
            <dc:creator><![CDATA[Tanishk Soni]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 08:42:48 GMT</pubDate>
            <atom:updated>2026-03-30T08:42:48.199Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fdQjkKgMRfeKsJ01y1zy9g.png" /></figure><blockquote><strong><em>Series:</em></strong><em> LangChain Mastery | </em><strong><em>Post:</em></strong><em> #16 of 34 </em><strong><em>Phase:</em></strong><em> 2 — LangChain Core Concepts | </em><strong><em>Section:</em></strong><em> 2.6 </em><strong><em>Level:</em></strong><em> Intermediate | </em><strong><em>Reading Time:</em></strong><em> ~9 min</em></blockquote><p>LLMs output text. Your application needs data.</p><p>That gap — between “a paragraph that mentions the answer” and “a Python dict you can work with” — is what output parsers solve. They sit at the end of your chain and convert raw model output into structured, typed, usable objects.</p><p>Get this right and your LLM app becomes a reliable data pipeline. Get it wrong and you’re writing fragile string parsing code that breaks whenever the model changes its phrasing.</p><p>This post covers every output parser you’ll actually use: StrOutputParser, JsonOutputParser, PydanticOutputParser, and CommaSeparatedListOutputParser — plus what to do when the model doesn&#39;t follow your format.</p><h3>StrOutputParser — The Default</h3><p>You’ve seen this in every post since #11. It does one thing: converts an AIMessage to a plain string.</p><pre>from langchain_core.output_parsers import StrOutputParser<br>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br><br>chain = (<br>    ChatPromptTemplate.from_template(&quot;Explain {concept} briefly.&quot;)<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>    | StrOutputParser()<br>)<br><br>result = chain.invoke({&quot;concept&quot;: &quot;caching&quot;})<br>print(type(result))   # &lt;class &#39;str&#39;&gt;<br>print(result)         # &quot;Caching is the process of storing...&quot;</pre><p>Without StrOutputParser, the chain returns an AIMessage object. With it, you get a plain string. That&#39;s the entire job.</p><p>Use StrOutputParser when:</p><ul><li>You’re displaying text directly to a user</li><li>The next step in your chain expects a string</li><li>You don’t need structured data from the response</li></ul><p>For everything else — structured data, typed objects, lists — use one of the parsers below.</p><h3>JsonOutputParser — Structured Dicts from LLMs</h3><p>JsonOutputParser instructs the model to return JSON and parses the response into a Python dict. It also handles streaming — it yields partial dicts as tokens arrive rather than waiting for the complete JSON.</p><pre>from langchain_core.output_parsers import JsonOutputParser<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_openai import ChatOpenAI<br><br>parser = JsonOutputParser()<br><br>chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Return a JSON object with these fields: &quot;<br>        &quot;name, category, difficulty (1-5), and a one-sentence description &quot;<br>        &quot;for this programming concept: {concept}\n\n&quot;<br>        &quot;{format_instructions}&quot;<br>    )<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br>    | parser<br>)<br><br>result = chain.invoke({<br>    &quot;concept&quot;: &quot;binary search&quot;,<br>    &quot;format_instructions&quot;: parser.get_format_instructions()<br>})<br><br>print(type(result))   # &lt;class &#39;dict&#39;&gt;<br>print(result)<br># {<br>#   &quot;name&quot;: &quot;Binary Search&quot;,<br>#   &quot;category&quot;: &quot;algorithms&quot;,<br>#   &quot;difficulty&quot;: 3,<br>#   &quot;description&quot;: &quot;An efficient search algorithm that...&quot;<br># }<br>print(result[&quot;difficulty&quot;])  # 3</pre><h4>get_format_instructions()</h4><p>Every structured parser has .get_format_instructions() — it generates text you inject into your prompt telling the model exactly what format to use:</p><pre>print(parser.get_format_instructions())<br># &quot;Return a markdown code snippet of a JSON object formatted to look like...&quot;</pre><p>Always inject format instructions into your prompt. Without them, the model might return valid JSON wrapped in markdown code fences, with extra explanation, or in a slightly wrong structure — all of which break parsing.</p><h4>Streaming with JsonOutputParser</h4><p>Unlike most parsers, JsonOutputParser supports streaming — it progressively builds the dict as tokens arrive:</p><pre>print(&quot;Streaming JSON: &quot;, end=&quot;&quot;)<br>for partial_result in chain.stream({&quot;concept&quot;: &quot;recursion&quot;,<br>                                     &quot;format_instructions&quot;: parser.get_format_instructions()}):<br>    print(partial_result)  # Prints partial dicts as they build up<br><br># Output (progressive):<br># {}<br># {&#39;name&#39;: &#39;&#39;}<br># {&#39;name&#39;: &#39;Recursion&#39;}<br># {&#39;name&#39;: &#39;Recursion&#39;, &#39;category&#39;: &#39;&#39;}<br># {&#39;name&#39;: &#39;Recursion&#39;, &#39;category&#39;: &#39;programming&#39;}<br># ... (builds up until complete)</pre><p>This is useful for showing progressive UI updates — you can render fields as they arrive rather than waiting for the full response.</p><h3>PydanticOutputParser — Typed &amp; Validated</h3><p>PydanticOutputParser is JsonOutputParser with type validation. You define a Pydantic model describing the exact shape you want, and LangChain ensures the output matches — raising a validation error if it doesn&#39;t.</p><pre>from langchain_core.output_parsers import PydanticOutputParser<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_openai import ChatOpenAI<br>from pydantic import BaseModel, Field<br>from typing import List<br><br># Define the exact shape you want<br>class ConceptAnalysis(BaseModel):<br>    name: str = Field(description=&quot;The name of the concept&quot;)<br>    category: str = Field(description=&quot;Category: algorithms, data-structures, patterns, etc.&quot;)<br>    difficulty: int = Field(description=&quot;Difficulty from 1 (easy) to 5 (expert)&quot;, ge=1, le=5)<br>    prerequisites: List[str] = Field(description=&quot;List of concepts to know first&quot;)<br>    description: str = Field(description=&quot;One sentence explanation&quot;)<br>    example_use_case: str = Field(description=&quot;A real-world use case&quot;)<br><br>parser = PydanticOutputParser(pydantic_object=ConceptAnalysis)<br><br>chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Analyze this programming concept: {concept}\n\n&quot;<br>        &quot;{format_instructions}&quot;<br>    )<br>    | ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br>    | parser<br>)<br><br>result = chain.invoke({<br>    &quot;concept&quot;: &quot;binary search tree&quot;,<br>    &quot;format_instructions&quot;: parser.get_format_instructions()<br>})<br><br># Result is a typed ConceptAnalysis object — not a dict<br>print(type(result))              # &lt;class &#39;ConceptAnalysis&#39;&gt;<br>print(result.name)               # &quot;Binary Search Tree&quot;<br>print(result.difficulty)         # 3<br>print(result.prerequisites)      # [&quot;arrays&quot;, &quot;recursion&quot;, &quot;trees&quot;]<br>print(result.example_use_case)   # &quot;Database indexing for fast lookups...&quot;<br><br># Full IDE autocompletion on result — no string key access</pre><h4>Field validation catches bad model output</h4><p>Pydantic validators run on the parsed output. If the model returns difficulty: 7 when you declared ge=1, le=5, parsing raises a ValidationError immediately rather than silently passing bad data into your application:</p><pre>from pydantic import BaseModel, Field, field_validator<br><br>class StrictAnalysis(BaseModel):<br>    difficulty: int = Field(ge=1, le=5)<br>    category: str<br><br>    @field_validator(&quot;category&quot;)<br>    @classmethod<br>    def category_must_be_valid(cls, v):<br>        valid = {&quot;algorithms&quot;, &quot;data-structures&quot;, &quot;patterns&quot;, &quot;concepts&quot;}<br>        if v.lower() not in valid:<br>            raise ValueError(f&quot;category must be one of {valid}&quot;)<br>        return v.lower()</pre><p>This is the key advantage of PydanticOutputParser over JsonOutputParser: you get type safety, range validation, and custom validators — not just a dict.</p><h4>When to use Pydantic vs Json parser</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/292f4f45065ea01f683e0a2115935d8f/href">https://medium.com/media/292f4f45065ea01f683e0a2115935d8f/href</a></iframe><h3>CommaSeparatedListOutputParser — Quick Lists</h3><p>For simple list outputs, CommaSeparatedListOutputParser is the fastest solution:</p><pre>from langchain_core.output_parsers import CommaSeparatedListOutputParser<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_openai import ChatOpenAI<br><br>parser = CommaSeparatedListOutputParser()<br><br>chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;List 5 popular Python libraries for {domain}.\n&quot;<br>        &quot;{format_instructions}&quot;<br>    )<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br>    | parser<br>)<br><br>result = chain.invoke({<br>    &quot;domain&quot;: &quot;data science&quot;,<br>    &quot;format_instructions&quot;: parser.get_format_instructions()<br>})<br><br>print(type(result))   # &lt;class &#39;list&#39;&gt;<br>print(result)         # [&#39;numpy&#39;, &#39;pandas&#39;, &#39;scikit-learn&#39;, &#39;matplotlib&#39;, &#39;seaborn&#39;]<br>print(result[0])      # &#39;numpy&#39;</pre><p>Simple, reliable for flat lists. For nested data or typed objects, use Pydantic.</p><h4>Handling Parser Failures</h4><p>Output parsers fail when the model doesn’t follow the format. This happens more than you’d like — especially with complex schemas or less capable models. Here’s how to handle it gracefully.</p><h4>OutputFixingParser — auto-retry on parse failure</h4><p>OutputFixingParser wraps another parser. When parsing fails, it sends the bad output back to the model with instructions to fix it:</p><pre>from langchain.output_parsers import OutputFixingParser<br>from langchain_openai import ChatOpenAI<br><br>base_parser = PydanticOutputParser(pydantic_object=ConceptAnalysis)<br><br># Wraps the base parser — retries automatically on failure<br>fixing_parser = OutputFixingParser.from_llm(<br>    parser=base_parser,<br>    llm=ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>)<br><br>chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Analyze: {concept}\n{format_instructions}&quot;<br>    )<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br>    | fixing_parser  # Auto-fixes bad output<br>)<br><br>result = chain.invoke({<br>    &quot;concept&quot;: &quot;quicksort&quot;,<br>    &quot;format_instructions&quot;: base_parser.get_format_instructions()<br>})</pre><p>If the model returns malformed JSON, OutputFixingParser sends it back with &quot;this JSON is invalid, fix it and return only valid JSON&quot; — usually resolving the issue in one retry.</p><h4>RetryOutputParser — retry with original prompt</h4><p>RetryOutputParser takes a different approach: instead of asking the model to fix bad output, it retries the entire original call:</p><pre>from langchain.output_parsers import RetryOutputParser<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_openai import ChatOpenAI<br><br>base_parser = PydanticOutputParser(pydantic_object=ConceptAnalysis)<br><br>retry_parser = RetryOutputParser.from_llm(<br>    parser=base_parser,<br>    llm=ChatOpenAI(model=&quot;gpt-4o-mini&quot;),<br>    max_retries=2<br>)<br><br>prompt = ChatPromptTemplate.from_template(<br>    &quot;Analyze: {concept}\n{format_instructions}&quot;<br>)<br><br># Use parse_with_prompt for retry context<br>chain_output = ChatOpenAI(model=&quot;gpt-4o-mini&quot;).invoke(<br>    prompt.format_prompt(<br>        concept=&quot;merge sort&quot;,<br>        format_instructions=base_parser.get_format_instructions()<br>    ).to_messages()<br>)<br><br>result = retry_parser.parse_with_prompt(<br>    chain_output.content,<br>    prompt.format_prompt(<br>        concept=&quot;merge sort&quot;,<br>        format_instructions=base_parser.get_format_instructions()<br>    )<br>)</pre><p><strong>When to use which:</strong></p><ul><li>OutputFixingParser — model output is almost right, just needs minor fixes</li><li>RetryOutputParser — model output is completely wrong, start fresh</li></ul><h4>Manual fallback with try/except</h4><p>For simple cases, a try/except with a sensible default is often cleaner than the retry parsers:</p><pre>from langchain_core.exceptions import OutputParserException<br><br>def safe_parse(chain, inputs: dict, default=None):<br>    &quot;&quot;&quot;Parse with graceful fallback on failure.&quot;&quot;&quot;<br>    try:<br>        return chain.invoke(inputs)<br>    except OutputParserException as e:<br>        print(f&quot;Parse failed: {e}&quot;)<br>        return default<br><br>result = safe_parse(<br>    chain,<br>    {&quot;concept&quot;: &quot;heap sort&quot;, &quot;format_instructions&quot;: parser.get_format_instructions()},<br>    default=ConceptAnalysis(<br>        name=&quot;Unknown&quot;,<br>        category=&quot;unknown&quot;,<br>        difficulty=3,<br>        prerequisites=[],<br>        description=&quot;Parse failed&quot;,<br>        example_use_case=&quot;N/A&quot;<br>    )<br>)</pre><h3>Prompting for Reliable Structured Output</h3><p>The best way to handle parser failures is to prevent them. These prompting practices dramatically reduce format errors:</p><p><strong>1. Inject format instructions — always</strong></p><pre># Always include parser.get_format_instructions() in your prompt<br># The model needs explicit instructions, not just an implicit expectation</pre><p><strong>2. Use low temperature for structured output</strong></p><pre># High temperature = creative = more likely to deviate from format<br>model = ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)  # Deterministic</pre><p><strong>3. Say “return ONLY the JSON” explicitly</strong></p><pre>template = (<br>    &quot;Analyze {concept}.\n\n&quot;<br>    &quot;{format_instructions}\n\n&quot;<br>    &quot;Return ONLY the JSON object. No explanation, no markdown fences, no preamble.&quot;<br>)</pre><p><strong>4. Use a stronger model for complex schemas</strong></p><pre># gpt-4o follows complex schemas more reliably than gpt-4o-mini<br># Worth the cost for schemas with 8+ fields or nested objects<br>model = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)</pre><p><strong>5. Keep schemas simple</strong></p><pre># ❌ Complex nested schema — model often gets confused<br>class ComplexOutput(BaseModel):<br>    analysis: NestedAnalysis<br>    metadata: MetadataObject<br>    results: List[ResultItem]<br><br># ✅ Flatten where possible<br>class SimpleOutput(BaseModel):<br>    summary: str<br>    score: int<br>    tags: List[str]<br>    recommendation: str</pre><h3>Choosing the Right Parser</h3><pre>Do you need plain text?<br>    YES → StrOutputParser<br><br>Do you need a flat list?<br>    YES → CommaSeparatedListOutputParser<br><br>Do you need structured data?<br>    Do you need type validation / IDE autocompletion?<br>        YES → PydanticOutputParser<br>        NO  → JsonOutputParser<br><br>    Does parsing fail frequently?<br>        Minor format errors → OutputFixingParser<br>        Complete failures   → RetryOutputParser or try/except fallback</pre><h3>What’s Next</h3><p><strong>Post #17</strong> combines two sections: LangChain Metadata &amp; Debugging with LangSmith — how to trace every chain invocation, inspect intermediate steps, and debug failures without console.log hell.</p><p><em>Next up: </em><strong><em>Post #17 — Metadata, Debugging &amp; LangSmith: Seeing Inside Your Chain</em></strong></p><p><em>Follow the series on Medium and find the complete curriculum and Revision Notes on the </em><a href="https://bitphonix.github.io/Langchain-Mastery/"><em>LangChain Mastery website</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5398ee6ee0fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prompt Templates & Variables: Stop Hardcoding Your Prompts]]></title>
            <link>https://medium.com/@sonitanishk2003/prompt-templates-variables-stop-hardcoding-your-prompts-d895631c1e12?source=rss-df11a530c327------2</link>
            <guid isPermaLink="false">https://medium.com/p/d895631c1e12</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Tanishk Soni]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 08:42:33 GMT</pubDate>
            <atom:updated>2026-03-30T08:42:33.237Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCPaYwjXn0aItmt3jQYSaQ.png" /></figure><blockquote><strong><em>Series:</em></strong><em> LangChain Mastery | </em><strong><em>Post:</em></strong><em> #15 of 34 </em><strong><em>Phase:</em></strong><em> 2 — LangChain Core Concepts | </em><strong><em>Section:</em></strong><em> 2.5 </em><strong><em>Level:</em></strong><em> Intermediate | </em><strong><em>Reading Time:</em></strong><em> ~9 min</em></blockquote><p>Hardcoded prompts are the LLM equivalent of hardcoded credentials. They work until they don’t — and when they don’t, you’re doing find-and-replace across a codebase instead of changing one variable.</p><p>LangChain’s prompt template system turns prompts into composable, reusable, testable components. A prompt template is just a string with named slots — but the tooling around it makes your chains dramatically more maintainable.</p><p>This post covers every template type you’ll actually use: PromptTemplate, ChatPromptTemplate, FewShotPromptTemplate, partial variables, and composition patterns that make your prompts flexible without becoming a mess.</p><h3>PromptTemplate — The Foundation</h3><p>PromptTemplate handles simple string prompts with {variable} placeholders:</p><pre>from langchain_core.prompts import PromptTemplate<br><br># Basic template<br>template = PromptTemplate.from_template(<br>    &quot;Explain {concept} to a {audience} in {style} style.&quot;<br>)<br><br># Invoke fills in the variables<br>result = template.invoke({<br>    &quot;concept&quot;: &quot;recursion&quot;,<br>    &quot;audience&quot;: &quot;10-year-old&quot;,<br>    &quot;style&quot;: &quot;storytelling&quot;<br>})<br>print(result.text)<br># &quot;Explain recursion to a 10-year-old in storytelling style.&quot;</pre><h4>Explicit variable declaration</h4><p>For templates used across a large codebase, declare variables explicitly — it makes the contract clear and catches typos early:</p><pre>template = PromptTemplate(<br>    template=&quot;Explain {concept} to a {audience} in {style} style.&quot;,<br>    input_variables=[&quot;concept&quot;, &quot;audience&quot;, &quot;style&quot;]<br>)<br><br># Trying to invoke with missing variable raises immediately<br>try:<br>    template.invoke({&quot;concept&quot;: &quot;recursion&quot;})<br>except KeyError as e:<br>    print(f&quot;Missing variable: {e}&quot;)  # Missing variable: &#39;audience&#39;</pre><h4>Validating templates at definition time</h4><pre># This raises immediately at definition — catches the typo before runtime<br>try:<br>    bad_template = PromptTemplate(<br>        template=&quot;Explain {consept} clearly.&quot;,  # Typo in variable name<br>        input_variables=[&quot;concept&quot;]              # Declared name doesn&#39;t match<br>    )<br>except ValueError as e:<br>    print(f&quot;Template error: {e}&quot;)</pre><p>Fail fast at definition time, not at runtime when a user is waiting.</p><h3>ChatPromptTemplate — For Chat Models</h3><p>ChatPromptTemplate builds structured message lists for chat models. It&#39;s what you&#39;ll use 90% of the time with modern LLMs.</p><pre>from langchain_core.prompts import ChatPromptTemplate<br><br># From a list of (role, content) tuples<br>prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, &quot;You are a {role}. Respond in {language}.&quot;),<br>    (&quot;human&quot;,  &quot;Question: {question}&quot;),<br>])<br><br>result = prompt.invoke({<br>    &quot;role&quot;: &quot;Python expert&quot;,<br>    &quot;language&quot;: &quot;English&quot;,<br>    &quot;question&quot;: &quot;What is a decorator?&quot;<br>})<br><br>print(result.to_messages())<br># [<br>#   SystemMessage(content=&quot;You are a Python expert. Respond in English.&quot;),<br>#   HumanMessage(content=&quot;Question: What is a decorator?&quot;)<br># ]</pre><h4>Using message objects directly</h4><p>When you need more control — custom metadata, specific message types — pass message objects instead of tuples:</p><pre>from langchain_core.messages import SystemMessage, HumanMessage<br><br>prompt = ChatPromptTemplate.from_messages([<br>    SystemMessage(content=&quot;You are a code reviewer. Be direct and specific.&quot;),<br>    (&quot;human&quot;, &quot;Review this code:\n\n{code}&quot;),<br>])</pre><p>Mix and match tuples and message objects freely — LangChain handles both.</p><h4>Multi-turn templates</h4><p>Pre-load example exchanges directly into the template:</p><pre>prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, &quot;You are a helpful assistant.&quot;),<br>    (&quot;human&quot;,  &quot;What&#39;s 2 + 2?&quot;),<br>    (&quot;ai&quot;,     &quot;4.&quot;),<br>    (&quot;human&quot;,  &quot;What&#39;s {number} + {number}?&quot;),<br>])<br><br>result = prompt.invoke({&quot;number&quot;: 5})<br># Includes the hardcoded example exchange + the dynamic question</pre><p>This is useful for few-shot prompting directly in the template — though FewShotPromptTemplate handles this more elegantly for larger example sets.</p><h3>Partial Variables — Pre-filling Templates</h3><p>Partial variables let you pre-fill some template slots while leaving others dynamic. This is invaluable when some values are known at chain construction time and others only at runtime.</p><pre>from langchain_core.prompts import ChatPromptTemplate<br><br>prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, &quot;You are a {role} working at {company}.&quot;),<br>    (&quot;human&quot;,  &quot;{question}&quot;),<br>])<br><br># Pre-fill role and company — these are fixed for this deployment<br>support_prompt = prompt.partial(<br>    role=&quot;customer support agent&quot;,<br>    company=&quot;Acme Corp&quot;<br>)<br><br># At runtime, only need the dynamic variable<br>result = support_prompt.invoke({<br>    &quot;question&quot;: &quot;How do I reset my password?&quot;<br>})<br># SystemMessage: &quot;You are a customer support agent working at Acme Corp.&quot;<br># HumanMessage:  &quot;How do I reset my password?&quot;</pre><h4>Callable partial variables</h4><p>Partial variables can be functions — called at invocation time, not at definition time. Perfect for dynamic values like timestamps or user context:</p><pre>from datetime import datetime<br><br>prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;,<br>     &quot;You are a helpful assistant. &quot;<br>     &quot;Current date: {date}. &quot;<br>     &quot;User tier: {tier}.&quot;),<br>    (&quot;human&quot;, &quot;{question}&quot;),<br>])<br><br># date is always fresh — computed at invocation time<br>dated_prompt = prompt.partial(date=lambda: datetime.now().strftime(&quot;%Y-%m-%d&quot;))<br><br># At runtime, only provide what&#39;s truly dynamic<br>result = dated_prompt.invoke({<br>    &quot;tier&quot;: &quot;premium&quot;,<br>    &quot;question&quot;: &quot;What are the latest AI models?&quot;<br>})<br># SystemMessage: &quot;...Current date: 2025-03-15. User tier: premium.&quot;</pre><p>Callable partials prevent stale values — the date is computed fresh on every call, not once at startup.</p><h3>FewShotPromptTemplate — Teaching by Example</h3><p>Few-shot prompting — showing the model examples of the desired input/output format — is one of the most effective ways to improve output quality and consistency. FewShotPromptTemplate structures this cleanly.</p><pre>from langchain_core.prompts import FewShotChatMessagePromptTemplate, ChatPromptTemplate<br><br># Define your examples<br>examples = [<br>    {<br>        &quot;input&quot;: &quot;2 + 2&quot;,<br>        &quot;output&quot;: &quot;4&quot;<br>    },<br>    {<br>        &quot;input&quot;: &quot;the sky during day&quot;,<br>        &quot;output&quot;: &quot;blue&quot;<br>    },<br>    {<br>        &quot;input&quot;: &quot;water&#39;s boiling point at sea level&quot;,<br>        &quot;output&quot;: &quot;100°C / 212°F&quot;<br>    },<br>]<br><br># Template for each example<br>example_prompt = ChatPromptTemplate.from_messages([<br>    (&quot;human&quot;, &quot;{input}&quot;),<br>    (&quot;ai&quot;,    &quot;{output}&quot;),<br>])<br><br># Few-shot template wraps the examples<br>few_shot_prompt = FewShotChatMessagePromptTemplate(<br>    examples=examples,<br>    example_prompt=example_prompt,<br>)<br><br># Final prompt: system + examples + actual question<br>final_prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, &quot;Answer questions as concisely as possible.&quot;),<br>    few_shot_prompt,               # Examples expand here<br>    (&quot;human&quot;, &quot;{question}&quot;),<br>])<br><br>from langchain_openai import ChatOpenAI<br>from langchain_core.output_parsers import StrOutputParser<br><br>chain = final_prompt | ChatOpenAI(model=&quot;gpt-4o-mini&quot;) | StrOutputParser()<br>result = chain.invoke({&quot;question&quot;: &quot;Mount Everest&#39;s height&quot;})<br>print(result)  # &quot;8,849 m / 29,032 ft&quot;</pre><p>The model sees the pattern from the examples and applies it to the new question — concise, consistent format with no extra explanation.</p><h4>Dynamic few-shot selection</h4><p>For large example sets, you don’t want to send all of them every time — that’s expensive. Select the most relevant examples dynamically using a SemanticSimilarityExampleSelector:</p><pre>from langchain_core.example_selectors import SemanticSimilarityExampleSelector<br>from langchain_openai import OpenAIEmbeddings<br>from langchain_community.vectorstores import FAISS<br><br>large_example_set = [<br>    {&quot;input&quot;: &quot;capital of France&quot;,       &quot;output&quot;: &quot;Paris&quot;},<br>    {&quot;input&quot;: &quot;capital of Japan&quot;,        &quot;output&quot;: &quot;Tokyo&quot;},<br>    {&quot;input&quot;: &quot;Python creator&quot;,          &quot;output&quot;: &quot;Guido van Rossum&quot;},<br>    {&quot;input&quot;: &quot;LangChain creator&quot;,       &quot;output&quot;: &quot;Harrison Chase&quot;},<br>    {&quot;input&quot;: &quot;speed of light&quot;,          &quot;output&quot;: &quot;299,792,458 m/s&quot;},<br>    # ... many more examples<br>]<br><br># Select the 2 most similar examples to the current question<br>selector = SemanticSimilarityExampleSelector.from_examples(<br>    large_example_set,<br>    OpenAIEmbeddings(),<br>    FAISS,<br>    k=2  # Select top 2 most relevant examples<br>)<br><br>dynamic_few_shot = FewShotChatMessagePromptTemplate(<br>    example_selector=selector,   # Use selector instead of static examples<br>    example_prompt=example_prompt,<br>)<br><br>final_prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, &quot;Answer concisely.&quot;),<br>    dynamic_few_shot,<br>    (&quot;human&quot;, &quot;{question}&quot;),<br>])</pre><p>For a geography question, the selector picks geography examples. For a programming question, it picks programming examples. Your prompt stays lean while the examples stay relevant.</p><h3>Prompt Composition — Building Reusable Pieces</h3><p>For large applications, build prompts from smaller reusable pieces rather than one giant template per use case:</p><pre>from langchain_core.prompts import ChatPromptTemplate<br><br># Reusable building blocks<br>SAFETY_INSTRUCTIONS = (<br>    &quot;Never reveal confidential information. &quot;<br>    &quot;If unsure, say you don&#39;t know rather than guessing.&quot;<br>)<br><br>TONE_INSTRUCTIONS = &quot;Be professional, concise, and friendly.&quot;<br><br>FORMAT_INSTRUCTIONS = (<br>    &quot;Structure responses with: &quot;<br>    &quot;1) Direct answer 2) Brief explanation 3) Example if helpful.&quot;<br>)<br><br>def build_prompt(role: str, domain: str) -&gt; ChatPromptTemplate:<br>    &quot;&quot;&quot;Build a consistent prompt for any role/domain combination.&quot;&quot;&quot;<br>    return ChatPromptTemplate.from_messages([<br>        (&quot;system&quot;,<br>         f&quot;You are a {role} specializing in {domain}.\n&quot;<br>         f&quot;{SAFETY_INSTRUCTIONS}\n&quot;<br>         f&quot;{TONE_INSTRUCTIONS}\n&quot;<br>         f&quot;{FORMAT_INSTRUCTIONS}&quot;),<br>        (&quot;human&quot;, &quot;{question}&quot;),<br>    ])<br><br># Reuse across different chains<br>support_prompt   = build_prompt(&quot;customer support agent&quot;, &quot;billing and accounts&quot;)<br>technical_prompt = build_prompt(&quot;technical advisor&quot;, &quot;API integration&quot;)<br>sales_prompt     = build_prompt(&quot;sales consultant&quot;, &quot;enterprise software&quot;)<br><br># Each chain shares the same structure, different specialization<br>from langchain_openai import ChatOpenAI<br>from langchain_core.output_parsers import StrOutputParser<br><br>model = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>parser = StrOutputParser()<br><br>support_chain   = support_prompt   | model | parser<br>technical_chain = technical_prompt | model | parser<br>sales_chain     = sales_prompt     | model | parser</pre><p>One build_prompt function. Three chains. When you update SAFETY_INSTRUCTIONS, all three chains get the update automatically.</p><h3>Inspecting &amp; Debugging Templates</h3><p>Always inspect what your template actually produces before connecting it to an expensive model call:</p><pre>prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, &quot;You are a {role}.&quot;),<br>    (&quot;human&quot;,  &quot;{question}&quot;),<br>])<br><br># Check input variables<br>print(prompt.input_variables)<br># [&#39;role&#39;, &#39;question&#39;]<br><br># Inspect the formatted output before sending to model<br>formatted = prompt.invoke({&quot;role&quot;: &quot;chef&quot;, &quot;question&quot;: &quot;How do I make pasta?&quot;})<br>print(formatted.to_messages())<br># [SystemMessage(content=&#39;You are a chef.&#39;),<br>#  HumanMessage(content=&#39;How do I make pasta?&#39;)]<br><br># Pretty print for debugging<br>for msg in formatted.to_messages():<br>    print(f&quot;[{msg.__class__.__name__}] {msg.content}&quot;)</pre><p>This is the LCEL advantage: every component is independently testable. Debug your prompt template without touching the model.</p><h3>The Prompt Template Cheat Sheet</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a951bdfac5b59348faef244cca0c692/href">https://medium.com/media/1a951bdfac5b59348faef244cca0c692/href</a></iframe><h3>What’s Next</h3><p><strong>Post #16</strong> covers Output Parsers — how to get structured, typed data out of LLMs reliably. StrOutputParser, JsonOutputParser, PydanticOutputParser, and what to do when the model doesn&#39;t follow your format.</p><p><em>Next up: </em><strong><em>Post #16 — Output Parsers: Getting Structured Data from LLMs</em></strong></p><p><em>Follow the series on Medium and find the complete curriculum and Revision Notes on the </em><a href="https://bitphonix.github.io/Langchain-Mastery/"><em>LangChain Mastery website</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d895631c1e12" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LangChain Memory & State: Building Conversations That Remember]]></title>
            <link>https://medium.com/@sonitanishk2003/langchain-memory-state-building-conversations-that-remember-06fa70f3b22f?source=rss-df11a530c327------2</link>
            <guid isPermaLink="false">https://medium.com/p/06fa70f3b22f</guid>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[chatbot-development]]></category>
            <dc:creator><![CDATA[Tanishk Soni]]></dc:creator>
            <pubDate>Sat, 28 Mar 2026 19:24:51 GMT</pubDate>
            <atom:updated>2026-03-28T19:24:51.583Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JTaDJ9hAxKkgk3QuNbJ2TQ.png" /></figure><blockquote><strong><em>Series:</em></strong><em> LangChain Mastery | </em><strong><em>Post:</em></strong><em> #14 of 46 </em><strong><em>Phase:</em></strong><em> 2 — LangChain Core Concepts | </em><strong><em>Section:</em></strong><em> 2.4 </em><strong><em>Level:</em></strong><em> Intermediate | </em><strong><em>Reading Time:</em></strong><em> ~9 min</em></blockquote><p>We established this in Post #1 and it’s worth repeating: <strong>LLMs are stateless</strong>. Every request is a fresh start. The model has no idea what you said five minutes ago unless you explicitly tell it.</p><p>This means “memory” in LangChain isn’t magic — it’s just a well-managed messages array. LangChain provides components to store conversation history, inject it into prompts cleanly, and manage its size so you don&#39;t silently overflow the context window.</p><p>That’s what this post is about. By the end you’ll have a fully stateful conversational chain with persistent history, automatic trimming, and a clear understanding of what’s happening at every layer.</p><h3>Why LLMs Are Stateless &amp; What Memory Really Means</h3><p>When a user has this conversation:</p><pre>User:      My name is Priya.<br>Assistant: Nice to meet you, Priya!<br>User:      What&#39;s my name?<br>Assistant: Your name is Priya.</pre><p>That last response isn’t the model “remembering” — it’s the model being shown the full transcript on every request. The third API call looks like this under the hood:</p><pre>messages = [<br>    {&quot;role&quot;: &quot;system&quot;,    &quot;content&quot;: &quot;You are a helpful assistant.&quot;},<br>    {&quot;role&quot;: &quot;user&quot;,      &quot;content&quot;: &quot;My name is Priya.&quot;},<br>    {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;Nice to meet you, Priya!&quot;},<br>    {&quot;role&quot;: &quot;user&quot;,      &quot;content&quot;: &quot;What&#39;s my name?&quot;},   # ← Current turn<br>]</pre><p>Memory = managing this list. Everything LangChain’s memory system does is:</p><ol><li>Store new messages as the conversation progresses</li><li>Retrieve previous messages when building the next request</li><li>Trim or summarize when the list gets too long</li></ol><p>Once you internalize this, the LangChain memory API becomes obvious rather than mysterious.</p><h3>MessagesPlaceholder — Injecting History into Prompts</h3><p>MessagesPlaceholder is a special prompt component that reserves a slot in your prompt template for a list of messages. It&#39;s how you wire conversation history into an LCEL chain.</p><pre>from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder<br><br>prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, &quot;You are a helpful assistant. Be concise.&quot;),<br>    MessagesPlaceholder(variable_name=&quot;history&quot;),  # ← History goes here<br>    (&quot;human&quot;, &quot;{input}&quot;),                          # ← Current user message<br>])</pre><p>When invoked, MessagesPlaceholder expands the history list inline between the system message and the current human message:</p><pre>from langchain_core.messages import HumanMessage, AIMessage<br><br># Simulate what the chain sees on turn 3<br>formatted = prompt.invoke({<br>    &quot;history&quot;: [<br>        HumanMessage(content=&quot;My name is Priya.&quot;),<br>        AIMessage(content=&quot;Nice to meet you, Priya!&quot;),<br>    ],<br>    &quot;input&quot;: &quot;What&#39;s my name?&quot;<br>})<br><br>print(formatted.to_messages())<br># [<br>#   SystemMessage(content=&quot;You are a helpful assistant. Be concise.&quot;),<br>#   HumanMessage(content=&quot;My name is Priya.&quot;),<br>#   AIMessage(content=&quot;Nice to meet you, Priya!&quot;),<br>#   HumanMessage(content=&quot;What&#39;s my name?&quot;)<br># ]</pre><p>The history slots in exactly where you placed MessagesPlaceholder. Empty history? The placeholder expands to nothing — no errors, no gaps.</p><pre># First turn — empty history works fine<br>formatted = prompt.invoke({<br>    &quot;history&quot;: [],  # Empty list — MessagesPlaceholder expands to nothing<br>    &quot;input&quot;: &quot;Hello!&quot;<br>})<br># [SystemMessage(...), HumanMessage(content=&quot;Hello!&quot;)]</pre><h3>ChatMessageHistory — Storing Conversation Turns</h3><p>ChatMessageHistory is a simple in-memory store for conversation messages. It tracks the full history for a single conversation session.</p><pre>from langchain_community.chat_message_histories import ChatMessageHistory<br>from langchain_core.messages import HumanMessage, AIMessage<br><br># Create a history store for one session<br>history = ChatMessageHistory()<br><br># Add messages as the conversation progresses<br>history.add_user_message(&quot;My name is Priya.&quot;)<br>history.add_ai_message(&quot;Nice to meet you, Priya!&quot;)<br>history.add_user_message(&quot;I&#39;m learning LangChain.&quot;)<br>history.add_ai_message(&quot;That&#39;s great! LangChain makes building LLM apps much easier.&quot;)<br><br># Inspect what&#39;s stored<br>print(history.messages)<br># [HumanMessage(...), AIMessage(...), HumanMessage(...), AIMessage(...)]<br><br>print(len(history.messages))  # 4<br><br># Clear if needed<br>history.clear()<br>print(history.messages)  # []</pre><p>ChatMessageHistory is intentionally simple — it&#39;s just a list with helper methods. The real power comes from combining it with RunnableWithMessageHistory.</p><h3>RunnableWithMessageHistory — Wiring It All Together</h3><p>RunnableWithMessageHistory is the component that connects a chain to a history store. It:</p><ol><li>Retrieves history for the current session before each invocation</li><li>Passes the history into the chain via MessagesPlaceholder</li><li>Saves the new human + AI messages back to the store after each invocation</li></ol><pre>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.runnables.history import RunnableWithMessageHistory<br>from langchain_community.chat_message_histories import ChatMessageHistory<br><br># Build the base chain (stateless — no memory logic here)<br>chain = (<br>    ChatPromptTemplate.from_messages([<br>        (&quot;system&quot;, &quot;You are a helpful assistant. Be concise.&quot;),<br>        MessagesPlaceholder(variable_name=&quot;history&quot;),<br>        (&quot;human&quot;, &quot;{input}&quot;),<br>    ])<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0.3)<br>    | StrOutputParser()<br>)<br><br># Session store — maps session_id → ChatMessageHistory<br>session_store = {}<br><br>def get_session_history(session_id: str) -&gt; ChatMessageHistory:<br>    &quot;&quot;&quot;Return existing history or create new one for this session.&quot;&quot;&quot;<br>    if session_id not in session_store:<br>        session_store[session_id] = ChatMessageHistory()<br>    return session_store[session_id]<br><br># Wrap the chain with memory management<br>chain_with_history = RunnableWithMessageHistory(<br>    chain,<br>    get_session_history,          # How to retrieve history for a session<br>    input_messages_key=&quot;input&quot;,   # Which key holds the current user message<br>    history_messages_key=&quot;history&quot; # Which key MessagesPlaceholder expects<br>)</pre><p>Now invoke it with a session_id in the config:</p><pre>config = {&quot;configurable&quot;: {&quot;session_id&quot;: &quot;user_priya_001&quot;}}<br><br># Turn 1<br>response = chain_with_history.invoke(<br>    {&quot;input&quot;: &quot;My name is Priya.&quot;},<br>    config=config<br>)<br>print(response)  # &quot;Nice to meet you, Priya!&quot;<br><br># Turn 2 — history is automatically loaded and saved<br>response = chain_with_history.invoke(<br>    {&quot;input&quot;: &quot;I&#39;m learning LangChain.&quot;},<br>    config=config<br>)<br>print(response)  # &quot;That&#39;s great! LangChain makes building LLM apps much easier.&quot;<br><br># Turn 3 — model remembers from turn 1<br>response = chain_with_history.invoke(<br>    {&quot;input&quot;: &quot;What&#39;s my name?&quot;},<br>    config=config<br>)<br>print(response)  # &quot;Your name is Priya.&quot;<br><br># Inspect what&#39;s been stored<br>history = session_store[&quot;user_priya_001&quot;]<br>print(f&quot;Stored {len(history.messages)} messages&quot;)  # 6</pre><p>The session_id is the key that separates conversations. Different users get different session_id values and never see each other&#39;s history:</p><pre># Two completely separate conversations<br>config_priya = {&quot;configurable&quot;: {&quot;session_id&quot;: &quot;user_priya_001&quot;}}<br>config_arjun  = {&quot;configurable&quot;: {&quot;session_id&quot;: &quot;user_arjun_002&quot;}}<br><br>chain_with_history.invoke({&quot;input&quot;: &quot;My name is Priya.&quot;}, config=config_priya)<br>chain_with_history.invoke({&quot;input&quot;: &quot;My name is Arjun.&quot;}, config=config_arjun)<br><br>chain_with_history.invoke({&quot;input&quot;: &quot;What&#39;s my name?&quot;}, config=config_priya)<br># → &quot;Your name is Priya.&quot;<br><br>chain_with_history.invoke({&quot;input&quot;: &quot;What&#39;s my name?&quot;}, config=config_arjun)<br># → &quot;Your name is Arjun.&quot;</pre><h3>Streaming with history</h3><p>RunnableWithMessageHistory supports all six execution modes. Streaming works exactly as expected:</p><pre>config = {&quot;configurable&quot;: {&quot;session_id&quot;: &quot;stream_session&quot;}}<br><br>for chunk in chain_with_history.stream(<br>    {&quot;input&quot;: &quot;Explain LCEL in two sentences.&quot;},<br>    config=config<br>):<br>    print(chunk, end=&quot;&quot;, flush=True)<br>print()</pre><p>History is loaded before the stream starts and saved after it completes.</p><h3>Trimming &amp; Managing History Length</h3><p>Here’s the silent problem with naive memory: every turn adds two messages to history. After 20 turns you’re sending 40+ messages on every request. After 50 turns you might hit the context window limit and get a BadRequestError.</p><p>You need a strategy for keeping history bounded.</p><h3>Strategy 1: Keep last N turns</h3><p>The simplest approach — discard everything older than the last N turns:</p><pre>from langchain_core.messages import BaseMessage, SystemMessage, trim_messages<br>from langchain_openai import ChatOpenAI<br><br>def trim_to_last_n_turns(messages: list, n: int = 10) -&gt; list:<br>    &quot;&quot;&quot;<br>    Keep system messages and the last n turns (n user + n assistant messages).<br>    One turn = 1 HumanMessage + 1 AIMessage = 2 messages.<br>    &quot;&quot;&quot;<br>    system_msgs = [m for m in messages if isinstance(m, SystemMessage)]<br>    conversation = [m for m in messages if not isinstance(m, SystemMessage)]<br><br>    # Keep last n*2 messages (n turns)<br>    trimmed = conversation[-(n * 2):]<br>    return system_msgs + trimmed<br><br><br># Use with RunnableWithMessageHistory via a preprocessing step<br>from langchain_core.runnables import RunnableLambda<br><br>trimmed_chain = (<br>    RunnableLambda(lambda x: {<br>        **x,<br>        &quot;history&quot;: trim_to_last_n_turns(x.get(&quot;history&quot;, []), n=10)<br>    })<br>    | chain<br>)</pre><h3>Strategy 2: LangChain’s built-in trim_messages</h3><p>LangChain provides a trim_messages utility that trims by token count — more precise than turn count:</p><pre>from langchain_core.messages import trim_messages<br>from langchain_openai import ChatOpenAI<br><br>trimmer = trim_messages(<br>    max_tokens=2000,               # Keep history within 2000 tokens<br>    strategy=&quot;last&quot;,               # Keep the most recent messages<br>    token_counter=ChatOpenAI(model=&quot;gpt-4o-mini&quot;),  # Count tokens accurately<br>    include_system=True,           # Always keep system message<br>    allow_partial=False,           # Don&#39;t split messages mid-way<br>    start_on=&quot;human&quot;,              # Always start history on a human message<br>)<br><br># Wire into the chain<br>chain_with_trimming = (<br>    ChatPromptTemplate.from_messages([<br>        (&quot;system&quot;, &quot;You are a helpful assistant.&quot;),<br>        MessagesPlaceholder(variable_name=&quot;history&quot;),<br>        (&quot;human&quot;, &quot;{input}&quot;),<br>    ])<br>    | RunnableLambda(lambda x: {**x, &quot;history&quot;: trimmer.invoke(x[&quot;history&quot;])})<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>    | StrOutputParser()<br>)</pre><h3>Strategy 3: Summarize old messages</h3><p>Instead of discarding old context, summarize it. More expensive but preserves information:</p><pre>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.messages import SystemMessage, HumanMessage, AIMessage<br><br>summarize_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Summarize this conversation history in 3-4 sentences, &quot;<br>        &quot;preserving key facts and context:\n\n{history}&quot;<br>    )<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br>    | StrOutputParser()<br>)<br><br>async def maybe_summarize_history(<br>    history: list,<br>    max_turns: int = 20<br>) -&gt; list:<br>    &quot;&quot;&quot;<br>    If history exceeds max_turns, summarize the oldest half<br>    and replace it with a single SystemMessage summary.<br>    &quot;&quot;&quot;<br>    if len(history) &lt;= max_turns * 2:<br>        return history  # Within limit — no action needed<br><br>    # Split: old half gets summarized, recent half is kept verbatim<br>    midpoint = len(history) // 2<br>    old_messages = history[:midpoint]<br>    recent_messages = history[midpoint:]<br><br>    # Summarize the old half<br>    history_text = &quot;\n&quot;.join(<br>        f&quot;{m.__class__.__name__}: {m.content}&quot;<br>        for m in old_messages<br>    )<br>    summary = await summarize_chain.ainvoke({&quot;history&quot;: history_text})<br><br>    # Replace old messages with a summary SystemMessage<br>    summary_message = SystemMessage(<br>        content=f&quot;[Earlier conversation summary]: {summary}&quot;<br>    )<br>    return [summary_message] + recent_messages</pre><p>Use strategy 1 for simple chatbots, strategy 2 for token-precise production apps, and strategy 3 when losing context would meaningfully hurt the user experience (e.g., long research or coding sessions).</p><h3>In-Memory vs Persistent Memory Stores</h3><p>ChatMessageHistory stores messages in RAM. When your process restarts, all conversations are gone. For a CLI demo that&#39;s fine. For a production web app it isn&#39;t.</p><p>LangChain supports multiple persistent backends via the langchain_community package:</p><h3>Redis (recommended for production)</h3><pre>from langchain_community.chat_message_histories import RedisChatMessageHistory<br><br>def get_session_history(session_id: str) -&gt; RedisChatMessageHistory:<br>    return RedisChatMessageHistory(<br>        session_id=session_id,<br>        url=&quot;redis://localhost:6379&quot;,<br>        ttl=3600  # Expire sessions after 1 hour of inactivity<br>    )<br><br>chain_with_history = RunnableWithMessageHistory(<br>    chain,<br>    get_session_history,<br>    input_messages_key=&quot;input&quot;,<br>    history_messages_key=&quot;history&quot;<br>)</pre><p>Everything else stays the same — RunnableWithMessageHistory doesn&#39;t care what backend you use, as long as get_session_history returns something with .messages, .add_user_message(), and .add_ai_message().</p><h3>SQLite (simple persistent storage)</h3><pre>from langchain_community.chat_message_histories import SQLChatMessageHistory<br><br>def get_session_history(session_id: str) -&gt; SQLChatMessageHistory:<br>    return SQLChatMessageHistory(<br>        session_id=session_id,<br>        connection_string=&quot;sqlite:///chat_history.db&quot;<br>    )</pre><h3>PostgreSQL (for existing Postgres setups)</h3><pre>from langchain_community.chat_message_histories import PostgresChatMessageHistory<br><br>def get_session_history(session_id: str) -&gt; PostgresChatMessageHistory:<br>    return PostgresChatMessageHistory(<br>        session_id=session_id,<br>        connection_string=&quot;postgresql://user:pass@localhost/dbname&quot;<br>    )</pre><h3>Choosing a backend</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2519cf9d18d985467bc5279d7791ff8/href">https://medium.com/media/d2519cf9d18d985467bc5279d7791ff8/href</a></iframe><p>The swap is a one-line change in get_session_history. Everything else — the chain, the memory wiring, the trimming — stays identical regardless of backend.</p><h3>The Full Picture</h3><p>Here’s the complete memory architecture in one place:</p><pre>User sends message<br>        ↓<br>RunnableWithMessageHistory<br>        ↓<br>  get_session_history(session_id)   ← Loads from Redis/SQLite/RAM<br>        ↓<br>  Trim/summarize if needed          ← Keep within context limits<br>        ↓<br>  Build prompt:<br>    [system msg]<br>    [history messages]              ← MessagesPlaceholder expands here<br>    [current user message]<br>        ↓<br>  Chain (prompt | model | parser)<br>        ↓<br>  Save human + AI messages          ← Written back to store<br>        ↓<br>Return response to user</pre><p>The chain itself remains stateless — it just processes a list of messages. RunnableWithMessageHistory handles the stateful bookkeeping around it. This separation keeps the chain testable and the memory management replaceable.</p><h3>What’s Next</h3><p><strong>Post #15</strong> covers Prompt Templates &amp; Variables in depth — ChatPromptTemplate, PromptTemplate, FewShotPromptTemplate, partial variables, and the prompt composition patterns that make your chains flexible and reusable.</p><p><em>Next up: </em><strong><em>Post #15 — Prompt Templates &amp; Variables: Stop Hardcoding Your Prompts</em></strong></p><p><em>Follow the series on Medium and find the complete curriculum and Revision Notes on the </em><a href="https://bitphonix.github.io/Langchain-Mastery/"><em>LangChain Mastery website</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=06fa70f3b22f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LCEL Execution Modes: invoke vs stream vs batch vs async]]></title>
            <link>https://medium.com/@sonitanishk2003/lcel-execution-modes-invoke-vs-stream-vs-batch-vs-async-7771045a69d1?source=rss-df11a530c327------2</link>
            <guid isPermaLink="false">https://medium.com/p/7771045a69d1</guid>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tanishk Soni]]></dc:creator>
            <pubDate>Sat, 28 Mar 2026 19:12:16 GMT</pubDate>
            <atom:updated>2026-03-28T19:12:16.756Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aY8yOSYf0rOr_PIhXge9SQ.png" /></figure><blockquote><strong><em>Series:</em></strong><em> LangChain Mastery | </em><strong><em>Post:</em></strong><em> #13 of 46 </em><strong><em>Phase:</em></strong><em> 2 — LangChain Core Concepts | </em><strong><em>Section:</em></strong><em> 2.3 </em><strong><em>Level:</em></strong><em> Intermediate | </em><strong><em>Reading Time:</em></strong><em> ~9 min</em></blockquote><p>One of LCEL’s most underappreciated features is that every chain — no matter how complex — automatically supports six execution modes out of the box.</p><p>You build the chain once. You get .invoke(), .stream(), .batch(), .ainvoke(), .astream(), and .abatch() for free. No extra code, no subclassing, no configuration.</p><p>But “free” doesn’t mean “use whichever feels right.” Each mode exists for a specific reason, and using the wrong one costs you either performance, UX quality, or both.</p><p>This post covers all six modes, how streaming propagates through complex chains, and a clear decision guide for choosing the right one.</p><h3>The Six Modes at a Glance</h3><p>Every LCEL chain inherits these six methods from the Runnable base class:</p><pre>Synchronous          Asynchronous<br>─────────────────    ─────────────────────<br>.invoke()      →     .ainvoke()<br>.stream()      →     .astream()<br>.batch()       →     .abatch()</pre><p>The sync and async versions are functionally identical — the async ones just use await and are safe to call from an async context without blocking the event loop.</p><p>Let’s go through each pair.</p><h3>invoke / ainvoke — Single Input, Single Output</h3><p>The default mode. Send one input, wait for the complete output, get it back.</p><pre>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br><br>chain = (<br>    ChatPromptTemplate.from_template(&quot;Explain {concept} in one sentence.&quot;)<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>    | StrOutputParser()<br>)<br><br># Synchronous — blocks until complete<br>result = chain.invoke({&quot;concept&quot;: &quot;recursion&quot;})<br>print(result)<br># &#39;Recursion is when a function calls itself...&#39;<br><br># Async — non-blocking, use inside async functions<br>import asyncio<br><br>async def main():<br>    result = await chain.ainvoke({&quot;concept&quot;: &quot;recursion&quot;})<br>    print(result)<br><br>asyncio.run(main())</pre><p>.invoke() returns the final output of the last step in the chain. For a chain ending in StrOutputParser(), that&#39;s a plain string. For a chain ending in a model with no parser, it&#39;s an AIMessage.</p><h3>What invoke actually does internally</h3><p>Understanding this prevents a lot of confusion later. When you call chain.invoke(input):</p><ol><li>Takes input and passes it to step[0].invoke(input)</li><li>Takes the output of step 0 and passes it to step[1].invoke(output_0)</li><li>Continues through all steps</li><li>Returns the output of the final step</li></ol><p>Each step’s output type must be compatible with the next step’s expected input type. If you chain a model (outputs AIMessage) directly into another prompt template (expects a dict), you&#39;ll get a type error. That&#39;s why parsers exist — they convert AIMessage → str, which flows cleanly into the next component.</p><h3>invoke with config</h3><p>Every execution method accepts an optional config parameter for runtime settings:</p><pre>from langchain_core.runnables import RunnableConfig<br><br>result = chain.invoke(<br>    {&quot;concept&quot;: &quot;recursion&quot;},<br>    config=RunnableConfig(<br>        tags=[&quot;production&quot;, &quot;explain-feature&quot;],     # For tracing/filtering<br>        metadata={&quot;user_id&quot;: &quot;u_123&quot;, &quot;session&quot;: &quot;abc&quot;},  # Custom metadata<br>        max_concurrency=5,                           # For batch operations<br>        run_name=&quot;explain-chain&quot;                     # Name in LangSmith traces<br>    )<br>)</pre><p>We’ll use config extensively in Post #17 when we cover LangSmith observability.</p><h3>stream / astream — Token-by-Token Output</h3><p>.stream() returns a generator that yields output chunks as they&#39;re produced — tokens from the model arrive immediately rather than waiting for the full response.</p><pre># Sync streaming — yields chunks as they arrive<br>for chunk in chain.stream({&quot;concept&quot;: &quot;neural networks&quot;}):<br>    print(chunk, end=&quot;&quot;, flush=True)<br>print()  # Newline when done<br><br># Async streaming — same but non-blocking<br>async def stream_async():<br>    async for chunk in chain.astream({&quot;concept&quot;: &quot;neural networks&quot;}):<br>        print(chunk, end=&quot;&quot;, flush=True)<br>    print()<br><br>asyncio.run(stream_async())</pre><h3>Collecting the full response while streaming</h3><p>Sometimes you need to stream to the user AND capture the complete response:</p><pre>full_response = &quot;&quot;<br><br>for chunk in chain.stream({&quot;concept&quot;: &quot;transformers&quot;}):<br>    print(chunk, end=&quot;&quot;, flush=True)<br>    full_response += chunk<br><br>print()<br>print(f&quot;\nFull response ({len(full_response)} chars): captured ✅&quot;)</pre><h3>stream_events — fine-grained event tracking</h3><p>For complex chains, .stream_events() gives you visibility into what&#39;s happening at every step — not just the final output tokens:</p><pre>async def trace_stream():<br>    async for event in chain.astream_events(<br>        {&quot;concept&quot;: &quot;embeddings&quot;},<br>        version=&quot;v2&quot;<br>    ):<br>        kind = event[&quot;event&quot;]<br>        name = event.get(&quot;name&quot;, &quot;&quot;)<br><br>        if kind == &quot;on_chain_start&quot;:<br>            print(f&quot;▶ Starting: {name}&quot;)<br>        elif kind == &quot;on_llm_stream&quot;:<br>            token = event[&quot;data&quot;][&quot;chunk&quot;].content<br>            if token:<br>                print(token, end=&quot;&quot;, flush=True)<br>        elif kind == &quot;on_chain_end&quot;:<br>            print(f&quot;\n■ Done: {name}&quot;)<br><br>asyncio.run(trace_stream())</pre><p>Output:</p><pre>▶ Starting: RunnableSequence<br>▶ Starting: ChatPromptTemplate<br>■ Done: ChatPromptTemplate<br>▶ Starting: ChatOpenAI<br>Neural...networks...are...<br>■ Done: ChatOpenAI<br>▶ Starting: StrOutputParser<br>■ Done: StrOutputParser</pre><p>This is the foundation of how LangSmith traces work. Every step emits events — you can hook into any of them.</p><h3>How Streaming Propagates Through Complex Chains</h3><p>This is where most developers get confused: <strong>does streaming work through the entire chain, or just the model?</strong></p><p>The answer: streaming propagates through every step that supports it. Most LangChain components are streaming-compatible. But some steps break the stream.</p><h3>Steps that propagate streaming ✅</h3><pre># All of these pass tokens through as they arrive:<br>chain = (<br>    ChatPromptTemplate.from_template(&quot;...&quot;)  # Emits once (not token by token)<br>    | ChatOpenAI(model=&quot;gpt-4o&quot;)             # Streams tokens ✅<br>    | StrOutputParser()                       # Passes tokens through ✅<br>    | RunnableLambda(lambda x: x.upper())    # Transforms each chunk ✅<br>)<br><br>for chunk in chain.stream({&quot;topic&quot;: &quot;AI&quot;}):<br>    print(chunk, end=&quot;&quot;, flush=True)<br># Tokens stream through the parser and lambda in real time</pre><h3>Steps that buffer and break streaming ⚠️</h3><pre>from langchain_core.output_parsers import JsonOutputParser<br><br># JsonOutputParser buffers until the full response arrives<br># because you can&#39;t parse partial JSON<br>chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Return a JSON object with &#39;title&#39; and &#39;summary&#39; for: {topic}&quot;<br>    )<br>    | ChatOpenAI(model=&quot;gpt-4o&quot;)<br>    | JsonOutputParser()  # ⚠️ Buffers — no streaming until JSON is complete<br>)<br><br># This waits for the full JSON before yielding anything<br>for chunk in chain.stream({&quot;topic&quot;: &quot;AI&quot;}):<br>    print(chunk)  # Gets complete dict, not tokens</pre><p>JsonOutputParser and PydanticOutputParser buffer the stream because partial JSON is invalid. You get the complete parsed object at the end, not incremental tokens.</p><h3>Streaming through RunnableParallel</h3><p>When streaming through a parallel step, output is buffered until all branches complete:</p><pre>parallel_chain = (<br>    {<br>        &quot;summary&quot;: summary_chain,<br>        &quot;topics&quot;: topics_chain<br>    }<br>    | combine_prompt<br>    | model<br>    | parser<br>)<br><br># The parallel step buffers until BOTH summary and topics complete<br># Then streaming resumes through combine_prompt → model → parser<br>for chunk in parallel_chain.stream({&quot;text&quot;: &quot;...&quot;}):<br>    print(chunk, end=&quot;&quot;, flush=True)</pre><p>The tokens you see streaming are from the final model step after the parallel branches have merged. The intermediate branches run to completion before streaming continues.</p><h3>batch / abatch — Multiple Inputs in Parallel</h3><p>.batch() takes a list of inputs and runs them concurrently, returning a list of outputs in the same order.</p><pre>inputs = [<br>    {&quot;concept&quot;: &quot;recursion&quot;},<br>    {&quot;concept&quot;: &quot;async/await&quot;},<br>    {&quot;concept&quot;: &quot;hash tables&quot;},<br>    {&quot;concept&quot;: &quot;REST APIs&quot;},<br>    {&quot;concept&quot;: &quot;docker containers&quot;},<br>]<br><br># Runs all 5 concurrently, returns results in input order<br>results = chain.batch(inputs)<br><br>for inp, result in zip(inputs, results):<br>    print(f&quot;{inp[&#39;concept&#39;]}: {result[:60]}...&quot;)</pre><h3>Controlling concurrency</h3><p>By default, .batch() runs all inputs simultaneously. For large batches or rate-limited APIs, control the max concurrency:</p><pre># Max 3 concurrent requests at a time<br>results = chain.batch(<br>    inputs,<br>    config=RunnableConfig(max_concurrency=3)<br>)</pre><p>This is equivalent to the asyncio.Semaphore pattern from Post #7 — but built into the method call.</p><h3>Handling partial failures in batch</h3><p>By default, if any input fails, .batch() raises immediately. Use return_exceptions=True to collect all results including errors:</p><pre>results = chain.batch(<br>    inputs,<br>    config=RunnableConfig(max_concurrency=3),<br>    return_exceptions=True  # Failed inputs return Exception objects<br>)<br><br>for i, result in enumerate(results):<br>    if isinstance(result, Exception):<br>        print(f&quot;Input {i} failed: {result}&quot;)<br>    else:<br>        print(f&quot;Input {i}: {result[:60]}...&quot;)</pre><h3>abatch — async batch</h3><p>For async contexts (FastAPI handlers, async pipelines), use .abatch():</p><pre>async def process_batch(inputs: list) -&gt; list:<br>    return await chain.abatch(<br>        inputs,<br>        config=RunnableConfig(max_concurrency=5)<br>    )<br><br>results = asyncio.run(process_batch(inputs))</pre><h3>batch vs RunnableParallel — what’s the difference?</h3><p>This confuses a lot of developers. Here’s the clear distinction:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d675904bbe72e5e69bc01f64ee5dddb/href">https://medium.com/media/6d675904bbe72e5e69bc01f64ee5dddb/href</a></iframe><pre># batch: same chain, 3 different inputs → [result1, result2, result3]<br>chain.batch([input1, input2, input3])<br><br># RunnableParallel: 3 different chains, same input → {&quot;a&quot;: r1, &quot;b&quot;: r2, &quot;c&quot;: r3}<br>{&quot;a&quot;: chain_a, &quot;b&quot;: chain_b, &quot;c&quot;: chain_c}.invoke(same_input)</pre><h3>Choosing the Right Execution Mode</h3><p>Here’s the decision framework:</p><pre>Is there a human watching a screen waiting for output?<br>    YES → Use .stream() or .astream()<br>    NO  →<br>        Are you processing multiple inputs?<br>            YES → Use .batch() or .abatch() with max_concurrency<br>            NO  →<br>                Are you in an async context (FastAPI, async function)?<br>                    YES → Use .ainvoke()<br>                    NO  → Use .invoke()</pre><p>In practice, this maps to:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0c53992ecc448280ee1ff8266375c4e/href">https://medium.com/media/a0c53992ecc448280ee1ff8266375c4e/href</a></iframe><h3>One important rule</h3><p><strong>Never call </strong><strong>.invoke() or </strong><strong>.batch() inside an already-running async event loop.</strong> This blocks the loop and causes deadlocks. Inside FastAPI route handlers, always use .ainvoke(), .astream(), or .abatch().</p><pre># ❌ Wrong — blocks the FastAPI event loop<br>@app.get(&quot;/explain&quot;)<br>async def explain(concept: str):<br>    return chain.invoke({&quot;concept&quot;: concept})  # Blocks!<br><br># ✅ Correct — non-blocking<br>@app.get(&quot;/explain&quot;)<br>async def explain(concept: str):<br>    return await chain.ainvoke({&quot;concept&quot;: concept})</pre><h3>A Complete Example: All Six Modes on One Chain</h3><pre>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br>import asyncio<br><br>chain = (<br>    ChatPromptTemplate.from_template(&quot;Define {term} in one sentence.&quot;)<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br>    | StrOutputParser()<br>)<br><br># 1. invoke<br>result = chain.invoke({&quot;term&quot;: &quot;API&quot;})<br>print(&quot;invoke:&quot;, result)<br><br># 2. stream<br>print(&quot;stream: &quot;, end=&quot;&quot;)<br>for chunk in chain.stream({&quot;term&quot;: &quot;SDK&quot;}):<br>    print(chunk, end=&quot;&quot;, flush=True)<br>print()<br><br># 3. batch<br>results = chain.batch([<br>    {&quot;term&quot;: &quot;API&quot;}, {&quot;term&quot;: &quot;SDK&quot;}, {&quot;term&quot;: &quot;CLI&quot;}<br>], config={&quot;max_concurrency&quot;: 3})<br>print(&quot;batch:&quot;, results)<br><br>async def async_modes():<br>    # 4. ainvoke<br>    result = await chain.ainvoke({&quot;term&quot;: &quot;REST&quot;})<br>    print(&quot;ainvoke:&quot;, result)<br><br>    # 5. astream<br>    print(&quot;astream: &quot;, end=&quot;&quot;)<br>    async for chunk in chain.astream({&quot;term&quot;: &quot;GraphQL&quot;}):<br>        print(chunk, end=&quot;&quot;, flush=True)<br>    print()<br><br>    # 6. abatch<br>    results = await chain.abatch([<br>        {&quot;term&quot;: &quot;HTTP&quot;}, {&quot;term&quot;: &quot;JSON&quot;}, {&quot;term&quot;: &quot;YAML&quot;}<br>    ], config={&quot;max_concurrency&quot;: 3})<br>    print(&quot;abatch:&quot;, results)<br><br>asyncio.run(async_modes())</pre><p>Same chain. Six modes. Zero extra configuration on the chain itself.</p><h3>What’s Next</h3><p><strong>Post #14</strong> covers LangChain Memory &amp; State — how to give chains and chatbots the ability to remember previous turns. We’ll look at MessagesPlaceholder, ChatMessageHistory, and how memory connects to everything we&#39;ve built so far.</p><p><em>Next up: </em><strong><em>Post #14 — LangChain Memory &amp; State: Building Conversations That Remember</em></strong></p><p><em>Follow the series on Medium and find the complete curriculum and Revision Notes on the </em><a href="https://bitphonix.github.io/Langchain-Mastery/"><em>LangChain Mastery website</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7771045a69d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LCEL Advanced Patterns: Parallel, Branch, and Lambda Runnables]]></title>
            <link>https://medium.com/@sonitanishk2003/lcel-advanced-patterns-parallel-branch-and-lambda-runnables-561cc53ebf9b?source=rss-df11a530c327------2</link>
            <guid isPermaLink="false">https://medium.com/p/561cc53ebf9b</guid>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Tanishk Soni]]></dc:creator>
            <pubDate>Tue, 24 Mar 2026 07:45:49 GMT</pubDate>
            <atom:updated>2026-03-24T07:45:49.831Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNSWsljIsIkmZVJDXC3C-g.png" /></figure><blockquote><strong><em>Series:</em></strong><em> LangChain Mastery | </em><strong><em>Post:</em></strong><em> #12 of 46 </em><strong><em>Phase:</em></strong><em> 2 — LangChain Core Concepts | </em><strong><em>Section:</em></strong><em> 2.2 </em><strong><em>Level:</em></strong><em>Advanced | </em><strong><em>Reading Time:</em></strong><em> ~9 min</em></blockquote><p>In Post #11 we covered the foundation of LCEL — the | operator, RunnableSequence, and how prompt + model + parser chain together. That linear flow covers a lot of ground.</p><p>But real pipelines aren’t always linear. Sometimes you need to run two LLM calls simultaneously and merge the results. Sometimes you need to route to different chains based on the input. Sometimes you need to inject a plain Python function into the middle of a chain without wrapping it in a class.</p><p>That’s what this post is for. Four Runnable primitives — RunnableParallel, RunnableBranch, RunnableLambda, and RunnablePassthrough — that unlock everything a linear sequence can&#39;t express.</p><h3>RunnableParallel — Running Steps Simultaneously</h3><p>RunnableParallel runs multiple Runnables on the same input <strong>at the same time</strong> and returns all their outputs as a dictionary.</p><pre>from langchain_core.runnables import RunnableParallel<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_openai import ChatOpenAI<br><br>model = ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0.3)<br>parser = StrOutputParser()<br><br># Two chains that run in parallel on the same input<br>explain_chain = (<br>    ChatPromptTemplate.from_template(&quot;Explain {concept} in simple terms.&quot;)<br>    | model | parser<br>)<br><br>example_chain = (<br>    ChatPromptTemplate.from_template(&quot;Give a real-world example of {concept}.&quot;)<br>    | model | parser<br>)<br><br># RunnableParallel runs both simultaneously<br>parallel = RunnableParallel(<br>    explanation=explain_chain,<br>    example=example_chain<br>)<br><br>result = parallel.invoke({&quot;concept&quot;: &quot;caching&quot;})<br>print(result[&quot;explanation&quot;])  # Simple explanation of caching<br>print(result[&quot;example&quot;])      # Real-world caching example</pre><p>Both chains receive {&quot;concept&quot;: &quot;caching&quot;} as input and run concurrently. The result is a dict with keys matching what you named them — explanation and example.</p><p><strong>Why this matters for performance:</strong> Without RunnableParallel, you&#39;d invoke these sequentially — adding their latencies together. With it, total time equals the slowest single chain, not the sum of all chains.</p><h3>The shorthand syntax</h3><p>RunnableParallel has a cleaner dict shorthand that you&#39;ll see in most LangChain code:</p><pre># These are identical:<br>parallel = RunnableParallel(explanation=explain_chain, example=example_chain)<br>parallel = {&quot;explanation&quot;: explain_chain, &quot;example&quot;: example_chain}</pre><p>When LangChain sees a plain dict in a chain, it automatically wraps it in RunnableParallel. This means you can write:</p><pre>chain = (<br>    {&quot;explanation&quot;: explain_chain, &quot;example&quot;: example_chain}<br>    | some_combining_prompt<br>    | model<br>    | parser<br>)</pre><p>The dict runs both chains in parallel, then feeds their combined output into the next step.</p><h3>Combining parallel outputs</h3><p>A common pattern: run multiple chains in parallel, then combine their outputs in a final step:</p><pre>combine_prompt = ChatPromptTemplate.from_template(<br>    &quot;Given this explanation: {explanation}\n\n&quot;<br>    &quot;And this example: {example}\n\n&quot;<br>    &quot;Write a concise summary that includes both.&quot;<br>)<br><br>full_chain = (<br>    {<br>        &quot;explanation&quot;: explain_chain,<br>        &quot;example&quot;: example_chain<br>    }<br>    | combine_prompt<br>    | model<br>    | parser<br>)<br><br>result = full_chain.invoke({&quot;concept&quot;: &quot;caching&quot;})<br>print(result)<br># A combined summary using both the explanation and example</pre><p>The parallel step produces {&quot;explanation&quot;: &quot;...&quot;, &quot;example&quot;: &quot;...&quot;} — exactly what combine_prompt expects as template variables. The keys align automatically.</p><h3>RunnablePassthrough — Passing Inputs Through Unchanged</h3><p>RunnablePassthrough is the simplest Runnable: it passes its input through to the output unchanged. This sounds useless until you need to carry an original input value alongside computed values through a chain.</p><pre>from langchain_core.runnables import RunnablePassthrough<br><br># Pass input through unchanged<br>passthrough = RunnablePassthrough()<br>result = passthrough.invoke({&quot;concept&quot;: &quot;caching&quot;})<br>print(result)  # {&quot;concept&quot;: &quot;caching&quot;} — unchanged</pre><p>The real power is RunnablePassthrough.assign() — it passes the input through AND adds new keys computed from it:</p><pre>chain = (<br>    RunnablePassthrough.assign(<br>        explanation=explain_chain,   # Adds &quot;explanation&quot; key<br>        example=example_chain        # Adds &quot;example&quot; key<br>    )<br>)<br><br>result = chain.invoke({&quot;concept&quot;: &quot;caching&quot;})<br># {&quot;concept&quot;: &quot;caching&quot;, &quot;explanation&quot;: &quot;...&quot;, &quot;example&quot;: &quot;...&quot;}<br># Original input preserved AND new keys added</pre><p>This is critically different from plain RunnableParallel — it preserves the original input keys alongside the new ones. When downstream steps need both the original input AND computed values, assign() is what you reach for.</p><h3>The pattern you’ll use constantly</h3><pre>chain = (<br>    RunnablePassthrough.assign(explanation=explain_chain)<br>    | RunnablePassthrough.assign(<br>        critique=lambda x: critique_chain.invoke(<br>            {&quot;explanation&quot;: x[&quot;explanation&quot;]}<br>        )<br>    )<br>    | final_prompt<br>    | model<br>    | parser<br>)</pre><p>Each .assign() adds a new key while carrying everything forward. By the time you reach final_prompt, the dict has all the keys you need — original input, explanation, and critique.</p><h3>RunnableLambda — Custom Python Functions in Chains</h3><p>RunnableLambda wraps any Python callable into a Runnable, letting you inject arbitrary logic into a chain without building a custom class.</p><pre>from langchain_core.runnables import RunnableLambda<br><br># Wrap any function as a Runnable<br>def word_count(text: str) -&gt; dict:<br>    return {&quot;text&quot;: text, &quot;word_count&quot;: len(text.split())}<br><br>count_runnable = RunnableLambda(word_count)<br><br>chain = (<br>    ChatPromptTemplate.from_template(&quot;Write a paragraph about {topic}.&quot;)<br>    | model<br>    | parser<br>    | count_runnable  # Custom function injected mid-chain<br>)<br><br>result = chain.invoke({&quot;topic&quot;: &quot;machine learning&quot;})<br>print(result[&quot;text&quot;])        # The generated paragraph<br>print(result[&quot;word_count&quot;])  # How many words it contains</pre><h3>The lambda shorthand</h3><p>For simple one-liners, you can use a plain lambda directly in a chain — LangChain auto-wraps it:</p><pre>chain = (<br>    explain_chain<br>    | (lambda text: text.upper())  # Inline lambda — no RunnableLambda needed<br>)</pre><p>Use RunnableLambda explicitly when your function is more than a one-liner, needs a proper name for debugging, or is reused across multiple chains.</p><h3>Async lambdas</h3><p>RunnableLambda supports async functions natively — important when your custom logic involves I/O like database lookups or external API calls:</p><pre>import asyncio<br><br>async def fetch_related_docs(query: str) -&gt; dict:<br>    # Simulate async database or vector store lookup<br>    await asyncio.sleep(0.1)<br>    return {&quot;query&quot;: query, &quot;docs&quot;: [f&quot;doc about {query}&quot;]}<br><br>chain = (<br>    RunnableLambda(fetch_related_docs)<br>    | ChatPromptTemplate.from_template(<br>        &quot;Using these docs: {docs}\nAnswer: {query}&quot;<br>    )<br>    | model<br>    | parser<br>)<br><br>result = await chain.ainvoke(&quot;what is LCEL?&quot;)</pre><p>The chain automatically uses the async path when you call .ainvoke() or .astream().</p><h3>RunnableBranch — Conditional Routing</h3><p>RunnableBranch routes inputs to different chains based on conditions — the if/elif/else of LCEL.</p><pre>from langchain_core.runnables import RunnableBranch<br><br># Three chains for different complexity levels<br>beginner_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Explain {concept} using a simple analogy a child would understand.&quot;<br>    ) | model | parser<br>)<br><br>intermediate_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Explain {concept} with technical accuracy but avoid deep math.&quot;<br>    ) | model | parser<br>)<br><br>expert_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Explain {concept} with full technical depth, equations welcome.&quot;<br>    ) | model | parser<br>)<br><br># Route based on the &quot;level&quot; key in the input<br>branch = RunnableBranch(<br>    (lambda x: x[&quot;level&quot;] == &quot;beginner&quot;,     beginner_chain),<br>    (lambda x: x[&quot;level&quot;] == &quot;intermediate&quot;, intermediate_chain),<br>    expert_chain  # Default — no condition needed<br>)<br><br># Test each route<br>print(branch.invoke({&quot;concept&quot;: &quot;neural networks&quot;, &quot;level&quot;: &quot;beginner&quot;}))<br>print(branch.invoke({&quot;concept&quot;: &quot;neural networks&quot;, &quot;level&quot;: &quot;intermediate&quot;}))<br>print(branch.invoke({&quot;concept&quot;: &quot;neural networks&quot;, &quot;level&quot;: &quot;expert&quot;}))</pre><p>RunnableBranch takes:</p><ul><li>Any number of (condition, runnable) tuples — evaluated in order, first match wins</li><li>A final default runnable with no condition — used when nothing matches</li></ul><h3>Dynamic routing based on LLM output</h3><p>A more powerful pattern: let the LLM itself decide which branch to take:</p><pre>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.runnables import RunnableBranch, RunnableLambda<br><br># Classifier chain — determines the type of question<br>classify_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Classify this question into exactly one category: &quot;<br>        &quot;TECHNICAL, CONCEPTUAL, or PRACTICAL.\n\n&quot;<br>        &quot;Question: {question}\n\n&quot;<br>        &quot;Reply with only the category name.&quot;<br>    )<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br>    | parser<br>)<br><br>technical_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Answer this technical question with code examples: {question}&quot;<br>    ) | model | parser<br>)<br><br>conceptual_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Explain the concept behind this question clearly: {question}&quot;<br>    ) | model | parser<br>)<br><br>practical_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Give practical, actionable advice for: {question}&quot;<br>    ) | model | parser<br>)<br><br># Full routing chain<br>routing_chain = (<br>    RunnablePassthrough.assign(category=classify_chain)<br>    | RunnableBranch(<br>        (lambda x: &quot;TECHNICAL&quot; in x[&quot;category&quot;],   technical_chain),<br>        (lambda x: &quot;CONCEPTUAL&quot; in x[&quot;category&quot;],  conceptual_chain),<br>        practical_chain  # Default<br>    )<br>)<br><br>result = routing_chain.invoke({<br>    &quot;question&quot;: &quot;How do I implement a binary search tree in Python?&quot;<br>})<br>print(result)  # Routes to technical_chain → answer with code</pre><p>The classifier runs first, adds a &quot;category&quot; key, then RunnableBranch routes to the right chain. The user sends one question — the pipeline handles everything else.</p><h3>Real-World Example: A Content Analysis Pipeline</h3><p>Let’s combine all four primitives into one coherent pipeline. The task: analyze a piece of text and produce a structured report covering sentiment, key topics, and a simplified summary — all in parallel, routed intelligently.</p><pre>from langchain_core.runnables import (<br>    RunnableParallel,<br>    RunnablePassthrough,<br>    RunnableLambda,<br>    RunnableBranch,<br>)<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_openai import ChatOpenAI<br><br>model = ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0.2)<br>parser = StrOutputParser()<br><br># ── Individual analysis chains ──────────────────────────────<br>sentiment_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Analyze the sentiment of this text. &quot;<br>        &quot;Reply with: POSITIVE, NEGATIVE, or NEUTRAL and one sentence why.\n\n{text}&quot;<br>    ) | model | parser<br>)<br><br>topics_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;List the 3 main topics in this text as a comma-separated list.\n\n{text}&quot;<br>    ) | model | parser<br>)<br><br>summary_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Summarize this text in exactly 2 sentences.\n\n{text}&quot;<br>    ) | model | parser<br>)<br><br># ── Word count utility (RunnableLambda) ─────────────────────<br>def get_text_stats(x: dict) -&gt; dict:<br>    text = x[&quot;text&quot;]<br>    return {<br>        **x,<br>        &quot;word_count&quot;: len(text.split()),<br>        &quot;char_count&quot;: len(text)<br>    }<br><br># ── Route long vs short texts to different summarizers ──────<br>short_summary = (<br>    ChatPromptTemplate.from_template(<br>        &quot;In one sentence, what is this about?\n\n{text}&quot;<br>    ) | model | parser<br>)<br><br>long_summary = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Write a 3-sentence executive summary of this text.\n\n{text}&quot;<br>    ) | model | parser<br>)<br><br>adaptive_summary = RunnableBranch(<br>    (lambda x: x[&quot;word_count&quot;] &lt; 100, short_summary),<br>    long_summary  # Default for longer texts<br>)<br><br># ── Full pipeline ────────────────────────────────────────────<br>analysis_pipeline = (<br>    RunnableLambda(get_text_stats)          # Add word/char count<br>    | RunnablePassthrough.assign(            # Run analyses in parallel<br>        sentiment=sentiment_chain,<br>        topics=topics_chain,<br>    )<br>    | RunnablePassthrough.assign(            # Route summary based on length<br>        summary=adaptive_summary<br>    )<br>)<br><br># ── Run it ───────────────────────────────────────────────────<br>text = &quot;&quot;&quot;<br>LangChain&#39;s expression language represents a significant shift in how<br>developers compose LLM applications. By treating every component as a<br>Runnable with a unified interface, it enables parallel execution,<br>conditional routing, and seamless streaming — all with the elegance of<br>the pipe operator. This makes complex pipelines both readable and maintainable.<br>&quot;&quot;&quot;<br><br>result = analysis_pipeline.invoke({&quot;text&quot;: text.strip()})<br><br>print(&quot;─&quot; * 50)<br>print(f&quot;Word count:  {result[&#39;word_count&#39;]}&quot;)<br>print(f&quot;Sentiment:   {result[&#39;sentiment&#39;]}&quot;)<br>print(f&quot;Topics:      {result[&#39;topics&#39;]}&quot;)<br>print(f&quot;Summary:     {result[&#39;summary&#39;]}&quot;)<br>print(&quot;─&quot; * 50)</pre><p>Output:</p><pre>──────────────────────────────────────────────────<br>Word count:  68<br>Sentiment:   POSITIVE — The text enthusiastically describes LCEL&#39;s benefits.<br>Topics:      LangChain, Runnable interface, pipeline composition<br>Summary:     LangChain&#39;s LCEL treats every component as a Runnable, enabling parallel and conditional pipelines via the pipe operator.<br>──────────────────────────────────────────────────</pre><p>Let’s trace what happened:</p><ol><li>RunnableLambda(get_text_stats) — added word_count and char_count without an API call</li><li>RunnablePassthrough.assign(sentiment=..., topics=...) — ran both analysis chains <strong>in parallel</strong>, preserving the original text and word_count keys</li><li>RunnablePassthrough.assign(summary=adaptive_summary) — routed to short_summary because word_count &lt; 100, added the summary key</li><li>Final result has all keys: text, word_count, char_count, sentiment, topics, summary</li></ol><p>Four primitives. One clean pipeline. Every step readable, every component independently testable.</p><h3>The LCEL Primitives Cheat Sheet</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f484a304bebbfe3fb74064f7492a3e20/href">https://medium.com/media/f484a304bebbfe3fb74064f7492a3e20/href</a></iframe><h3>What’s Next</h3><p><strong>Post #13</strong> covers LCEL execution modes in depth — the full .invoke(), .stream(), .batch(), and their async counterparts. We&#39;ll look at when to use each one, how streaming propagates through complex chains, and how .batch() handles concurrency under the hood.</p><p><em>Next up: </em><strong><em>Post #13 — LCEL Execution Modes: invoke vs stream vs batch vs async</em></strong></p><p><em>Follow the series on Medium and find the complete curriculum and Revision Notes on the </em><a href="https://bitphonix.github.io/Langchain-Mastery/"><em>LangChain Mastery website</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=561cc53ebf9b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LCEL & The Pipe Operator: Composing LLM Chains Like a Pro]]></title>
            <link>https://medium.com/@sonitanishk2003/lcel-the-pipe-operator-composing-llm-chains-like-a-pro-dd27aa37d496?source=rss-df11a530c327------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd27aa37d496</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[generative-ai]]></category>
            <dc:creator><![CDATA[Tanishk Soni]]></dc:creator>
            <pubDate>Tue, 24 Mar 2026 07:34:32 GMT</pubDate>
            <atom:updated>2026-03-24T07:34:32.308Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ik5PzUQ3v5vjxX6XVtbQOQ.png" /></figure><blockquote><strong><em>Series:</em></strong><em> LangChain Mastery | </em><strong><em>Post:</em></strong><em> #11 of 46 </em><strong><em>Phase:</em></strong><em> 2 — LangChain Core Concepts | </em><strong><em>Section:</em></strong><em> 2.1 </em><strong><em>Level:</em></strong><em>Intermediate | </em><strong><em>Reading Time:</em></strong><em> ~9 min</em></blockquote><p>You’ve spent two phases writing LLM calls by hand. You understand what happens at the HTTP level, how tokens are counted, how streaming works, how errors are handled. That foundation is what makes this phase valuable — because now we add the abstraction that makes all of it composable.</p><p>Welcome to Phase 2. Welcome to LangChain.</p><p>The first thing to understand about LangChain is that its core idea is deceptively simple: <strong>everything is a Runnable, and Runnables can be chained with the </strong><strong>| operator.</strong></p><p>That’s it. That’s LCEL — the LangChain Expression Language. One interface. One operator. Endless composability.</p><p>Let’s unpack what that actually means.</p><h3>What LCEL Is &amp; Why It Exists</h3><p>Before LCEL (introduced in LangChain v0.1), building a chain looked like this:</p><pre># Old LangChain — verbose, hard to customize<br>from langchain.chains import LLMChain<br>from langchain.prompts import PromptTemplate<br>from langchain.llms import OpenAI<br><br>chain = LLMChain(<br>    llm=OpenAI(),<br>    prompt=PromptTemplate(<br>        input_variables=[&quot;topic&quot;],<br>        template=&quot;Explain {topic} in simple terms.&quot;<br>    )<br>)<br>result = chain.run(&quot;recursion&quot;)</pre><p>It worked, but it was opaque. You couldn’t easily inspect intermediate steps, add custom logic between components, run steps in parallel, or stream output through the chain. Every chain type (LLMChain, SequentialChain, RetrievalQA) was a separate class with its own API.</p><p>LCEL replaced all of that with a single unified interface. Every component — prompts, models, parsers, retrievers, custom functions — implements the same Runnable interface. And any two Runnables can be connected with |.</p><pre># LCEL — composable, transparent, streamable<br>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br><br>chain = (<br>    ChatPromptTemplate.from_template(&quot;Explain {topic} in simple terms.&quot;)<br>    | ChatOpenAI(model=&quot;gpt-4o&quot;)<br>    | StrOutputParser()<br>)<br><br>result = chain.invoke({&quot;topic&quot;: &quot;recursion&quot;})</pre><p>Same result. But now the chain is transparent — you can see every component. It’s composable — you can swap any piece. And it’s streamable — .stream() works on the whole chain, not just the model.</p><h3>The | Pipe Operator Explained</h3><p>If you’ve used Unix pipes, the concept is identical:</p><pre># Unix: output of ls becomes input of grep<br>ls -la | grep &quot;.py&quot;</pre><pre># LCEL: output of prompt becomes input of model, output of model becomes input of parser<br>chain = prompt | model | parser</pre><p>The | operator is Python&#39;s bitwise OR operator, overloaded by LangChain&#39;s Runnable base class. When you write a | b, LangChain creates a RunnableSequence that:</p><ol><li>Calls a.invoke(input)</li><li>Takes the output of a and passes it as input to b.invoke(...)</li><li>Returns the output of b</li></ol><p>The entire chain is itself a Runnable — so you can pipe chains into other chains:</p><pre>chain_a = prompt_a | model<br>chain_b = prompt_b | model | parser<br><br># Chains compose just like individual components<br>full_chain = chain_a | chain_b</pre><p>This composability is LCEL’s killer feature. You build small, focused chains and connect them — rather than trying to fit everything into one monolithic chain class.</p><h3>Building Your First Chain Step by Step</h3><p>Let’s build up a chain from scratch, one component at a time.</p><h3>Step 1: Install and import</h3><pre>pip install langchain langchain-openai</pre><pre>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.messages import HumanMessage, SystemMessage</pre><h3>Step 2: The prompt template</h3><pre>prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, &quot;You are an expert at explaining technical concepts clearly.&quot;),<br>    (&quot;human&quot;, &quot;Explain {concept} in {style} terms. Keep it under 100 words.&quot;)<br>])<br><br># Inspect what the prompt produces before connecting anything<br>formatted = prompt.invoke({&quot;concept&quot;: &quot;recursion&quot;, &quot;style&quot;: &quot;simple&quot;})<br>print(formatted)<br># Output: ChatPromptValue with SystemMessage + HumanMessage<br>print(formatted.to_messages())<br># [SystemMessage(...), HumanMessage(content=&#39;Explain recursion in simple terms...&#39;)]</pre><p>Notice we can call .invoke() on the prompt alone and inspect the output. Every component in LCEL is independently testable — you don&#39;t need the full chain to debug a prompt.</p><h3>Step 3: The model</h3><pre>model = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0.3)<br><br># Call the model directly with the formatted prompt output<br>response = model.invoke(formatted)<br>print(response)<br># AIMessage(content=&#39;Recursion is when a function calls itself...&#39;, ...)<br>print(type(response))<br># &lt;class &#39;langchain_core.messages.ai.AIMessage&#39;&gt;</pre><p>The model returns an AIMessage object — not a plain string. That&#39;s where the output parser comes in.</p><h3>Step 4: The output parser</h3><pre>parser = StrOutputParser()<br><br># Parser turns AIMessage → plain string<br>text = parser.invoke(response)<br>print(text)<br># &#39;Recursion is when a function calls itself...&#39;<br>print(type(text))<br># &lt;class &#39;str&#39;&gt;</pre><h3>Step 5: Chain them with |</h3><pre>chain = prompt | model | parser<br><br># Now invoke the full chain<br>result = chain.invoke({<br>    &quot;concept&quot;: &quot;recursion&quot;,<br>    &quot;style&quot;: &quot;simple&quot;<br>})<br>print(result)<br># &#39;Recursion is when a function calls itself...&#39;</pre><p>The input flows through each component automatically:</p><pre>{&quot;concept&quot;: &quot;recursion&quot;, &quot;style&quot;: &quot;simple&quot;}<br>    → prompt  → ChatPromptValue (formatted messages)<br>    → model   → AIMessage (raw model response)<br>    → parser  → str (clean text)</pre><h3>RunnableSequence Under the Hood</h3><p>When you write prompt | model | parser, LangChain creates a RunnableSequence. Let&#39;s look at what that actually is:</p><pre>from langchain_core.runnables import RunnableSequence<br><br>chain = prompt | model | parser<br>print(type(chain))<br># &lt;class &#39;langchain_core.runnables.base.RunnableSequence&#39;&gt;<br><br># Inspect the steps<br>print(chain.steps)<br># [ChatPromptTemplate, ChatOpenAI, StrOutputParser]<br><br># Each step is independently accessible<br>print(chain.first)   # ChatPromptTemplate<br>print(chain.last)    # StrOutputParser</pre><p>RunnableSequence is itself a Runnable — it implements the same interface as every individual component. This is why chains compose: a RunnableSequence can be a step inside another RunnableSequence.</p><p>The Runnable interface defines four execution methods, all available on any chain:</p><pre># Single input → single output<br>result = chain.invoke({&quot;concept&quot;: &quot;APIs&quot;, &quot;style&quot;: &quot;beginner&quot;})<br><br># Single input → token stream<br>for token in chain.stream({&quot;concept&quot;: &quot;APIs&quot;, &quot;style&quot;: &quot;beginner&quot;}):<br>    print(token, end=&quot;&quot;, flush=True)<br><br># List of inputs → list of outputs (parallel)<br>results = chain.batch([<br>    {&quot;concept&quot;: &quot;APIs&quot;, &quot;style&quot;: &quot;beginner&quot;},<br>    {&quot;concept&quot;: &quot;recursion&quot;, &quot;style&quot;: &quot;simple&quot;},<br>    {&quot;concept&quot;: &quot;async&quot;, &quot;style&quot;: &quot;technical&quot;},<br>])<br><br># Async versions of all three<br>result = await chain.ainvoke({&quot;concept&quot;: &quot;APIs&quot;, &quot;style&quot;: &quot;beginner&quot;})<br>async for token in chain.astream({&quot;concept&quot;: &quot;APIs&quot;, &quot;style&quot;: &quot;beginner&quot;}):<br>    print(token, end=&quot;&quot;, flush=True)<br>results = await chain.abatch([...])</pre><p>Every LCEL chain gets .invoke(), .stream(), .batch(), .ainvoke(), .astream(), and .abatch() for free — without any extra work. This is what &quot;unified interface&quot; means in practice.</p><h3>Chaining Prompts, Models &amp; Parsers — Real Examples</h3><h3>Example 1: A translation chain</h3><pre>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br><br>translate_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Translate the following text to {language}:\n\n{text}&quot;<br>    )<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br>    | StrOutputParser()<br>)<br><br>result = translate_chain.invoke({<br>    &quot;language&quot;: &quot;French&quot;,<br>    &quot;text&quot;: &quot;LangChain makes it easy to build LLM applications.&quot;<br>})<br>print(result)<br># &#39;LangChain facilite la création d&#39;applications LLM.&#39;</pre><h3>Example 2: A streaming chain</h3><p>Streaming works on the full chain — not just the model:</p><pre>for chunk in translate_chain.stream({<br>    &quot;language&quot;: &quot;Spanish&quot;,<br>    &quot;text&quot;: &quot;Streaming makes LLM apps feel instant.&quot;<br>}):<br>    print(chunk, end=&quot;&quot;, flush=True)<br># &#39;El streaming hace que las aplicaciones LLM se sientan instantáneas.&#39;</pre><p>The stream passes through the parser — you get plain string chunks, not AIMessage chunks. The parser handles the conversion transparently.</p><h3>Example 3: A two-step chain (chain of chains)</h3><pre># Step 1: Generate a concept explanation<br>explain_chain = (<br>    ChatPromptTemplate.from_template(&quot;Explain {concept} in one paragraph.&quot;)<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>    | StrOutputParser()<br>)<br><br># Step 2: Simplify whatever text comes in<br>simplify_prompt = ChatPromptTemplate.from_template(<br>    &quot;Simplify this explanation for a 10-year-old:\n\n{explanation}&quot;<br>)<br>simplify_chain = (<br>    simplify_prompt<br>    | ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>    | StrOutputParser()<br>)<br><br># Chain them — output of explain becomes input of simplify<br># We need to map the output to the expected input key<br>from langchain_core.runnables import RunnablePassthrough<br><br>full_chain = (<br>    explain_chain<br>    | (lambda text: {&quot;explanation&quot;: text})  # Map str → dict<br>    | simplify_chain<br>)<br><br>result = full_chain.invoke({&quot;concept&quot;: &quot;quantum entanglement&quot;})<br>print(result)</pre><p>The lambda (lambda text: {&quot;explanation&quot;: text}) bridges the output of the first chain (a plain string) to the expected input of the second chain (a dict with key &quot;explanation&quot;). In the next post, we&#39;ll see cleaner ways to do this with RunnablePassthrough and RunnableLambda.</p><h3>LCEL vs Manual Chaining: The Real Comparison</h3><p>Let’s do the honest comparison. Here’s the same task — explain a concept, then evaluate the explanation’s clarity — done both ways.</p><p><strong>Manual chaining (what we’d write in Phase 1):</strong></p><pre>from openai import OpenAI<br><br>client = OpenAI()<br><br>def explain_and_evaluate(concept: str) -&gt; dict:<br>    # Step 1: Generate explanation<br>    explain_response = client.chat.completions.create(<br>        model=&quot;gpt-4o-mini&quot;,<br>        messages=[<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Explain {concept} in one paragraph.&quot;}<br>        ],<br>        max_tokens=200<br>    )<br>    explanation = explain_response.choices[0].message.content<br><br>    # Step 2: Evaluate the explanation<br>    evaluate_response = client.chat.completions.create(<br>        model=&quot;gpt-4o-mini&quot;,<br>        messages=[<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: (<br>                f&quot;Rate this explanation&#39;s clarity from 1-10 &quot;<br>                f&quot;and explain why:\n\n{explanation}&quot;<br>            )}<br>        ],<br>        max_tokens=150<br>    )<br>    evaluation = evaluate_response.choices[0].message.content<br><br>    return {&quot;explanation&quot;: explanation, &quot;evaluation&quot;: evaluation}<br><br>result = explain_and_evaluate(&quot;neural networks&quot;)</pre><p><strong>LCEL chaining:</strong></p><pre>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.runnables import RunnablePassthrough<br><br>model = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>parser = StrOutputParser()<br><br>explain_chain = (<br>    ChatPromptTemplate.from_template(&quot;Explain {concept} in one paragraph.&quot;)<br>    | model<br>    | parser<br>)<br><br>evaluate_chain = (<br>    ChatPromptTemplate.from_template(<br>        &quot;Rate this explanation&#39;s clarity from 1-10 and explain why:\n\n{explanation}&quot;<br>    )<br>    | model<br>    | parser<br>)<br><br>full_chain = (<br>    RunnablePassthrough.assign(explanation=explain_chain)<br>    | RunnablePassthrough.assign(<br>        evaluation=lambda x: evaluate_chain.invoke({&quot;explanation&quot;: x[&quot;explanation&quot;]})<br>    )<br>)<br><br>result = full_chain.invoke({&quot;concept&quot;: &quot;neural networks&quot;})<br>print(result[&quot;explanation&quot;])<br>print(result[&quot;evaluation&quot;])</pre><p>Honestly? For this specific case the manual version isn’t dramatically worse. Where LCEL wins decisively:</p><p><strong>Streaming.</strong> Add .stream() to the LCEL chain and you get streaming through every step. To stream the manual version, you&#39;d rewrite both API calls.</p><p><strong>Batch processing.</strong> full_chain.batch([...]) runs multiple inputs in parallel automatically. Manual version needs asyncio.</p><p><strong>Observability.</strong> LangSmith (LangChain’s tracing tool) hooks into every LCEL chain automatically — you get full traces of every step, input, and output with zero extra code. We’ll cover this in Post #17.</p><p><strong>Composability at scale.</strong> When chains have 6–8 steps, branch on conditions, run steps in parallel, and feed into agents — LCEL’s composability becomes essential, not just nice-to-have. The manual version becomes unmaintainable.</p><h3>The Mental Model to Carry Forward</h3><p>Before the next post, lock in this mental model:</p><pre>Everything in LangChain is a Runnable.<br>Every Runnable has: .invoke() .stream() .batch() .ainvoke() .astream() .abatch()<br>Any two Runnables connect with |<br>The result of | is a RunnableSequence, which is itself a Runnable.</pre><p>That’s the entire foundation of LCEL. Everything in Phase 2 builds on these four facts.</p><h3>What’s Next</h3><p><strong>Post #12</strong> goes deeper into LCEL — RunnableParallel for running steps simultaneously, RunnableBranch for conditional routing, and RunnableLambda for injecting custom Python functions anywhere in a chain. These are the patterns that make complex pipelines possible.</p><p><em>Next up: </em><strong><em>Post #12 — LCEL Advanced Patterns: Parallel, Branch, and Lambda Runnables</em></strong></p><p><em>Follow the series on Medium and find the complete curriculum and Revision Notes on the </em><a href="https://bitphonix.github.io/Langchain-Mastery/"><em>LangChain Mastery website</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd27aa37d496" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Production SDK Chat App: The Phase 1 Capstone]]></title>
            <link>https://medium.com/@sonitanishk2003/production-sdk-chat-app-the-phase-1-capstone-18b7130e4aa5?source=rss-df11a530c327------2</link>
            <guid isPermaLink="false">https://medium.com/p/18b7130e4aa5</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Tanishk Soni]]></dc:creator>
            <pubDate>Tue, 10 Mar 2026 05:13:17 GMT</pubDate>
            <atom:updated>2026-03-10T05:13:17.424Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MYW-zYAtpZTFvp1r-Vkb-w.png" /></figure><blockquote><strong><em>Series:</em></strong><em> LangChain Mastery | </em><strong><em>Post:</em></strong><em> #10 of 46 </em><strong><em>Phase:</em></strong><em> 1 — Official SDK Usage | </em><strong><em>Section:</em></strong><em> 1.5 (Capstone)</em><strong><em>Level:</em></strong><em> Advanced | </em><strong><em>Reading Time:</em></strong><em> ~15 min</em></blockquote><p>Ten posts in. Two complete phases behind us.</p><p>In Phase 0, we built a CLI chat app using nothing but raw HTTP — every header, every JSON parse, every SSE line by hand. It worked, and building it taught us exactly what happens at the wire level.</p><p>In Phase 1, we learned what the SDK gives us over that raw foundation: typed responses, clean streaming iterators, a structured exception hierarchy, async concurrency, and resilient retry logic.</p><p>Now we put it all together.</p><p>This post rebuilds the Phase 0 CLI chat app using the full Phase 1 SDK stack. Same features, same structure — but cleaner, more capable, and production-ready in ways the Phase 0 version couldn’t be. By the end, you’ll have a working app and a clear sense of what the SDK upgrade actually buys you.</p><h3>What’s Changing — The Phase 0 → Phase 1 Diff</h3><p>Before writing a line of code, let’s be clear about what’s actually changing and why.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b3115d70b07832656d3a274c51c66076/href">https://medium.com/media/b3115d70b07832656d3a274c51c66076/href</a></iframe><p>The app structure stays identical — config.py, costs.py, client.py, chat.py, main.py. What changes is what&#39;s inside each file.</p><h3>Project Structure</h3><pre>sdk_chat/<br>├── config.py      # Settings and constants (unchanged from Phase 0)<br>├── costs.py       # Cost tracking with typed SDK usage objects<br>├── client.py      # ResilientClient wrapping AsyncOpenAI<br>├── chat.py        # Async conversation loop and history management<br>└── main.py        # Entry point</pre><p>Same five files. Let’s walk through each one.</p><h3>config.py — Mostly Unchanged, One Addition</h3><pre># config.py<br>import os<br>from dataclasses import dataclass, field<br><br>@dataclass<br>class Config:<br>    # API settings<br>    api_key: str<br><br>    # Model settings — now a priority chain, not just primary + fallback<br>    model_chain: tuple = (&quot;gpt-4o&quot;, &quot;gpt-4o-mini&quot;)<br>    temperature: float = 0.7<br>    max_tokens: int = 1000<br><br>    # Retry settings (used by ResilientClient)<br>    max_retries: int = 3<br>    base_delay: float = 1.0<br><br>    # Concurrency<br>    max_concurrent: int = 1  # Chat is sequential; set higher for batch use<br><br>    # History<br>    max_history_turns: int = 20<br><br>    # System prompt<br>    system_prompt: str = (<br>        &quot;You are a helpful, concise assistant. &quot;<br>        &quot;Respond clearly and directly. &quot;<br>        &quot;If you don&#39;t know something, say so.&quot;<br>    )<br><br>    # Pricing per 1M tokens<br>    pricing: dict = field(default_factory=lambda: {<br>        &quot;gpt-4o&quot;:      {&quot;input&quot;: 2.50,  &quot;output&quot;: 10.00},<br>        &quot;gpt-4o-mini&quot;: {&quot;input&quot;: 0.15,  &quot;output&quot;: 0.60},<br>    })<br><br><br>def load_config() -&gt; Config:<br>    api_key = os.environ.get(&quot;OPENAI_API_KEY&quot;)<br>    if not api_key:<br>        raise EnvironmentError(<br>            &quot;OPENAI_API_KEY environment variable not set.\n&quot;<br>            &quot;Run: export OPENAI_API_KEY=&#39;sk-your-key&#39;&quot;<br>        )<br>    return Config(api_key=api_key)</pre><p>The key change: primary_model + fallback_models from Phase 0 are now a single model_chain tuple. The client works through the chain automatically — no separate fallback logic needed at the call site.</p><h3>costs.py — Upgraded to Use Typed SDK Objects</h3><p>Phase 0’s costs.py worked with raw integers from a JSON dict. Phase 1 works with the SDK&#39;s typed CompletionUsage object directly — cleaner and less error-prone:</p><pre># costs.py<br>from dataclasses import dataclass, field<br>from typing import Optional<br>from openai.types.completion_usage import CompletionUsage<br><br><br>@dataclass<br>class MessageCost:<br>    model: str<br>    prompt_tokens: int<br>    completion_tokens: int<br>    total_tokens: int<br>    cost_usd: float<br>    estimated: bool = False  # True if tiktoken estimate, False if from API<br><br>    def display(self):<br>        source = &quot;~&quot; if self.estimated else &quot;&quot;  # ~ prefix for estimates<br>        print(<br>            f&quot;\n  📊 [{self.model}] &quot;<br>            f&quot;in: {source}{self.prompt_tokens} | &quot;<br>            f&quot;out: {source}{self.completion_tokens} | &quot;<br>            f&quot;cost: ${self.cost_usd:.5f}&quot;<br>        )<br><br><br>@dataclass<br>class SessionCost:<br>    costs: list = field(default_factory=list)<br><br>    def add(self, cost: MessageCost):<br>        self.costs.append(cost)<br><br>    @property<br>    def total_usd(self) -&gt; float:<br>        return sum(c.cost_usd for c in self.costs)<br><br>    @property<br>    def total_tokens(self) -&gt; int:<br>        return sum(c.total_tokens for c in self.costs)<br><br>    @property<br>    def message_count(self) -&gt; int:<br>        return len(self.costs)<br><br>    def display_summary(self):<br>        if not self.costs:<br>            return<br>        print(&quot;\n&quot; + &quot;─&quot; * 50)<br>        print(&quot;  💰 Session Summary&quot;)<br>        print(&quot;─&quot; * 50)<br>        print(f&quot;  Messages:     {self.message_count}&quot;)<br>        print(f&quot;  Total tokens: {self.total_tokens:,}&quot;)<br>        print(f&quot;  Total cost:   ${self.total_usd:.5f}&quot;)<br>        print(f&quot;  Avg per msg:  ${self.total_usd / self.message_count:.5f}&quot;)<br>        print(&quot;─&quot; * 50)<br><br><br>def from_sdk_usage(<br>    model: str,<br>    usage: CompletionUsage,  # Typed SDK object, not a raw dict<br>    pricing: dict<br>) -&gt; MessageCost:<br>    &quot;&quot;&quot;Build a MessageCost from the SDK&#39;s typed CompletionUsage object.&quot;&quot;&quot;<br>    price = pricing.get(model, {&quot;input&quot;: 0.0, &quot;output&quot;: 0.0})<br>    cost = (<br>        (usage.prompt_tokens / 1_000_000) * price[&quot;input&quot;] +<br>        (usage.completion_tokens / 1_000_000) * price[&quot;output&quot;]<br>    )<br>    return MessageCost(<br>        model=model,<br>        prompt_tokens=usage.prompt_tokens,<br>        completion_tokens=usage.completion_tokens,<br>        total_tokens=usage.total_tokens,<br>        cost_usd=cost,<br>        estimated=False<br>    )<br><br><br>def from_estimate(<br>    model: str,<br>    messages: list,<br>    response_text: str,<br>    pricing: dict<br>) -&gt; MessageCost:<br>    &quot;&quot;&quot;Fallback: estimate cost via tiktoken when usage data isn&#39;t available.&quot;&quot;&quot;<br>    import tiktoken<br>    try:<br>        enc = tiktoken.encoding_for_model(model)<br>    except KeyError:<br>        enc = tiktoken.get_encoding(&quot;cl100k_base&quot;)<br><br>    prompt_tokens = sum(<br>        len(enc.encode(m[&quot;content&quot;])) + 4 for m in messages<br>    ) + 2<br>    completion_tokens = len(enc.encode(response_text))<br>    total = prompt_tokens + completion_tokens<br><br>    price = pricing.get(model, {&quot;input&quot;: 0.0, &quot;output&quot;: 0.0})<br>    cost = (<br>        (prompt_tokens / 1_000_000) * price[&quot;input&quot;] +<br>        (completion_tokens / 1_000_000) * price[&quot;output&quot;]<br>    )<br>    return MessageCost(<br>        model=model,<br>        prompt_tokens=prompt_tokens,<br>        completion_tokens=completion_tokens,<br>        total_tokens=total,<br>        cost_usd=cost,<br>        estimated=True<br>    )<br></pre><p>The estimated flag on MessageCost is a small but important addition — when cost is shown with a ~ prefix, users know it&#39;s an approximation rather than the exact count from the API.</p><h3>client.py — The Heart of Phase 1</h3><p>This is the biggest change from Phase 0. What was api.py (raw HTTP calls, manual SSE parsing, custom retry logic) is now a clean ResilientClient built on AsyncOpenAI:</p><pre># client.py<br>import asyncio<br>import time<br>from typing import Optional, AsyncIterator<br>import httpx<br>from openai import (<br>    AsyncOpenAI,<br>    RateLimitError,<br>    InternalServerError,<br>    APIConnectionError,<br>    APITimeoutError,<br>    AuthenticationError,<br>    BadRequestError,<br>    APIStatusError,<br>)<br>from openai.types.chat import ChatCompletionChunk<br>from config import Config<br>from costs import MessageCost, from_sdk_usage, from_estimate<br><br>RETRYABLE = (RateLimitError, InternalServerError, APIConnectionError, APITimeoutError)<br><br><br>class ResilientClient:<br>    &quot;&quot;&quot;<br>    AsyncOpenAI wrapper with:<br>    - Automatic retry with backoff on transient errors<br>    - Multi-model fallback chain<br>    - Streaming with real-time token output<br>    - Exact usage tracking via stream_options<br>    - Structured exception handling<br>    &quot;&quot;&quot;<br><br>    def __init__(self, config: Config):<br>        self.config = config<br>        self.client = AsyncOpenAI(<br>            api_key=config.api_key,<br>            max_retries=0,  # We handle retries ourselves for full control<br>            timeout=httpx.Timeout(connect=10.0, read=60.0, write=10.0, pool=5.0)<br>        )<br><br>    async def _stream_single_model(<br>        self,<br>        messages: list,<br>        model: str,<br>    ) -&gt; tuple[str, Optional[MessageCost]]:<br>        &quot;&quot;&quot;<br>        Stream a completion from a single model.<br>        Returns (response_text, cost) on success.<br>        Raises on any error — caller handles retry/fallback.<br>        &quot;&quot;&quot;<br>        full_response = &quot;&quot;<br>        usage_data = None<br>        ttft = None<br>        start = time.perf_counter()<br><br>        async with self.client.chat.completions.create(<br>            model=model,<br>            messages=messages,<br>            max_tokens=self.config.max_tokens,<br>            temperature=self.config.temperature,<br>            stream=True,<br>            stream_options={&quot;include_usage&quot;: True}<br>        ) as stream:<br>            print(f&quot;\n  🤖 Assistant: &quot;, end=&quot;&quot;, flush=True)<br><br>            async for chunk in stream:<br>                # Capture usage from final chunk<br>                if chunk.usage:<br>                    usage_data = chunk.usage<br><br>                if not chunk.choices:<br>                    continue<br><br>                token = chunk.choices[0].delta.content<br>                finish = chunk.choices[0].finish_reason<br><br>                if token:<br>                    if ttft is None:<br>                        ttft = time.perf_counter() - start<br>                    print(token, end=&quot;&quot;, flush=True)<br>                    full_response += token<br><br>                if finish == &quot;length&quot;:<br>                    print(&quot;\n  ⚠️  [Response truncated — increase max_tokens]&quot;,<br>                          end=&quot;&quot;, flush=True)<br><br>        print()  # Newline after streamed response<br><br>        # Build cost object<br>        if usage_data:<br>            cost = from_sdk_usage(model, usage_data, self.config.pricing)<br>        else:<br>            cost = from_estimate(<br>                model, messages, full_response, self.config.pricing<br>            )<br><br>        return full_response, cost<br><br>    async def _with_retry(self, model: str, fn) -&gt; tuple[str, Optional[MessageCost]]:<br>        &quot;&quot;&quot;Execute fn with retry logic for a single model.&quot;&quot;&quot;<br>        last_error = None<br><br>        for attempt in range(self.config.max_retries + 1):<br>            try:<br>                return await fn()<br><br>            except AuthenticationError:<br>                raise SystemExit(<br>                    &quot;❌ Authentication failed. Check your OPENAI_API_KEY.&quot;<br>                )<br><br>            except BadRequestError as e:<br>                raise ValueError(f&quot;Bad request: {e.message}&quot;) from e<br><br>            except RETRYABLE as e:<br>                last_error = e<br>                if attempt &gt;= self.config.max_retries:<br>                    break<br><br>                # Respect Retry-After if present<br>                retry_after = None<br>                if hasattr(e, &#39;response&#39;) and e.response:<br>                    retry_after = e.response.headers.get(&quot;Retry-After&quot;)<br>                wait = float(retry_after) if retry_after else (<br>                    self.config.base_delay * (2 ** attempt)<br>                )<br>                print(f&quot;\n  ⚠️  {type(e).__name__} — retrying in {wait:.1f}s &quot;<br>                      f&quot;(attempt {attempt + 1}/{self.config.max_retries})...&quot;)<br>                await asyncio.sleep(wait)<br><br>            except APIStatusError as e:<br>                raise RuntimeError(<br>                    f&quot;API error {e.status_code}: {e.message}&quot;<br>                ) from e<br><br>        raise RuntimeError(<br>            f&quot;Model {model} failed after {self.config.max_retries + 1} attempts. &quot;<br>            f&quot;Last error: {last_error}&quot;<br>        )<br><br>    async def stream_with_fallback(<br>        self,<br>        messages: list,<br>    ) -&gt; tuple[str, MessageCost]:<br>        &quot;&quot;&quot;<br>        Stream a completion, falling back through model_chain on failure.<br>        Always returns (response_text, cost) or raises RuntimeError.<br>        &quot;&quot;&quot;<br>        last_error = None<br><br>        for model in self.config.model_chain:<br>            try:<br>                result = await self._with_retry(<br>                    model=model,<br>                    fn=lambda m=model: self._stream_single_model(messages, m)<br>                )<br>                # Log fallback usage if not primary model<br>                if model != self.config.model_chain[0]:<br>                    print(f&quot;  ℹ️  [Used fallback model: {model}]&quot;)<br>                return result<br><br>            except ValueError:<br>                raise  # Bad request — don&#39;t try other models<br><br>            except SystemExit:<br>                raise  # Auth failure — no point trying other models<br><br>            except RuntimeError as e:<br>                last_error = e<br>                if model != self.config.model_chain[-1]:<br>                    next_model = self.config.model_chain[<br>                        self.config.model_chain.index(model) + 1<br>                    ]<br>                    print(f&quot;\n  ↩️  Falling back to {next_model}...&quot;)<br>                continue<br><br>        raise RuntimeError(<br>            f&quot;All models in chain failed. Last: {last_error}&quot;<br>        )</pre><p>Compare this to Phase 0’s api.py. The Phase 0 version had:</p><ul><li>Manual requests.post() with headers</li><li>15 lines of SSE parsing</li><li>A separate errors.py for retry logic</li><li>Manual fallback chain implementation</li></ul><p>The Phase 1 version has:</p><ul><li>AsyncOpenAI client (auth handled automatically)</li><li>async for chunk in stream (SSE parsing gone)</li><li>Retry logic in _with_retry() using typed exceptions</li><li>Fallback chain built into stream_with_fallback()</li></ul><p>Same behavior. Half the code. Fully async.</p><h3>chat.py — Async Conversation Loop</h3><p>The conversation loop upgrades from sync to async, and gains a cleaner command handler:</p><pre># chat.py<br>import asyncio<br>from config import Config<br>from costs import SessionCost<br>from client import ResilientClient<br><br>COMMANDS = {<br>    &quot;/quit&quot;:  &quot;Exit the chat&quot;,<br>    &quot;/exit&quot;:  &quot;Exit the chat&quot;,<br>    &quot;/cost&quot;:  &quot;Show session cost summary&quot;,<br>    &quot;/clear&quot;: &quot;Clear conversation history&quot;,<br>    &quot;/model&quot;: &quot;Show current model chain&quot;,<br>    &quot;/help&quot;:  &quot;Show this help message&quot;,<br>}<br><br><br>def trim_history(<br>    messages: list,<br>    system_message: dict,<br>    max_turns: int<br>) -&gt; list:<br>    conversation = [m for m in messages if m[&quot;role&quot;] != &quot;system&quot;]<br>    trimmed = conversation[-(max_turns * 2):]<br>    return [system_message] + trimmed<br><br><br>async def handle_command(<br>    command: str,<br>    config: Config,<br>    session_cost: SessionCost,<br>    messages: list,<br>    system_message: dict<br>) -&gt; bool:<br>    &quot;&quot;&quot;Handle a slash command. Returns True if app should exit.&quot;&quot;&quot;<br>    cmd = command.strip().lower()<br><br>    if cmd in (&quot;/quit&quot;, &quot;/exit&quot;):<br>        return True<br><br>    elif cmd == &quot;/cost&quot;:<br>        session_cost.display_summary()<br><br>    elif cmd == &quot;/clear&quot;:<br>        messages.clear()<br>        messages.append(system_message)<br>        print(&quot;  ✅ Conversation history cleared.&quot;)<br><br>    elif cmd == &quot;/model&quot;:<br>        print(f&quot;  Model chain: {&#39; → &#39;.join(config.model_chain)}&quot;)<br>        print(f&quot;  Temperature: {config.temperature}&quot;)<br>        print(f&quot;  Max tokens:  {config.max_tokens}&quot;)<br><br>    elif cmd == &quot;/help&quot;:<br>        print(&quot;\n  Available commands:&quot;)<br>        for c, desc in COMMANDS.items():<br>            print(f&quot;    {c:&lt;10} — {desc}&quot;)<br><br>    else:<br>        print(f&quot;  ❓ Unknown command: {cmd}. Type /help for options.&quot;)<br><br>    return False<br><br><br>async def run_chat(config: Config):<br>    &quot;&quot;&quot;Main async chat loop.&quot;&quot;&quot;<br>    session_cost = SessionCost()<br>    system_message = {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: config.system_prompt}<br>    messages = [system_message]<br>    llm = ResilientClient(config)<br><br>    print(&quot;\n&quot; + &quot;═&quot; * 55)<br>    print(&quot;  🦜 LangChain Mastery — SDK Chat (Phase 1)&quot;)<br>    print(&quot;═&quot; * 55)<br>    print(f&quot;  Models:  {&#39; → &#39;.join(config.model_chain)}&quot;)<br>    print(f&quot;  Temp:    {config.temperature} | Max tokens: {config.max_tokens}&quot;)<br>    print(f&quot;  Type /help for commands, /quit to exit&quot;)<br>    print(&quot;═&quot; * 55 + &quot;\n&quot;)<br><br>    while True:<br>        try:<br>            # asyncio-compatible input (non-blocking in async context)<br>            user_input = await asyncio.get_event_loop().run_in_executor(<br>                None, lambda: input(&quot;  You: &quot;).strip()<br>            )<br>        except (KeyboardInterrupt, EOFError):<br>            print(&quot;\n\n  👋 Interrupted. Goodbye!&quot;)<br>            break<br><br>        if not user_input:<br>            continue<br><br>        # Handle slash commands<br>        if user_input.startswith(&quot;/&quot;):<br>            should_exit = await handle_command(<br>                user_input, config, session_cost, messages, system_message<br>            )<br>            if should_exit:<br>                break<br>            continue<br><br>        # Add user message<br>        messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_input})<br><br>        # Trim history to stay within context window<br>        messages = trim_history(messages, system_message, config.max_history_turns)<br><br>        # Stream response<br>        try:<br>            response_text, cost = await llm.stream_with_fallback(messages)<br><br>            # Add to history<br>            messages.append({&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: response_text})<br><br>            # Track and display cost<br>            session_cost.add(cost)<br>            cost.display()<br><br>        except ValueError as e:<br>            print(f&quot;\n  ❌ Request error (fix required): {e}&quot;)<br>            messages.pop()  # Remove failed user message<br><br>        except RuntimeError as e:<br>            print(f&quot;\n  ❌ Service unavailable: {e}&quot;)<br>            print(&quot;  All models in the fallback chain failed. Try again shortly.&quot;)<br>            messages.pop()<br><br>        print()  # Spacing between turns<br><br>    # Final summary on exit<br>    session_cost.display_summary()<br>    print(&quot;\n  👋 Goodbye!\n&quot;)</pre><p>One important async detail: input() is a blocking call. In an async context, it blocks the event loop. We wrap it with run_in_executor(None, ...) to run it in a thread pool, keeping the event loop free. This matters when the app is extended to handle background tasks.</p><h3>main.py — Async Entry Point</h3><pre># main.py<br>import asyncio<br>import sys<br>from config import load_config<br>from chat import run_chat<br><br><br>def main():<br>    try:<br>        config = load_config()<br>    except EnvironmentError as e:<br>        print(f&quot;\n  ❌ {e}\n&quot;)<br>        sys.exit(1)<br><br>    asyncio.run(run_chat(config))<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h3>Running the App</h3><pre>pip install openai tiktoken httpx<br>export OPENAI_API_KEY=&quot;sk-your-key-here&quot;<br>python main.py</pre><h3>Full Demo Walkthrough</h3><p>Here’s a real session showing all the Phase 1 features in action:</p><pre>═══════════════════════════════════════════════════════<br>  🦜 LangChain Mastery — SDK Chat (Phase 1)<br>═══════════════════════════════════════════════════════<br>  Models:  gpt-4o → gpt-4o-mini<br>  Temp:    0.7 | Max tokens: 1000<br>  Type /help for commands, /quit to exit<br>═══════════════════════════════════════════════════════<br><br>  You: What&#39;s the difference between async and threading in Python?<br><br>  🤖 Assistant: Both achieve concurrency but work differently:<br><br>**Threading** runs multiple threads in the same process. Due to Python&#39;s<br>GIL, threads don&#39;t run Python code truly in parallel — but they do run<br>concurrently for I/O-bound tasks (network, disk), where the GIL is released.<br><br>**Async (asyncio)** uses a single thread with an event loop. Instead of<br>context-switching between threads, it cooperatively yields control at<br>`await` points. This is more efficient for I/O-heavy workloads with many<br>concurrent operations.<br><br>**Rule of thumb:**<br>- Many concurrent I/O operations → async<br>- CPU-bound work → multiprocessing<br>- Legacy code or blocking libraries → threading<br><br>  📊 [gpt-4o] in: 42 | out: 112 | cost: $0.00121<br><br>  You: /model<br>  Model chain: gpt-4o → gpt-4o-mini<br>  Temperature: 0.7<br>  Max tokens:  1000<br><br>  You: Give me a one-line Python example of asyncio.gather<br><br>  🤖 Assistant: `results = await asyncio.gather(fetch(url1), fetch(url2), fetch(url3))`<br><br>  📊 [gpt-4o] in: 162 | out: 24 | cost: $0.00065<br><br>  You: /cost<br><br>──────────────────────────────────────────────────<br>  💰 Session Summary<br>──────────────────────────────────────────────────<br>  Messages:     2<br>  Total tokens: 340<br>  Total cost:   $0.00186<br>  Avg per msg:  $0.00093<br>──────────────────────────────────────────────────<br><br>  You: /quit<br><br>──────────────────────────────────────────────────<br>  💰 Session Summary<br>──────────────────────────────────────────────────<br>  Messages:     2<br>  Total tokens: 340<br>  Total cost:   $0.00186<br>  Avg per msg:  $0.00093<br>──────────────────────────────────────────────────<br><br>  👋 Goodbye!</pre><p>Notice what’s different from the Phase 0 demo:</p><p><strong>Exact token counts.</strong> Phase 0 showed ~ estimated counts from tiktoken. Phase 1 shows exact counts from the API via stream_options. The numbers are trustworthy.</p><p><strong>The </strong><strong>/model command</strong> shows the full fallback chain. If gpt-4o goes down mid-session, the app transparently switches to gpt-4o-mini and tells you.</p><p><strong>Input tokens grow naturally.</strong> Message 1 had 42 input tokens. Message 2 had 162 — because the first exchange is now part of the history being sent. The stateless API, working exactly as explained in Post #1.</p><h3>Phase 0 vs Phase 1: The Final Scorecard</h3><p>Let’s close with a concrete line count comparison between the two implementations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6b12bcaa80e2fe7f97ef3d24396f2a3/href">https://medium.com/media/c6b12bcaa80e2fe7f97ef3d24396f2a3/href</a></iframe><p>The total line count is actually <em>lower</em> in Phase 1 — despite being more capable — because errors.py disappears entirely. All retry and exception logic lives inside ResilientClient now, and the SDK&#39;s typed exception hierarchy makes it cleaner than the manual status code checking in Phase 0.</p><p><strong>Capabilities gained in Phase 1:</strong></p><ul><li>Fully async (concurrent requests when needed)</li><li>Exact token counts via stream_options</li><li>Typed IDE autocompletion everywhere</li><li>Structured exception hierarchy</li><li>estimated flag on cost display</li><li>/model command showing fallback chain</li><li>run_in_executor for non-blocking input</li></ul><p><strong>Code removed in Phase 1:</strong></p><ul><li>All manual HTTP headers</li><li>All SSE parsing logic</li><li>All JSON decoding of responses</li><li>errors.py entirely</li><li>Manual response.json()[&quot;choices&quot;][0][&quot;message&quot;][&quot;content&quot;] chains</li></ul><h3>What We’ve Built Across Two Phases</h3><p>Step back for a moment. Over 10 posts, starting from a single curl command, we&#39;ve built:</p><p><strong>Phase 0 — raw HTTP foundation:</strong></p><ul><li>Understanding of the full LLM API request/response cycle</li><li>Token economics and cost calculation</li><li>Streaming via SSE protocol</li><li>Error handling and exponential backoff with jitter</li><li>A complete CLI chat app from scratch</li></ul><p><strong>Phase 1 — SDK upgrade:</strong></p><ul><li>Typed SDK client with clean async/await</li><li>Concurrent request handling with semaphores</li><li>Streaming with exact usage tracking</li><li>Structured exception hierarchy and per-request error tracking</li><li>A ResilientClient wrapper that hides complexity cleanly</li><li>A production-grade SDK chat app</li></ul><p>Every concept we’ve covered — statelessness, tokens, streaming, retries, fallbacks, async concurrency — is going to appear again in Phase 2. But when it does, it’ll be inside LangChain abstractions. And because you understand what those abstractions are hiding, you’ll never be confused by what’s happening underneath.</p><h3>Phase 2 Starts Next</h3><p><strong>Post #11</strong> begins Phase 2: LangChain Core Concepts. We start with LCEL — the LangChain Expression Language — and the pipe | operator that makes chain composition feel elegant. After two phases of writing everything by hand, you&#39;re going to appreciate what LCEL gives you.</p><p><em>Next up: </em><strong><em>Post #11 — LCEL &amp; The Pipe Operator: Composing LLM Chains Like a Pro</em></strong></p><p><em>Follow the series on Medium and find the complete curriculum and Revision Notes on the </em><a href="https://bitphonix.github.io/Langchain-Mastery/"><em>LangChain Mastery website</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18b7130e4aa5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>