<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Afaf Athar on Medium]]></title>
        <description><![CDATA[Stories by Afaf Athar on Medium]]></description>
        <link>https://medium.com/@afafathar?source=rss-486083ff911c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZZhMd2aEUqrq4pb7UukcOA.jpeg</url>
            <title>Stories by Afaf Athar on Medium</title>
            <link>https://medium.com/@afafathar?source=rss-486083ff911c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:37:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@afafathar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From HyDE to RAG-Fusion: Optimizing Vector Retrieval in Python]]></title>
            <link>https://afafathar.medium.com/from-hyde-to-rag-fusion-optimizing-vector-retrieval-in-python-52ad39c04191?source=rss-486083ff911c------2</link>
            <guid isPermaLink="false">https://medium.com/p/52ad39c04191</guid>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[vector]]></category>
            <category><![CDATA[hyde]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Afaf Athar]]></dc:creator>
            <pubDate>Sat, 22 Aug 2026 07:45:24 GMT</pubDate>
            <atom:updated>2026-08-22T07:45:24.497Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i7_GEj9f6SX54bkm.jpg" /><figcaption>RAG Fusion</figcaption></figure><h3>1. The Vector Space Mismatch</h3><p>The core failure mode of naive Retrieval-Augmented Generation (RAG) lies in a fundamental mathematical assumption: that a user’s short, often poorly phrased query will share high semantic similarity with a dense, multi-sentence document chunk.</p><p>In high-dimensional vector space, queries and documents inhabit distinct distributions. A user asking <em>“vendor payment terms”</em> generates a sparse, short vector. The policy document containing the answer is dense, formal, and structured. Cosine similarity between these two vectors often yields low precision, pulling in irrelevant chunks or missing the target entirely.</p><p>To solve this, we must implement <strong>Query Translation</strong> — modifying or expanding the input prompt before vector encoding to optimize retrieval performance.</p><h3>2. Technique 1: Hypothetical Document Embeddings (HyDE)</h3><h4>The Concept</h4><p>Introduced in research by Gao et al., <strong>HyDE</strong> bypasses query-to-document matching by translating the query into a synthetic document first.</p><ol><li>The user submits a short query.</li><li>An LLM generates a hypothetical, plausible answer (even if factually inaccurate or containing hallucinated details).</li><li>The system embeds the <em>hypothetical answer</em> rather than the original query.</li><li>The vector store searches for real document chunks that resemble the synthetic answer.</li></ol><p>Because embedding models group semantically similar structures together, searching with a generated document aligns much closer to real source chunks in vector space.</p><h4>Python Implementation</h4><pre>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.prompts import PromptTemplate<br>from langchain_openai import ChatOpenAI, OpenAIEmbeddings<br>from langchain_community.vectorstores import Chroma<br><br>1. Define the HyDE Prompt<br>hyde_prompt = PromptTemplate.from_template(<br>    &quot;&quot;&quot;Please write a passage to answer the question below. <br>Try to include key technical terms and structural details that a formal document would contain.<br>Question: {question}<br>Passage:&quot;&quot;&quot;<br>)<br>llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br>hyde_chain = hyde_prompt | llm | StrOutputParser()<br># 2. Function to retrieve using HyDE<br>def hyde_retrieval(query: str, vectorstore: Chroma, k: int = 4):<br>    # Generate the hypothetical document<br>    hypothetical_doc = hyde_chain.invoke({&quot;question&quot;: query})<br>    <br>    # Embed the hypothetical document and perform similarity search<br>    retrieved_docs = vectorstore.similarity_search(hypothetical_doc, k=k)<br>    return retrieved_docs<br></pre><h3>3. Technique 2: RAG-Fusion and Reciprocal Rank Fusion (RRF)</h3><h4>The Concept</h4><p>Users rarely ask questions from an optimal angle on the first try. <strong>RAG-Fusion</strong> addresses prompt ambiguity through multi-query expansion and algorithmic re-ranking:</p><ul><li><strong>Query Expansion:</strong> An LLM generates 4 different variations of the user’s initial prompt, capturing multiple perspectives and keywords.</li><li><strong>Parallel Retrieval:</strong> The system executes vector searches across all generated query variations in parallel.</li><li><strong>Reciprocal Rank Fusion (RRF):</strong> The retrieved document sets are merged and re-ranked using an RRF algorithm to compute a unified relevance score for every unique document.</li></ul><h4>The Mathematics of RRF</h4><p>Reciprocal Rank Fusion scores each document $d$ across a set of query result lists Q using the formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*BT4fHPxWW9bJC57_WOivIg.png" /></figure><p>$$RRF\_Score(d \in D) = \sum_{q \in Q} \frac{1}{k + r(q, d)}$$</p><p>Where:</p><ul><li><em>r(q, d)</em> is the rank position of document <em>d</em> in the result list for query <em>q</em></li><li><em>k</em> is a smoothing constant (typically set to <em>60</em>) to prevent top-ranked documents from dominating the score excessively.</li></ul><h4>Python Implementation</h4><pre>from typing import List<br>from langchain_core.pydantic_v1 import BaseModel, Field<br><br># 1. Multi-Query Generation Schema<br>class QueryExpansion(BaseModel):<br>    queries: List[str] = Field(<br>        description=&quot;Generate 4 distinct rephrasings or sub-questions of the original query.&quot;<br>    )<br><br>expansion_prompt = PromptTemplate.from_template(<br>    &quot;&quot;&quot;You are an AI assistant. Generate 4 different versions or angles of the given user question <br>to help retrieve relevant documents from a vector database.<br><br>Original Question: {question}&quot;&quot;&quot;<br>)<br><br>expansion_llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0).with_structured_output(QueryExpansion)<br>expansion_chain = expansion_prompt | expansion_llm<br><br># 2. Reciprocal Rank Fusion Algorithm<br>def reciprocal_rank_fusion(results: List[List], k_constant: int = 60):<br>    fused_scores = {}<br>    <br>    for docs in results:<br>        for rank, doc in enumerate(docs):<br>            doc_str = doc.page_content<br>            if doc_str not in fused_scores:<br>                fused_scores[doc_str] = {&quot;doc&quot;: doc, &quot;score&quot;: 0.0}<br>            <br>            # Apply RRF Formula<br>            fused_scores[doc_str][&quot;score&quot;] += 1.0 / (rank + 1 + k_constant)<br>            <br>    # Sort documents by calculated RRF score in descending order<br>    reranked_results = sorted(<br>        fused_scores.values(), <br>        key=lambda x: x[&quot;score&quot;], <br>        reverse=True<br>    )<br>    <br>    return [item[&quot;doc&quot;] for item in reranked_results]<br><br># 3. RAG-Fusion Execution Workflow<br>def rag_fusion_retrieval(original_query: str, vectorstore: Chroma, top_k: int = 4):<br>    # Step 1: Expand queries<br>    expanded = expansion_chain.invoke({&quot;question&quot;: original_query})<br>    all_queries = [original_query] + expanded.queries<br>    <br>    # Step 2: Execute parallel vector retrievals<br>    raw_results = []<br>    for q in all_queries:<br>        raw_results.append(vectorstore.similarity_search(q, k=top_k))<br>        <br>    # Step 3: Apply RRF Re-ranking<br>    final_docs = reciprocal_rank_fusion(raw_results)<br>    return final_docs[:top_k]</pre><h3>4. Architectural Comparison: HyDE vs. RAG-Fusion</h3><p><strong>StrategyBest Use CasePrimary AdvantageTrade-offHyDE</strong>Short, conceptual, or abstract questionsConverts query vector space into document vector spaceAdds latency due to synthetic document generation; risk if LLM generates completely wild domain assumptions<strong>RAG-Fusion</strong>Complex, ambiguous, or multi-faceted queriesMaximizes recall by searching across multiple semantic anglesMultiple vector DB calls increase read throughput and compute overhead</p><h3>5. Conclusion</h3><p>Relying on raw user queries for vector search is a major source of low contextual precision in production RAG systems. By implementing a Query Translation layer with <strong>HyDE</strong> or <strong>RAG-Fusion</strong>, you align the semantic structure of your inputs with your database index, dramatically improving retrieval accuracy before the generation step ever begins.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52ad39c04191" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecting Logical and Semantic Routers for Enterprise Agentic Workflows]]></title>
            <link>https://afafathar.medium.com/architecting-logical-and-semantic-routers-for-enterprise-agentic-workflows-bd977a3bf122?source=rss-486083ff911c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd977a3bf122</guid>
            <category><![CDATA[logical-thinking]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Afaf Athar]]></dc:creator>
            <pubDate>Thu, 23 Jul 2026 14:08:15 GMT</pubDate>
            <atom:updated>2026-07-23T14:10:26.003Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*aNB7dGUGQ0D9DCT0sXCNqQ.png" /></figure><h4>1. Why a Flat RAG Architecture Fails Under Real Traffic</h4><p>Benchmark and demo queries are adversarially clean — they’re written by the same person who built the retriever, so they align with the embedding space by construction. Production traffic doesn’t share that property.</p><p>In domains like procurement, ITSM, or claims processing, incoming prompts are underspecified, multi-intent, or reference structured identifiers (ticket numbers, SKUs, case IDs) that have no meaningful semantic neighborhood.</p><p>A single-entry-point architecture — embed everything, run top-k retrieval against one vector index — has three concrete failure modes:</p><ul><li><strong>Precision collapse on structured lookups.</strong> A query like <strong>status of ticket</strong> <strong>#8892</strong> embeds close to <em>other tickets about status</em>, not to the literal record with ID 8892. Vector similarity is not a substitute for an equality filter or SQL predicate.</li><li><strong>Hallucination under distribution mismatch.</strong> If the retriever returns weak or irrelevant context, the generator will often still produce a fluent, confident, and wrong answer rather than abstaining.</li><li><strong>Unnecessary cost and latency.</strong> Every query pays for embedding + ANN search + full generation, even ones that map trivially to a deterministic function call (e.g., <em>cancel my order</em>).</li></ul><p>The architectural fix is to decouple <em>intent classification</em> from <em>task execution</em> by inserting a <strong>Routing Layer</strong> as the entry node of the orchestration graph. This is a standard triage pattern from classical NLU pipelines (intent classification → slot filling → dialogue management), re-applied to LLM-orchestrated systems.</p><h4>2. Logical Routing: Classification via Structured Outputs</h4><p>Logical routing repurposes the LLM as a constrained classifier rather than a free-text generator. Instead of prompting the model to answer the user’s question, you constrain its output space to a fixed schema and ask it to select a category.</p><p>Mechanically, this relies on the model provider’s structured-output or tool-calling interface (JSON mode with a schema, or a single forced tool call).</p><p>This matters because unconstrained generation followed by regex-based parsing is fragile — schema-constrained decoding (via grammar-based sampling or forced function calls) gives you near-deterministic parse-ability, which is the property you actually need at the entry point of a state machine.</p><p><strong>Basic single-label router:</strong></p><pre>from langchain_core.pydantic_v1 import BaseModel, Field<br>from langchain_openai import ChatOpenAI<br><br>class RouteQuery(BaseModel):<br><br>    &quot;&quot;&quot;Route a user query to the most relevant data source or pipeline.&quot;&quot;&quot;<br>    datasource: str = Field(<br>        description=&quot;Given a user query, choose to route it to &#39;vector_store&#39;, &#39;sql_db&#39;, or &#39;deterministic_api&#39;.&quot;<br>    )<br>def logical_router(state):<br><br>    question = state[&quot;question&quot;]<br>    llm = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br>    structured_router = llm.with_structured_output(RouteQuery)<br>    route = structured_router.invoke(question)<br>    return {&quot;destination&quot;: route.datasource}</pre><p><strong>Extending to multi-intent / compound queries:</strong> Real prompts often carry more than one intent (<em>cancel order 404 and tell me your refund policy</em>). A single-label schema silently drops the second intent. Model this as a list of routes with per-route extracted arguments instead:</p><pre>from typing import List, Literal<br>from pydantic import BaseModel, Field<br><br>class RouteItem(BaseModel):<br>    datasource: Literal[&quot;vector_store&quot;, &quot;sql_db&quot;, &quot;deterministic_api&quot;]<br>    sub_query: str = Field(description=&quot;The portion of the user&#39;s message relevant to this route&quot;)<br>    confidence: float = Field(ge=0.0, le=1.0)<br><br>class MultiRouteQuery(BaseModel):<br>    routes: List[RouteItem]<br><br>def logical_router_multi(state):<br>    question = state[&quot;question&quot;]<br>    llm = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br>    structured_router = llm.with_structured_output(MultiRouteQuery)<br>    result = structured_router.invoke(question)<br>    # Downstream: fan out each RouteItem to its destination node,<br>    # then merge partial results before final response synthesis.<br>    return {&quot;routes&quot;: [r.dict() for r in result.routes]}</pre><p>This turns your graph’s conditional edge into a <strong>fan-out/fan-in</strong> pattern rather than a single branch — each sub-query goes to its own agent, and a merge node reconciles partial responses before final synthesis.</p><p><strong>Cost/latency profile:</strong> logical routing adds one LLM round-trip (typically 150–800ms with a small/fast model like gpt-4o-mini or a distilled classifier) plus token cost proportional to prompt length. At high QPS, this is often the dominant latency contributor in the whole pipeline if you’re not careful about model choice.</p><h4>3. Semantic Routing: Embedding-Space Classification</h4><p>Semantic routing replaces the LLM call with a nearest-centroid or nearest-neighbor lookup in embedding space. You maintain a small set of <strong>route profiles: </strong>either single representative utterances or a <em>cluster</em> of utterances per route: embed them offline, and at request time embed the incoming query and compute similarity against each profile.</p><p><strong>Single-centroid version (what most tutorials show):</strong></p><pre>import numpy as np<br>from langchain_openai import OpenAIEmbeddings<br><br>embeddings = OpenAIEmbeddings(model=&quot;text-embedding-3-small&quot;)<br><br>route_profiles = {<br>    &quot;procurement_policy&quot;: embeddings.embed_query(<br>        &quot;What are the rules for vendor onboarding and contract compliance?&quot;<br>    ),<br>    &quot;technical_support&quot;: embeddings.embed_query(<br>        &quot;My system is down, how do I reset my password or fix a bug?&quot;<br>    ),<br>}<br>def cosine_sim(a, b):<br><br>    a, b = np.array(a), np.array(b)<br>    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))<br><br>def semantic_router(state, threshold: float = 0.80):<br><br>    question = state[&quot;question&quot;]<br>    q_embedding = embeddings.embed_query(question)<br>    scored = {<br>        name: cosine_sim(q_embedding, profile)<br>        for name, profile in route_profiles.items()<br>    }<br>    best_route, highest_sim = max(scored.items(), key=lambda x: x[1])<br>    if highest_sim &lt; threshold:<br>        return {&quot;destination&quot;: &quot;general_llm_agent&quot;, &quot;route_confidence&quot;: highest_sim}<br>    return {&quot;destination&quot;: best_route, &quot;route_confidence&quot;: highest_sim}</pre><p><strong>Production hardening — multi-utterance centroids and margin-based fallback:</strong> A single reference embedding per route is noisy and overfits to that one phrasing. In practice, embed 10–50 representative utterances per route, average (or store all and take max-similarity), and critically, check the <em>margin</em> between the top two scores, not just the absolute top score. A high top score with a close second-best is a genuine ambiguity signal that an absolute threshold alone will miss:</p><pre>def semantic_router_v2(state, threshold: float = 0.80, min_margin: float = 0.05):<br>    question = state[&quot;question&quot;]<br>    q_embedding = embeddings.embed_query(question)<br><br>    scored = {}<br>    for route_name, utterance_embeddings in route_profiles.items():<br>        sims = [cosine_sim(q_embedding, u) for u in utterance_embeddings]<br>        scored[route_name] = max(sims)  # or np.mean(sorted(sims)[-3:]) for top-k avg<br>    ranked = sorted(scored.items(), key=lambda x: x[1], reverse=True)<br>    (best_route, best_sim), (_, second_sim) = ranked[0], ranked[1]<br>    if best_sim &lt; threshold or (best_sim - second_sim) &lt; min_margin:<br>        return {&quot;destination&quot;: &quot;general_llm_agent&quot;, &quot;route_confidence&quot;: best_sim}<br>    return {&quot;destination&quot;: best_route, &quot;route_confidence&quot;: best_sim}</pre><p><strong>Embedding model choice matters here.</strong> Cheaper/smaller embedding models (e.g., text-embedding-3-small) are usually sufficient for route classification since you&#39;re separating coarse-grained clusters, not doing fine-grained semantic search — reserve larger embedding models for your actual retrieval index where recall quality directly affects answer correctness.</p><p><strong>Cost/latency profile:</strong> one embedding call (often &lt;20ms with a local/cached model, or ~30–50ms against a hosted embedding API) plus O(n_routes × n_utterances) dot products, which is negligible until you have hundreds of routes — at which point an ANN index (FAISS/HNSW) over route utterances instead of brute-force cosine is worth it.</p><h4><strong>4. Hybrid Cascade: Combining Both for Cost-Accuracy Tradeoff</strong></h4><p>Neither router alone is optimal in production. Semantic routing is cheap but brittle at the margins; logical routing is accurate but expensive at scale. The standard pattern is a <strong>cascade</strong>: semantic router handles the confident majority of traffic, logical router arbitrates the ambiguous remainder.</p><pre>def hybrid_router(state, semantic_threshold: float = 0.85):<br>    semantic_result = semantic_router_v2(state, threshold=semantic_threshold)<br><br>if semantic_result[&quot;destination&quot;] != &quot;general_llm_agent&quot;:<br>        # High-confidence semantic match - skip the LLM call entirely<br>        return semantic_result<br>    # Low-confidence or ambiguous - escalate to LLM-based classification<br>    logical_result = logical_router(state)<br>    return logical_result</pre><p>In a well-tuned system, this pushes 70–90% of traffic through the sub-50ms semantic path and reserves the LLM call for the genuinely ambiguous tail — meaningfully cutting p50/p95 latency and per-query cost versus routing everything through an LLM.</p><p><strong>Threshold calibration</strong> shouldn’t be a guess. Collect a labeled validation set of real (or synthetic) queries per route, sweep the threshold and margin parameters, and pick the operating point on a precision-recall curve that matches your tolerance for misrouted queries — a misroute to deterministic_api (which might mutate state) is far costlier than a misroute to general_llm_agent (which just costs an extra hop).</p><h4>5. Orchestrating the Flow in LangGraph</h4><p>Once you have a router function — semantic, logical, or hybrid — it becomes the entry node of the state graph, and its output key drives a conditional edge:</p><pre>workflow.set_entry_point(&quot;triage_router&quot;)<br>workflow.add_conditional_edges(<br>    &quot;triage_router&quot;,<br>    lambda state: state[&quot;destination&quot;],<br>    {<br>        &quot;vector_store&quot;: &quot;rag_agent&quot;,<br>        &quot;sql_db&quot;: &quot;data_analytics_agent&quot;,<br>        &quot;deterministic_api&quot;: &quot;action_execution_pipeline&quot;,<br>        &quot;general_llm_agent&quot;: &quot;fallback_conversational_agent&quot;,<br>    },<br>)</pre><p>For the multi-intent fan-out case, replace the single conditional edge with a Send API call (LangGraph&#39;s mechanism for dynamic parallel branching) so each extracted sub-query is dispatched to its own node instance, with a downstream merge node collecting results before final response synthesis.</p><h4>6. Observability: Routing Is a Metric, Not a One-Time Decision</h4><p>Treat the router as a monitored component, not a fire-and-forget classifier. At minimum, log per-request: the chosen route, the confidence/similarity score, and (sampled) human or LLM-judged correctness of that routing decision. This lets you:</p><ul><li>Detect <strong>distribution drift</strong> — if average confidence scores trend downward over weeks, your route profiles no longer match live traffic and need re-embedding against fresh utterance samples.</li><li>Catch <strong>silent misrouting</strong> to deterministic_api, which is the highest-blast-radius failure mode since it can trigger real side effects (cancellations, updates) on a misclassified intent — consider requiring a higher confidence threshold specifically for routes that mutate state, versus routes that only read.</li><li>Feed misrouted examples back into your route-profile utterance sets (for semantic routing) or few-shot examples (for logical routing) as a lightweight continuous-improvement loop.</li></ul><h4>7. Conclusion</h4><p>Stop routing every input through a single monolithic RAG chain and hoping the retriever or the generator absorbs the ambiguity. Decouple intent classification from execution with an explicit Routing Layer, choose semantic, logical, or a hybrid cascade based on your latency/cost/accuracy constraints, and instrument the router itself as a first-class, monitored component of the system — not a static function you write once and forget.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd977a3bf122" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adaptive RAG: Integrating In-Flow Evaluation into LangGraph State Machines]]></title>
            <link>https://afafathar.medium.com/adaptive-rag-integrating-in-flow-evaluation-into-langgraph-state-machines-0f0362e3e52a?source=rss-486083ff911c------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f0362e3e52a</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[metrcis]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[adaptive-rag]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Afaf Athar]]></dc:creator>
            <pubDate>Fri, 17 Jul 2026 12:42:20 GMT</pubDate>
            <atom:updated>2026-07-17T12:42:20.907Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*1n4y8BDyvuEpeonq.jpg" /></figure><p>Moving an LLM application from a flashy local prototype to an enterprise-grade production system is one of the steepest challenges in modern AI engineering.</p><p>In a traditional engineering stack, we protect our systems using comprehensive CI/CD pipelines and unit tests. Yet, in the world of Generative AI, many teams treat evaluation as an afterthought — running batch scripts offline or checking monitoring dashboards long after a user has already received a hallucinated answer.</p><p>If your application handles strict domain tasks, like automated procurement triaging or complex ticketing tool workflows, offline evaluations aren’t enough. You need to catch failures before they reach the user.</p><p>The solution is <strong>Adaptive RAG</strong>: pulling traditional evaluation metrics out of the dashboard and embedding them directly into your live inference loop as binary unit tests using a state machine.</p><h3>The Blind Spot of Naive RAG</h3><p>A standard, linear RAG pipeline is completely blind to its own failures. It embeds a query, executes a similarity search against a vector store, and blindly pipes the retrieved chunks into an LLM context window.</p><p>If the retrieval step pulls irrelevant data or out-of-domain context, the generation node has no choice but to hallucinate or provide a low-quality response.</p><p>To build a resilient system, we must pivot from a linear chain to a <strong>state-driven orchestrator</strong> built with a graph architecture. By utilising a framework like LangGraph, we can treat the RAG pipeline as a state machine where execution nodes (Retrieval, Generation) alternate with deterministic evaluation nodes (Graders).</p><h3>Architectural Design: The Self-Correcting Graph</h3><p>Before diving into the code, let’s map out the core components of this architecture. Our state machine relies on a single shared State object—a centralised dictionary that tracks the user’s query, the retrieved document chunks, the evaluation status, and the generated response.</p><pre>                  +------------------+<br>                  |  1. USER QUERY   |<br>                  +--------+---------+<br>                           |<br>                           v<br>                  +------------------+<br>                  |   2. RETRIEVE    |<br>                  +--------+---------+<br>                           |<br>                           v<br>                  +------------------+<br>                  | 3. GRADE CONTEXT |<br>                  +--------+---------+<br>                           |<br>            +--------------+--------------+<br>            |                             |<br>      (If Irrelevant)                (If Relevant)<br>            |                             |<br>            v                             v<br>  +-------------------+         +-------------------+<br>  | 4. FALLBACK SEARCH|         |    5. GENERATE    |<br>  +---------+---------+         +---------+---------+<br>            |                             |<br>            +------------&gt; &lt;--------------+<br>                           |<br>                           v<br>                  +------------------+<br>                  |  6. FAITHFULNESS |<br>                  |      GRADER      |<br>                  +--------+---------+<br>                           |<br>            +--------------+--------------+<br>            |                             |<br>     (Hallucinated)                (Passed Test)<br>            |                             |<br>            v                             v<br>  +-------------------+         +-------------------+<br>  | 7. RE-WRITE QUERY |         |     8. RETURN     |<br>  +---------+---------+         |     RESPONSE      |<br>            |                   +-------------------+<br>            +-&gt; [Loop back]</pre><p>Our orchestrator implements two crucial in-flow unit tests:</p><ul><li><strong>Contextual Precision Check:</strong> Evaluating if the retrieved data is relevant to the prompt. If it fails, it bypasses generation entirely and routes the state to a fallback node (like a dynamic web search or an alternative internal database).</li><li><strong>Faithfulness Check:</strong> Evaluating the generated draft against the source chunks to look for hallucinations. If it fails, the graph catches the exception, rewrites the query or prompt context, and loops back to try again.</li></ul><h3>Python Implementation</h3><p>Let’s build this step-by-step using Python and LangGraph.</p><h4>Step 1: Defining the Core State</h4><p>Our graph state maintains the historical context of the execution.</p><pre>from typing import TypedDict, List<br><br>class AdaptiveRAGState(TypedDict):<br>    question: str<br>    documents: List[str]<br>    generation: str<br>    is_relevant: str<br>    is_faithful: str</pre><h4>Step 2: Creating Structured LLM Graders</h4><p>To ensure our graph transitions are strictly deterministic, we must force our evaluation nodes to output explicit binary scores (yes or no). We achieve this by binding Pydantic schemas to our LLM nodes via structured outputs.</p><pre>from langchain_core.pydantic_v1 import BaseModel, Field<br>from langchain_openai import ChatOpenAI<br><br># Pydantic schema for context evaluation<br>class ContextPrecisionScore(BaseModel):<br>    &quot;&quot;&quot;Binary score for checking if retrieved context is relevant.&quot;&quot;&quot;<br>    binary_score: str = Field(description=&quot;Are the documents relevant to the question? &#39;yes&#39; or &#39;no&#39;&quot;)<br><br># Pydantic schema for hallucination evaluation<br>class FaithfulnessScore(BaseModel):<br>    &quot;&quot;&quot;Binary score for hallucination check on generated text.&quot;&quot;&quot;<br>    binary_score: str = Field(description=&quot;Is the answer grounded strictly in the context? &#39;yes&#39; or &#39;no&#39;&quot;)<br><br># Node: Context Precision Grader<br>def grade_context(state: AdaptiveRAGState):<br>    question = state[&quot;question&quot;]<br>    documents = state[&quot;documents&quot;]<br>    <br>    llm = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br>    structured_grader = llm.with_structured_output(ContextPrecisionScore)<br>    <br>    # In real application, pass documents and question via a structured prompt<br>    is_relevant = &quot;yes&quot;<br>    for doc in documents:<br>        response = structured_grader.invoke({&quot;question&quot;: question, &quot;context&quot;: doc})<br>        if response.binary_score == &quot;no&quot;:<br>            is_relevant = &quot;no&quot;<br>            break # Trigger fallback if any critical chunk is irrelevant<br>            <br>    return {&quot;is_relevant&quot;: is_relevant}</pre><h4>Step 3: Wiring the Conditional Edge Logic</h4><p>Conditional edges evaluate the metrics stored in the state dictionary and dictate the next logical routing step.</p><pre>def route_post_retrieval(state: AdaptiveRAGState):<br>    &quot;&quot;&quot;Routes to generation or fallback based on precision test.&quot;&quot;&quot;<br>    if state[&quot;is_relevant&quot;] == &quot;no&quot;:<br>        return &quot;fallback_search&quot;<br>    return &quot;generate_answer&quot;<br><br>def route_post_generation(state: AdaptiveRAGState):<br>    &quot;&quot;&quot;Routes to completion or triggers a rewrite loop if a hallucination is caught.&quot;&quot;&quot;<br>    if state[&quot;is_faithful&quot;] == &quot;no&quot;:<br>        return &quot;rewrite_query&quot;<br>    return &quot;end_workflow&quot;</pre><h4>Step 4: Compiling the State Machine</h4><p>With our nodes and conditional routing logic ready, we assemble the graph topology.</p><pre>from langgraph.graph import END, StateGraph<br>workflow = StateGraph(AdaptiveRAGState)<br><br># Register the functional nodes<br>workflow.add_node(&quot;retrieve&quot;, retrieve_node)<br>workflow.add_node(&quot;grade_context&quot;, grade_context)<br>workflow.add_node(&quot;fallback_search&quot;, fallback_search_node)<br>workflow.add_node(&quot;generate_answer&quot;, generate_answer_node)<br>workflow.add_node(&quot;grade_faithfulness&quot;, grade_faithfulness_node)<br>workflow.add_node(&quot;rewrite_query&quot;, rewrite_query_node)<br><br># Set up the strict edge sequence<br>workflow.set_entry_point(&quot;retrieve&quot;)<br>workflow.add_edge(&quot;retrieve&quot;, &quot;grade_context&quot;)<br><br># Implement live unit test routing<br>workflow.add_conditional_edges(<br>    &quot;grade_context&quot;,<br>    route_post_retrieval,<br>    {<br>        &quot;fallback_search&quot;: &quot;fallback_search&quot;,<br>        &quot;generate_answer&quot;: &quot;generate_answer&quot;<br>    }<br>)<br>workflow.add_edge(&quot;fallback_search&quot;, &quot;generate_answer&quot;)<br>workflow.add_edge(&quot;generate_answer&quot;, &quot;grade_faithfulness&quot;)<br>workflow.add_conditional_edges(<br>    &quot;grade_faithfulness&quot;,<br>    route_post_generation,<br>    {<br>        &quot;rewrite_query&quot;: &quot;rewrite_query&quot;,<br>        &quot;end_workflow&quot;: END<br>    }<br>)<br>workflow.add_edge(&quot;rewrite_query&quot;, &quot;retrieve&quot;) # Loop back for re-retrieval<br><br># Compile into an executable application<br>app = workflow.compile()</pre><h3>Production Observability and Continuous Evaluation</h3><p>When you shift to an active, cyclic RAG loop, monitoring tools change roles. Instead of just showing logs, platforms like LangSmith act as your production debugger, mapping out every node execution, conditional path, and retry cycle.</p><p>Furthermore, you can utilise the outputs of your in-flow evaluation nodes to feed metrics seamlessly into validation frameworks like RAGAs and DeepEval. Over time, tracking the percentage of queries that hit the <strong><em>rewrite loop</em></strong> or the <strong><em>fallback search node</em></strong> becomes a powerful indicator of how clean and robust your baseline vector database index truly is.</p><h3>Conclusion</h3><p>Building complex AI applications requires adopting a software engineering mindset. By treating RAG pipelines as deterministic state machines on guardrails, we transition away from unreliable prompt-tweaking. We can confidently capture exceptions, dynamically clean context, and guarantee reliable generations before they ever hit a production endpoint.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f0362e3e52a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing RAPTOR: How to Build Hierarchical RAG for Complex Document Processing]]></title>
            <link>https://afafathar.medium.com/implementing-raptor-how-to-build-hierarchical-rag-for-complex-document-processing-a5fe6cad3d71?source=rss-486083ff911c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5fe6cad3d71</guid>
            <category><![CDATA[raptor]]></category>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[trees]]></category>
            <category><![CDATA[knn-algorithm]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Afaf Athar]]></dc:creator>
            <pubDate>Sun, 12 Jul 2026 10:40:37 GMT</pubDate>
            <atom:updated>2026-07-12T10:40:37.637Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/954/0*th6r0hulvj4Ivz7o.png" /><figcaption>Raptor Workflow</figcaption></figure><p>Standard RAG retrieves flat chunks with nearest-neighbor search. That works for single-fact lookup, but it breaks down on questions that require synthesizing information spread across a document — or across many documents. RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) fixes this by building a <strong>tree of summaries</strong> over your corpus, so retrieval can pull context at whatever level of abstraction the question needs.</p><p>This guide walks through building a working RAPTOR pipeline from scratch: chunking, embedding, recursive clustering, summarization, and tree-aware retrieval. We’ll also cover evaluation, production considerations, and a full end-to-end script you can adapt directly.</p><p>RAPTOR was introduced as a way to address a structural gap in retrieval-augmented generation: embedding models are good at finding <em>locally</em> similar text, but they have no built-in concept of document hierarchy — sections, chapters, or cross-document themes. Most production RAG failures on complex queries trace back to this gap, not to a bad embedding model or a weak LLM. The fix isn’t a better embedding model; it’s giving the retrieval system a structure to search over in the first place.</p><h3>Why flat retrieval falls short</h3><p>A standard pipeline embeds every chunk and retrieves the top-k nearest to the query vector. This works when the answer lives in one chunk. It fails when:</p><ul><li>The answer requires combining facts from multiple, non-adjacent chunks</li><li>The question is about the <em>document as a whole</em> (<strong><em>what’s the overall sentiment of this filing?</em></strong>)</li><li>The corpus is large enough that semantically similar-but-irrelevant chunks crowd out the right ones</li></ul><p>RAPTOR solves this by clustering chunks, summarizing each cluster, and repeating the process recursively — building layers of increasingly abstract summaries. At query time, you search across all layers at once, so the system can pull a leaf-level detail or a top-level summary depending on what’s needed.</p><h3>Architecture overview</h3><pre>Layer 2 (root):      [Summary of Summaries]<br>Layer 1:         [Cluster A Summary]  [Cluster B Summary]  [Cluster C Summary]<br>Layer 0 (leaves): [chunk1] [chunk2] [chunk3] [chunk4] [chunk5] [chunk6] ...</pre><p>Each layer is built by:</p><ol><li>Embedding the nodes in the layer below</li><li>Clustering those embeddings</li><li>Summarizing each cluster with an LLM</li><li>Treating those summaries as the nodes of the new layer</li><li>Repeating until you hit a single root node or a stopping condition</li></ol><p>All nodes across all layers get embedded and stored in the same vector index — retrieval isn’t restricted to one layer.</p><h3>How RAPTOR compares to other approaches</h3><p>Before building, it’s worth knowing where RAPTOR sits relative to other strategies people reach for when flat retrieval isn’t enough:</p><p>Approach How it works Strength Weakness Flat KNN retrieval Embed chunks, retrieve top-k nearest to query Simple, cheap, fast to build No structure; fails on synthesis questions Larger chunk sizes Reduce fragmentation by embedding bigger chunks Slightly more context per chunk Dilutes embeddings, hits context limits, doesn’t solve cross-document synthesis Query decomposition / multi-hop retrieval Break the query into sub-questions, retrieve for each Handles some multi-step questions Requires knowing the sub-questions upfront; expensive at query time Reranking (cross-encoders) Retrieve broadly, then rerank with a more precise model Improves precision within retrieved set Doesn’t fix a candidate set that’s missing the right chunks entirely <strong>RAPTOR (hierarchical summarization)</strong> Pre-build a tree of summaries at multiple abstraction levels, index all layers Handles both granular and broad queries against the same index, no query-time routing Higher indexing cost; tree needs periodic rebuilding</p><p>RAPTOR isn’t a replacement for reranking or query decomposition — they solve different problems and often get combined. RAPTOR fixes the <em>structure of the index :</em> reranking and decomposition improve what you do with a candidate set once you have one.</p><h3>Step 1: Chunk your documents</h3><pre>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>def chunk_documents(documents, chunk_size=400, chunk_overlap=50):<br>    splitter = RecursiveCharacterTextSplitter(<br>        chunk_size=chunk_size,<br>        chunk_overlap=chunk_overlap,<br>        separators=[&quot;\n\n&quot;, &quot;\n&quot;, &quot;. &quot;, &quot; &quot;, &quot;&quot;]<br>    )<br>    chunks = []<br>    for doc in documents:<br>        chunks.extend(splitter.split_text(doc))<br>    return chunks</pre><p>Keep chunks small enough that clusters form around genuinely related content, not just adjacent text.</p><h3>Step 2: Embed the chunks</h3><pre>from sentence_transformers import SentenceTransformer<br>import numpy as np<br>embedder = SentenceTransformer(&quot;all-MiniLM-L6-v2&quot;)<br>def embed_texts(texts):<br>    return embedder.encode(texts, show_progress_bar=True)</pre><p>Any embedding model works — swap in OpenAI, Cohere, or a domain-specific model as needed.</p><h3>Step 3: Cluster with Gaussian Mixture Models</h3><p>RAPTOR’s original implementation uses <strong>soft clustering</strong> (GMM) instead of hard clustering like k-means, because a chunk can legitimately belong to more than one topic. UMAP is used first to reduce dimensionality, since GMM struggles in high-dimensional embedding space.</p><pre>import umap<br>from sklearn.mixture import GaussianMixture<br><br>def reduce_dimensions(embeddings, n_components=10):<br>    reducer = umap.UMAP(n_components=n_components, metric=&quot;cosine&quot;)<br>    return reducer.fit_transform(embeddings)<br><br>def cluster_embeddings(embeddings, max_clusters=10, threshold=0.1):<br>    reduced = reduce_dimensions(embeddings)<br>    best_n, best_bic = 1, np.inf<br>    for n in range(1, min(max_clusters, len(reduced))):<br>        gmm = GaussianMixture(n_components=n, random_state=42)<br>        gmm.fit(reduced)<br>        bic = gmm.bic(reduced)<br>        if bic &lt; best_bic:<br>            best_bic, best_n = bic, n<br>    gmm = GaussianMixture(n_components=best_n, random_state=42)<br>    gmm.fit(reduced)<br>    probs = gmm.predict_proba(reduced)<br>    # Soft assignment: a point can belong to multiple clusters above threshold<br>    cluster_labels = [np.where(row &gt; threshold)[0].tolist() for row in probs]<br>    return cluster_labels, best_n</pre><p>BIC (Bayesian Information Criterion) picks a reasonable number of clusters automatically, so you don’t need to hardcode it per document.</p><h3>Step 4: Summarize each cluster</h3><p>Each cluster’s member texts get concatenated and passed to an LLM for summarization. This is what turns a group of related chunks into a single higher-level node.</p><pre>import anthropic<br><br>client = anthropic.Anthropic()<br>def summarize_cluster(texts):<br>    combined = &quot;\n\n&quot;.join(texts)<br>    response = client.messages.create(<br>        model=&quot;claude-sonnet-4-5&quot;,<br>        max_tokens=500,<br>        messages=[{<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: f&quot;Summarize the following text into a dense, &quot;<br>                       f&quot;information-preserving paragraph. Retain key facts, &quot;<br>                       f&quot;numbers, and entities:\n\n{combined}&quot;<br>        }]<br>    )<br>    return response.content[0].text</pre><h3>Step 5: Build the tree recursively</h3><pre>class RaptorNode:<br><br>    def __init__(self, text, embedding, layer, children=None):<br>        self.text = text<br>        self.embedding = embedding<br>        self.layer = layer<br>        self.children = children or []<br><br>def build_raptor_tree(chunks, max_layers=4, min_cluster_size=2):<br><br>    layer_0 = [RaptorNode(text=c, embedding=e, layer=0)<br>               for c, e in zip(chunks, embed_texts(chunks))]<br>    all_nodes = list(layer_0)<br>    current_layer_nodes = layer_0<br>    for layer_num in range(1, max_layers + 1):<br>        if len(current_layer_nodes) &lt;= min_cluster_size:<br>            break<br>        embeddings = np.array([n.embedding for n in current_layer_nodes])<br>        cluster_labels, n_clusters = cluster_embeddings(embeddings)<br>        next_layer_nodes = []<br>        for cluster_id in range(n_clusters):<br>            members = [n for n, labels in zip(current_layer_nodes, cluster_labels)<br>                       if cluster_id in labels]<br>            if not members:<br>                continue<br>            summary_text = summarize_cluster([m.text for m in members])<br>            summary_embedding = embed_texts([summary_text])[0]<br>            new_node = RaptorNode(<br>                text=summary_text,<br>                embedding=summary_embedding,<br>                layer=layer_num,<br>                children=members<br>            )<br>            next_layer_nodes.append(new_node)<br>        all_nodes.extend(next_layer_nodes)<br>        current_layer_nodes = next_layer_nodes<br>        if len(next_layer_nodes) &lt;= 1:<br>            break<br><br>    return all_nodes</pre><p>This builds the full tree bottom-up and returns every node — leaves and summaries — as a flat list ready for indexing.</p><h3>Step 6: Index all layers together</h3><p>The key design choice in RAPTOR: leaf chunks and summary nodes go into the <strong>same</strong> vector store. Retrieval doesn’t need to know in advance which layer holds the answer.</p><pre>import chromadb<br><br>def build_index(all_nodes):<br>    client = chromadb.Client()<br>    collection = client.create_collection(&quot;raptor_index&quot;)<br>    collection.add(<br>        ids=[str(i) for i in range(len(all_nodes))],<br>        embeddings=[n.embedding.tolist() for n in all_nodes],<br>        documents=[n.text for n in all_nodes],<br>        metadatas=[{&quot;layer&quot;: n.layer} for n in all_nodes]<br>    )<br><br>    return collection</pre><h3>Step 7: Query the tree</h3><pre>def query_raptor(collection, query, top_k=8):<br><br>    query_embedding = embed_texts([query])[0]<br>    results = collection.query(<br>        query_embeddings=[query_embedding.tolist()],<br>        n_results=top_k<br>    )<br><br>    return results[&quot;documents&quot;][0]<br><br>def answer_question(collection, query):<br><br>    retrieved = query_raptor(collection, query)<br>    context = &quot;\n\n&quot;.join(retrieved)<br>    response = client.messages.create(<br>        model=&quot;claude-sonnet-4-5&quot;,<br>        max_tokens=1000,<br>        messages=[{<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: f&quot;Using the following context, answer the question.\n\n&quot;<br>                       f&quot;Context:\n{context}\n\nQuestion: {query}&quot;<br>        }]<br>    )<br><br>    return response.content[0].text</pre><p>Because summary nodes and leaf chunks are embedded in the same space, a broad question naturally retrieves top-layer summaries, while a specific question naturally retrieves leaf-level detail — no separate routing logic needed.</p><h3>Evaluating whether it’s actually working</h3><p>Don’t just ship this and assume it’s better — measure it against your flat baseline. A few practical checks:</p><pre>def evaluate_retrieval(collection_raptor, collection_flat, eval_set):<br>    &quot;&quot;&quot;<br>    eval_set: list of {&quot;question&quot;: str, &quot;expected_facts&quot;: list[str]}<br>    Checks whether retrieved context contains the expected facts.<br>    &quot;&quot;&quot;<br>    results = {&quot;raptor&quot;: 0, &quot;flat&quot;: 0}<br><br>    for item in eval_set:<br>        raptor_context = &quot; &quot;.join(query_raptor(collection_raptor, item[&quot;question&quot;]))<br>        flat_context = &quot; &quot;.join(query_raptor(collection_flat, item[&quot;question&quot;]))<br>        raptor_hit = all(fact.lower() in raptor_context.lower() for fact in item[&quot;expected_facts&quot;])<br>        flat_hit = all(fact.lower() in flat_context.lower() for fact in item[&quot;expected_facts&quot;])<br>        results[&quot;raptor&quot;] += int(raptor_hit)<br>        results[&quot;flat&quot;] += int(flat_hit)<br>    total = len(eval_set)<br><br>    return {k: v / total for k, v in results.items()}</pre><p>Build your eval set with a deliberate mix:</p><ul><li><strong>Single-fact questions</strong> (should perform similarly on both — this is your sanity check that RAPTOR isn’t regressing simple cases)</li><li><strong>Multi-hop / synthesis questions</strong> (this is where RAPTOR should clearly win)</li><li><strong>Document-level questions</strong> (<strong><em>what’s the overall theme/sentiment of X</em></strong>) that require summary-layer retrieval</li></ul><p>If RAPTOR isn’t outperforming flat retrieval on the synthesis and document-level categories, the tree isn’t summarizing at the right granularity — revisit chunk size or cluster threshold before assuming the architecture itself is wrong.</p><h3>Production considerations</h3><p>A few things the toy pipeline above glosses over that matter once this is running against real traffic:</p><ul><li><strong>Cost accounting</strong>: every layer of the tree costs one LLM summarization call per cluster. For a corpus with N leaf chunks and a branching factor of ~5, expect roughly N/5 + N/25 + … summarization calls total — budget for this at indexing time, not query time.</li><li><strong>Incremental updates</strong>: rebuilding the whole tree on every document update doesn’t scale. Track which leaf chunks changed, identify the clusters (and therefore summary nodes) they belong to, and only re-summarize the affected subtree up to the root.</li><li><strong>Failure handling</strong>: LLM summarization calls can fail or time out mid-build. Checkpoint progress layer-by-layer so a failure doesn’t force a full rebuild from leaf chunks.</li><li><strong>Latency</strong>: query-time latency is unaffected by tree depth — you’re still doing a single top-k vector search. The cost of RAPTOR is entirely at indexing time, which is why it fits offline/batch corpora (filings, wikis, contracts) better than corpora updated every few seconds.</li><li><strong>Metadata for debugging</strong>: store layer and children references (or IDs) in your vector store metadata. When retrieval returns a summary node, being able to trace back to its source chunks is invaluable for debugging hallucinated or overly compressed summaries.</li></ul><h3>Practical tuning notes</h3><ul><li><strong>Chunk size</strong>: smaller chunks (300–500 tokens) give clustering more granularity to work with. Too large and everything looks similar.</li><li><strong>Cluster threshold</strong>: lowering the soft-assignment threshold lets more chunks belong to multiple clusters, which helps when topics overlap — but raises compute cost.</li><li><strong>Max layers</strong>: 3–4 layers is typically enough even for large corpora. More layers add retrieval flexibility but also indexing cost and summarization latency.</li><li><strong>Summarization model</strong>: use a cheaper/faster model for lower layers (many small summaries) and reserve your best model for the root layers (fewer, higher-stakes summaries).</li><li><strong>Re-indexing</strong>: RAPTOR trees need to be rebuilt when the underlying corpus changes significantly. For frequently updated corpora, consider rebuilding only the affected subtree rather than the whole index.</li></ul><h3>When RAPTOR is worth the added complexity</h3><p>RAPTOR adds real cost: extra embedding calls, clustering compute, and LLM summarization calls per layer, all at indexing time. It’s worth it when your corpus has:</p><ul><li>Long documents where questions span multiple sections</li><li>Multi-document corpora where answers require cross-document synthesis</li><li>A mix of granular (<strong><em>what number…</em></strong>) and broad (<strong><em>summarize…</em></strong>) queries against the same index</li></ul><p>If your use case is genuinely single-fact lookup over short documents, flat retrieval is simpler and cheaper — save the hierarchy for where it earns its keep.</p><h3>Putting it all together</h3><p>Here’s the full pipeline end-to-end, from raw documents to an answered question:</p><pre>def run_raptor_pipeline(documents, question):<br>    # 1-2: chunk and embed<br>    chunks = chunk_documents(documents)<br># 3-5: recursively cluster, summarize, build tree<br>    all_nodes = build_raptor_tree(chunks)<br>    # 6: index every layer together<br>    collection = build_index(all_nodes)<br>    # 7: query and answer<br>    return answer_question(collection, question)<br># Example usage<br>documents = [open(f&quot;docs/{f}&quot;).read() for f in os.listdir(&quot;docs&quot;)]<br>answer = run_raptor_pipeline(documents, &quot;What are the common risk factors across these filings?&quot;)<br>print(answer)</pre><p>For a corpus of a few hundred pages, expect indexing to take minutes, not seconds — most of that time is LLM summarization calls, not clustering. Parallelising summarization calls within a layer (they’re independent of each other) is the easiest win if indexing time becomes a bottleneck.</p><h3>Further reading</h3><ul><li>The original RAPTOR paper (Sarthi et al.) for the full method and benchmark results against baseline retrieval</li><li>LangChain and LlamaIndex both ship reference RAPTOR implementations if you want to compare against a battle-tested version before building your own</li><li>If you’re evaluating against production traffic, log which layer each retrieved node came from — over time this tells you whether your query mix is mostly granular, mostly broad, or genuinely needs both, which should inform how many layers you actually need</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5fe6cad3d71" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 4: From Concept to Production: Implementing Multi-Representation Indexing]]></title>
            <link>https://afafathar.medium.com/part-4-from-concept-to-production-implementing-multi-representation-indexing-ddaf92d93bac?source=rss-486083ff911c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddaf92d93bac</guid>
            <category><![CDATA[deepeval]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[raga]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Afaf Athar]]></dc:creator>
            <pubDate>Sat, 11 Jul 2026 13:06:00 GMT</pubDate>
            <atom:updated>2026-07-11T13:06:00.810Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the past three parts, we’ve challenged the assumption that long-context models eliminate the need for retrieval. We’ve explored why traditional chunk-based RAG struggles with complex enterprise knowledge bases and how <strong>Multi-Representation Indexing (MRI)</strong> separates retrieval from generation by optimising each stage independently. The architecture is compelling in theory — but how does it translate into a real-world system?</p><p>In this final section, we’ll walk through a practical implementation, discuss operational best practices, highlight common pitfalls, and explore where retrieval architectures are headed next.</p><h3>Building the Pipeline</h3><p>Although the underlying idea is simple, a production implementation consists of several distinct stages that work together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*kWgYyuvCnlYMI9kRRS8Y3Q.png" /><figcaption>RAG Pipeline</figcaption></figure><p>Notice that embeddings are <strong>never generated from the entire production corpus during inference</strong>.</p><p>The expensive work happens during ingestion. Inference becomes lightweight:</p><ul><li>retrieve</li><li>fetch</li><li>reason</li></ul><p>This separation is one of the biggest contributors to scalability.</p><h3>Example Implementation with LangChain</h3><p>A simplified ingestion workflow might look like this:</p><pre>from langchain_openai import ChatOpenAI<br>from langchain_openai import OpenAIEmbeddings<br><br>llm = ChatOpenAI(model=&quot;gpt-4.1&quot;)<br>embeddings = OpenAIEmbeddings(model=&quot;text-embedding-3-large&quot;)<br><br>document = load_document()<br><br>summary = llm.invoke(<br>    f&quot;&quot;&quot;<br>    Create a retrieval-optimized representation.<br><br>    Include:<br>    - entities<br>    - concepts<br>    - business terminology<br>    - likely search intents<br><br>    Document:<br><br>    {document}<br>    &quot;&quot;&quot;<br>)<br><br>vector = embeddings.embed_query(summary.content)<br><br>vector_store.add(<br>    embedding=vector,<br>    metadata={<br>        &quot;doc_id&quot;: doc_id,<br>        &quot;version&quot;: version,<br>        &quot;department&quot;: department<br>    }<br>)<br><br>object_store.upload(doc_id, document)</pre><p>The important observation here is that <strong>the vector store never contains the full document</strong>. Instead, it stores a compact representation that points back to the source of truth through metadata.</p><h3>Retrieval Workflow</h3><p>At query time, the process becomes straightforward:</p><pre>query_embedding = embeddings.embed_query(user_query)<br><br>results = vector_store.search(<br>    embedding=query_embedding,<br>    top_k=10<br>)<br><br>doc_ids = [r.metadata[&quot;doc_id&quot;] for r in results]<br><br>documents = object_store.fetch(doc_ids)<br><br>answer = llm.invoke(<br>    build_prompt(<br>        query=user_query,<br>        documents=documents<br>    )<br>)</pre><p>Notice what the language model receives: Not chunks. Not summaries. The complete original documents. The summaries disappear after retrieval has done its job.</p><h3>Implementation with LlamaIndex</h3><p>LlamaIndex offers similar capabilities through document transformations and indexing pipelines.</p><p>A typical workflow looks like:</p><pre>                          documents<br>                          <br>                              ↓<br>                          <br>                          Transformation Pipeline<br>                          <br>                              ↓<br>                          <br>                          Retrieval Summary<br>                          <br>                              ↓<br>                          <br>                          Embedding<br>                          <br>                              ↓<br>                          <br>                          Vector Index<br>                          <br>                              ↓<br>                          <br>                          Retriever<br>                          <br>                              ↓<br>                          <br>                          Parent Document Loader<br>                          <br>                              ↓<br>                          <br>                          Response Synthesizer</pre><p>One advantage of LlamaIndex is its support for composing retrieval pipelines with node parsers, retrievers, rerankers, and response synthesisers, making it straightforward to experiment with different indexing strategies while keeping the overall architecture modular.</p><p>Regardless of the framework, the underlying principle remains the same: <strong>optimise the representation used for retrieval without compromising the context used for generation.</strong></p><h3>Best Practices for Production Systems</h3><p>Through deploying retrieval systems across evolving knowledge bases, a few practices consistently prove valuable.</p><h3>Treat Object Storage as the Source of Truth</h3><p>Never overwrite original documents. If a better embedding model becomes available, you should be able to regenerate representations without needing to reconstruct the underlying data. Immutable storage simplifies debugging and ensures reproducibility.</p><h3>Version Everything</h3><p>Many teams version their models but forget to version their embeddings. Every indexed representation should record:</p><ul><li>embedding model</li><li>LLM version used for summary generation</li><li>ingestion timestamp</li><li>document version</li><li>prompt version</li></ul><p>When retrieval quality changes, version metadata makes it much easier to identify the cause.</p><h3>Cache Retrieval Representations</h3><p>Generating retrieval summaries requires LLM inference, which can be expensive for large document collections.</p><p>Rather than regenerating summaries every time you rebuild your index, cache them and refresh only when the source document changes. This reduces cost while keeping the retrieval layer consistent.</p><h3>Monitor Retrieval Failures</h3><p>When a user reports an incorrect answer, avoid assuming the language model is responsible.</p><p>Instead, investigate the pipeline systematically:</p><ol><li>Was the correct document retrieved?</li><li>Was it ranked highly enough?</li><li>Was the correct context passed to the LLM?</li><li>Did the model faithfully answer using that context?</li></ol><p>Breaking the pipeline into stages makes failures easier to diagnose and resolve.</p><h3>Common Pitfalls</h3><p>Even well-designed retrieval architectures can underperform if a few key principles are overlooked.</p><h3>Using Summaries for Generation</h3><p>Retrieval summaries are intentionally compressed. They should help identify the right document not replace it. Generating responses directly from summaries often strips away the nuance, supporting evidence, and surrounding context needed for accurate answers.</p><h3>Ignoring Metadata</h3><p>Semantic search is powerful, but it should not operate in isolation. Metadata filters such as document type, publication date, language, and user permissions can dramatically improve retrieval precision while reducing unnecessary search.</p><h3>Optimising Only for Benchmarks</h3><p>A retrieval pipeline that performs well on a static evaluation dataset may struggle with evolving enterprise content. Real-world knowledge bases change continuously. Evaluation should therefore be integrated into the development lifecycle rather than treated as a one-time exercise.</p><h3>Where Retrieval Is Heading</h3><p>Retrieval architectures are evolving rapidly. While summary-based indexing is a powerful approach today, future systems are likely to combine multiple retrieval representations simultaneously.</p><p>A single document may be represented as:</p><ul><li>a semantic summary</li><li>extracted entities</li><li>hypothetical user questions</li><li>structured metadata</li><li>a knowledge graph</li><li>and multimodal embeddings</li></ul><p>At query time, these complementary representations can be searched together, improving recall across a wide range of query types — from natural language questions to exact identifier lookups.</p><p>This reflects a broader trend in AI engineering: <strong>retrieval is becoming an orchestration problem rather than a single-vector search problem.</strong></p><h3>Final Thoughts</h3><p>Long-context models have fundamentally changed how we think about Retrieval-Augmented Generation but they haven’t made retrieval obsolete. Instead, they’ve shifted its purpose. The role of retrieval is no longer to squeeze as much information as possible into a limited prompt.</p><p>Its role is to ensure the language model spends its context window reasoning over the <strong>right information</strong>, rather than searching through irrelevant information. That distinction may seem subtle, but it changes how we design production AI systems. Multi-Representation Indexing embraces this shift by separating search from generation.</p><p>Retrieval becomes faster because it operates on compact, retrieval-optimized representations. Generation becomes more reliable because it operates on complete, unmodified source documents. As enterprise knowledge bases continue to grow and long-context models become even more capable, the winning architectures won’t be those that retrieve the most text.</p><p>They’ll be the ones that retrieve the <strong>right</strong> text efficiently, consistently, and with measurable quality. The future of RAG isn’t about abandoning retrieval. It’s about making retrieval intelligent enough that the language model can focus on what it does best: <strong>Reasoning.</strong></p><h3>References &amp; Further Reading</h3><p>If readers want to explore these ideas further, consider linking to:</p><ul><li>The original <strong>RAG</strong> paper by Patrick Lewis et al. (2020)</li><li><strong>RAGAs</strong>: Automated Evaluation of Retrieval-Augmented Generation</li><li><strong>DeepEval</strong> documentation</li><li>LangChain documentation on Multi-Vector and Parent Document Retrievers</li><li>LlamaIndex documentation on document transformations and retrievers</li><li>Research on long-context LLMs and retrieval-aware architectures</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddaf92d93bac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 3: From Architecture to Production: Building RAG Systems You Can Trust]]></title>
            <link>https://afafathar.medium.com/part-3-from-architecture-to-production-building-rag-systems-you-can-trust-1f9c2e05ec60?source=rss-486083ff911c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f9c2e05ec60</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[metrics]]></category>
            <dc:creator><![CDATA[Afaf Athar]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 15:26:00 GMT</pubDate>
            <atom:updated>2026-07-10T15:26:00.914Z</atom:updated>
            <content:encoded><![CDATA[<p>Designing a better retrieval architecture is only half the challenge. The other half begins the moment your application reaches production.</p><p>A proof-of-concept that works on a handful of documents can quickly struggle when faced with thousands of documents, millions of queries, continuously changing knowledge bases, and users who ask questions in ways you never anticipated. At that point, retrieval quality becomes just as important as the capabilities of the language model itself. One lesson becomes obvious after deploying enterprise RAG systems:</p><blockquote><strong>Most incorrect answers are not caused by the LLM. They originate much earlier in the retrieval pipeline.</strong></blockquote><p>If the wrong document is retrieved, even the most capable language model has little chance of producing the correct answer. This is why production-grade RAG systems invest heavily in improving retrieval before a prompt ever reaches the LLM.</p><h3>Retrieval Is No Longer Just Vector Search</h3><p>Early RAG architectures relied almost exclusively on semantic similarity. The process was straightforward:</p><ol><li>Embed the user’s query.</li><li>Search the vector database.</li><li>Retrieve the nearest neighbors.</li><li>Generate an answer.</li></ol><p>While effective for many use cases, vector search alone has limitations. Imagine a user searching for:</p><blockquote><strong>SOC 2 Type II policy updated in January 2025.</strong></blockquote><p>A purely semantic search understands the intent behind the question, but it may not prioritise the most recent policy document. Similarly, a query like:</p><blockquote><strong>PCI DSS v4.0 encryption requirements.</strong></blockquote><p>contains specific keywords and version numbers that exact lexical matching often handles better than semantic embeddings. This is why modern retrieval systems rarely depend on a single retrieval strategy.</p><h3>Hybrid Search: Combining Semantic and Lexical Retrieval</h3><p>Production systems increasingly adopt <strong>hybrid search</strong>, combining dense vector embeddings with traditional keyword-based retrieval.</p><p>Instead of choosing between semantic similarity and lexical matching, they leverage both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*IQx76PVQ3DaynG3j9YBwEg.png" /><figcaption>Hybrid Retrieval</figcaption></figure><p>Dense retrieval captures meaning, intent, and conceptual similarity. Lexical retrieval excels at finding:</p><ul><li>product names</li><li>API endpoints</li><li>version numbers</li><li>IDs</li><li>acronyms</li><li>regulatory standards</li><li>exact terminology.</li></ul><p>Together, they significantly improve recall.</p><p>Many modern vector databases — including Pinecone, Weaviate, ElasticSearch, Azure AI Search, and OpenSearch — support hybrid retrieval natively, making this approach increasingly common in production environments.</p><h3>Rerankers: The Unsung Hero of Modern RAG</h3><p>Even with hybrid retrieval, the top-ranked documents are not always the best documents. Embedding similarity measures closeness in vector space — not true relevance.</p><p>That’s where rerankers come in. A reranker evaluates each retrieved document <strong>in the context of the user’s query</strong>, assigning a more accurate relevance score before generation begins. The workflow becomes:</p><pre>                            User Query<br>                                  │<br>                                  ▼<br>                            Hybrid Retrieval<br>                                  │<br>                                  ▼<br>                            Top 20 Documents<br>                                  │<br>                                  ▼<br>                            Cross-Encoder Reranker<br>                                  │<br>                                  ▼<br>                            Top 5 Documents<br>                                  │<br>                                  ▼<br>                            Fetch Full Documents<br>                                  │<br>                                  ▼<br>                            LLM Generation</pre><p>Unlike embedding models, which encode queries and documents independently, rerankers process them together. This allows them to capture subtler relationships and often leads to noticeably better retrieval precision.</p><p>Although reranking adds a small amount of latency, the improvement in answer quality is often well worth the trade-off — especially for enterprise search applications where accuracy is more important than shaving a few milliseconds off response time.</p><h3>Metadata Is More Powerful Than Most Teams Realize</h3><p>One of the most overlooked aspects of production RAG is metadata. Many teams focus entirely on embeddings, yet metadata often eliminates large portions of irrelevant search space before semantic search even begins.</p><p>During ingestion, every document should include structured metadata such as:</p><ul><li>document type</li><li>department</li><li>author</li><li>publication date</li><li>version</li><li>security classification</li><li>language</li><li>region</li><li>access permissions.</li></ul><p>Instead of searching across an entire knowledge base, the system can first narrow the candidate set.</p><p>For example:</p><ul><li>Search only engineering documentation.</li><li>Exclude archived documents.</li><li>Return policies published after January 2025.</li><li>Retrieve only documents the current user is authorized to access.</li></ul><p>This combination of structured filtering and semantic retrieval is both faster and more accurate than relying on embeddings alone.</p><h3>Observability: You Can’t Improve What You Can’t Measure</h3><p>One of the biggest mistakes teams make is treating retrieval as a black box. Users report that an answer is incorrect, and engineers immediately start tuning prompts or changing models.</p><p>But the first question should always be:</p><blockquote><strong><em>Where did the failure occur?</em></strong></blockquote><ul><li>Was the wrong document retrieved? Did the reranker discard the correct result?</li><li>Did the language model hallucinate despite receiving the right context?</li><li>These are fundamentally different problems that require different solutions.</li></ul><p>A useful way to think about a RAG system is as two connected but independent pipelines:</p><pre>                            Retrieval Pipeline<br>                                    │<br>                                    ▼<br>                            Generation Pipeline</pre><p>Each stage deserves its own monitoring, metrics, and debugging tools. Without this separation, teams often spend weeks optimising prompts for what is actually a retrieval problem.</p><h3>Evaluating Retrieval Separately from Generation</h3><p>Traditional software systems are tested using unit tests and integration tests. RAG systems require something more nuanced. A correct answer depends on two independent components:</p><ol><li>Did retrieval provide the right context?</li><li>Did the LLM generate a faithful response from that context?</li></ol><p>Measuring only the final answer hides where failures originate. Instead, evaluate retrieval and generation independently.</p><h3>Retrieval Metrics</h3><p>The first stage asks:</p><blockquote><strong><em>Did we retrieve the right information?</em></strong></blockquote><p>Useful metrics include:</p><ul><li><strong>Context Precision</strong> — How much of the retrieved context is actually relevant?</li><li><strong>Context Recall</strong> — Did retrieval surface all the information needed to answer the question?</li><li><strong>Hit Rate</strong> — How often does the correct document appear in the retrieved set?</li><li><strong>Mean Reciprocal Rank (MRR)</strong> — How highly is the correct document ranked?</li><li><strong>Normalized Discounted Cumulative Gain (NDCG)</strong> — How well are relevant documents ordered?</li></ul><p>These metrics tell you whether your search pipeline is doing its job before the LLM is involved.</p><h3>Generation Metrics</h3><p>Once the correct context has been retrieved, evaluate the model’s response.</p><p>Key metrics include:</p><ul><li><strong>Faithfulness</strong> — Is every claim supported by the retrieved documents?</li><li><strong>Answer Relevancy</strong> — Does the answer actually address the user’s question?</li><li><strong>Context Utilisation</strong> — Is the model using the retrieved context effectively?</li><li><strong>Hallucination Rate</strong> — How often does the model invent unsupported information?</li></ul><p>Separating these metrics makes debugging far more systematic.</p><h3>Using RAGAs and DeepEval in Practice</h3><p>Two popular open-source frameworks make this process much easier:</p><ul><li><strong>RAGAs</strong> focuses on evaluating retrieval-augmented generation pipelines without requiring manually labeled answers. It provides metrics such as Context Precision, Context Recall, Faithfulness, and Answer Relevancy, helping teams assess both retrieval quality and response quality.</li><li><strong>DeepEval</strong> extends evaluation by allowing developers to create automated test suites, define custom evaluation criteria, and integrate quality checks into continuous integration pipelines. This makes it easier to catch regressions before they reach production.</li></ul><p>Rather than running evaluations only during development, many teams incorporate these frameworks directly into their deployment workflows.</p><p>For example:</p><pre>                            New Knowledge Base<br>                                    │<br>                                    ▼<br>                            Generate Retrieval Representations<br>                                    │<br>                                    ▼<br>                            Run Automated Retrieval Tests<br>                                    │<br>                                    ▼<br>                            Evaluate Faithfulness<br>                                    │<br>                                    ▼<br>                            Quality Threshold Met?<br>                                   / \<br>                                 Yes  No<br>                                  │    │<br>                                  ▼    ▼<br>                             Deploy  Investigate</pre><p>By treating evaluation as a continuous process rather than a one-time benchmark, organisations can maintain confidence that changes to documents, embedding models, or prompts do not silently degrade system performance.</p><h3>Production Is an Iterative Process</h3><p>No retrieval architecture is perfect from day one. As your knowledge base evolves, so will user behavior. New document types will appear. Terminology will change. Products will be renamed. Embedding models will improve.</p><p>The most successful RAG systems are not those with the most sophisticated prompts or the largest language models. They are the systems that continuously measure, evaluate, and refine retrieval quality. Because at scale, <strong>retrieval is not a one-time engineering task — it is an ongoing capability.</strong></p><h3><em>Coming Up in Part 4</em></h3><p><em>We’ve now explored why traditional chunking struggles, how Multi-Representation Indexing separates search from generation, and what it takes to build a production-ready retrieval pipeline.</em></p><p><em>In the final part, we’ll bring everything together with implementation guidance, LangChain and LlamaIndex examples, best practices for maintaining large-scale RAG systems, common pitfalls to avoid, and a look at where retrieval architectures are heading as long-context models continue to evolve.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f9c2e05ec60" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 2: Designing a Production-Grade Multi-Representation Indexing Pipeline]]></title>
            <link>https://afafathar.medium.com/part-2-designing-a-production-grade-multi-representation-indexing-pipeline-b9718b0c73d5?source=rss-486083ff911c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9718b0c73d5</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[indexing]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Afaf Athar]]></dc:creator>
            <pubDate>Thu, 09 Jul 2026 17:26:01 GMT</pubDate>
            <atom:updated>2026-07-09T17:26:01.045Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Search and generation are fundamentally different problems.</strong></blockquote><p>Yet most traditional RAG pipelines still assume that the same piece of text should serve both purposes. A document is chunked, embedded, stored in a vector database, retrieved during inference, and finally passed to the LLM for answer generation.</p><p>The workflow is simple — and for many use cases, it’s sufficient. But as document collections grow larger and user queries become more complex, this <strong>one representation fits all </strong>approach starts to reveal its limitations.</p><p>A chunk that is ideal for embedding isn’t necessarily ideal for reasoning. Conversely, the complete document that provides the richest context for generation often contains too much information to retrieve efficiently. Instead of forcing a single representation to do both jobs, we can optimize each stage independently.</p><p>That’s the core principle behind <strong>Multi-Representation Indexing (MRI).</strong></p><h3>What Is Multi-Representation Indexing?</h3><p>Multi-Representation Indexing is a retrieval architecture where a document is represented in multiple ways, each optimized for a specific purpose.</p><p>Rather than indexing the original document directly, the system creates specialized representations that improve retrieval quality while preserving the original document for answer generation.</p><p>Think of it like organising a library. A book has:</p><ul><li>a title</li><li>a table of contents</li><li>an abstract</li><li>keywords</li><li>an index</li><li>the complete book itself</li></ul><p>Each representation serves a different purpose. You don’t search the full text of every book to find one about distributed systems. You search its metadata, title, abstract, or index. Only after identifying the correct book do you start reading it.</p><p>Enterprise RAG systems benefit from the same philosophy. Instead of asking embeddings to understand hundreds of pages of documentation, we create a representation specifically designed for semantic retrieval.</p><h3>One Document: Multiple Representations</h3><p>A production system might generate several retrieval-friendly views of the same document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*pXrbNHym1Mte-NRS4aRhgQ.png" /></figure><p>Each representation answers a different retrieval challenge. For example: a user searching for <em>“OAuth refresh token expiration”</em> might match:</p><ul><li>keywords</li><li>extracted entities</li><li>a generated hypothetical question</li><li>or a semantic summary</li></ul><p>All of them ultimately point to the same source document. This dramatically improves recall without requiring the LLM to process unnecessary information.</p><h3>Why Retrieval Summaries Work Surprisingly Well</h3><p>One of the most effective representations is the <strong>retrieval summary</strong>. At first glance, this feels counterintuitive. Wouldn’t embedding the entire document preserve more information.</p><p>Surprisingly, not always. Embedding models work best when the input contains a dense concentration of meaningful concepts. Unfortunately, enterprise documents rarely look like that.</p><p>A typical technical document contains:</p><ul><li>repeated explanations</li><li>navigation headers</li><li>formatting artifacts</li><li>examples</li><li>implementation snippets</li><li>changelogs</li><li>legal disclaimers</li><li>tables</li><li>references</li><li>and a significant amount of redundant language</li></ul><p>All of this contributes to semantic noise. A retrieval summary removes that noise. Instead of describing <em>everything</em>, it focuses on what users are likely to search for:</p><ul><li>important entities</li><li>business terminology</li><li>technical concepts</li><li>domain-specific vocabulary</li><li>relationships</li><li>workflows</li><li>and constraints</li></ul><p>In other words:</p><blockquote><strong>A retrieval summary isn’t optimized for readability. It’s optimized for discoverability.</strong></blockquote><p>That distinction makes a significant difference. The embedding generated from a carefully crafted summary often captures the semantic intent of a document more effectively than embedding dozens of isolated chunks.</p><h3>Designing the Pipeline</h3><p>Once we separate search from generation, the overall architecture becomes much cleaner. Instead of storing chunks as the primary retrieval unit, we store <strong>representations</strong>. The original document remains untouched.</p><p>The pipeline looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*klTdIDf8ckrRnOzhqYLEOw.png" /></figure><p>Notice something important. The vector database never stores the document itself. It stores only the retrieval representation.</p><p>The original document lives elsewhere: typically object storage such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. The vector database simply acts as a fast routing layer.</p><h3>Step 1: Store the Original Document</h3><p>Every ingestion pipeline should begin by storing the complete document exactly as it was received. Whether it’s:</p><ul><li>PDFs</li><li>Confluence pages</li><li>SharePoint documents</li><li>Jira tickets</li><li>customer support conversations</li><li>policy manuals</li><li>or engineering documentation</li></ul><p>the original version should remain immutable. Assign each document a unique identifier such as doc_id. This becomes the primary key connecting every representation back to its source. Keeping the original document unchanged provides several advantages:</p><ul><li>reproducibility</li><li>easier debugging</li><li>version control</li><li>and future re-indexing without data loss</li></ul><p>Object storage effectively becomes the <strong>source of truth</strong> for your knowledge base.</p><h3>Step 2: Generate Retrieval Representations</h3><p>Rather than immediately splitting documents into chunks, the next stage uses an LLM to generate one or more retrieval-friendly representations.</p><p>A prompt might ask the model to produce:</p><ul><li>a concise semantic summary</li><li>important entities</li><li>domain-specific terminology</li><li>likely user questions</li><li>related technologies</li><li>business processes</li><li>and key constraints</li></ul><p>Unlike user-facing summaries, these outputs are designed specifically to improve retrieval. A typical retrieval summary might look something like:</p><blockquote>This document explains OAuth 2.0 authentication, access tokens, refresh tokens, token expiration, JWT validation, authorisation flows, identity providers, and session management.</blockquote><p>Notice how it intentionally concentrates searchable concepts.</p><h3>Step 3: Embed Only What Matters</h3><p>Once the retrieval summary has been generated, it is passed to an embedding model. The resulting vector is stored in the vector database alongside metadata such as:</p><ul><li>doc_id</li><li>document type</li><li>author</li><li>creation date</li><li>department</li><li>version</li><li>security permissions</li></ul><p>This metadata becomes incredibly valuable later. Modern retrieval systems rarely rely solely on semantic similarity. Metadata filtering allows us to answer questions like:</p><ul><li>Search only engineering documents</li><li>Exclude archived policies</li><li>Retrieve documents published after 2024</li><li>Restrict results based on user permissions</li></ul><p>Semantic search and structured filtering complement each other remarkably well.</p><h3>Step 4: Retrieve First, Read Later</h3><p>When a user submits a question, the workflow changes significantly from traditional RAG. Instead of retrieving chunks to answer the question directly, the system retrieves the <strong>best document identifiers</strong>.</p><p>The sequence looks like this:</p><ol><li>Embed the user query.</li><li>Search against the retrieval summaries.</li><li>Identify the most relevant documents.</li><li>Retrieve their doc_ids.</li><li>Fetch the complete source documents.</li><li>Pass the original documents to the LLM.</li></ol><p>Notice that the retrieval summary has already completed its job. It never reaches the language model. Its only responsibility is helping the system locate the correct document efficiently. Generation always happens using the original source material.</p><p>This separation is one of the biggest architectural differences between traditional chunk-based RAG and Multi-Representation Indexing.</p><h3>Why This Improves Faithfulness</h3><p>One of the biggest sources of hallucination in RAG systems isn’t the language model itself. It’s missing context. If retrieval returns only fragments of a document, the model often has to infer relationships that were present in the original text but lost during chunking.</p><p>By retrieving the complete document after search succeeds, the model gains access to:</p><ul><li>surrounding context</li><li>complete workflows</li><li>section hierarchy</li><li>tables</li><li>definitions</li><li>and supporting explanations.</li></ul><p>Instead of reconstructing a story from scattered pieces, the LLM can reason over the source as it was originally written. The result is often more coherent, more accurate, and easier to verify.</p><p><em>In </em><strong><em>Part 3</em></strong><em>, we’ll move from architecture to production engineering. We’ll explore hybrid search, rerankers, metadata filtering, observability, and how to evaluate retrieval and generation separately using </em><strong><em>RAGAs</em></strong><em>, </em><strong><em>DeepEval</em></strong><em>, and other metrics that help ensure your RAG system remains accurate as it evolves.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9718b0c73d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1: Beyond Naive Chunking: Why Production RAG Needs Multi-Representation Indexing]]></title>
            <link>https://afafathar.medium.com/part-1-beyond-naive-chunking-why-production-rag-needs-multi-representation-indexing-d34d410c80c1?source=rss-486083ff911c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d34d410c80c1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[multi-indexing]]></category>
            <category><![CDATA[agentic-ai-architecture]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Afaf Athar]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 17:16:30 GMT</pubDate>
            <atom:updated>2026-07-08T17:35:41.628Z</atom:updated>
            <content:encoded><![CDATA[<p>Every few months, the AI community declares <strong>Retrieval-Augmented Generation (RAG)</strong> dead.</p><p>When GPT-4 Turbo introduced larger context windows, people predicted that retrieval pipelines would soon become obsolete. Then came models capable of processing hundreds of thousands — and eventually millions — of tokens. Once again, the same conclusion echoed across blog posts, conference talks, and social media:</p><blockquote>“Why build retrieval systems when the model can simply read everything?”</blockquote><p>On paper, the argument sounds reasonable.</p><p>If an LLM can ingest an entire product manual, a company’s internal documentation, or months of customer support conversations in a single prompt, why bother maintaining vector databases, embeddings, chunking strategies, or rerankers?</p><p>Unfortunately, this assumption falls apart the moment you move from a demo to a production system. The real limitation isn’t how much information an LLM can <strong>accept</strong>. It’s how effectively it can <strong>locate, prioritize, and reason over the right information</strong>. Those are three fundamentally different problems. Long-context models solve the first. Production retrieval systems solve the other two. And that’s precisely why retrieval is evolving — not disappearing.</p><h3>The Evolution of RAG</h3><p>To understand where retrieval is heading, it’s worth revisiting why RAG existed in the first place.</p><p>Early language models had context windows measured in just a few thousand tokens. Feeding entire documents into a prompt simply wasn’t possible. The only practical solution was to split documents into smaller pieces, convert those pieces into embeddings, and retrieve the most relevant chunks during inference.</p><p>The architecture looked something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h-UlLROdaFuV-9Ob9FZM0w.png" /><figcaption>RAG Architecture</figcaption></figure><p>For its time, this architecture was revolutionary. Instead of forcing the model to memorise proprietary knowledge during training, organisations could dynamically retrieve relevant information at inference time. Customer support systems could answer questions using internal documentation.</p><ul><li>Legal assistants could search contracts.</li><li>Healthcare copilots could retrieve clinical guidelines.</li><li>Enterprise search suddenly became conversational.</li></ul><p>RAG transformed static language models into systems capable of reasoning over external knowledge.</p><h3>Why Chunking Was Never the Goal</h3><p>Over time, many engineers began treating chunking as a best practice. In reality, it was always an engineering compromise.Embedding models accepted limited input sizes. <strong><em>Vector databases worked best with fixed-length vectors. Context windows were constrained. Chunking solved those constraints — but it introduced new ones. Imagine indexing a software architecture document.</em></strong></p><p>A section describing authentication might begin with an overview, continue with a sequence diagram, explain OAuth flows, discuss token expiration, and conclude with implementation examples.</p><blockquote>After chunking, that coherent explanation becomes several unrelated fragments. One chunk might contain the sequence diagram. Another contains the explanation of refresh tokens. A third contains implementation details. Each fragment is embedded independently, unaware of its relationship to the others.</blockquote><p>When a user asks:</p><blockquote>“How does authentication work in this platform?”</blockquote><p>Vector search may retrieve three technically relevant chunks — but none provide the complete picture.</p><p>The LLM is then forced to reconstruct the missing context. Sometimes it succeeds. Sometimes it hallucinates the connections. Sometimes it simply answers incompletely. The problem isn’t the language model. The problem is that we’ve asked it to assemble a puzzle after we’ve already thrown away the box.</p><h3>The Million-Token Illusion</h3><p>At first glance, long-context models appear to solve this problem. Instead of retrieving fragments, why not provide the entire document? After all, today’s frontier models can comfortably process hundreds of pages in a single request. In practice, however, larger context windows introduce a different set of challenges.</p><h3>1. Reasoning doesn’t scale linearly</h3><p>Reading isn’t the same as reasoning. Finding one fact inside a million tokens is relatively straightforward. Synthesizing evidence scattered across dozens of sections is significantly harder. As context grows, models spend more attention locating relevant information before they can begin reasoning about it. The bottleneck shifts from <strong>capacity</strong> to <strong>attention allocation</strong>.</p><h3>2. Bigger prompts are expensive</h3><p>Enterprise applications rarely search one document. They search thousands. Feeding entire document collections into every query dramatically increases latency and inference cost. A support chatbot answering hundreds of requests per minute cannot afford to process gigabytes of context for every interaction. Retrieval still matters because it reduces the search space.</p><h3>3. More context can introduce more noise</h3><p>Not every paragraph contributes to the answer. Large documents contain boilerplate, repeated explanations, change-logs, legal notices, examples, appendices, and formatting artifacts. Providing all of that context forces the model to distinguish signal from noise. Ironically, giving the model <em>more</em> information can sometimes reduce answer quality. The challenge isn’t maximizing context. It’s maximizing <strong>relevant context</strong>.</p><h3>Search and Generation Are Different Problems</h3><p>This is the insight that changed how I think about RAG. Most traditional pipelines assume the same representation should serve both retrieval and generation. They embed raw chunks. Retrieve raw chunks. Generate from raw chunks. But search engines and language models optimise for different objectives. A retrieval system wants semantic density. A language model wants rich contextual information. Those aren’t the same thing. Imagine walking into a library.</p><p>If someone asks for a book about distributed systems, the librarian doesn’t hand you every page individually. They search the catalog. Once the correct book is identified, <strong>then</strong> they give you the complete book. The catalog exists to optimise discovery. The book exists to optimize understanding. Those are two different representations of the same knowledge.</p><p>Production RAG systems should work the same way. Instead of forcing one document representation to satisfy both search and generation, we should optimise each stage independently.</p><p>That idea leads us to <strong>Multi-Representation Indexing</strong>.</p><p><a href="https://medium.com/@afafathar/b9718b0c73d5"><em>In Part 2</em></a><em>, we’ll design a production-grade Multi-Representation Indexing pipeline, explore how retrieval summaries improve semantic search, build the end-to-end architecture, and discuss why separating search representations from generation representations leads to more faithful and scalable RAG systems.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d34d410c80c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Deterministic RAG Orchestrators with LangGraph and State Machines]]></title>
            <link>https://afafathar.medium.com/building-deterministic-rag-orchestrators-with-langgraph-and-state-machines-7f5796400da5?source=rss-486083ff911c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f5796400da5</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[agentic-ai-systems]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[orchestration]]></category>
            <category><![CDATA[agentic-ai-architecture]]></category>
            <dc:creator><![CDATA[Afaf Athar]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 07:26:59 GMT</pubDate>
            <atom:updated>2026-07-07T07:26:59.431Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why Flow Engineering is the missing layer between brittle pipelines and unpredictable agents</em></p><h4>1. Introduction: The Limits of Linear RAG and Open Agents</h4><p>If you’ve shipped a Retrieval-Augmented Generation system past the prototype stage, you’ve probably hit the same wall twice — from opposite directions.</p><p><strong>The first wall is the linear pipeline.</strong> The classic RAG flow — embed query, search vector store, stuff top-k chunks into a prompt, generate — is easy to build and easy to demo. It is also brittle in production. It has no mechanism to notice when retrieval failed. If the vector store returns four irrelevant chunks because the query used domain vocabulary that doesn’t exist in your corpus, the pipeline doesn’t know that happened. It generates anyway, confidently, on garbage context. There’s no self-correction because there’s no <em>decision point</em> — just a straight line from input to output.</p><p><strong>The second wall is the open agent.</strong> The natural instinct after hitting pipeline brittleness is to reach for a ReAct-style agent: give an LLM a retrieval tool, a web search tool, and a loop, and let it reason about what to do next. This solves the self-correction problem in theory, but introduces a new one — the agent’s control flow now lives inside the model’s token generation. For open-ended research assistants, that unpredictability is a feature. For a domain-constrained enterprise RAG system — say, one answering questions against a compliance corpus, or a support knowledge base with SLAs — it’s a liability. The same query can take a different path through the same tools on two different runs. Failure loops (the agent re-invoking the same tool with the same arguments) are common. And because the <strong>graph</strong> of possible execution paths is implicit rather than explicit, you can’t audit it, test it, or guarantee its worst-case latency.</p><p>What both extremes are missing is the same thing: <strong>an explicit, inspectable control flow that still has room for conditional logic.</strong></p><p>This is the core idea behind <strong>Flow Engineering</strong> — treating the RAG system not as a single prompt chain, nor as a fully autonomous agent, but as a <strong>state machine</strong>. You define the nodes (discrete operations), you define the edges (transitions), and you define the conditions under which one edge is taken over another. The LLM still makes decisions — a grader model still classifies document relevance, a router still decides whether to fall back to web search — but those decisions are constrained to specific, structured decision points in a graph you designed and can visualize.</p><p><strong>LangGraph</strong> is the current tool of choice for this pattern, because it gives you a low-level primitive — a typed, shared state object passed between nodes — with full control over cycles, conditional edges, and persistence, without hiding the control flow behind a “smart” agent loop.</p><p>In this article we’ll implement a working example of this pattern: a <strong>Corrective RAG (CRAG)</strong> system, where retrieved documents are graded for relevance before generation, and a web search fallback kicks in automatically when the retrieved context isn’t good enough.</p><h4>2. System Design: Architecting the State Machine</h4><p>Before writing any code, it’s worth drawing the system, because the entire value proposition of this architecture is that the diagram <em>is</em> the implementation — there’s no hidden reasoning loop translating intent into execution.</p><h4>The State</h4><p>At the center of a LangGraph application is a single object: the <strong>State</strong>. Every node in the graph receives the current state, does some work, and returns a partial update to that state. Nothing is implicit — if a node needs the retrieved documents, they must exist as a field on the state object; if it needs to accumulate a value, it must explicitly write to it.</p><p>For our CRAG flow, state needs to track:</p><ul><li>The original question</li><li>The list of documents currently held (retrieved, possibly replaced by web search)</li><li>A web_search flag, set by the grader, indicating whether at least one retrieved document failed the relevance check</li><li>The final generation output</li></ul><h4>The Nodes</h4><p>Node Responsibility retrieve Query the vector store for the top-k documents relevant to the question grade_documents For each retrieved document, invoke an LLM constrained to a binary structured output (yes/no) judging relevance to the question web_search If any document was graded irrelevant, query a web search tool and append/replace context generate Produce the final answer using whatever context survived grading (and/or web search)</p><h4>The Conditional Edge</h4><p>The interesting part of this graph isn’t the nodes — it’s the single decision point after grading. This is the entire <strong>correction</strong> in Corrective RAG:</p><pre>retrieve → grade_documents → [conditional] → generate<br>                                    ↓<br>                              web_search → generate</pre><p>If the grader marks <strong>any</strong> document as irrelevant, the graph routes to web_search before generation. If all documents pass, it routes straight to generate. This decision is made by a plain Python function reading a field off the state — not by an LLM improvising a tool call. That&#39;s the deterministic part: the <em>routing logic</em> is code, even though the <em>judgment</em> that feeds it comes from a model.</p><h4>3. Python Implementation: Building the LangGraph</h4><h4>3.1 Defining the State</h4><p>LangGraph state is typically defined as a TypedDict. Each key becomes a slot that nodes read from and write to.</p><pre>from typing import List<br>from typing_extensions import TypedDict<br><br>class GraphState(TypedDict):<br>    &quot;&quot;&quot;<br>    Represents the state of our CRAG graph.<br><br>    Attributes:<br>        question: the user&#39;s original query<br>        generation: the LLM&#39;s final generated answer<br>        web_search: &quot;Yes&quot; / &quot;No&quot; flag set by the grader node<br>        documents: list of retrieved (or web-searched) document contents<br>    &quot;&quot;&quot;<br>    question: str<br>    generation: str<br>    web_search: str<br>    documents: List[str]</pre><p>Keeping this flat and explicit is deliberate. Every node’s function signature becomes self-documenting: you can tell exactly what a node depends on and what it’s allowed to mutate just by reading its type hints.</p><h4>3.2 Building the Nodes</h4><p>Each node is a plain Python function with the signature (state: GraphState) -&gt; dict. It reads what it needs from state and returns only the keys it wants to update — LangGraph merges the returned dict back into the state.</p><p><strong>Retrieve node:</strong></p><pre>def retrieve(state: GraphState) -&gt; dict:<br>    &quot;&quot;&quot;Retrieve documents from the vector store.&quot;&quot;&quot;<br>    question = state[&quot;question&quot;]<br>    documents = retriever.invoke(question)  # your configured vector store retriever<br>    return {&quot;documents&quot;: [doc.page_content for doc in documents], &quot;question&quot;: question}</pre><p><strong>Grader node — the core of CRAG.</strong></p><p>The key architectural trick here is that we don’t ask the LLM to <strong>assess relevance</strong> in free text. We force it into a structured, binary output using a Pydantic schema bound to the model via tool calling / structured output. This is what makes the downstream conditional edge deterministic — the routing function only ever has to check a literal string, never parse prose.</p><pre>from pydantic import BaseModel, Field<br>from langchain_openai import ChatOpenAI  # swap for your provider of choice<br>from langchain_core.prompts import ChatPromptTemplate<br><br>class GradeDocuments(BaseModel):<br>    &quot;&quot;&quot;Binary relevance score for a retrieved document.&quot;&quot;&quot;<br>    binary_score: str = Field(<br>        description=&quot;Document is relevant to the question, &#39;yes&#39; or &#39;no&#39;&quot;<br>    )<br><br>    grade_llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br>    structured_grader = grade_llm.with_structured_output(GradeDocuments)<br><br>    grade_prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;,<br>     &quot;You are a grader assessing relevance of a retrieved document to a user question. &quot;<br>     &quot;Give a binary score &#39;yes&#39; or &#39;no&#39; to indicate whether the document is relevant.&quot;),<br>    (&quot;human&quot;, &quot;Retrieved document:\n\n{document}\n\nUser question: {question}&quot;)<br>])<br><br>    grader_chain = grade_prompt | structured_grader<br><br>def grade_documents(state: GraphState) -&gt; dict:<br>    &quot;&quot;&quot;Grade each retrieved document; flag web_search if any fail relevance.&quot;&quot;&quot;<br>    question = state[&quot;question&quot;]<br>    documents = state[&quot;documents&quot;]<br><br>    filtered_docs = []<br>    web_search_needed = &quot;No&quot;<br><br>for doc in documents:<br>        result = grader_chain.invoke({&quot;question&quot;: question, &quot;document&quot;: doc})<br>        if result.binary_score.lower() == &quot;yes&quot;:<br>            filtered_docs.append(doc)<br>        else:<br>            web_search_needed = &quot;Yes&quot;  # even one failure triggers correction<br><br>return {&quot;documents&quot;: filtered_docs, &quot;web_search&quot;: web_search_needed, &quot;question&quot;: question}</pre><p><strong>Web search fallback node:</strong></p><pre>from langchain_community.tools.tavily_search import TavilySearchResults<br><br>web_search_tool = TavilySearchResults(k=3)<br><br>def web_search(state: GraphState) -&gt; dict:<br>    &quot;&quot;&quot;Append web search results to the document set when retrieval is insufficient.&quot;&quot;&quot;<br>    question = state[&quot;question&quot;]<br>    documents = state[&quot;documents&quot;]<br><br>    search_results = web_search_tool.invoke({&quot;query&quot;: question})<br>    web_content = &quot;\n&quot;.join([r[&quot;content&quot;] for r in search_results])<br>    documents.append(web_content)<br><br>    return {&quot;documents&quot;: documents, &quot;question&quot;: question}</pre><p><strong>Generate node:</strong></p><pre>from langchain_core.output_parsers import StrOutputParser<br>from langchain import hub<br><br>rag_prompt = hub.pull(&quot;rlm/rag-prompt&quot;)<br>gen_llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;, temperature=0)<br>rag_chain = rag_prompt | gen_llm | StrOutputParser()<br><br>def generate(state: GraphState) -&gt; dict:<br>    &quot;&quot;&quot;Generate the final answer from surviving context.&quot;&quot;&quot;<br>    question = state[&quot;question&quot;]<br>    documents = state[&quot;documents&quot;]<br><br>    generation = rag_chain.invoke({&quot;context&quot;: documents, &quot;question&quot;: question})<br>    return {&quot;generation&quot;: generation}</pre><h4>3.3 Wiring the Conditional Edge</h4><p>This is the routing logic that replaces an agent’s improvised tool selection. It’s a plain function that inspects one field on the state and returns a string naming the next node.</p><pre>def decide_to_generate(state: GraphState) -&gt; str:<br>    &quot;&quot;&quot;<br>    Deterministic router: read the grader&#39;s flag, decide the next node.<br>    No LLM call happens here — this is pure control flow.<br>    &quot;&quot;&quot;<br>    web_search_needed = state[&quot;web_search&quot;]<br>  <br>    if web_search_needed == &quot;Yes&quot;:<br>        return &quot;web_search&quot;<br>    return &quot;generate&quot;</pre><h4>3.4 Compiling the Graph</h4><p>Now we assemble everything using LangGraph’s StateGraph. Nodes are registered, a fixed edge connects retrieve → grade_documents, and a <strong>conditional edge</strong> connects grade_documents to either web_search or generate based on decide_to_generate.</p><pre>from langgraph.graph import StateGraph, START, END<br><br>workflow = StateGraph(GraphState)<br><br># Register nodes<br>workflow.add_node(&quot;retrieve&quot;, retrieve)<br>workflow.add_node(&quot;grade_documents&quot;, grade_documents)<br>workflow.add_node(&quot;web_search&quot;, web_search)<br>workflow.add_node(&quot;generate&quot;, generate)<br><br># Fixed edges<br>workflow.add_edge(START, &quot;retrieve&quot;)<br>workflow.add_edge(&quot;retrieve&quot;, &quot;grade_documents&quot;)<br><br># Conditional edge — the correction mechanism<br>workflow.add_conditional_edges(<br>    &quot;grade_documents&quot;,<br>    decide_to_generate,<br>    {<br>        &quot;web_search&quot;: &quot;web_search&quot;,<br>        &quot;generate&quot;: &quot;generate&quot;,<br>    },<br>)<br><br>workflow.add_edge(&quot;web_search&quot;, &quot;generate&quot;)<br>workflow.add_edge(&quot;generate&quot;, END)<br><br>app = workflow.compile()</pre><p>That app object is now an executable graph. You invoke it like any LangChain runnable:</p><pre>result = app.invoke({&quot;question&quot;: &quot;What are the data residency requirements for our EU customers?&quot;})<br>print(result[&quot;generation&quot;])</pre><p>Notice what you get for free here: if you print app.get_graph().draw_mermaid(), LangGraph will generate an actual Mermaid diagram of this exact structure — meaning the diagram in Section 2 isn&#39;t just documentation, it&#39;s a rendering of the compiled object itself.</p><blockquote><strong>There is no drift between the architecture we designed and the code that runs, because the graph <em>is</em> the architecture.</strong></blockquote><h4>4. Tracing and Observability</h4><p>Determinism in the control-flow graph is only half the story — the other half is being able to prove, after the fact, <em>why</em> the graph took the path it took on a specific run. This is where cyclic, conditional flows like CRAG earn their complexity: a linear pipeline barely needs tracing because there’s only one path through it, but a graph with a correction loop has at least two, and you need to know which one fired and why.</p><p>LangSmith integrates with LangGraph with essentially zero setup — setting the standard environment variables is enough to get a full trace of every node invocation:</p><pre>import os<br><br>os.environ[&quot;LANGCHAIN_TRACING_V2&quot;] = &quot;true&quot;<br>os.environ[&quot;LANGCHAIN_API_KEY&quot;] = &quot;your-langsmith-key&quot;<br>os.environ[&quot;LANGCHAIN_PROJECT&quot;] = &quot;crag-deterministic-orchestrator&quot;</pre><p>With tracing enabled, every call to app.invoke() produces a trace tree in LangSmith that mirrors the graph structure exactly: a retrieve span, a grade_documents span containing one child LLM call per document graded, and — critically — you can see the <strong>exact structured output</strong> the grader returned for each document, including which one was scored &quot;no&quot; and triggered the correction.</p><p>This is the auditability payoff. When a stakeholder asks <strong>why did this answer cite a web result instead of our internal knowledge base,</strong> you don’t have to guess or re-run the query hoping to reproduce the behavior. You open the trace, see that document #3 was graded irrelevant with a specific reasoning trace behind the structured call, see the decide_to_generate router read web_search: &quot;Yes&quot;, and see exactly which Tavily results were appended before generation. The failure (or correction) is a first-class, inspectable event — not an emergent property of an opaque agent loop you have to reverse-engineer.</p><p>This is also where the latency argument from the conclusion gets its evidence: because every node is a bounded, named span, you can see in LangSmith’s timeline view exactly how much wall-clock time the grading loop costs versus generation, and optimize the actual bottleneck instead of guessing at <strong>the agent being slow</strong>.</p><h4>5. Conclusion</h4><p>Corrective RAG built on LangGraph’s state machine primitive gives you something that neither a linear pipeline nor an open agent can offer on its own: <strong>a system that can correct itself, and prove exactly how it did so.</strong></p><p>Three concrete benefits fall out of this architecture:</p><ul><li><strong>Reduced latency variance.</strong> Because the control flow is a fixed graph with a bounded number of conditional branches — not an open-ended agent loop that might call a tool three times or ten — worst-case latency is predictable and testable ahead of deployment.</li><li><strong>Complete auditability.</strong> Every decision point (the grader’s binary classification, the router’s branch selection) is a discrete, traceable event with a structured output behind it, not a token sequence you have to parse after the fact to guess what the model <strong>decided</strong>.</li><li><strong>Reliable scaling.</strong> Adding a new correction path — say, a second grader that checks for hallucination in the final generation before returning it to the user — means adding one more node and one more conditional edge to an existing, well-understood graph. It does not mean redesigning a prompt that was implicitly responsible for the entire agent’s judgment.</li></ul><blockquote>The broader lesson is that <strong>agentic</strong> and <strong>deterministic</strong> aren’t opposites — they’re dials. LangGraph lets you decide, node by node, exactly how much of the decision-making you hand to the model and how much you keep as explicit, testable code. For domain-constrained production RAG, that’s usually the right trade to make.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f5796400da5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing Production-Ready Retrieval-Augmented Generation (RAG) Pipelines with LlamaIndex and…]]></title>
            <link>https://afafathar.medium.com/implementing-production-ready-retrieval-augmented-generation-rag-pipelines-with-llamaindex-and-69686f892f1e?source=rss-486083ff911c------2</link>
            <guid isPermaLink="false">https://medium.com/p/69686f892f1e</guid>
            <category><![CDATA[pinecone]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[llamaindex]]></category>
            <dc:creator><![CDATA[Afaf Athar]]></dc:creator>
            <pubDate>Fri, 12 Dec 2025 15:35:43 GMT</pubDate>
            <atom:updated>2025-12-12T15:35:43.840Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Implementing Production-Ready Retrieval-Augmented Generation (RAG) Pipelines with LlamaIndex and Vector Stores</strong></h3><p>As a data scientist specializing in MLOps, I’ve seen countless RAG prototypes fail the transition to production. The primary bottleneck is rarely the LLM itself; it is almost always the fragile, poorly governed data ingestion and retrieval pipeline.</p><p>Retrieval-Augmented Generation (RAG) is the definitive solution for grounding large language models (LLMs) in proprietary, external knowledge, drastically reducing hallucinations and pushing knowledge cutoffs into the present day. However, achieving production-grade stability requires moving beyond simple in-memory indexing.</p><p>This guide focuses on how to leverage <strong>LlamaIndex</strong> — an indispensable data framework for LLM applications — to architect resilient, persistent RAG systems using robust Vector Stores, ensuring high relevance, scalability, and MLOps compliance.</p><h4><strong>The Production RAG Architecture: Decoupling Pipelines</strong></h4><p>A production RAG system must be architecturally split into two highly decoupled pipelines:</p><p>1. <strong>The Indexing/Ingestion Pipeline (Offline):</strong> Handles ETL, document parsing, chunking, embedding, and persistence to the Vector Store. This runs periodically or based on data triggers (e.g., a new document uploaded to S3).</p><p>2. <strong>The Query Pipeline (Online): </strong>Takes the user query, converts it to an embedding, retrieves the top-$K$ contexts from the Vector Store, and feeds them into the LLM prompt. This pipeline must be optimized for low latency.</p><p>LlamaIndex excels at bridging these two components while abstracting away the complexities of the underlying vector store integrations.</p><h4><strong>Ingestion Pipeline: Robust Chunking and Metadata Management</strong></h4><p>The quality of your retrieval is determined by the quality of your chunks. In production, we cannot rely on naive fixed-size chunking. We must prioritize semantics and leverage LlamaIndex’s advanced `NodeParser` capabilities.</p><p>A key engineering step is defining a comprehensive `ServiceContext`. This context object bundles your critical production configurations — the LLM, the embedding model, and the node parser — ensuring consistency across ingestion and query stages.</p><p><strong>Code Block 1: Setting Up the Ingestion Pipeline</strong></p><blockquote>We use the <strong>SentenceSplitter</strong> (a specific <strong>NodeParser</strong>) for intelligent chunking and attach critical metadata, which is key for advanced filtering later on.</blockquote><pre>from llama_index.core.ingestion import IngestionPipeline<br>from llama_index.core.node_parser import SentenceSplitter<br>from llama_index.core import ServiceContext, Settings<br>from llama_index.core.readers import SimpleDirectoryReader<br>from llama_index.embeddings.huggingface import HuggingFaceEmbedding<br>import os<br># 1. Define Production Settings (Consistent across the system)<br>Settings.embed_model = HuggingFaceEmbedding(model_name=&quot;BAAI/bge-small-en-v1.5&quot;)<br># Note: In production, LLM should point to an API endpoint (e.g., OpenAI, Anthropic)<br># 2. Configure the Node Parser for Semantic Chunking<br>splitter = SentenceSplitter(<br>chunk_size=512,<br>chunk_overlap=50,<br>)<br># 3. Load Documents (Assuming data is in a local directory for simplicity)<br>documents = SimpleDirectoryReader(&quot;./data&quot;).load_data()<br># 4. Define the Ingestion Pipeline<br>pipeline = IngestionPipeline(<br>transformations=[<br>splitter,<br>Settings.embed_model<br># Add metadata extractors here if needed (e.g., TitleExtractor)<br>]<br>)<br># Run the pipeline to generate Nodes (ready for persistence)<br>nodes = pipeline.run(documents=documents)<br>print(f&quot;Generated {len(nodes)} indexable nodes.&quot;)</pre><h4><strong>Persistence and Production Vector Stores</strong></h4><p>For a system to be truly production-ready, its knowledge base must be persistent, scalable, and accessible via a network endpoint. Local filesystem storage is insufficient.</p><p>LlamaIndex provides deep integration with leading cloud-native Vector Stores (e.g., Pinecone, Weaviate, AstraDB, Qdrant). When initializing the index, you simply pass the instantiated Vector Store client object. LlamaIndex handles the low-level communication and vector uploads.</p><h4><strong>Code Block 2: Initializing a Persistent Vector Store (Example with Chroma)</strong></h4><p>For MLOps environments, using a managed or self-hosted Vector Store ensures that the index can be queried concurrently by multiple service instances and easily backed up or versioned.</p><pre>from llama_index.vector_stores.chroma import ChromaVectorStore<br>from llama_index.core import VectorStoreIndex<br>import chromadb<br><br># 1. Initialize the Chroma Client (or Pinecone/Weaviate client)<br># In production, this client connects to a remote server endpoint<br>db = chromadb.PersistentClient(path=&quot;./chroma_db_prod&quot;)<br>collection = db.get_or_create_collection(&quot;production_rag_index&quot;)<br><br># 2. Integrate the Vector Store with LlamaIndex<br>vector_store = ChromaVectorStore(collection=collection)<br><br># 3. Create the Index and Persist the Nodes<br># This pushes the embedded nodes generated in the previous step to Chroma<br>prod_index = VectorStoreIndex(<br>    nodes=nodes,<br>    vector_store=vector_store,<br>)<br><br>print(&quot;Index successfully built and persisted to the Vector Store.&quot;)</pre><h4><strong>The Query Engine: Advanced Retrieval Strategies</strong></h4><p>The final layer is the Query Pipeline, which must ensure that only the most relevant context is retrieved, minimizing unnecessary tokens passed to the LLM (saving cost and improving response speed).</p><p>A basic RAG implementation uses simple top-$K$ vector similarity. A production system must use <strong>retrieval post-processing</strong>. LlamaIndex offers powerful techniques like <strong>Contextual Compression </strong>(using a smaller, faster model to filter retrieved chunks) or <strong>Sentence Window Retrieval</strong> (retrieving a small chunk, but expanding the context window just before synthesis).</p><h4><strong>Code Block 3: Defining the Optimized Query Engine</strong></h4><pre>from llama_index.core.retrievers import VectorIndexRetriever<br>from llama_index.llms.openai import OpenAI # Example LLM<br>from llama_index.core.query_engine import RetrieverQueryEngine<br><br># 1. Re-load the Index from the persistent Vector Store<br># Note: We only need the index definition, the data is pulled from Chroma on demand<br>loaded_index = VectorStoreIndex.from_vector_store(vector_store=vector_store)<br><br># 2. Define the Retriever (setting K=5 for high relevance)<br>retriever = VectorIndexRetriever(<br>    index=loaded_index,<br>    similarity_top_k=5,<br>)<br><br># 3. Instantiate the Query Engine with a production LLM<br># This engine handles the final prompt assembly and synthesis<br>query_engine = RetrieverQueryEngine.from_args(<br>    retriever=retriever,<br>    llm=OpenAI(model=&quot;gpt-4o-mini&quot;), # Using a robust, low-latency model<br>)<br># 4. Run the Query<br>response = query_engine.query(&quot;What were the Q3 operational metrics for the new project?&quot;)<br>print(response)</pre><p>By decoupling the ingestion process, leveraging persistent Vector Stores, and applying advanced retrieval strategies provided by LlamaIndex, you transition your RAG application from a proof-of-concept to a scalable, reliable service ready for high-throughput production environments. Remember, MLOps success in RAG starts with the indexing architecture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69686f892f1e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>