<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Uros Boskovic on Medium]]></title>
        <description><![CDATA[Stories by Uros Boskovic on Medium]]></description>
        <link>https://medium.com/@urosboskovic?source=rss-8e88b063c52e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BA0HL6wf19Q9M-Yx_EoUVw.jpeg</url>
            <title>Stories by Uros Boskovic on Medium</title>
            <link>https://medium.com/@urosboskovic?source=rss-8e88b063c52e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:48:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@urosboskovic/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why RAG Systems Fail in Production]]></title>
            <link>https://medium.com/@urosboskovic/why-rag-systems-fail-in-production-85cd9e332a73?source=rss-8e88b063c52e------2</link>
            <guid isPermaLink="false">https://medium.com/p/85cd9e332a73</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[artificial-inteligence]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Uros Boskovic]]></dc:creator>
            <pubDate>Mon, 18 May 2026 12:09:47 GMT</pubDate>
            <atom:updated>2026-05-18T12:09:47.686Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Lessons from building large-scale LLM systems: failure modes, trade-offs, and what actually works</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WBhfDa4pD2_OThO4O-bQjg.png" /></figure><h3>Introduction</h3><p>RAG (Retrieval-Augmented Generation) systems often look deceptively simple in demos: retrieve a few relevant documents, pass them to a language model, and generate a grounded answer.</p><p>In production, things are very different. Here is a concrete example of what failure looks like.<br>A user asks<em>: “How do I reset my password?”.</em> The system retrieves a support article about account security — topically similar but written for a different product version. The model generates a step-by-step answer that is fluent, confident, and completely wrong for the user’s context. No error is thrown. No flag is raised.</p><p>This is the defining characteristic of RAG failures: the system often returns answers that are coherent, plausible, and partially correct — but subtly wrong. This makes RAG systems difficult to debug and even harder to trust.</p><p>In this post, I’ll walk through the most common failure modes I’ve seen when building RAG systems at scale — and the concrete patterns that actually improved reliability in production.</p><h3>What RAG Promises (and Where It Breaks)</h3><p>Conceptually, RAG combines retrieval of relevant context with generation from a language model. The implicit assumptions are:</p><ul><li>the retrieved context is relevant</li><li>the model will use it correctly</li></ul><p>In practice, both assumptions often fail. In many production systems, issues are not caused by a single component, but by interactions between data quality, retrieval strategy, and model behavior.</p><h3>Failure Modes in Production RAG Systems</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5BRksZ58uK4N8_NWldW4kg.png" /></figure><h3>1. Data Quality Issues</h3><p>As in any ML system, data quality is essential. However, due to the nature of RAG systems, data issues can go undetected while still harming the entire product. This is often the most underestimated failure mode.</p><p>Real-world RAG systems usually rely on complex ETL pipelines that run periodically and combine multiple data sources. These pipelines perform chunking, summarization, embedding calculation, and relationship extraction. The number of transformation stages makes these pipelines highly susceptible to silent failures.</p><p>In practice, issues often include:</p><ul><li>duplicate documents skewing retrieval (top-K results returning only a few distinct chunks)</li><li>usage of incompatible or poorly suited embedding models (for example, indexing documents with one embedding model but later querying with another, or using a generic text embedding model for code, logs, or structured data)</li><li>silent changes in the summarization process (for example, a new prompt or model that starts dropping key fields from summaries)</li><li>missing relationships or metadata</li><li>incomplete or inconsistent fields (missing identifiers, empty values, schema changes)</li></ul><p>Even a well-designed retrieval pipeline will return misleading context if the underlying data is inconsistent.</p><p>It is important to understand potential failure modes in your data. Invest early in defining data health metrics and building monitoring systems that track metrics for each data load — for example, the percentage of documents with missing required fields, the distribution of embedding norms over time, or the share of records changed in each run. Even a simple report can make your system significantly more reliable and save hours of debugging.</p><h3>2. Naive Chunking and the Loss of Structure</h3><p>Most RAG systems treat documents as collections of independent chunks.<br> This works for simple use cases but breaks down at scale.</p><p>The core issue is not chunking itself — it is the lack of an explicit data model. By “data model” I mean representing documents as structured objects (for example, document → section → paragraph) and explicitly encoding how parts relate to each other.</p><p>Documents are not flat text. They contain:</p><ul><li>hierarchy</li><li>relationships</li><li>dependencies across sections</li></ul><p>Chunking removes this structure.</p><p><strong>Example: Why Code Breaks Naive Chunking</strong></p><p>This limitation is especially clear when working with code. Code is not just text — it is a system of relationships:</p><ul><li>functions call other functions</li><li>components depend on each other</li></ul><pre>def process_order(order):<br>    if not validate_payment(order):<br>        raise ValueError(&quot;Invalid payment&quot;)<br>    return fulfill_order(order)<br><br>def validate_payment(order):<br>    return order.payment_status == &quot;confirmed&quot;<br><br>def fulfill_order(order):<br>    order.status = &quot;fulfilled&quot;<br>    dispatch_shipment(order)<br>    return order</pre><p>If these functions are chunked separately, their relationships are lost and reasoning about the system becomes incomplete. A query matching <strong>process_order</strong> would retrieve only that chunk — with no knowledge that it depends on <strong>validate_payment</strong> and <strong>fulfill_order</strong> to function correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hoO_UyaYWjSn24Za66qiYw.png" /></figure><p>Semantic similarity alone may occasionally surface related code, but it does not reliably recover execution flow or dependency structure.</p><p>Mature RAG systems benefit significantly from introducing explicit structure into the data layer:</p><ul><li>hierarchical modeling (document → section → paragraph)</li><li>explicit relationships encoding (dependencies, references, references between modules)</li></ul><p>This enables a combination of semantic retrieval for entry points and structured expansion for context. For code, for example, we can store a call graph and, once a relevant function is identified, expand context deterministically along that graph so that its callers and callees are retrieved as a unit.</p><h3>3. Retrieval Strategy Mismatch</h3><p>Another common mistake in RAG systems is an over-reliance on vector search. Semantic similarity works well in many cases, but fails in scenarios where:</p><ul><li>exact matches are required (e.g. retrieving specific lines of code)</li><li>queries are keyword-driven (e.g. searching for a specific error code)</li></ul><p>In these cases, the system retrieves context that appears relevant but is logically incorrect. A search for error code <em>E1234</em>, for example, may return documents about <em>E123 </em>because the documents appear contextually related.</p><p>It is important to detect scenarios where other search methods are better suited. For identifier-heavy queries over code, logs, or error messages, simple keyword search can provide more precise and reliable results than pure semantic search. Consider also:</p><ul><li>hybrid search (semantic + keyword), where keyword search acts as an implicit reranker or hard filter</li><li>graph or structured retrieval for relationship-heavy data</li></ul><p>Selecting the right search mechanism based on the query can significantly improve retrieval precision — for instance, routing natural language questions to semantic-heavy retrieval and identifier queries to keyword-heavy retrieval.</p><h3>4. Irrelevant but Plausible Context</h3><p>Some of the most common failures occur when retrieved context is coherent and fluent, but logically irrelevant to the user’s query.</p><p>This is a natural consequence of similarity-based retrieval. For example, if a user asks a question about <em>Azure SQL Database</em>, content about <em>MySQL </em>may be semantically similar and included in the context. The model then generates an answer grounded in misleading information.</p><p>The model is often reasoning over the wrong context rather than hallucinating from scratch. Many so-called hallucinations are retrieval failures upstream.</p><p>To avoid this, the system must route the query to the correct subset of the knowledge base before retrieval begins.</p><p>Some of the most effective approaches include:</p><ul><li>enriching documents with structured metadata (e.g. <em>ProductName</em>, <em>DocumentType</em>) and filtering based on it</li><li>using re-ranking algorithms to improve relevance ordering</li><li>validating retrieved documents before passing them to the model</li></ul><p>Filtering and narrowing the search space is often more effective than improving the model itself.</p><h3>5. Partial Context and Coverage Gaps</h3><p>Unlike the previous failure mode, the issue here is not irrelevant context, but insufficient breadth of context. Most real-world questions require context that spans multiple areas of a knowledge base rather than a single highly relevant document. For example, the question:</p><p><em>“What are the main components of this codebase?”</em></p><p>cannot be answered from one isolated chunk. It typically requires visibility across multiple subsystems such as authentication, payments, orders, notifications, and shared infrastructure.</p><p>With standard vector search, top-K retrieval often fails to capture this breadth. Results tend to cluster around closely related chunks, reinforcing a single subsystem while excluding others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2saN0XZFxWQmTqWGiV43A.png" /></figure><p>This introduces a coverage bias: parts of the knowledge base become overrepresented in retrieval, while other relevant areas are never surfaced. As a result, downstream generation is based on an incomplete view of the system.</p><p>At a high level, this happens because vector retrieval optimizes for “more of the same” semantic similarity rather than global coverage. While this works well for narrow factual queries, it is less effective for questions that require system-wide understanding across multiple components.</p><p>Mitigating this issue requires retrieval strategies that explicitly optimize for diversity and coverage, not just relevance. Common approaches include:</p><ul><li>query rephrasing and decomposition (for example, breaking “What are the main components of this codebase?” into separate queries for authentication, payments, orders, and infrastructure)</li><li>multiple retrieval passes for complex queries (running several searches with different emphases and then merging results)</li><li>diversity-aware ranking across sources or components (penalizing chunks from sources that are already overrepresented in the current set)</li><li>hierarchical or graph-based retrieval expansion (retrieving high-level modules first, then expanding into representative files or sections within each module)</li></ul><p>Ultimately, retrieval quality is not only about surfacing the most relevant documents, but also about ensuring sufficient coverage of the problem space so the model can reason about the system as a whole.</p><h3>6. The Model Doesn’t Always Use the Context</h3><p>Language models do not consistently utilize all provided context. In practice, they may ignore relevant chunks, over-weight prior knowledge, or focus disproportionately on the most recent or salient tokens. For example, even with updated documentation in the prompt, a model may keep recommending a deprecated API because it has seen it many times during pretraining.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mMCASg3VUftnObpFkFOrgA.png" /></figure><p>The result is a subtle but important failure mode: the system appears grounded in the retrieved context, yet the final answer is still incorrect.</p><p>These issues become more pronounced as context length increases. Additional tokens do not necessarily improve reasoning; instead, they can introduce noise and dilute critical signals.</p><p>Mitigating this requires careful system design beyond retrieval quality alone. Key strategies include:</p><ul><li>structuring and ordering context deliberately</li><li>limiting inputs to only the most relevant information</li><li>evaluating not just retrieval quality, but how effectively the model uses the retrieved context</li></ul><p>Even with high-quality data and retrieval, model behavior remains a fundamental source of uncertainty.</p><h3>Conclusion</h3><p>Building reliable RAG systems is not about plugging in better models. It is about understanding how systems fail — and designing around those failures.</p><p>In many cases, the biggest improvements come not from the model itself, but from:</p><ul><li>better data</li><li>better retrieval strategies</li><li>better system design</li></ul><p>RAG failures often look like model problems, but their root cause is usually system design: data quality, chunking, retrieval strategy, and how context is assembled before generation.</p><p>The hard part is not getting a model to answer questions — it is getting the entire system to provide the right information, in the right form, at the right time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85cd9e332a73" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>