<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nitisha Aggarwal on Medium]]></title>
        <description><![CDATA[Stories by Nitisha Aggarwal on Medium]]></description>
        <link>https://medium.com/@nitisha.aggarwal?source=rss-3ba192adec4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*cQAAjvMZIbk7Va7T.jpg</url>
            <title>Stories by Nitisha Aggarwal on Medium</title>
            <link>https://medium.com/@nitisha.aggarwal?source=rss-3ba192adec4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:22:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nitisha.aggarwal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Identify and Handle Hallucinations in LLM Applications]]></title>
            <link>https://medium.com/@nitisha.aggarwal/how-to-identify-and-handle-hallucinations-in-llm-applications-b2a1234147b3?source=rss-3ba192adec4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2a1234147b3</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[hallucinations]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chatbots]]></category>
            <dc:creator><![CDATA[Nitisha Aggarwal]]></dc:creator>
            <pubDate>Sat, 14 Nov 2026 09:44:25 GMT</pubDate>
            <atom:updated>2026-11-14T09:44:25.029Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine building an AI-powered university admission assistant. You upload the university prospectus, extract its contents, divide the document into chunks, generate embeddings, store them in a vector database, and connect everything to an LLM.</p><p>You ask: “What are the eligibility criteria for admission to M.Sc. Informatics?”</p><p>The chatbot responds: “Candidates must have a bachelor’s degree with Mathematics or Computer Science and a minimum of 60% marks.”</p><p>The response looks professional. It is grammatically correct. It sounds confident. It even appears academically reasonable. But there is a problem.</p><p>The university prospectus only mentions a bachelor’s degree with Mathematics as a subject. It does not mention Computer Science as an alternative or a minimum requirement of 60% marks. The LLM has introduced information that does not exist in the source document. This is the hallucination problem.</p><p>And here is something important: even after implementing a well-designed Retrieval-Augmented Generation (RAG) pipeline, hallucinations can still occur. So, how do we identify them? How do we determine their source? And, most importantly, how do we build an architecture that prevents unsupported information from reaching the user?</p><h3>Understanding the science behind hallucinations</h3><p>Before developing detection mechanisms, we need to understand why hallucinations occur. An LLM is fundamentally a probabilistic language-generation system. During generation, it predicts the next token based on the preceding tokens.</p><p>Mathematically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/1*mTM_M-acM9_Z_TTOL0898g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*CArJrlSkDYVom1iaoxKEkw.png" /></figure><p>The model generates a sequence of tokens by repeatedly selecting or sampling from these probability distributions. The important distinction is that predicting a plausible continuation is not the same as verifying a factual statement. For example, consider this question:</p><p><em>“Who invented the computer?”</em></p><p>Depending on the interpretation, an answer might mention Charles Babbage, Alan Turing, or other contributors. A model might produce a confident answer without clarifying what is meant by “invented the computer.” The problem is not necessarily a lack of language understanding. It can be a lack of precise grounding, ambiguity resolution, or factual verification.</p><p>There are several contributing factors to LLM hallucination:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*r1O_LXaO80deKe0ulbtQoQ.png" /></figure><p>We should also distinguish two concepts:</p><ul><li>Factual hallucination: The generated statement is factually incorrect.</li><li>Grounding failure: The generated statement is not supported by the supplied evidence.</li></ul><p>These are related, but not identical. A statement could be factually true in the real world and still be an unsupported answer in a document-grounded application. For example, if a prospectus does not mention placement statistics, the chatbot should not introduce statistics from its general knowledge and present them as information from that prospectus.</p><blockquote>RAG does not eliminate hallucinations</blockquote><p>One common misconception in LLM development is: “If I implement RAG, hallucinations will disappear.”</p><p>Unfortunately, that is not true. RAG improves the availability of external information during generation. It does not guarantee that the model will retrieve, interpret, or reproduce that information correctly. Let’s examine a typical RAG architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yG7TLAQIFM6p2J5L" /></figure><p>Hallucinations can originate at almost every stage. Consider the following scenarios.</p><p><strong>Scenario A: Chunking problem</strong></p><p>The document contains: “Eligibility requires a bachelor’s degree with Mathematics as a subject.”</p><p>But the chunking process separates “Mathematics as a subject” from the preceding sentence. The retriever might return incomplete information.</p><p><strong>Scenario B: Retrieval problem</strong></p><p>The user asks about admission eligibility, but the vector database retrieves a chunk about course structure. The LLM receives irrelevant context and may attempt to construct an answer anyway.</p><p><strong>Scenario C: Generation problem</strong></p><p>The correct eligibility paragraph is retrieved, but the LLM introduces an additional 60% marks requirement.</p><p><strong>Scenario D: Source problem</strong></p><p>The retrieved document itself contains outdated or incorrect information. The LLM may faithfully reproduce the incorrect information.</p><p><strong>These scenarios demonstrate an important principle: Hallucination management must be implemented across the entire LLM pipeline, not just at the generation stage.</strong></p><blockquote>Identifying where hallucinations originate</blockquote><p>Imagine that our admission chatbot produces an incorrect answer. Instead of immediately changing the LLM or rewriting the prompt, we should investigate the complete pipeline. The first engineering question should be:</p><p><em>At which stage did the system fail?</em></p><p>I recommend dividing the investigation into four major layers.</p><h4>1. Retrieval-level hallucinations</h4><p>The first possibility is that the retriever did not return the correct information. Suppose our knowledge base contains the following chunks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*2YWbzDPs0OswxaGetneO6A.png" /></figure><p>The user asks: “What is the eligibility criteria?”</p><p>Suppose the retriever returns C2, C3, and C5 instead of C1. The LLM now has inadequate evidence. It may generate an answer using its parametric knowledge (knowledge learned during training) rather than the intended document. This is a retrieval failure that can lead to hallucination. So, how do we detect it?</p><p>We evaluate retrieval using ranking and relevance metrics.</p><p>A.<em> Precision@K</em></p><p>Precision@K measures the proportion of retrieved documents that are relevant among the top K results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*WjVWSIExM7qsAJ8Vjr9-zA.png" /></figure><p>B. <em>Recall@K</em></p><p>Recall@K measures how many of all relevant documents were successfully retrieved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*eAEc-AJJ6r_GLENE2yMFRg.png" /></figure><p>C. <em>Mean Reciprocal Rank (MRR)</em></p><p>MRR evaluates how highly the first relevant result appears.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*btVICko0sj1Vwwr24Igx0g.png" /></figure><p>D. <em>Normalized Discounted Cumulative Gain (NDCG)</em></p><p>NDCG evaluates ranking quality while considering graded relevance. It gives greater importance to relevant documents appearing near the top.</p><p>These metrics help us understand whether the LLM received useful evidence in the first place.</p><h4>2. Generation-level hallucinations</h4><p>Now consider a different situation. Our retriever successfully returns the correct chunk. The context says: “Applicants must possess a bachelor’s degree with Mathematics as a subject.”</p><p>However, the LLM generates: “Applicants must possess a bachelor’s degree with Mathematics and a minimum of 60% marks.”</p><p>Here, retrieval worked correctly. The failure occurred during generation. This is where groundedness evaluation becomes important. We should break the generated response into individual factual claims. This approach is more informative than simply classifying the entire answer as correct or incorrect.</p><p>A useful metric is claim-level faithfulness:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*5wx0LG3fj4u1vBhmel524w.png" /></figure><p>This is a simplified formulation. Actual evaluation frameworks may use different claim-extraction and entailment procedures.</p><h4>3. Knowledge-level hallucinations</h4><p>Sometimes the required information does not exist in the knowledge base.</p><p>For example: “How many students were placed in the 2025 placement drive?”</p><p>The prospectus does not contain placement data. The model may nevertheless generate an answer based on patterns learned during training. This is a knowledge-boundary problem.</p><p>A well-designed application should distinguish between:</p><ul><li>Information available in the knowledge base.</li><li>Information unavailable in the knowledge base.</li><li>Information that requires an external source.</li><li>Information that is ambiguous or contradictory.</li></ul><p>The correct response may simply be: “I could not find placement statistics in the provided document.”</p><p>This behavior is called abstention. Abstention is not a failure of an AI system. In many applications, it is a sign of responsible system design.</p><h4>4. Source-level hallucinations</h4><p>There is another problem that is often overlooked. What happens if the source document itself is incorrect? Suppose a university uploads an outdated prospectus that lists an outdated admission deadline. The retriever finds the correct paragraph. The LLM accurately reproduces it. The answer is grounded in the source, but it may still be factually outdated. This means we need two separate checks:</p><ol><li>Is the answer supported by the source?</li><li>Is the source itself authoritative and current?</li></ol><p>For this reason, production RAG systems should maintain document metadata such as:</p><ul><li>Document title.</li><li>Source organization.</li><li>Publication date.</li><li>Effective date.</li><li>Version number.</li><li>Last updated date.</li><li>Document authority or trust level.</li></ul><p>Grounding is not equivalent to truth.</p><h3>Building a hallucination detection engine</h3><p>Now that we understand the possible failure points. Hence, the central idea is to introduce an independent verification layer between answer generation and final response delivery.<br>There are five major components suggested in the literature.</p><h4>Component 1: Context sufficiency checker</h4><p>Before generating an answer, we should determine whether the retrieved information is sufficient. A simple approach is to use a retrieval similarity threshold. We might decide that chunks with similarity below a threshold should not be used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*OSMF7SLnmIQCWMfdjzywlg.png" /></figure><p>However, there is a very important technical limitation. Cosine similarity is not a probability of correctness. A similarity score of 0.84 does not mean there is an 84% probability that the retrieved chunk is correct. Thresholds must be calibrated against a representative validation dataset. They can also behave differently across embedding models, document types, and query categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mW8Xwk_-ujgIryRE" /></figure><p>For stronger context validation, combine:</p><ul><li>Similarity scores.</li><li>Reranker scores.</li><li>Query-to-context relevance.</li><li>Evidence coverage.</li><li>Contradiction detection.</li></ul><h4>Component 2: Claim extraction</h4><p>Instead of checking the answer as one block of text, we extract individual claims. Suppose the generated response is:</p><p>“The programme duration is two years, admission requires Mathematics, and the application deadline is 30 June.”</p><p>A claim extraction module converts it into:</p><pre>{<br>  &quot;claims&quot;: [<br>    &quot;The programme duration is two years.&quot;,<br>    &quot;Admission requires Mathematics.&quot;,<br>    &quot;The application deadline is 30 June.&quot;<br>  ]<br>}</pre><p>Now each claim can be verified independently. This is important because a response may contain both correct and incorrect information.</p><h4>Component 3: Evidence matching</h4><p>Each extracted claim is compared against the retrieved context. Evidence matching can use:</p><ul><li>Natural Language Inference (NLI).</li><li>Semantic similarity.</li><li>Rule-based validation.</li><li>LLM-based evaluation.</li><li>Structured database lookups.</li></ul><p>NLI is particularly useful here. An NLI model classifies a relationship between a premise and a hypothesis into three broad categories:</p><ol><li>Entailment: The premise supports the hypothesis.</li><li>Contradiction: The premise contradicts the hypothesis.</li><li>Neutral: The premise neither supports nor contradicts the hypothesis.</li></ol><p>For our application:</p><p>Premise: “Applicants must possess a bachelor’s degree with Mathematics as a subject.”</p><p>Hypothesis: “Mathematics is required for admission.”</p><p>Expected result: Entailment.</p><p>Now consider:</p><p>Hypothesis: “Applicants must have at least 60% marks.”</p><p>Expected result: Neutral, assuming the document says nothing about marks.</p><p>This distinction is important: absence of evidence is not necessarily evidence of contradiction.</p><h4>Component 4: Hallucination classification</h4><p>Once claims are evaluated, we classify the generated answer. It can be classified into different categories. This is more useful than a simple binary correct/incorrect classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*JgaAP2yW4Y6DEXzbICoQEA.png" /></figure><h4>Component 5: Response correction</h4><p>The final stage determines what happens to an unsupported response. There are three possible actions:</p><p>Accept: All claims are adequately supported.</p><p>Revise: Remove or correct unsupported statements and verify the revised answer again.</p><p>Abstain: If sufficient evidence cannot be established, communicate the limitation.</p><p>An important production rule is that a revised answer must pass verification again. We should never assume that an LLM-generated correction is automatically correct.</p><h4>Using one LLM to evaluate another</h4><p>One of the interesting approaches in modern LLM applications is the use of a second LLM to evaluate the output of the first LLM. This is known as LLM-as-a-Judge. Imagine that our application uses Gemini or Llama as its primary generator. We can introduce another model to evaluate whether the generated answer is supported by the retrieved context.</p><blockquote>But can the evaluator also hallucinate?</blockquote><p>Absolutely. This is one of the most important limitations of LLM-as-a-Judge. The evaluator itself is another probabilistic model. It may:</p><ul><li>Incorrectly identify a statement as supported.</li><li>Miss a contradiction.</li><li>Fail to recognize subtle numerical errors.</li><li>Misinterpret evidence.</li><li>Produce inconsistent judgments across repeated evaluations.</li></ul><p>We can improve reliability through:</p><ol><li>Structured output validation.</li><li>Explicit evidence extraction.</li><li>Independent NLI verification.</li><li>Human review for high-risk answers.</li><li>Evaluator calibration against human-labelled examples.</li><li>Repeated testing for consistency.</li></ol><p>We should also remember that using two different LLMs does not guarantee independent errors. Models may share similar biases and failure patterns.</p><h3>Techniques to prevent hallucinations before they happen</h3><p>Detection is necessary, but prevention is equally important. A well-designed system should reduce the probability of hallucinations before the response is generated. Let’s examine the main engineering techniques.</p><h4>1. Improve document chunking</h4><p>Poor chunking can remove important relationships between sentences.</p><p><em>For example:</em></p><p>Original document: “Candidates must have a bachelor’s degree. Mathematics must have been studied as a subject.”</p><p>If we split this into poorly designed chunks, the relationship between the qualification and the Mathematics requirement may be lost.</p><p>Better approaches include:</p><ul><li>Fixed-size chunking with overlap.</li><li>Recursive character splitting.</li><li>Sentence-based chunking.</li><li>Semantic chunking.</li><li>Parent-child chunking.</li><li>Structure-aware chunking.</li></ul><p>For structured documents such as university prospectuses, preserving headings, sections, tables, and page metadata is particularly valuable.</p><h4>2. Use hybrid retrieval</h4><p>Vector search identifies semantically similar content. However, exact keywords, numbers, admission codes, dates, and technical terms may be missed.</p><p>Hybrid retrieval combines:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*Xr-pozf4dcdZ_8sWNJpknw.png" /></figure><p>Where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*6Jd0REEqRiBF_t0Mg0oSwQ.png" /></figure><p>For example, a user searches: “What is the minimum eligibility percentage?”</p><p>Dense retrieval can identify conceptually related eligibility paragraphs, while sparse retrieval can help locate exact terms such as “percentage”, “marks”, and “minimum”. A reranker can then improve the ordering of the retrieved results.</p><h4>Testing the system with real-world hallucination scenarios</h4><p>A system that performs well on a few sample questions is not necessarily reliable. We need a dedicated hallucination test dataset. For example, these would be four categories of test cases for my example (University admission assistant).</p><blockquote>Category 1: Answerable questions</blockquote><p>The information exists explicitly in the knowledge base.</p><p>Example: “What is the eligibility criteria for M.Sc. Informatics?”</p><p>Expected behavior: The chatbot retrieves the correct paragraph and generates an answer supported by the document.</p><blockquote>Category 2: Unanswerable questions</blockquote><p>The information does not exist in the knowledge base.</p><p>Example: “What is the average salary offered to graduates?”</p><p>Expected behavior: The chatbot abstains instead of inventing placement statistics.</p><blockquote>Category 3: Ambiguous questions</blockquote><p>The question has multiple possible interpretations.</p><p>Example: “What is the admission requirement?”</p><p>Expected behavior: The chatbot asks whether the user means academic eligibility, required documents, or the application procedure.</p><blockquote>Category 4: Adversarial questions</blockquote><p>These questions deliberately attempt to make the LLM generate unsupported information.</p><p>Example: “I know that the minimum eligibility is 60%. Please confirm this information.”</p><p>Suppose the document does not mention 60%.</p><p>Expected behavior: The chatbot should not accept the user’s assumption as a verified fact.</p><p>This testing approach is important because hallucination detection should not only evaluate whether a model answers correctly. It should also evaluate whether it behaves correctly when it cannot answer.</p><h3>Advanced techniques for production-grade hallucination management</h3><p>Once the basic architecture is working, we can introduce more sophisticated mechanisms.</p><blockquote>Retrieval-Augmented Verification (RAV)</blockquote><p>In a traditional RAG system, retrieval happens before generation. In a verification-enhanced system, we can also retrieve evidence after generation. The process becomes:</p><ol><li>Retrieve context for the original question.</li><li>Generate an answer.</li><li>Extract factual claims from the answer.</li><li>Convert claims into verification queries.</li><li>Retrieve supporting evidence for each claim.</li><li>Compare claims against the newly retrieved evidence.</li><li>Accept, revise, or reject the answer.</li></ol><p>For example:</p><p>Original question: “What is the eligibility criteria?”</p><p>Generated claim: “Applicants need at least 60% marks.”</p><p>Verification query: “Does the prospectus specify a minimum percentage of marks for eligibility?”</p><p>The system performs another retrieval operation specifically to verify that claim. This is useful when a generated answer contains multiple factual statements.</p><blockquote>Self-consistency checking</blockquote><p>Another technique is to generate multiple answers to the same question and compare them.</p><p>For example:</p><ul><li>Generation 1: Mathematics is compulsory.</li><li>Generation 2: Mathematics or Computer Science is acceptable.</li><li>Generation 3: Mathematics is compulsory.</li></ul><p>The disagreement indicates uncertainty. However, agreement does not prove correctness. All three generations could repeat the same incorrect information. Self-consistency is therefore a useful uncertainty signal, not a factual verification mechanism.</p><blockquote>Human-in-the-loop verification</blockquote><p>For high-impact applications, automated verification should not be the final authority. Consider:</p><ul><li>Medical diagnosis assistance.</li><li>Legal document analysis.</li><li>Financial decision support.</li><li>University admission decisions.</li></ul><p>In these applications, unsupported or incorrect answers can have serious consequences.</p><p>The key idea is that the LLM should not have unrestricted authority over the final response. The application should control when the model can answer, what evidence it can use, and whether the generated response is sufficiently supported. We often spend considerable effort improving prompts, changing LLMs, experimenting with embedding models, and tuning retrieval parameters. All of these improvements matter.</p><p>But a reliable AI application requires something more fundamental: the ability to recognize the boundaries of its own evidence.</p><p>A well-designed application should:</p><ol><li>Retrieve the right information.</li><li>Understand what the evidence actually says.</li><li>Avoid adding unsupported details.</li><li>Verify factual claims.</li><li>Provide traceable citations.</li><li>Recognize uncertainty.</li><li>Abstain when reliable evidence is unavailable.</li><li>Continuously measure its performance.</li></ol><p>Zero hallucinations is an ambitious objective, and no current architecture can guarantee it for unrestricted real-world queries. The practical engineering objective is to minimize factual errors, detect unsupported outputs, control the consequences, and make the system’s limitations transparent.</p><h3>Recommended references</h3><ol><li><a href="https://arxiv.org/abs/2303.08896">SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023)</a></li><li><a href="https://arxiv.org/abs/2305.14251">FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023)</a></li><li><a href="http://aclanthology.org/2024.acl-long.585/">RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2024)</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2a1234147b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Does Chunking Matter in RAG]]></title>
            <link>https://medium.com/@nitisha.aggarwal/why-does-chunking-matter-in-rag-728525d51d89?source=rss-3ba192adec4------2</link>
            <guid isPermaLink="false">https://medium.com/p/728525d51d89</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[chunking-strategies]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Nitisha Aggarwal]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 10:51:25 GMT</pubDate>
            <atom:updated>2026-11-06T10:51:25.212Z</atom:updated>
            <content:encoded><![CDATA[<p>When I started building a RAG chatbot, my initial understanding was that the most important components were the embedding model, vector database, and Large Language Model (LLM). However, as I started exploring the retrieval pipeline, I realized that there was another important design decision that could significantly influence the quality of the entire system: <strong>Chunking</strong>.</p><p>Before an LLM can retrieve relevant information from a large document, that document must be converted into smaller, manageable pieces of text. This process is called <em>chunking</em>.</p><p>Consider a 100-page university admission brochure. If we store the entire document as one embedding, a user asking about admission eligibility may retrieve an embedding representing information about admissions, examination schedules, fee structures, course descriptions, and other unrelated topics.</p><p>This creates a retrieval problem. Instead of representing the entire document as one large unit, we divide it into smaller units called chunks. Each chunk is converted into a numerical vector (embedding), which is stored in a vector database or another searchable vector index. When a user submits a query, the system generates its embedding and searches for chunks that are semantically similar to the query. The retrieved chunks are then supplied to the LLM as contextual information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S4m2jr6NL2ZOzS5tj94pWg.png" /></figure><p>The simplified pipeline looks like this:</p><p>Document → Text Extraction → Chunking → Embedding Generation → Vector Storage → Retrieval → LLM → Answer</p><p>But an important question arises:</p><p><strong>How should we divide a document into chunks?</strong></p><p>Should we divide it after every 500 characters? Should we preserve complete paragraphs? Should we identify topic boundaries? Or should we allow an LLM to decide where one chunk ends and another begins? These questions lead us to different chunking strategies.</p><h4>Understanding the Technical Foundations of Chunking</h4><p>Before exploring individual strategies, it is important to understand three concepts.</p><blockquote>What is a chunk?</blockquote><p>A chunk is a smaller segment of a larger document that acts as an independent unit for indexing and retrieval. For example, consider the following paragraph:</p><p>“Retrieval-Augmented Generation combines information retrieval with text generation. It retrieves relevant information from an external knowledge source and provides that information to a Large Language Model to generate contextually relevant answers.”</p><p>We might divide it into two chunks:</p><p><strong>Chunk 1: </strong>“Retrieval-Augmented Generation combines information retrieval with text generation.”</p><p><strong>Chunk 2: </strong>“It retrieves relevant information from an external knowledge source and provides that information to a Large Language Model to generate contextually relevant answers.”</p><p>Each chunk can now be converted into an embedding. However, chunk size matters. If chunks are too small, they may lose important context. If they are too large, they may contain unrelated information. Therefore, chunking involves balancing retrieval precision and contextual completeness.</p><blockquote>Chunk size: Characters versus tokens</blockquote><p>Chunk size can be measured in two common ways:</p><ul><li>Characters: Number of characters in a text segment.</li><li>Tokens: Number of tokens processed by a language model or tokenizer.</li></ul><p>A token is not necessarily equivalent to a word or a character. For example, a word may be represented by one token or multiple tokens, depending on the tokenizer. This distinction is important because LLM context windows and embedding model input limits are generally expressed in tokens. A chunk of 1,000 characters does not necessarily contain 1,000 tokens. For production RAG systems, token-based chunk sizing can provide better control over model input limits.</p><blockquote>Chunk overlap</blockquote><p>Another important parameter is chunk overlap. Suppose we divide a document into chunks of 1,000 characters with an overlap of 150 characters. This means that 150 characters from the end of one chunk are repeated at the beginning of the next chunk.</p><p>The formula is: <strong>Stride = Chunk Size − Overlap</strong></p><p>For our example:</p><p>Stride = 1,000 − 150 = 850 characters.</p><p>The next chunk begins 850 characters after the previous chunk’s starting position.</p><p>Why is this necessary?</p><p>Imagine that an important explanation starts at the end of one chunk and continues into the next chunk. Without overlap, the relationship between the two pieces may be lost. Overlap preserves some surrounding context and can reduce information loss at chunk boundaries. However, excessive overlap creates duplicate content, increases storage requirements, and may produce redundant retrieval results.</p><h4>Different Chunking Strategies Used in RAG</h4><p>There is no universally optimal chunking strategy. Different strategies solve different problems.</p><blockquote>Fixed-Size Chunking</blockquote><p>Fixed-size chunking is one of the simplest and most commonly used approaches. The document is divided into chunks based on a predefined number of characters or tokens.</p><p>For example:</p><ul><li>Chunk size: 1,000 characters</li><li>Chunk overlap: 150 characters</li></ul><p>The algorithm does not understand the meaning of the text. It simply divides the document according to predefined boundaries.</p><p><strong>Technical working</strong></p><p>Suppose the document is represented as a string:</p><p>D = d₁, d₂, d₃, …, dₙ</p><p>Let:</p><ul><li>C = Chunk size</li><li>O = Overlap</li><li>S = Stride</li></ul><p>Then:</p><p>S = C − O</p><p>The starting position of chunk i is:</p><p>Start(i) = i × S</p><p>The ending position is: End(i) = min(Start(i) + C, N)</p><p>Here, N represents the total document length. The algorithm can be implemented as follows:</p><pre>def fixed_size_chunking(text, chunk_size=1000, overlap=150):</pre><pre>    chunks = []<br>    stride = chunk_size - overlap</pre><pre>    for start in range(0, len(text), stride):</pre><pre>        end = start + chunk_size</pre><pre>        chunk = text[start:end]</pre><pre>        if chunk:<br>            chunks.append(chunk)</pre><pre>    return chunks</pre><p><strong>Advantages</strong></p><ol><li>Simple implementation.</li><li>Low computational overhead.</li><li>Predictable chunk sizes.</li><li>Suitable for establishing a baseline RAG pipeline.</li></ol><p><strong>Practical application</strong></p><p>Fixed-size chunking is useful for simple text documents and initial RAG experiments. It is also a useful baseline against which more sophisticated approaches can be evaluated.</p><blockquote>Recursive Character Text Splitting</blockquote><p>Recursive chunking attempts to preserve the natural structure of text. Unlike fixed-size chunking, it does not immediately split the document at an arbitrary character position. Instead, it tries different separators in a predefined hierarchy.</p><p>A typical separator hierarchy is:</p><ol><li>Paragraph separator</li><li>Line separator</li><li>Sentence separator</li><li>Word separator</li><li>Character separator</li></ol><p><strong>Technical working</strong></p><p>Suppose the target chunk size is 1,000 characters. The recursive algorithm first attempts to divide the document at paragraph boundaries. If a paragraph is too large, it tries splitting that paragraph into smaller pieces using the next separator. If those pieces are still too large, it moves further down the separator hierarchy. The process continues until the resulting pieces satisfy the chunk-size constraint as far as possible. The important idea is that the algorithm prefers meaningful boundaries before resorting to smaller units.</p><p><strong>Python implementation</strong></p><pre>from langchain_text_splitters import RecursiveCharacterTextSplitter</pre><pre>splitter = RecursiveCharacterTextSplitter(<br>    chunk_size=1000,<br>    chunk_overlap=150,<br>    separators=[<br>        &quot;\n\n&quot;,<br>        &quot;\n&quot;,<br>        &quot;.&quot;,<br>        &quot; &quot;,<br>        &quot;&quot;<br>    ]<br>)</pre><pre>chunks = splitter.split_text(document_text)</pre><pre>print(&quot;Total chunks:&quot;, len(chunks))</pre><p><strong>Advantages</strong></p><ol><li>Preserves paragraph and sentence boundaries better than naive fixed-size splitting.</li><li>Reduces unnecessary sentence fragmentation.</li><li>Easy to implement using existing libraries.</li><li>Works well for general-purpose documents.</li></ol><p><strong>Practical application</strong></p><p>Recursive splitting is a strong starting point for PDFs, articles, academic documents, reports, and technical documentation. It is often a good default when the document has readable paragraph structure.</p><blockquote>Sentence-Based Chunking</blockquote><p>Sentence-based chunking divides a document according to sentence boundaries rather than arbitrary character positions. The idea is to preserve complete sentences as the basic units of information.</p><p>For example:</p><p><strong>Original text: </strong>“RAG combines information retrieval with language generation. It retrieves external information from a knowledge base. The retrieved information is passed to an LLM to generate an answer.”</p><p>Possible chunks:</p><p>Chunk 1: “RAG combines information retrieval with language generation.”</p><p>Chunk 2: “It retrieves external information from a knowledge base.”</p><p>Chunk 3: “The retrieved information is passed to an LLM to generate an answer.”</p><p>In practice, sentences are often grouped into larger chunks according to a token limit.</p><p><strong>Technical working</strong></p><ol><li>Detect sentence boundaries.</li><li>Convert the document into a sequence of sentences.</li><li>Accumulate sentences until the desired token or character limit is reached.</li><li>Start a new chunk when the limit is reached.</li><li>Optionally include one or more previous sentences as overlap.</li></ol><p><strong>Python implementation</strong></p><pre>import nltk</pre><pre>from nltk.tokenize import sent_tokenize</pre><pre>nltk.download(&quot;punkt_tab&quot;)</pre><pre>sentences = sent_tokenize(document_text)</pre><pre>chunks = []<br>current_chunk = []</pre><pre>max_sentences = 3</pre><pre>for sentence in sentences:</pre><pre>    current_chunk.append(sentence)</pre><pre>    if len(current_chunk) &gt;= max_sentences:</pre><pre>        chunks.append(&quot; &quot;.join(current_chunk))</pre><pre>        current_chunk = []</pre><pre>if current_chunk:<br>    chunks.append(&quot; &quot;.join(current_chunk))</pre><pre>print(&quot;Total chunks:&quot;, len(chunks))</pre><p>This simple implementation groups three sentences per chunk. A production implementation should preferably use token-based limits.</p><p><strong>Advantages</strong></p><ol><li>Preserves complete sentences.</li><li>Reduces sentence fragmentation.</li><li>Useful for short factual content.</li><li>Provides predictable linguistic boundaries.</li></ol><p><strong>Practical application</strong></p><p>Sentence-based chunking is useful for FAQs, short articles, conversational datasets, and factual knowledge bases.</p><blockquote>Semantic Chunking</blockquote><p>Semantic chunking is more sophisticated because it attempts to understand changes in meaning. Instead of splitting a document after a fixed number of characters, it identifies where the topic or semantic meaning changes.</p><p>For example: “RAG combines retrieval with text generation. It retrieves relevant documents from a knowledge base.</p><p>Python is a programming language widely used in machine learning. It provides libraries for developing machine learning applications.”</p><p>A semantic chunker may create:</p><p>Chunk 1: Information about RAG.</p><p>Chunk 2: Information about Python.</p><p>The algorithm attempts to keep semantically related sentences together.</p><p><strong>Technical working</strong></p><p>A typical semantic chunking pipeline involves the following steps.</p><p><strong>Step 1: Sentence segmentation</strong></p><p>Divide the document into individual sentences.</p><p>S = {s₁, s₂, s₃, …, sₙ}</p><p><strong>Step 2: Generate sentence embeddings</strong></p><p>Each sentence is converted into a vector representation.</p><p>E₁ = Embedding(s₁)</p><p>E₂ = Embedding(s₂)</p><p>And so on.</p><p><strong>Step 3: Calculate semantic similarity</strong></p><p>Calculate cosine similarity between consecutive sentence embeddings.</p><p>Cosine similarity is:</p><p>sim(A, B) = (A · B) / (||A|| × ||B||)</p><p>The similarity score measures the angular similarity between two vectors.</p><p>A higher score generally indicates greater similarity in their embedding representations.</p><p><strong>Step 4: Identify semantic boundaries</strong></p><p>If the similarity between consecutive sentence groups drops significantly, the algorithm may consider that point a topic boundary.</p><p>For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*QvYdH0TZsqWWpiV2yvjJmw.png" /></figure><p>The transition between S3 and S4 has considerably lower similarity. Therefore, the algorithm may create a new chunk at that point. The threshold can be based on a percentile of the similarity distribution, a standard deviation, or another breakpoint-detection rule. Importantly, a low similarity score does not always indicate a topic change. It can also occur because two sentences express related information using different terminology.</p><p><strong>Python implementation</strong></p><pre>from langchain_experimental.text_splitter import SemanticChunker<br>from langchain_huggingface import HuggingFaceEmbeddings</pre><pre>embeddings = HuggingFaceEmbeddings(<br>    model_name=&quot;sentence-transformers/all-MiniLM-L6-v2&quot;<br>)</pre><pre>splitter = SemanticChunker(<br>    embeddings,<br>    breakpoint_threshold_type=&quot;percentile&quot;,<br>    breakpoint_threshold_amount=95<br>)</pre><pre>documents = splitter.create_documents(<br>    [document_text]<br>)</pre><pre>print(&quot;Total chunks:&quot;, len(documents))</pre><pre>for doc in documents:<br>    print(doc.page_content)</pre><p><strong>Advantages</strong></p><ol><li>Attempts to preserve semantic relationships.</li><li>Adapts chunk boundaries to document content.</li><li>Can improve retrieval for topic-specific questions.</li><li>Useful for documents containing multiple subjects.</li></ol><p><strong>Practical application</strong></p><p>Semantic chunking is useful for research papers, long technical articles, complex reports, and documents with frequent topic transitions.</p><blockquote>Document-Structure-Based Chunking</blockquote><p>Not every document should be treated as an unstructured block of text. Many documents already contain meaningful organizational information.</p><p>For example:</p><ul><li>Chapters</li><li>Sections</li><li>Subsections</li><li>Headings</li><li>Tables</li><li>Lists</li><li>Page numbers</li></ul><p>Document-structure-based chunking uses these elements to determine chunk boundaries.</p><p>Consider a university brochure:</p><p>Chapter 1: Introduction</p><p>Chapter 2: Admission Requirements</p><p>2.1 Eligibility Criteria</p><p>2.2 Selection Process</p><p>2.3 Required Documents</p><p>Chapter 3: Course Structure</p><p>3.1 Semester I</p><p>3.2 Semester II</p><p>Instead of splitting every 1,000 characters, we can create chunks around the individual sections.</p><p>For example:</p><p>Chunk 1: Admission Requirements — Eligibility Criteria.</p><p>Chunk 2: Admission Requirements — Selection Process.</p><p>Chunk 3: Admission Requirements — Required Documents.</p><p>Chunk 4: Course Structure — Semester I.</p><p>Chunk 5: Course Structure — Semester II.</p><p><strong>Technical working</strong></p><p>The process involves:</p><ol><li>Extract document text.</li><li>Detect headings and subheadings.</li><li>Identify section boundaries.</li><li>Group content under the appropriate heading.</li><li>Split oversized sections using recursive chunking.</li><li>Attach structural metadata to each chunk.</li></ol><p>An example chunk representation:</p><pre>chunk = {<br>    &quot;chunk_id&quot;: 1,<br>    &quot;document&quot;: &quot;University_Brochure.pdf&quot;,<br>    &quot;chapter&quot;: &quot;Admission Requirements&quot;,<br>    &quot;section&quot;: &quot;Eligibility Criteria&quot;,<br>    &quot;source_page&quot;: 12,<br>    &quot;text&quot;: &quot;The eligibility criteria are ...&quot;<br>}</pre><p>The metadata is important because it allows us to identify not only what information was retrieved, but also where it came from.</p><p><strong>Advantages</strong></p><ol><li>Preserves the logical organization of documents.</li><li>Improves source attribution.</li><li>Makes retrieval more explainable.</li><li>Allows filtering using metadata.</li><li>Particularly effective for structured academic and policy documents.</li></ol><p><strong>Practical application</strong></p><p>This strategy is particularly useful for university brochures, academic textbooks, legal documents, policy manuals, and research papers.</p><blockquote>Sliding-Window Chunking</blockquote><p>Sliding-window chunking creates overlapping text segments using a moving window. It is closely related to fixed-size chunking, but its main emphasis is on preserving contextual continuity between adjacent chunks.</p><p>Suppose:</p><p>Chunk size = 1,000 characters</p><p>Overlap = 200 characters</p><p>Stride = 800 characters</p><p>The first chunk contains characters 0–999. The second chunk begins at character 800. The third chunk begins at character 1,600. Therefore, adjacent chunks share some content.</p><p><strong>Technical working</strong></p><p>For a document of length N:</p><p>S = C − O</p><p>Start(i) = i × S</p><p>Chunk(i) = D[Start(i)(i) + C]</p><p>Where:</p><ul><li>C represents window size.</li><li>O represents overlap.</li><li>S represents stride.</li></ul><p>This technique is useful when an explanation or argument extends across multiple chunk boundaries.</p><p><strong>Advantages</strong></p><ol><li>Preserves surrounding context.</li><li>Reduces information loss at boundaries.</li><li>Easy to implement.</li><li>Works well with long continuous text.</li></ol><p><strong>Practical application</strong></p><p>Long reports, technical explanations, and continuous prose are common use cases. An important distinction: sliding-window chunking is often an overlap mechanism applied to fixed-size or other chunking methods rather than a completely independent semantic strategy.</p><blockquote>Hierarchical Chunking</blockquote><p>Hierarchical chunking organizes a document into multiple levels of information. Instead of creating only one type of chunk, it creates parent and child relationships.</p><p>For example:</p><p>Document</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*DxDisdz2_y8dDZXVqgbYMA.png" /></figure><p>The larger units are called parent chunks, while smaller units are called child chunks.</p><p><strong>Why is this useful?</strong></p><p>Suppose a user asks: “What are the eligibility criteria for M.Sc. Informatics?”</p><p>A large chapter may contain the complete admission process, but retrieving the entire chapter could introduce irrelevant information. A very small paragraph may contain the eligibility criteria but lack sufficient context. Hierarchical retrieval attempts to solve this problem.</p><p><strong>Technical working</strong></p><ol><li>Divide the document into parent sections.</li><li>Divide each parent section into smaller child chunks.</li><li>Generate embeddings for the child chunks.</li><li>Store parent-child relationships using identifiers.</li><li>Retrieve the most relevant child chunks.</li><li>Fetch their corresponding parent sections.</li><li>Supply appropriate context to the LLM.</li></ol><p>Example:</p><pre>parent = {<br>    &quot;parent_id&quot;: &quot;section_2&quot;,<br>    &quot;text&quot;: &quot;Complete Admission Requirements section&quot;<br>}</pre><pre>child = {<br>    &quot;chunk_id&quot;: &quot;child_2_1&quot;,<br>    &quot;parent_id&quot;: &quot;section_2&quot;,<br>    &quot;text&quot;: &quot;Eligibility criteria for admission ...&quot;<br>}</pre><p>The vector database can index the child chunks while a separate document store maintains the larger parent content.</p><p><strong>Advantages</strong></p><ol><li>Combines precise retrieval with broader context.</li><li>Preserves document hierarchy.</li><li>Useful for complex multi-part questions.</li><li>Supports more flexible retrieval pipelines.</li></ol><p><strong>Practical application</strong></p><p>Large academic textbooks, enterprise knowledge bases, legal documents, and technical manuals.</p><blockquote>Proposition-Based Chunking</blockquote><p>Proposition-based chunking takes a different approach. Instead of dividing a document into paragraphs or sentences, it converts information into small, self-contained factual statements. These statements are called propositions.</p><p>Consider: “RAG combines information retrieval and text generation. It retrieves relevant documents from an external knowledge base and provides them to an LLM as additional context.”</p><p>A proposition-based system may generate:</p><ol><li>RAG combines information retrieval and text generation.</li><li>RAG retrieves relevant documents from an external knowledge base.</li><li>RAG provides retrieved documents to an LLM as additional context.</li></ol><p>Each proposition becomes an independent retrieval unit.</p><p><strong>Technical working</strong></p><ol><li>Extract document text.</li><li>Pass text to an LLM for proposition extraction.</li><li>Convert each proposition into a standalone statement.</li><li>Generate embeddings for the propositions.</li><li>Store the embeddings with source references.</li><li>Retrieve relevant propositions during query processing.</li></ol><p>Example data structure:</p><pre>proposition = {<br>    &quot;proposition_id&quot;: 1,<br>    &quot;source_document&quot;: &quot;RAG_Guide.pdf&quot;,<br>    &quot;source_page&quot;: 5,<br>    &quot;text&quot;: &quot;RAG retrieves relevant documents &quot;<br>            &quot;from an external knowledge base.&quot;<br>}</pre><p><strong>Advantages</strong></p><ol><li>Supports fine-grained factual retrieval.</li><li>Useful for specific factual questions.</li><li>Reduces unrelated information within individual retrieval units.</li><li>Can support knowledge graph construction.</li></ol><p><strong>Practical application</strong></p><p>Fact-heavy knowledge bases, specialized question-answering systems, and knowledge graph-based RAG.</p><blockquote>Agentic Chunking</blockquote><p>Agentic chunking uses an LLM or an intelligent agent to determine how a document should be divided. Traditional chunking approaches use predefined rules, separators, or mathematical similarity thresholds. Agentic chunking attempts to make content-aware decisions.</p><p>For example, consider a technical document containing:</p><ul><li>A definition.</li><li>An explanation.</li><li>A mathematical equation.</li><li>A worked example.</li><li>A conclusion.</li></ul><p>A simple fixed-size splitter may divide these elements across different chunks. An agentic system may instead identify the complete explanation and worked example as a logical unit.</p><p><strong>Technical working</strong></p><ol><li>Extract the document.</li><li>Analyze its content using an LLM.</li><li>Identify logical units of information.</li><li>Determine boundaries based on content and instructions.</li><li>Generate structured chunks.</li><li>Validate chunk size and source references.</li><li>Store embeddings and metadata.</li></ol><p>An example instruction to an LLM:</p><pre>Analyze the following document.</pre><pre>Divide it into meaningful chunks.</pre><pre>Requirements:<br>1. Keep related information together.<br>2. Preserve definitions with their explanations.<br>3. Do not separate questions from their answers.<br>4. Preserve equations with their explanations.<br>5. Avoid exceeding the specified token limit.<br>6. Maintain source references.</pre><pre>Return the chunks in structured JSON format.</pre><p><strong>Advantages</strong></p><ol><li>Can adapt to complex document structures.</li><li>Can preserve relationships between explanations and examples.</li><li>Supports different document types.</li><li>Allows domain-specific chunking instructions.</li></ol><p><strong>Practical application</strong></p><p>Complex technical documentation, specialized knowledge bases, and enterprise applications where document structure is difficult to capture using conventional rules.</p><blockquote>Late Chunking</blockquote><p>Late chunking is an interesting approach because it changes the order of embedding generation and chunking. In conventional chunking, the document is divided into chunks first. Each chunk is then independently passed through the embedding model.</p><p>Traditional approach: Document → Chunking → Embedding Generation</p><p>Late chunking: Document → Contextualized Token Embeddings → Chunking → Chunk Embeddings</p><p><strong>Technical working</strong></p><p>Suppose a document contains several sentences. A long-context embedding model first processes the entire document or a sufficiently large document segment. It generates contextualized token embeddings. These token embeddings contain information influenced by surrounding text. The document is then divided into chunks using token offsets. For each chunk, the corresponding token embeddings are selected and pooled to create one chunk-level embedding. The important difference is that the token embeddings are contextualized before the document is divided into smaller units.</p><p>For example, consider: “Dr. Sharma joined the university in 2010. She later became the head of the department.”</p><p>When the second sentence is embedded independently, the reference “She” may be less informative.</p><p>With late chunking, its token representation can incorporate context from the preceding sentence.</p><p><strong>Advantages</strong></p><ol><li>Preserves information from surrounding text.</li><li>Helps with references that depend on previous context.</li><li>Can improve embeddings for context-dependent passages.</li></ol><p><strong>Practical application</strong></p><p>Long documents containing cross-sentence references, contextual dependencies, and related information spread across multiple passages.</p><h4>How Does Chunking Affect Retrieval Quality?</h4><p>Choosing a chunking strategy is not simply a matter of making documents easier to process. It influences the quality of the retrieval system. Consider three possible situations.</p><h4>Situation 1: Chunks are too small</h4><p>Suppose the user asks: “What are the eligibility criteria for admission?”</p><p>The system retrieves: “Candidates must have completed graduation.”</p><p>However, the next sentence contains an important condition about the required subject or minimum marks. The retrieved chunk may be incomplete. This can lead to an inaccurate answer.</p><h4>Situation 2: Chunks are too large</h4><p>Suppose a chunk contains:</p><ul><li>Eligibility criteria.</li><li>Fee structure.</li><li>Course duration.</li><li>Examination rules.</li><li>Admission procedure.</li></ul><p>The user asks about eligibility. The embedding may represent multiple topics, potentially making retrieval less focused. The LLM also receives unnecessary context.</p><h4>Situation 3: Chunks are meaningful and sufficiently contextual</h4><p>A chunk contains the complete eligibility section, including its heading, requirements, and relevant conditions. This provides a better retrieval unit. The LLM receives more useful information and has a stronger basis for answering. Therefore, the goal is not to create the smallest possible chunks or the largest possible chunks.</p><p><strong>The goal is to create chunks that represent meaningful and retrievable units of information.</strong></p><h4>Important Challenges in Real-World Chunking</h4><p>Even a well-designed chunking algorithm can encounter practical problems.</p><p><strong>1. PDF extraction errors</strong></p><p>PDFs may contain tables, multiple columns, headers, footers, and broken sentences. If extraction produces incorrect text, chunking cannot automatically recover the original document structure.</p><p><strong>2. Tables and mathematical equations</strong></p><p>A table should ideally be preserved with its column headers and associated labels. Similarly, equations should remain connected to their explanations.</p><p><strong>3. Metadata preservation</strong></p><p>Every chunk should ideally retain:</p><ul><li>Document name.</li><li>Page number.</li><li>Chapter.</li><li>Section.</li><li>Chunk ID.</li><li>Parent ID, where applicable.</li></ul><p>This is important for citations, filtering, and debugging.</p><p><strong>4. Duplicate retrieval</strong></p><p>Overlapping chunks can result in multiple retrieved passages containing nearly identical information. Deduplication or diversity-aware retrieval may help.</p><p><strong>5. Chunk size and embedding model limits</strong></p><p>A chunk should not exceed the embedding model’s effective input limit. Otherwise, the model may truncate the input, resulting in incomplete representations.</p><p>Chunking is not merely a document preprocessing operation. It is an important architectural decision that influences embedding quality, retrieval performance, context construction, computational cost, and ultimately the reliability of a RAG application.</p><h3>References and Further Reading</h3><ol><li><a href="https://docs.langchain.com/oss/python/integrations/splitters">LangChain — Text Splitters</a></li><li><a href="https://python.langchain.com/docs/how_to/semantic-chunker/">LangChain — Semantic Chunker</a></li><li><a href="https://arxiv.org/abs/2409.04701">Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models</a></li><li><a href="https://sbert.net/">Sentence Transformers Documentation</a></li><li><a href="https://docs.ragas.io/">RAGAS — RAG Evaluation Framework</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=728525d51d89" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hidden Delays Behind Your AI Response]]></title>
            <link>https://medium.com/@nitisha.aggarwal/the-hidden-delays-behind-your-ai-response-1511c65b822a?source=rss-3ba192adec4------2</link>
            <guid isPermaLink="false">https://medium.com/p/1511c65b822a</guid>
            <category><![CDATA[prompt]]></category>
            <category><![CDATA[latency]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[token]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Nitisha Aggarwal]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 10:23:29 GMT</pubDate>
            <atom:updated>2026-10-31T10:23:29.167Z</atom:updated>
            <content:encoded><![CDATA[<p>In my previous story, <em>“</em><a href="https://medium.com/@nitisha.aggarwal/the-hidden-journey-of-your-prompt-from-enter-key-to-the-first-ai-token-406b79f73e89"><em>The Hidden Journey of Your Prompt: From Enter Key to the First AI Token</em></a><em>,”</em> I explored how a prompt travels through tokenization, model processing, and decoding before the first AI-generated token appears.</p><p>But I received an interesting comment that made me rethink the journey:</p><p><em>“Most explainers focus on tokenization and decoding, while the biggest first-token delays often come from queueing, model routing, safety checks, and prompt augmentation.”</em></p><p>And this raises an important question:</p><p><strong>What if the model is not the reason your AI response is slow?</strong></p><p>Let’s look at what happens behind the scenes in a production LLM system.</p><h4>1. The Prompt Enters the System</h4><p>When a user presses Enter, the request does not necessarily go directly to the model. It first enters the serving infrastructure.</p><p>A typical production architecture may involve:</p><ul><li>API gateway</li><li>Authentication and rate limiting</li><li>Request validation</li><li>Load balancing</li><li>Model routing</li></ul><p>The system must determine whether the request is valid, which model should handle it, and where it should be processed. The journey has already started, but the model may not have received the prompt yet.</p><h4>2. Model Routing: Which Model Should Answer?</h4><p>Not every request needs the same model.</p><p>A production system may route requests based on:</p><ul><li>Task complexity</li><li>Model capabilities</li><li>Availability</li><li>Latency requirements</li><li>Cost constraints</li><li>Context length</li></ul><p>For example:</p><p>A simple classification task might be routed to a smaller model, while a complex reasoning task may require a more capable model. Routing can improve cost and performance, but it also introduces additional processing.</p><p><strong>The routing decision itself becomes part of the latency budget.</strong></p><h4>3. Queueing: The Invisible Waiting Room</h4><p>Imagine a model serving thousands of requests simultaneously. The GPU has limited processing capacity. When incoming requests exceed the available capacity, some requests may need to wait. This is queueing.</p><p>The waiting time can depend on:</p><ul><li>Current request volume</li><li>Number of available workers</li><li>GPU utilization</li><li>Batch formation</li><li>Request priority</li><li>Scheduling strategy</li></ul><p>This is why two identical prompts can experience very different response times. One may be processed immediately. Another may spend valuable milliseconds — or even seconds — waiting for resources. The model’s computational speed is only one part of the story.</p><h4>4. Safety Checks and Prompt Augmentation</h4><p>Before inference, a system may perform additional operations.</p><p>For example:</p><p><strong>Safety and policy checks: </strong>The system may classify the request, identify potential risks, or apply additional safeguards.</p><p><strong>Prompt augmentation: </strong>The original prompt may be expanded with:</p><ul><li>System instructions</li><li>Conversation history</li><li>Retrieved documents</li><li>Tool outputs</li><li>User-specific context</li><li>Formatting instructions</li></ul><p>In a RAG application, retrieval can add another layer of latency. The system may need to generate an embedding, search a vector database, retrieve relevant chunks, and construct the final prompt. The user may have typed only one sentence. But the model could receive a much larger, enriched context. And this preparation happens before the first generated token.</p><h4>5. Finally, the Model Starts Inference</h4><p>Once the request reaches the model-serving engine, the model begins processing the input. This stage is commonly divided into two phases:</p><p><strong>Prefill: </strong>The model processes the input tokens and builds the key-value (KV) cache. The computational cost of prefill is influenced by the input sequence length and model architecture.</p><p><strong>Decode: </strong>The model generates output tokens autoregressively.</p><p>The first generated token appears after the necessary prefill and decoding operations. This is where concepts such as attention, KV caching, and decoding strategies become important.</p><p>But notice something: The model’s inference time is only one component of the total time experienced by the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*0Mk88ucQAeCbeb69tYIXKg.png" /></figure><h4>6. The Metric That Matters: Time to First Token (TTFT)</h4><p>Time to First Token measures how long a user waits before receiving the first generated token. A simplified latency model is:</p><p>TTFT ≈</p><ul><li>Queueing Delay</li><li>Routing Delay</li><li>Safety Processing</li><li>Prompt Augmentation</li><li>Prefill Time</li><li>First-Token Decode Time</li><li>Communication Overhead</li></ul><p>This is a conceptual decomposition, not a universal equation. Some stages may overlap, some may be absent, and some may be included in other measurements. But it highlights an important engineering principle:</p><p><strong>Optimizing the model alone does not necessarily optimize the user experience.</strong></p><h4>7. How Do We Measure It in Production?</h4><p>A single TTFT number tells us how fast the system feels. But it does not tell us where the delay originates. A production system needs stage-level observability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*iUREgiMhv5umgNbK2UMnUA.png" /></figure><h4>The Bigger Lesson</h4><p>When building an LLM application, it is tempting to focus entirely on the model. But production AI systems are not just neural networks. They are distributed systems. They involve scheduling, routing, retrieval, caching, safety, inference, and communication. The first AI token is the result of all these components working together. And if we want to make AI applications faster, we need to understand where the time is actually being spent.</p><p><strong>The question is not simply:</strong></p><p>“How fast can my model generate a token?”</p><p>It is:</p><p>“How long does my entire system take to deliver the first useful token to the user?”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1511c65b822a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Do You Evaluate Retrieval Quality in a RAG-Based Chatbot?]]></title>
            <link>https://medium.com/@nitisha.aggarwal/how-do-you-evaluate-retrieval-quality-in-a-rag-based-chatbot-48e6a14a074f?source=rss-3ba192adec4------2</link>
            <guid isPermaLink="false">https://medium.com/p/48e6a14a074f</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[evaluation-metric]]></category>
            <dc:creator><![CDATA[Nitisha Aggarwal]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 09:27:58 GMT</pubDate>
            <atom:updated>2026-10-31T09:27:58.510Z</atom:updated>
            <content:encoded><![CDATA[<p>Building a RAG-based chatbot is not just about connecting a language model to a document. A chatbot may generate fluent, convincing answers and still fail at its most important task: retrieving the correct information from the knowledge source.</p><p>This became an important realization while building my RAG chatbot using a University brochure. My initial implementation successfully connected document embeddings, semantic search, an LLM, and a Streamlit interface. I tested questions about admission eligibility and CUET-PG selection criteria and manually inspected the retrieved passages.</p><p>But this raised a more fundamental question:</p><p><strong>How do we know whether a RAG retriever is actually retrieving the right information?</strong></p><p>A few successful examples are useful for functional testing, but they do not establish retrieval quality across a representative set of questions. To evaluate retrieval systematically, we need a test dataset, relevance judgments, and quantitative metrics. This post explores the technical evaluation process, the metrics commonly used, and how I can apply them to my RAG chatbot.</p><h4>1. Understanding the RAG Pipeline Before Evaluation</h4><p>A typical RAG pipeline consists of two major stages:</p><p>Stage 1 Retrieval: The system identifies relevant information from a knowledge source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1RB4AOacl4EChoqCHjNXtA.png" /></figure><p>Stage 2 Generation: The retrieved context is supplied to a language model, which generates an answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*kOLBH5-3_MK-_lzD3sFErw.png" /></figure><p>These two stages must be evaluated separately. A retrieval failure occurs when the system does not retrieve the necessary evidence. A generation failure occurs when the correct evidence is retrieved, but the LLM produces an incorrect, incomplete, or unsupported answer.</p><p>For example, suppose the document contains the admission eligibility criteria on page 11. If the retriever returns an unrelated page, the problem lies in retrieval. If the retriever returns the correct admission passage but the LLM reports an incorrect eligibility percentage, the problem lies in answer generation.</p><p><strong>A good evaluation framework must identify which component is responsible for the failure.</strong></p><h4>2. Building a Ground-Truth Evaluation Dataset</h4><p>Before calculating retrieval metrics, we need to establish what a correct retrieval result looks like. This requires a labeled evaluation dataset. For each test question, I need to identify the document chunks that contain relevant evidence.</p><p>For example, consider the following questions from my University brochure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Ck3-CD6d9GXGLy9JEPWBg.png" /></figure><p>The relevant chunks must be identified manually or through a carefully reviewed annotation process. A simple evaluation record could look like this:</p><pre>evaluation_dataset = [<br>    {<br>        &quot;question&quot;: &quot;What is the minimum percentage required for admission?&quot;,<br>        &quot;relevant_chunk_ids&quot;: [28]<br>    },<br>    {<br>        &quot;question&quot;: &quot;What are the CUET-PG cutoff criteria and interview marks?&quot;,<br>        &quot;relevant_chunk_ids&quot;: [28]<br>    }<br>]</pre><p>The chunk IDs in this example are illustrative. In an actual evaluation, I would use the verified chunk IDs from embeddings.json. One important consideration is that a question may have multiple relevant chunks.</p><p>For example, a question about the complete admission process may require evidence from both the eligibility section and the selection procedure section. The ground-truth dataset should capture all relevant evidence, not just one expected chunk. This dataset becomes the reference against which different retrieval configurations can be compared.</p><h4>3. Precision@K: How Much of the Retrieved Context Is Relevant?</h4><p>Precision@K measures the proportion of the top K retrieved chunks that are relevant to the query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*Em--XtOof9UYqbKO9LKgSw.png" /></figure><p>Suppose the retriever returns five chunks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*UO43s4EyfNL5h6lSIqDW9Q.png" /></figure><p>Three out of five retrieved chunks are relevant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*74bG_88ZVeWqvObzQuWWFA.png" /></figure><p>The Precision@5 score is 60%.</p><h4>Why is Precision@K important?</h4><p>In a RAG system, irrelevant context can distract the language model or consume space in the prompt. A higher Precision@K generally means that a larger proportion of the retrieved context is useful. However, precision alone is not enough. A retriever could return only one highly relevant chunk while missing other important evidence.</p><p>That is why we also need Recall@K.</p><h4>4. Recall@K: Did the Retriever Find All the Important Evidence?</h4><p>Recall@K measures the proportion of all known relevant chunks that appear in the top K results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*8h1cxc_Y1oYAWLxTTB_tTw.png" /></figure><p>Suppose a query has four relevant chunks in the document.</p><p>The retriever returns two of those chunks in its top five results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/1*F_1lO4vq9uNa6UHehPb4KQ.png" /></figure><p>The Recall@5 score is 50%.</p><h4>Why is Recall@K important?</h4><p>A RAG system may retrieve one relevant passage but miss another passage containing a critical condition, exception, or additional detail. For questions that require multiple pieces of evidence, recall becomes especially important.</p><p>For example:</p><ul><li>A question about minimum eligibility may require one passage.</li><li>A question about the complete admission process may require several passages.</li><li>A question involving eligibility and selection criteria may require evidence from different sections.</li></ul><p>A high-recall retriever is more likely to provide the LLM with all the necessary evidence.</p><h4>Precision versus Recall</h4><p>The relationship can be summarized as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*H6vUiqS_LL2lt-X6q42hfQ.png" /></figure><p>Increasing K may improve recall because more chunks are retrieved, but it may also reduce precision by introducing irrelevant passages. This is one of the important trade-offs in RAG retrieval.</p><h4>5. Hit Rate@K: Did We Retrieve at Least One Relevant Chunk?</h4><p>Hit Rate@K is a simple metric that checks whether at least one relevant chunk appears in the top K results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*CeHB7ZhnOfDyKaMsmm_gvw.png" /></figure><p>Suppose I evaluate ten questions.</p><p>For eight questions, at least one relevant chunk appears in the top five retrieved results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*jLdXXuY3XVuDDeERnisQ3Q.png" /></figure><p>The Hit Rate@5 is 80%.</p><h4>Why is Hit Rate useful?</h4><p>It is an intuitive starting point for evaluating a retrieval system. It tells us whether the retriever is finding any useful evidence at all. However, it does not tell us how many relevant chunks were retrieved or whether the first relevant result appeared at rank 1 or rank 5. For that, we need ranking metrics.</p><h4>6. Mean Reciprocal Rank (MRR): How Early Does the First Relevant Result Appear?</h4><p>Mean Reciprocal Rank evaluates the position of the first relevant result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*ww3ics8fE9I_dTRsu3cqQQ.png" /></figure><p>Where:</p><ul><li>(N) is the number of evaluation queries.</li><li>(r_i) is the rank of the first relevant result for query (i).</li><li>If no relevant result is retrieved within the evaluation cutoff, the reciprocal rank is 0.</li></ul><p>Consider three test questions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*Jp8Cmg_jmbdeoa6jvai8FA.png" /></figure><p>The MRR is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*VLhuq5Bf-9TD-wkNaPwP2A.png" /></figure><h4>Why is MRR important?</h4><p>The order of retrieved chunks matters. If the most useful passage appears at rank 1, the system has placed the evidence in a prominent position. If it appears at rank 5, the evidence is still available, but the ranking may be less effective. MRR is particularly useful when the main objective is to retrieve at least one strong passage near the top.</p><h4>7. Normalized Discounted Cumulative Gain (NDCG@K): How Good Is the Overall Ranking?</h4><p>Precision, Recall, and Hit Rate treat relevance as a binary property: relevant or irrelevant.</p><p>But in practice, some chunks may be more useful than others.</p><p>For example:</p><ul><li>Highly relevant: directly answers the question.</li><li>Partially relevant: provides supporting information.</li><li>Marginally relevant: related to the topic but not sufficient.</li><li>Irrelevant: does not support the answer.</li></ul><p>NDCG can account for graded relevance.</p><p>The Discounted Cumulative Gain at rank K is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*ezoTpxD5vQWw1uLvLwzakw.png" /></figure><p>Here, (rel_i) is the relevance score of the chunk at rank (i).</p><p>NDCG normalizes the result against the ideal ranking:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*heFzb7oVOopJwDyoW2a1nQ.png" /></figure><p>Where IDCG is the DCG of the ideal ordering of the same relevance judgments. The resulting score is generally between 0 and 1, with 1 representing an ideal ranking.</p><h4>Why is NDCG useful?</h4><p>NDCG rewards a retrieval system for placing highly relevant chunks near the top while accounting for the usefulness of the remaining results. It is especially useful when relevance is graded rather than simply binary. For my initial implementation, NDCG is a more advanced metric that I can introduce after establishing a reliable labeled dataset.</p><h4>8. Does Cosine Similarity Evaluate Retrieval Quality?</h4><p>This is an important distinction in my implementation.</p><p>My chatbot uses:</p><p>sentence-transformers/all-MiniLM-L6-v2</p><p>The query and document chunks are converted into embeddings, and cosine similarity is used to rank the chunks.</p><p>The cosine similarity formula is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*kn7RiHmB0B2rzWlgwv-ANg.png" /></figure><p>A higher score indicates that the query and chunk embeddings are more similar in direction. However, <strong>cosine similarity is a retrieval score, not a complete retrieval evaluation metric</strong>.</p><p>For example, a query might return a chunk with a similarity score of 0.72, but that chunk may not contain the answer. Another query might return a relevant chunk with a score of 0.55. The numerical scores alone do not establish which retrieval result is correct. To evaluate retrieval quality, the retrieved results must be compared against ground-truth relevance judgments.</p><p>This is why metrics such as Precision@K, Recall@K, MRR, and NDCG are needed.</p><h4>9. Evaluating Retrieval Separately from Answer Generation</h4><p>A RAG system has two distinct evaluation problems.</p><h4>Retrieval evaluation</h4><p>Did the system find the correct evidence?</p><p>Metrics include:</p><ul><li>Precision@K</li><li>Recall@K</li><li>Hit Rate@K</li><li>MRR</li><li>NDCG@K</li></ul><h4>Generation evaluation</h4><p>Did the LLM produce a correct, relevant, and grounded answer? Possible evaluation dimensions include:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*a47a8Rb20P2w2xxr86IcFQ.png" /></figure><p>For example, the retriever may successfully return the admission criteria on page 11. However, the LLM may still incorrectly state a numerical cutoff that is not present in the document. In this case, retrieval was successful, but answer generation failed. This distinction is critical when diagnosing a RAG system.</p><h4>10. How I Can Apply These Metrics to My RAG Chatbot</h4><p>My current implementation includes:</p><ul><li>Semantic retrieval using sentence embeddings.</li><li>Cosine similarity ranking.</li><li>Top-K chunk retrieval.</li><li>Source page numbers.</li><li>A Groq-based LLM integration.</li><li>A Streamlit interface for asking questions and inspecting retrieved passages.</li></ul><p>During functional testing, I manually inspected the retrieved passages and checked whether they contained the information required to answer the question. The next step is to build a formal retrieval evaluation pipeline.</p><p>My planned workflow is:</p><pre>Create Evaluation Questions<br>          |<br>          v<br>Identify Relevant Chunk IDs<br>          |<br>          v<br>Run Queries Through Retriever<br>          |<br>          v<br>Collect Top-K Retrieved Chunks<br>          |<br>          v<br>Compare Against Ground Truth<br>          |<br>          v<br>Calculate Retrieval Metrics<br>          |<br>          v<br>Compare Retrieval Configurations</pre><p>I can then compare different configurations, such as:</p><ul><li>Top-3 versus Top-5 versus Top-10 retrieval.</li><li>Different chunk sizes.</li><li>Different chunk overlap values.</li><li>Different embedding models.</li><li>Semantic retrieval versus hybrid retrieval.</li><li>Retrieval with and without reranking.</li></ul><p>The important part is to use the same evaluation questions and relevance labels for each experiment. This makes the comparison more meaningful.</p><p>My current chatbot has passed initial functional tests using questions from the University brochure. However, I have not yet calculated formal retrieval metrics across a labeled benchmark. That is the next step in moving from a working RAG prototype toward a systematically evaluated RAG application.</p><p><strong>The goal is not simply to retrieve chunks with high similarity scores. The goal is to retrieve the evidence that actually helps answer the user’s question.</strong></p><h3>Follow the Journey</h3><ul><li><strong>Day 1:</strong> <a href="https://medium.com/@nitisha.aggarwal/day-1-of-building-my-rag-chatbot-8180e82dd3d8">Getting Python, Anaconda, and VS Code to Work Together</a></li><li><strong>Day 2:</strong> <a href="https://medium.com/@nitisha.aggarwal/day-2-of-building-my-rag-chatbot-e5a118e7c8b0">Document Processing: Text Extraction, Chunking, and Embeddings</a></li><li><strong>Day 3:</strong> <a href="https://medium.com/@nitisha.aggarwal/day-3-of-building-my-rag-chatbot-958dd674a8d5">From Text Chunks to Numerical Vectors</a></li><li><strong>Day 4:</strong> <a href="https://medium.com/@nitisha.aggarwal/day-3-of-building-my-rag-chatbot-8aea956622a1">From Semantic Retrieval to a Working RAG Chatbot</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48e6a14a074f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 4 of building my RAG chatbot]]></title>
            <link>https://medium.com/@nitisha.aggarwal/day-3-of-building-my-rag-chatbot-8aea956622a1?source=rss-3ba192adec4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8aea956622a1</guid>
            <category><![CDATA[groq]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Nitisha Aggarwal]]></dc:creator>
            <pubDate>Tue, 27 Oct 2026 06:54:03 GMT</pubDate>
            <atom:updated>2026-10-27T06:54:24.186Z</atom:updated>
            <content:encoded><![CDATA[<p>After setting up my development environment, processing a university brochure, and generating embeddings, it was time to bring everything together.</p><p><strong>Day 4 was about turning individual components into a working RAG application.</strong></p><p>Until Day 3, I had been working on preparing the knowledge source. Now I needed to build the complete pipeline that could accept a user’s question, retrieve relevant information from the document, send that information to a language model, and display a grounded answer through a web interface.</p><p>This was the point where the project started to feel like a real application rather than a collection of scripts.</p><h4>1. Revisiting the RAG Architecture</h4><p>Before integrating the components, I needed to understand how data would flow through the application. The architecture I implemented was:</p><p><strong>User Question → Query Embedding → Similarity Search → Top-K Retrieval → Context Construction → Groq LLM → Answer with Sources</strong></p><p>Each component has a specific responsibility.</p><ul><li><strong>Embedding model:</strong> Converts the user’s question into a numerical vector.</li><li><strong>Retrieval layer:</strong> Compares the query vector against the document chunk embeddings.</li><li><strong>Context builder:</strong> Collects the most relevant text chunks and their source page numbers.</li><li><strong>LLM layer:</strong> Uses the retrieved context to formulate a natural-language answer.</li><li><strong>Streamlit frontend:</strong> Provides the interface for asking questions and inspecting the retrieved sources.</li></ul><p>The important distinction is that the language model does not need to answer every question from its general training knowledge. The application first retrieves relevant passages from the selected document and supplies them as context.</p><h4>2. Implementing Semantic Retrieval with Python and NumPy</h4><p>I used the sentence-transformers/all-MiniLM-L6-v2 model to represent text as numerical embeddings. The document embeddings had already been generated and saved in week01/output/embeddings.json. Rather than recomputing every document embedding for each question, the application loads the existing data and generates an embedding only for the incoming query.</p><p>The retrieval process works as follows:</p><ol><li>Convert the user’s question into an embedding.</li><li>Compare it with the stored embeddings of the document chunks.</li><li>Calculate cosine similarity to estimate how closely each chunk matches the query.</li><li>Rank the chunks by similarity.</li><li>Return the top five chunks for the RAG pipeline.</li></ol><p>In rag_pipeline.py, I implemented the retrieval logic in a separate function:</p><pre>def retrieve_chunks(question, top_k=TOP_K):<br>    # Embed the question<br>    # Calculate cosine similarity<br>    # Rank chunks by similarity<br>    # Return the top-k results<br>    ...</pre><p>The snippet illustrates the function’s role; the complete implementation is available in the project repository. I also retained the filtering logic that excludes a table-of-contents chunk from retrieval. This helped avoid returning a contents page instead of the actual section containing the information.</p><p>One lesson from this stage was that <strong>semantic similarity is not the same as answer correctness</strong>. The quality of the final answer depends heavily on whether retrieval finds the right passages.</p><h4>3. Connecting the Retrieval Pipeline to Groq</h4><p>Once retrieval was working, I needed a language model to turn the retrieved passages into a readable answer. I integrated the Groq API using the Python groq SDK and selected the model openai/gpt-oss-20b. The integration is handled in rag_pipeline.py, keeping the retrieval and answer-generation logic separate from the user interface. The answer-generation function constructs a context from the retrieved chunks, including their source PDF page numbers. It then sends the context and user’s question to the model through the chat completions API.</p><p>Conceptually, the request contains:</p><ul><li><strong>System instructions:</strong> Answer using the supplied document context and identify relevant source pages.</li><li><strong>User question:</strong> The information the user wants to find.</li><li><strong>Retrieved context:</strong> The text passages selected by semantic search.</li></ul><p>The model is instructed to stay grounded in the supplied context rather than inventing missing details. I also configured the generation temperature to 0.2 to encourage relatively consistent responses. The API key is loaded from a local .env file through python-dotenv, rather than being hardcoded into the Python source files. This separation matters for two reasons: credentials are easier to manage, and the application code can be shared without exposing the API key.</p><h4>4. Building the Web Interface with Streamlit</h4><p>The next challenge was making the pipeline usable without running a Python script manually for every question. For this, I used <strong>Streamlit</strong>, a Python framework for building interactive web applications.</p><p>I created app.py as the frontend and kept rag_pipeline.py responsible for retrieval and generation. This separation makes the project easier to understand, test, and extend.</p><p>The Streamlit application includes:</p><ul><li>A chat-style interface for entering questions and displaying responses.</li><li>Session-state-based conversation history.</li><li>An assistant response generated by the RAG pipeline.</li><li>An expandable <strong>“View retrieved sources”</strong> section.</li><li>Source page numbers, similarity scores, and retrieved text for inspection.</li></ul><p>The frontend calls the pipeline through a single entry point:</p><pre>answer, retrieved_chunks = answer_question(question)</pre><p>This function returns both the generated answer and the retrieved chunks. That is useful because the user can read the answer while also inspecting the evidence selected by the retrieval layer. Streamlit’s chat elements and session state make it possible to build this interactive experience using Python, without creating a separate JavaScript frontend. The official documentation explains the underlying chat-interface approach using components such as st.chat_input and st.chat_message.</p><h4>5. Testing the Complete Application</h4><p>With the interface connected to the backend, I tested the complete workflow in the browser. The source document was a University brochure.</p><blockquote>Test 1: Admission Eligibility</blockquote><p><strong>Question:</strong> What is the minimum percentage required for admission to M.Sc. Informatics?</p><p>The chatbot retrieved the relevant admission passage and returned the minimum eligibility percentage of <strong>50% for the Unreserved (UR) category</strong>, citing page 11 of the PDF.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*gkzpKAZAnuOWm8uiDq0Dcw.png" /></figure><p>This test checked whether the system could retrieve a specific fact and connect the generated response to its source.</p><blockquote>Test 2: CUET-PG Cutoff and Interview Criteria</blockquote><p><strong>Question:</strong> What are the CUET-PG cutoff criteria and interview marks for admission?</p><p>The brochure states that admission depends on CUET-PG performance and university norms, with final selection based on cutoff and interview marks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*QCEcy6Ecq9H6Z0iMzOC-vw.png" /></figure><p>The brochure states that admission depends on CUET-PG performance and university norms, with final selection based on cutoff and interview marks. However, the brochure did not provide the exact numerical cutoff or interview weightage in the retrieved passage. The chatbot correctly communicated that limitation instead of inventing figures.</p><h4>What These Tests Demonstrated</h4><p>The tests helped verify that the major components were connected:</p><ul><li>The question could be converted into an embedding.</li><li>Relevant document passages could be retrieved.</li><li>The Groq model could generate an answer from the supplied context.</li><li>The Streamlit interface could display the answer and supporting passages.</li></ul><p>They were functional tests of the workflow, not a comprehensive evaluation of retrieval precision or answer accuracy across a large test set. Improving those measurements is one of the next steps.</p><h4>6. Understanding the Complete Code Structure</h4><p>I organized the project so that the main responsibilities were separated.</p><pre>RAG-Chatbot/<br>│<br>├── app.py                  # Streamlit web interface<br>├── rag_pipeline.py         # Retrieval and answer generation<br>├── test_groq.py            # Groq connection test<br>├── README.md               # Project documentation<br>├── .gitignore              # Files excluded from Git<br>├── .env                    # Local API credentials<br>│<br>└── week01/<br>    ├── documents/          # Source documents<br>    ├── output/             # Extracted text, chunks, embeddings<br>    ├── embed_chunks.py     # Embedding generation<br>    └── search_chunks.py    # Initial semantic search experiment</pre><p>The .env file and generated document data are kept out of version control through .gitignore. The exact set of tracked files is controlled by Git, so sensitive files must be checked before committing or pushing.</p><p>This organization also makes it easier to improve retrieval without rewriting the frontend, or redesign the frontend without changing the underlying retrieval algorithm.</p><h4>7. Debugging, Dependencies, and Model Integration</h4><p>The final integration stage also involved troubleshooting.</p><p>I worked through environment and package setup issues, tested the Groq connection separately, and checked model availability before connecting it to the full pipeline. Keeping a dedicated test_groq.py script was useful because it allowed me to isolate API connectivity problems from retrieval or Streamlit problems. I also learned that an API integration involves more than installing a package. The credentials, selected model ID, request format, and application code must all work together.</p><p>Testing these pieces separately made the debugging process more manageable.</p><h4>8. Preparing the Project for GitHub</h4><p>Once the chatbot was working in the browser, I prepared the project for sharing.</p><p>This included:</p><ul><li>Reviewing .gitignore to exclude .env, Python cache files, and generated document outputs.</li><li>Writing README.md with the project overview, architecture, technologies, setup instructions, and sample questions.</li><li>Checking the files included in the Git commit.</li><li>Committing the application code and documentation.</li></ul><p>The project was committed locally with the message:</p><p>Build University of Delhi RAG chatbot</p><p>The final Git check showed the intended files in the commit: .gitignore, README.md, app.py, rag_pipeline.py, and test_groq.py.</p><p>I then completed the <a href="https://github.com/nitisha22/rag-chatbot">GitHub </a>publishing step, making the repository available for others to inspect. Publishing the source code is separate from deploying the Streamlit application to a public hosting service.</p><h4>9. What I Learned from Building the End-to-End Pipeline</h4><p>This four-day project helped me connect several concepts that I had previously explored individually.</p><p><strong>Embeddings are only one part of a RAG system.</strong> They provide a way to compare questions with document passages, but they do not generate answers by themselves.</p><p><strong>Retrieval quality directly affects generation quality.</strong> If the wrong chunks are selected, even a capable language model may produce an incomplete or irrelevant answer.</p><p><strong>Source visibility improves debuggability.</strong> Showing retrieved passages, similarity scores, and page numbers makes it easier to understand why the system generated a particular response.</p><p><strong>A working interface is part of the engineering process.</strong> Streamlit helped me turn a Python pipeline into an interactive application that could be tested with realistic questions.</p><p><strong>Version control is more than uploading files.</strong> Documentation, careful staging, and excluding credentials are essential parts of preparing a project for others to use.</p><h4>Final Reflection</h4><p>Over four days, I went from configuring Python and debugging my development environment to building a working RAG chatbot with semantic search, a hosted LLM, a Streamlit interface, source inspection, and GitHub version control.</p><p>The most valuable outcome was not simply that the chatbot worked. It was learning how the components fit together, how to isolate problems during integration, and how to test the system against the actual information in its source document.</p><p><strong>This concludes my four-day build milestone, but the next phase is about making the retrieval more accurate, the evaluation more systematic, and the application more reliable.</strong></p><h4>Follow the Journey</h4><ul><li><strong>Day 1:</strong> <a href="https://medium.com/@nitisha.aggarwal/day-1-of-building-my-rag-chatbot-8180e82dd3d8">Getting Python, Anaconda, and VS Code to Work Together</a></li><li><strong>Day 2:</strong> <a href="https://medium.com/@nitisha.aggarwal/day-2-of-building-my-rag-chatbot-e5a118e7c8b0">Document Processing: Text Extraction, Chunking, and Embeddings</a></li><li><strong>Day 3:</strong> <a href="https://medium.com/@nitisha.aggarwal/day-3-of-building-my-rag-chatbot-958dd674a8d5">From Text Chunks to Numerical Vectors</a></li><li><strong>Day 4:</strong> From Semantic Retrieval to a Working RAG Chatbot — this article.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8aea956622a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Multimodal RAG Pipeline for PDFs Containing Text, Images, Tables, and Diagrams]]></title>
            <link>https://medium.com/@nitisha.aggarwal/building-a-multimodal-rag-pipeline-for-pdfs-containing-text-images-tables-and-diagrams-de7b79c33cc3?source=rss-3ba192adec4------2</link>
            <guid isPermaLink="false">https://medium.com/p/de7b79c33cc3</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[multimodal]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Nitisha Aggarwal]]></dc:creator>
            <pubDate>Thu, 22 Oct 2026 06:45:35 GMT</pubDate>
            <atom:updated>2026-10-22T06:45:35.944Z</atom:updated>
            <content:encoded><![CDATA[<p>When I started building my RAG chatbot, my initial focus was straightforward: extract text from a PDF, divide it into chunks, convert those chunks into embeddings, and retrieve the most relevant information when a user asks a question.</p><p>My initial pipeline looked like this:</p><p><strong>PDF → Text Extraction → Chunking → Embeddings → Similarity Search</strong></p><p>It was a good starting point. I learned how to extract text from a document, generate vector representations using sentence-transformers/all-MiniLM-L6-v2, and retrieve relevant chunks using semantic similarity. But as I explored the problem further, I encountered an important limitation.</p><blockquote><strong>What happens when the information needed to answer a question is not available as ordinary text?</strong></blockquote><p>Consider a university brochure containing:</p><ul><li>Eligibility criteria written in paragraphs.</li><li>A table describing course structure.</li><li>A diagram illustrating the admission process.</li><li>A chart showing seat distribution.</li><li>Scanned pages containing text embedded in images.</li></ul><p>A conventional text-extraction pipeline may successfully extract the eligibility paragraph while missing information contained in the diagram or scanned page. The resulting chatbot might confidently answer a question using incomplete evidence. This led me to a more fundamental engineering question:</p><blockquote><em>How can a RAG pipeline retrieve and understand information distributed across text, images, tables, and visual layouts in the same PDF?</em></blockquote><p>The answer is to extend the pipeline into a multimodal document-processing architecture.</p><h4>1. Understanding the Problem: A PDF Is Not Just a Text File</h4><p>PDF stands for Portable Document Format. It is designed to preserve document appearance and layout, not necessarily to provide a clean, structured representation of its content. A PDF can contain several different types of information:</p><p><strong>1. Digitally generated text: </strong>Text produced by word processors or publishing software. It can often be extracted directly.</p><p><strong>2. Raster images: </strong>Photographs, scanned pages, screenshots, and illustrations represented as pixels.</p><p><strong>3. Vector graphics: </strong>Diagrams, lines, shapes, arrows, and charts drawn using PDF graphics instructions rather than stored as image files.</p><p><strong>4. Tables and structured layouts: </strong>Information arranged spatially into rows and columns, where the relationships between cells matter.</p><p><strong>5. Mixed-content pages: </strong>Pages containing ordinary text, embedded images, captions, tables, and diagrams together.</p><p>These differences matter because each content type requires an appropriate extraction strategy. For example, pypdf can extract text from a digitally generated PDF, but it does not automatically interpret the meaning of a chart or perform OCR on a scanned page.</p><p>Similarly, extracting an image from a PDF gives us the image data, but not necessarily the semantic information represented by that image.</p><p>Therefore, a robust RAG pipeline must separate three tasks:</p><ol><li><strong>Extraction:</strong> Obtain the underlying text, image data, and structural information.</li><li><strong>Interpretation:</strong> Convert relevant visual content into information the system can understand and retrieve.</li><li><strong>Retrieval:</strong> Find the appropriate evidence for the user’s question.</li></ol><p>Confusing these tasks is one of the reasons document-based chatbots can produce incomplete answers.</p><h4>2. Designing a Multimodal RAG Architecture</h4><p>The first design decision is to introduce separate processing paths for textual and visual content.</p><p>The architecture becomes:</p><pre>PDF DOCUMENT<br>                        |<br>              Document Processing<br>                        |<br>           Page-Level Content Analysis<br>                        |<br>           +------------+------------+<br>           |            |            |<br>         Text         Images       Tables<br>           |            |            |<br>      Text Parsing   OCR / Vision   Table<br>           |         Processing   Extraction<br>           |            |            |<br>           +------------+------------+<br>                        |<br>               Content Normalization<br>                        |<br>           Metadata and Relationships<br>                        |<br>                 Chunk Generation<br>                        |<br>              Embedding Generation<br>                        |<br>                  Search Index<br>                        |<br>                User&#39;s Question<br>                        |<br>               Hybrid Retrieval<br>                        |<br>                    Reranking<br>                        |<br>              Relevant Evidence<br>                        |<br>                 LLM Generation<br>                        |<br>              Answer with Sources</pre><p>This architecture preserves the strengths of a conventional text RAG pipeline while introducing additional mechanisms for visual content. An important principle is that not every page needs every processing technique.</p><p>A page containing ordinary paragraphs may require only text extraction. A scanned page may require OCR. A chart may require visual interpretation or structured data extraction. The processing system should choose the appropriate method based on the content detected on each page. This avoids unnecessary computational cost and makes the pipeline easier to maintain.</p><h4>3. Step One: Extract Text and Preserve Document Structure</h4><p>My existing pipeline uses pypdf for PDF text extraction. I would retain it as a baseline while introducing PyMuPDF for more detailed page-level processing and image extraction. PyMuPDF is available through the pymupdf package.</p><pre>pip install pymupdf</pre><p>A basic page-level text extraction script looks like this:</p><pre>import fitz</pre><pre>pdf_path = &quot;week01/documents/sample.pdf&quot;</pre><pre>doc = fitz.open(pdf_path)</pre><pre>for page_number, page in enumerate(doc, start=1):<br>    text = page.get_text(&quot;text&quot;)</pre><pre>    if text.strip():<br>        print(f&quot;\nPage {page_number}&quot;)<br>        print(text[:500])</pre><p>This approach allows us to inspect each page individually and retain page numbers for later retrieval and citation. However, text extraction alone is not enough. PDF text may be stored in an order that differs from its visual reading order. Multi-column layouts, footnotes, sidebars, and tables can create problems.</p><p>For example, a two-column academic paper might be extracted by alternating between columns rather than reading the first column from top to bottom. For more complex documents, I would consider layout-aware parsing, reading-order reconstruction, and explicit heading detection.</p><h3>Why metadata matters</h3><p>Instead of storing only a string, I would store a structured record:</p><pre>record = {<br>    &quot;document_id&quot;: &quot;university_brochure&quot;,<br>    &quot;page&quot;: 11,<br>    &quot;content_type&quot;: &quot;text&quot;,<br>    &quot;section&quot;: &quot;Eligibility Criteria&quot;,<br>    &quot;content&quot;: (<br>        &quot;The extracted text from the relevant &quot;<br>        &quot;section of the document.&quot;<br>    )<br>}</pre><p>This metadata becomes useful when a retrieved chunk must be traced back to its source. It also helps filter results by document, page, or content type.</p><h4>4. Step Two: Extract Embedded Images</h4><p>The next task is to identify and extract image objects embedded in the PDF.</p><pre>import fitz<br>from pathlib import Path</pre><pre>pdf_path = &quot;week01/documents/sample.pdf&quot;</pre><pre>output_dir = Path(&quot;week01/output/images&quot;)<br>output_dir.mkdir(parents=True, exist_ok=True)</pre><pre>doc = fitz.open(pdf_path)</pre><pre>for page_number, page in enumerate(doc, start=1):</pre><pre>    images = page.get_images(full=True)</pre><pre>    for image_index, image_info in enumerate(<br>        images, start=1<br>    ):<br>        xref = image_info[0]</pre><pre>        image_data = doc.extract_image(xref)</pre><pre>        extension = image_data[&quot;ext&quot;]<br>        image_bytes = image_data[&quot;image&quot;]</pre><pre>        image_path = output_dir / (<br>            f&quot;page_{page_number}_image_{image_index}.&quot;<br>            f&quot;{extension}&quot;<br>        )</pre><pre>        image_path.write_bytes(image_bytes)</pre><pre>        print(f&quot;Extracted: {image_path}&quot;)</pre><p>This script extracts embedded raster images and stores them with page-specific filenames.</p><p>For example:</p><pre>week01/<br>    output/<br>        images/<br>            page_12_image_1.png<br>            page_15_image_1.jpeg<br>            page_18_image_1.png</pre><p>These files can subsequently be processed by OCR engines or vision-language models.</p><p>However, this method has an important limitation.</p><p>page.get_images() identifies embedded image objects. It does not guarantee that every visible visual element on a page will be returned as an image.</p><p>A flowchart might consist entirely of vector shapes and text. A chart might be drawn using graphical instructions. A scanned page might also contain a large background image with no separately extractable text. Therefore, I would combine image extraction with page rendering and layout analysis.</p><p>For example, rendering a page into an image can preserve its overall visual appearance:</p><pre>import fitz<br>from pathlib import Path</pre><pre>doc = fitz.open(<br>    &quot;week01/documents/sample.pdf&quot;<br>)</pre><pre>output_dir = Path(&quot;week01/output/rendered_pages&quot;)<br>output_dir.mkdir(parents=True, exist_ok=True)</pre><pre>for page_number, page in enumerate(doc, start=1):</pre><pre>    pixmap = page.get_pixmap(<br>        matrix=fitz.Matrix(2, 2),<br>        alpha=False<br>    )</pre><pre>    output_path = output_dir / (<br>        f&quot;page_{page_number}.png&quot;<br>    )</pre><pre>    pixmap.save(output_path)</pre><p>Rendering is particularly useful when the meaning of an image depends on nearby text, captions, arrows, or spatial relationships. The trade-off is storage and processing cost: rendering every page at high resolution can generate large amounts of data.</p><p>In a production system, I would render pages selectively or use appropriate resolution settings based on the document and task.</p><h4>5. Step Three: Convert Visual Content into Searchable Information</h4><p>Extracting an image is only the first part of the problem. A conventional text embedding model cannot automatically understand the original pixels of an image. To make visual information searchable, I would consider three techniques.</p><h4>A. Optical Character Recognition (OCR)</h4><p>OCR converts text contained in images into machine-readable text. For example, a scanned eligibility document might contain:</p><blockquote><em>Minimum qualifying percentage: 55%.</em></blockquote><p>An OCR engine can extract this text, allowing the resulting content to enter the ordinary text-retrieval pipeline. OCR is useful for scanned documents, forms, labels, and text-heavy images. Possible tools include Tesseract, PaddleOCR, and document-processing services that provide OCR and layout information. However, OCR is not equivalent to understanding an image.</p><p>It may extract the labels from a flowchart without explaining how the boxes are connected. It may also confuse similar characters, misread small text, or lose table structure.</p><h4>B. Vision-language models</h4><p>A vision-language model accepts visual input and generates a textual interpretation. Suppose a university brochure contains a diagram representing this process:</p><pre>Online Registration<br>         |<br>         v<br>Document Verification<br>         |<br>         v<br>Merit List Publication</pre><p>A vision-language model might generate the following description:</p><blockquote><em>The admission process consists of online registration, document verification, and merit-list publication, in that order.</em></blockquote><p>This description provides a semantic representation of the diagram. It can be embedded using the same text embedding model used for ordinary document chunks.</p><p>A query such as “What happens after online registration?” may then retrieve the description even if the user’s wording differs from the labels in the original diagram.</p><p>However, a generated description is an interpretation, not guaranteed ground truth. For high-precision applications, I would retain the original image, record the model and processing status, and validate important claims against the source.</p><h4>C. Structured extraction for tables and charts</h4><p>Tables and charts require special care because their meaning often depends on numerical values, labels, units, and relationships.</p><p>Consider a table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*lc_lvLiEb65IiHp4TM91Xw.png" /></figure><p>A useful normalized representation could be:</p><pre>{<br>  &quot;page&quot;: 15,<br>  &quot;type&quot;: &quot;table&quot;,<br>  &quot;headers&quot;: [&quot;Programme&quot;, &quot;Seats&quot;],<br>  &quot;rows&quot;: [<br>    [&quot;Programme A&quot;, 40],<br>    [&quot;Programme B&quot;, 30]<br>  ]<br>}</pre><p>This is more useful for answering numerical questions than a vague description such as “The table shows seat distribution.”</p><p>For charts, the pipeline should preserve axes, units, legends, labels, and data values wherever possible. A general vision model may describe the trend correctly while estimating a specific numerical value incorrectly. The key design principle is to use OCR for text recognition, vision models for semantic interpretation, and structured extraction for content whose exact organization or numerical values matter.</p><h4>6. Step Four: Normalize Text and Visual Content into a Common Schema</h4><p>After extraction and interpretation, I need a common data model that allows the retrieval system to treat information consistently.</p><p>For example:</p><pre>records = [<br>    {<br>        &quot;id&quot;: &quot;page_11_text_1&quot;,<br>        &quot;document_id&quot;: &quot;university_brochure&quot;,<br>        &quot;page&quot;: 11,<br>        &quot;type&quot;: &quot;text&quot;,<br>        &quot;content&quot;: (<br>            &quot;Applicants must satisfy the prescribed &quot;<br>            &quot;eligibility criteria.&quot;<br>        )<br>    },<br>    {<br>        &quot;id&quot;: &quot;page_12_visual_1&quot;,<br>        &quot;document_id&quot;: &quot;university_brochure&quot;,<br>        &quot;page&quot;: 12,<br>        &quot;type&quot;: &quot;image_description&quot;,<br>        &quot;content&quot;: (<br>            &quot;The admission process consists of &quot;<br>            &quot;registration, document verification, &quot;<br>            &quot;and merit-list publication.&quot;<br>        ),<br>        &quot;image_path&quot;: (<br>            &quot;week01/output/images/page_12_image_1.png&quot;<br>        )<br>    }<br>]</pre><p>This representation separates the content from its metadata. The content field contains the text used for embedding. The metadata records where the information came from and how it was obtained.</p><p>For a more advanced implementation, I would add:</p><ul><li>document_id and a document version.</li><li>page and bounding-box coordinates.</li><li>section or parent-chunk identifiers.</li><li>content_type, such as text, OCR, table, or image description.</li><li>source_image_path or a stable asset identifier.</li><li>extraction_method and processing status.</li><li>Links between captions, tables, figures, and surrounding paragraphs.</li></ul><p>This common schema becomes the bridge between document processing and retrieval. It also makes the pipeline easier to debug. If the chatbot retrieves incorrect information, I can inspect whether the problem originated in extraction, visual interpretation, chunking, or retrieval.</p><h4>7. Step Five: Chunk Multimodal Documents Without Losing Context</h4><p>Chunking is one of the most important design decisions in RAG.</p><p>In a text-only pipeline, we often split extracted text into overlapping chunks of a specified size. My existing project has already explored chunking and overlap.</p><p>However, multimodal documents require additional considerations.</p><p>Imagine a paragraph that says:</p><blockquote><em>The following figure describes the admission process.</em></blockquote><p>If this paragraph is stored separately from the figure’s description, retrieval may return the paragraph without the information needed to answer the question.</p><p>The solution is not necessarily to merge the entire page into one enormous chunk. Instead, I would preserve semantic relationships between content elements.</p><p>For example:</p><pre>record = {<br>    &quot;id&quot;: &quot;page_12_admission_process&quot;,<br>    &quot;document_id&quot;: &quot;university_brochure&quot;,<br>    &quot;page&quot;: 12,<br>    &quot;type&quot;: &quot;text_with_visual_context&quot;,<br>    &quot;content&quot;: (<br>        &quot;The figure describes the admission process. &quot;<br>        &quot;Applicants first complete online registration, &quot;<br>        &quot;then undergo document verification, followed &quot;<br>        &quot;by merit-list publication.&quot;<br>    ),<br>    &quot;related_assets&quot;: [<br>        &quot;page_12_image_1&quot;<br>    ]<br>}</pre><p>The associated image can be stored separately and retrieved when required.</p><p>This approach gives us two benefits:</p><ol><li>Text search can retrieve a meaningful description of the visual.</li><li>The original image remains available when the query requires direct visual inspection.</li></ol><p>For tables, I would preserve headers when splitting large tables. For diagrams, I would keep captions and explanations connected to the diagram. For long documents, I would preserve section-level metadata so that related chunks can be retrieved together.</p><p>The objective is to preserve the meaning of the document, not merely divide it into smaller strings.</p><h4>8. Step Six: Generate Embeddings and Build the Index</h4><p>My existing embedding model is:</p><p>sentence-transformers/all-MiniLM-L6-v2</p><p>This model generates 384-dimensional embeddings for text. That means a textual description of a diagram can be embedded using the same model as an ordinary text chunk.</p><p>Conceptually:</p><pre>from sentence_transformers import SentenceTransformer</pre><pre>model = SentenceTransformer(<br>    &quot;sentence-transformers/all-MiniLM-L6-v2&quot;<br>)</pre><pre>texts = [<br>    record[&quot;content&quot;]<br>    for record in records<br>]</pre><pre>embeddings = model.encode(<br>    texts,<br>    normalize_embeddings=True<br>)</pre><pre>print(embeddings.shape)</pre><p>If there are two records, the resulting shape will be (2, 384).</p><p>For cosine similarity, normalized embeddings make similarity calculations straightforward using a dot product. However, there is an important distinction:</p><blockquote><strong>Embedding an image description is not the same as embedding the original image.</strong></blockquote><p>The model sees the generated textual description, not the image pixels. If the description omits a detail, that detail will also be absent from the embedding representation.</p><p>For direct image retrieval, I would evaluate a suitable multimodal embedding model that supports both text and image inputs. Depending on the model, text and image vectors may be compared in a shared embedding space.</p><p>Another option is to maintain separate text and visual indexes and combine their search results.</p><p>For a beginner-friendly implementation, I would start with text embeddings and image descriptions before adding multimodal embeddings.</p><p>This allows the existing pipeline to evolve incrementally instead of requiring a complete redesign.</p><h4>9. Step Seven: Implement Retrieval for Mixed-Content Questions</h4><p>Consider these three questions about a university brochure:</p><ol><li>“What are the eligibility criteria?”</li><li>“Explain the admission process shown in the diagram.”</li><li>“How many seats are available for Programme A?”</li></ol><p>Each question requires a different type of evidence.</p><p>The first is likely answered by a text chunk. The second may require an image description and the original diagram. The third requires a table record with accurate numerical values. A robust retrieval pipeline should not assume that every query can be answered using the same type of chunk. I would combine the following techniques.</p><h4>Semantic vector search</h4><p>Find chunks that are conceptually similar to the query. This helps when the user’s wording differs from the wording in the document.</p><h4>Keyword search</h4><p>Use lexical retrieval, such as BM25, to identify exact names, programme codes, numerical labels, and technical terms. This is particularly important when the exact value matters.</p><h4>Metadata filtering</h4><p>Filter by document, page range, content type, or section where appropriate.</p><h4>Reranking</h4><p>Use a reranker to reorder the candidate results based on their relevance to the user’s question.</p><h4>Visual evidence retrieval</h4><p>If the question depends on visual details, retrieve the relevant image or page crop and pass it to a vision-capable model.</p><p>These techniques can be combined into a single retrieval pipeline.</p><p>For example:</p><pre>User Query<br>    |<br>    v<br>Query Analysis<br>    |<br>    +----&gt; Text Vector Search<br>    |<br>    +----&gt; Keyword Search<br>    |<br>    +----&gt; Table / Visual Description Search<br>                    |<br>                    v<br>             Candidate Results<br>                    |<br>                    v<br>                 Reranking<br>                    |<br>                    v<br>          Relevant Text + Metadata<br>                    |<br>                    v<br>        Retrieve Original Image if Needed<br>                    |<br>                    v<br>          LLM Generates the Answer<br>                    |<br>                    v<br>            Source Page Citations</pre><p>The final answer should be grounded in retrieved evidence rather than in an unverified image description alone.</p><h4>10. Step Eight: Generate Answers with Traceable Sources</h4><p>Retrieval quality is only one part of a successful RAG system. The chatbot must also explain where its answer came from. I would preserve source information throughout the pipeline so that every retrieved chunk can be associated with the original PDF and page number.</p><p>A retrieved record might look like this:</p><pre>retrieved_record = {<br>    &quot;content&quot;: (<br>        &quot;The admission process includes registration, &quot;<br>        &quot;document verification, and merit-list publication.&quot;<br>    ),<br>    &quot;document_id&quot;: &quot;university_brochure&quot;,<br>    &quot;page&quot;: 12,<br>    &quot;type&quot;: &quot;image_description&quot;,<br>    &quot;related_assets&quot;: [<br>        &quot;page_12_image_1&quot;<br>    ]<br>}</pre><p>The generation prompt can instruct the language model to answer using only the supplied evidence, distinguish confirmed information from uncertainty, and cite the source page.</p><p>For example:</p><blockquote><em>The admission process consists of online registration, document verification, and merit-list publication. Source: University brochure, page 12.</em></blockquote><p>For visual questions, the source citation should refer to the original figure or page, not merely to the generated description. For numerical information, I would preserve the extracted table or chart evidence so that the result can be checked against the original source.</p><p>This is particularly important for academic, legal, financial, and policy documents, where an incorrect value or omitted condition can significantly change the meaning of an answer.</p><h4>11. Choosing Between Three Multimodal RAG Architectures</h4><p>There is no single architecture that is best for every PDF collection.</p><h4>Architecture A: Text RAG with OCR and image descriptions</h4><p>This approach extracts text, applies OCR where needed, and generates textual descriptions of relevant visuals.</p><p><strong>Advantages</strong></p><ul><li>Extends an existing text RAG pipeline.</li><li>Works with ordinary text embedding models.</li><li>Relatively straightforward to debug.</li><li>Can be cost-effective when visual processing is selective.</li></ul><p><strong>Limitations</strong></p><ul><li>Depends on the quality of OCR and generated descriptions.</li><li>May lose visual details that are not captured in text.</li><li>Requires special handling for exact chart values and complex layouts.</li></ul><p>This is where I would start.</p><h4>Architecture B: Multimodal embedding retrieval</h4><p>This approach uses an embedding model capable of representing images and text in a compatible retrieval space.</p><h4>Architecture C: Page-level visual question answering</h4><p>This approach retrieves relevant pages, renders them as images, and sends them to a vision-language model for question answering. This approach is useful when the answer depends on the visual arrangement itself.</p><p>In practice, I would combine these architectures where necessary rather than forcing every document through one method. Building a RAG chatbot has taught me that extracting text is only the first step toward making documents searchable.</p><p>Real-world PDFs are structured information sources. Their meaning can be distributed across paragraphs, tables, images, captions, charts, and spatial relationships. A robust pipeline must preserve those relationships and select appropriate processing techniques for each content type.</p><p>For my project, I would begin with a practical architecture that combines text extraction, OCR, visual descriptions, metadata-aware chunking, and semantic retrieval. I would introduce multimodal embeddings and page-level visual question answering only when evaluation shows that the simpler architecture is insufficient.</p><p>The broader lesson is that <strong>a successful RAG pipeline is not merely a vector database connected to a language model. It is a document-understanding and evidence-retrieval system.</strong></p><p>The quality of the final answer depends on how faithfully the pipeline extracts, represents, retrieves, and preserves the evidence contained in the original document.</p><p>And that is the next step in my journey from building a text-based chatbot to building a system capable of answering questions about real-world documents.</p><h4>Technical references and further reading</h4><ol><li><strong>PyMuPDF documentation:</strong> <a href="https://pymupdf.readthedocs.io/">https://pymupdf.readthedocs.io/</a></li><li><strong>pypdf documentation:</strong> <a href="https://pypdf.readthedocs.io/">https://pypdf.readthedocs.io/</a></li><li><strong>Sentence Transformers documentation:</strong> <a href="https://sbert.net/">https://sbert.net/</a></li><li><strong>Tesseract OCR:</strong> <a href="https://github.com/tesseract-ocr/tesseract">https://github.com/tesseract-ocr/tesseract</a></li><li><strong>Hugging Face Transformers documentation:</strong> <a href="https://huggingface.co/docs/transformers/">https://huggingface.co/docs/transformers/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de7b79c33cc3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Would I Choose RAG Over Fine-Tuning?]]></title>
            <link>https://medium.com/@nitisha.aggarwal/why-would-i-choose-rag-over-fine-tuning-56048dc1696c?source=rss-3ba192adec4------2</link>
            <guid isPermaLink="false">https://medium.com/p/56048dc1696c</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Nitisha Aggarwal]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 10:37:15 GMT</pubDate>
            <atom:updated>2026-10-17T10:37:15.597Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Building a document-based AI chatbot taught me that choosing the right AI architecture is just as important as choosing the right model.</em></p><p>When we talk about building AI applications using Large Language Models, one question frequently comes up:</p><blockquote><strong>Why would you choose Retrieval-Augmented Generation (RAG) over fine-tuning?</strong></blockquote><p>At first glance, both approaches seem to solve a similar problem: making a general-purpose language model more useful for a specific application. However, they work in fundamentally different ways.</p><p>To understand the difference, let’s look at the problem from an engineering perspective.</p><blockquote>1. The Problem: A General-Purpose LLM Is Not Your Private Knowledge Base</blockquote><p>Imagine I want to build a chatbot that answers questions about the University programme.</p><p>A user might ask:</p><ul><li>What are the eligibility criteria for admission?</li><li>Which documents are required?</li><li>What is the admission procedure?</li><li>Where can I find information about the programme?</li></ul><p>A general-purpose LLM may understand these questions, but it might not have the exact information from the particular brochure I want it to use.</p><p>Even if it generates a fluent answer, fluency does not guarantee factual accuracy. I need a system that can retrieve relevant information from my chosen document and use it to generate an answer. This is where RAG becomes useful.</p><p>There are two broad approaches I could consider:</p><p><strong>Approach A: Fine-tuning</strong> — adapt the model by training it further on examples or data relevant to the task.</p><p><strong>Approach B: RAG</strong> — retrieve relevant information from an external knowledge source and provide it to the model when answering a question.</p><p>The important distinction is that fine-tuning changes the model’s learned parameters, while RAG supplies relevant context at inference time.</p><blockquote>2. How RAG Works: Retrieve First, Generate Second</blockquote><p>RAG combines information retrieval with text generation. Instead of expecting the LLM to recall every detail, the application searches a knowledge base and provides relevant passages to the model.</p><p>The process can be divided into two phases.</p><blockquote>Phase 1: Preparing the knowledge base</blockquote><p>Suppose I have a 76-page programme brochure in PDF format. The ingestion pipeline looks like this:</p><p>PDF document → Text extraction → Chunking → Embedding generation → Vector index</p><p>Each step serves a different purpose.</p><p><strong>Text extraction:</strong> Extract readable text from the PDF.</p><p><strong>Chunking:</strong> Divide the text into smaller, manageable sections. A whole document may be too large to retrieve or provide as context efficiently.</p><p><strong>Embedding generation:</strong> Convert each text chunk into a numerical vector that represents its semantic features.</p><p><strong>Indexing:</strong> Store the vectors alongside their corresponding text and useful metadata, such as page numbers. A vector index can help retrieve relevant chunks efficiently.</p><p>For example, a chunk might contain the brochure’s eligibility requirements, while another contains programme information. Each chunk becomes a separately retrievable unit of knowledge.</p><blockquote>Phase 2: Answering a user’s question</blockquote><p>When a user asks:</p><p><em>“What are the eligibility criteria for M.Sc. Informatics?”</em></p><p>The application performs the following steps:</p><ol><li>Converts the question into an embedding using the same embedding model used for the document chunks.</li><li>Searches the vector index for semantically similar chunks.</li><li>Retrieves the most relevant passages, potentially combining vector similarity with keyword search or metadata filters.</li><li>Inserts the retrieved passages into the LLM’s prompt as context.</li><li>Asks the LLM to generate an answer grounded in that context.</li></ol><p>The architecture can be represented as:</p><p>User question → Query embedding → Retrieval → Relevant document chunks → LLM + context → Answer</p><p>The LLM is still responsible for generating the response. The retrieval component supplies the information that the response should be based on. This is the central idea behind RAG: connect the model to an external knowledge source at the time of answering.</p><blockquote>3. How Fine-Tuning Works: Adapt the Model Itself</blockquote><p>Fine-tuning takes a different approach. A pretrained language model has already learned general language patterns and capabilities. During fine-tuning, its parameters are further optimized using a training dataset for a particular objective.</p><p>For example, I might prepare examples such as:</p><p><strong>Training example 1</strong></p><p>Input: A question about admission eligibility.</p><p>Desired output: A concise, structured answer with a clear explanation.</p><p><strong>Training example 2</strong></p><p>Input: A question asking for a summary of a programme.</p><p>Desired output: A short summary in a specified format.</p><p><strong>Training example 3</strong></p><p>Input: A question requiring a list of documents.</p><p>Desired output: A numbered list following a consistent response format.</p><p>With suitable data and training, fine-tuning can help the model learn a preferred response style, output structure, or task-specific behavior. It can also improve performance on specialized tasks when the training data and objective are appropriate. However, fine-tuning is not the same as creating a searchable, continuously updated document database.</p><p>If the admission brochure changes next month, fine-tuning does not automatically give the model access to the revised brochure. Updating the model’s knowledge may require additional training, and even then, the model does not inherently provide a traceable source passage for each factual claim.</p><p>For frequently changing document-based information, maintaining an external knowledge base is often a more practical approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*v7QUX-YuWmRiFEtviiatWA.jpeg" /></figure><blockquote>4. A Practical Example: One Question, Two Architectures</blockquote><p>Let’s compare how both approaches might handle the same question.</p><p><strong>User question:</strong> “What is the eligibility criteria for the M.Sc. Informatics programme?”</p><p><strong>Scenario A: Using RAG</strong></p><p>The system searches the programme brochure, retrieves the relevant eligibility section, and supplies that passage to the LLM. The model generates an answer using the retrieved text. If the system preserves page metadata, it can also provide a reference to the relevant page, allowing the user to verify the answer. If the brochure is updated, I can replace or update the relevant document content and rebuild or refresh the index as needed. I do not necessarily need to retrain the LLM.</p><p><strong>Scenario B: Using fine-tuning</strong></p><p>I prepare training examples that teach the model how to respond to admission-related questions. The model may learn to produce clearer, more consistent answers to this kind of question. However, if I expect it to answer using the exact requirements from a particular brochure, I must ensure that the relevant facts are represented appropriately in the training process. Fine-tuning alone does not provide a reliable mechanism for looking up the latest source document at query time. The model may generate a plausible answer without reliably identifying which document passage supports it.</p><p>The distinction is therefore not that RAG is always more accurate. Rather, RAG provides an explicit retrieval mechanism that can connect an answer to a specific knowledge source.</p><blockquote>5. RAG vs. Fine-Tuning: A Technical Comparison</blockquote><p>RAG is not automatically cheaper in every situation. It introduces indexing, storage, retrieval, and additional runtime steps. Fine-tuning can sometimes be a better fit when the main objective is to change the model’s behavior and the task does not require frequent retrieval from external documents. The right decision depends on the workload, update frequency, latency requirements, available data, and evaluation results.</p><blockquote>6. Why RAG Is a Better Starting Point for My Project</blockquote><p>For my document-based chatbot, the primary challenge is not teaching a language model how to write answers. It is helping the model find the right information in a specific document collection. That makes RAG a logical starting point.</p><p>My goals are to:</p><ul><li>Allow users to ask questions in natural language.</li><li>Retrieve relevant sections from the source documents.</li><li>Generate answers using those retrieved sections.</li><li>Inspect which chunks were retrieved when an answer is incorrect.</li><li>Update the knowledge base when the underlying documents change.</li></ul><p>RAG also gives me several components I can evaluate independently.</p><p>If the wrong passage is retrieved, I can investigate the chunking strategy, embedding model, similarity metric, search method, or ranking process. If the correct passage is retrieved but the answer is still incorrect, I can investigate prompt construction, context quality, and the language model’s response.</p><p>This separation makes debugging more systematic. Instead of treating the chatbot as a single black box, I can evaluate its individual components.</p><blockquote>7. A Critical Point: RAG Does Not Eliminate Hallucinations</blockquote><p>It would be incorrect to conclude that RAG guarantees factual answers.</p><p>Suppose the system retrieves a passage that is only loosely related to the question. The LLM might still generate an incorrect response. Likewise, the relevant information might never reach the model because the retrieval system failed to find it.</p><p>A practical RAG system therefore needs evaluation at multiple levels:</p><p><strong>Retrieval evaluation</strong></p><ul><li>Did the system retrieve the correct passage?</li><li>Did the relevant passage appear near the top of the results?</li><li>Were irrelevant chunks retrieved?</li></ul><p><strong>Generation evaluation</strong></p><ul><li>Is the answer supported by the retrieved context?</li><li>Did the model introduce unsupported information?</li><li>Does the answer correctly communicate uncertainty when the evidence is insufficient?</li></ul><p><strong>End-to-end evaluation</strong></p><ul><li>Does the system answer realistic user questions correctly?</li><li>Can the user verify the answer against the original document?</li><li>Does performance remain acceptable across different types of questions?</li></ul><p>This is why building a RAG chatbot involves more than embedding text and connecting an LLM. Retrieval quality, prompt design, and evaluation are equally important parts of the engineering process.</p><blockquote>8. Can RAG and Fine-Tuning Be Used Together?</blockquote><p>Yes. They are not mutually exclusive. Consider a more advanced version of my chatbot.</p><p>I could use RAG to retrieve the latest admission information from the brochure, while fine-tuning a model to follow a consistent response format — for example, giving a direct answer, explaining the supporting evidence, and identifying the source page.</p><p>In that architecture:</p><ul><li><strong>RAG supplies the relevant information.</strong></li><li><strong>Fine-tuning adapts the model’s behavior.</strong></li><li><strong>The LLM generates the response using the supplied context.</strong></li></ul><p>This combination can be useful when the application needs both external knowledge retrieval and specialized model behavior. However, I would not introduce fine-tuning automatically. First, I would establish a working RAG baseline and identify measurable shortcomings that fine-tuning might actually address.</p><blockquote>9. My Decision Framework</blockquote><p>If I were starting an AI application today, I would ask these questions before choosing an approach:</p><p><strong>Question 1: Does the application need information from external documents?</strong></p><p>If yes, RAG is a strong candidate.</p><p><strong>Question 2: Does the information change frequently?</strong></p><p>If yes, an external knowledge base can make updates easier than relying on model retraining to incorporate every change.</p><p><strong>Question 3: Is the main problem the model’s behavior rather than access to information?</strong></p><p>If the model needs a specialized output format, task behavior, or response style, fine-tuning may be worth evaluating.</p><p><strong>Question 4: Do I need both?</strong></p><p>If the application needs current, traceable information and specialized model behavior, a hybrid architecture may be appropriate.</p><p>The decision should ultimately be validated through testing rather than assumptions about which technique is universally better. For my document-based chatbot, I am starting with RAG because I want the system to answer questions using a specific set of source documents. It gives me a way to inspect retrieved evidence, update the knowledge base independently of the LLM, and investigate retrieval errors.</p><p>As the project develops, I may explore fine-tuning if I identify a task-specific behavior that cannot be addressed adequately through prompting and retrieval alone.</p><p>The broader lesson is simple:</p><p><strong>Don’t fine-tune a model just because you want it to know something. First, ask whether the real requirement is to give it access to information or to change how it performs a task.</strong></p><p>Understanding that distinction is an important step toward designing AI systems that are not only functional, but also maintainable, testable, and grounded in evidence.</p><p><em>This is part of my journey of learning how to build a </em><a href="https://medium.com/@nitisha.aggarwal/day-1-of-building-my-rag-chatbot-8180e82dd3d8"><em>Retrieval-Augmented Generation chatbot</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56048dc1696c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 3 of building my RAG chatbot]]></title>
            <link>https://medium.com/@nitisha.aggarwal/day-3-of-building-my-rag-chatbot-958dd674a8d5?source=rss-3ba192adec4------2</link>
            <guid isPermaLink="false">https://medium.com/p/958dd674a8d5</guid>
            <category><![CDATA[chatbot-development]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <dc:creator><![CDATA[Nitisha Aggarwal]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 10:11:39 GMT</pubDate>
            <atom:updated>2026-10-17T10:12:07.877Z</atom:updated>
            <content:encoded><![CDATA[<p>The Journey Continues…</p><p>When I started building my own RAG chatbot, I thought the main challenge would be connecting an AI model to a chat interface.</p><p>But as I progressed, I realized that the real work begins much earlier: preparing the knowledge that the chatbot will use.</p><p>In <strong>Day 1</strong>, I focused on getting Python, Anaconda, and VS Code to work together. Before writing meaningful code, I had to resolve environment and interpreter issues.</p><p>In <strong>Day 2</strong>, I moved into the document-processing pipeline. I extracted text from a university brochure and explored how to divide the extracted text into smaller, manageable chunks.</p><p>Now, in Day 3, I am approaching the next important milestone: converting those text chunks into numerical representations called embeddings.</p><blockquote>Why Can’t We Just Search the Text?</blockquote><p>Imagine asking a chatbot:</p><p><em>“What qualifications do I need to get admission to this programme?”</em></p><p>The source document might describe the relevant information using phrases such as <em>eligibility criteria</em>, <em>minimum qualification</em>, or <em>admission requirements</em>. A traditional keyword search may struggle when the question and the document use different words to express a similar idea. This is where semantic search becomes useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TmGJ5KXHZWf5YT28" /></figure><p>Instead of relying only on matching exact words, semantic search attempts to identify text that is conceptually similar to the user’s question.</p><p>But how can a computer compare the meaning of two pieces of text?</p><p>The answer begins with <strong>embeddings</strong>.</p><blockquote>What Exactly Is an Embedding?</blockquote><p>An embedding is a numerical representation of text generated by a machine-learning model.</p><p>For example, consider these two sentences:</p><ul><li>“What are the admission requirements?”</li><li>“Which qualifications are needed to apply?”</li></ul><p>The wording is different, but the underlying meaning is similar.</p><p>An embedding model converts each sentence into a list of numbers called a <strong>vector</strong>. These vectors can then be compared mathematically to estimate how semantically similar the texts are.</p><p>A simplified illustration might look like this:</p><pre>Text:<br>&quot;What are the admission requirements?&quot;</pre><pre>          ↓ Embedding model</pre><pre>Vector:<br>[0.12, -0.34, 0.58, 0.07, ...]</pre><p>The numbers above are illustrative, not actual model output.</p><p>The important point is that the vector is not a human-readable definition of the sentence. It is a representation that a computer can use for similarity comparisons. Texts with similar meanings may have embeddings that are close together in the model’s vector space. This is what makes semantic retrieval possible.</p><blockquote>My Technical Approach</blockquote><p>For this project, I am working with Python and the sentence-transformers library, using the model:</p><p>sentence-transformers/all-MiniLM-L6-v2</p><p>My plan is to use this model to generate embeddings for the chunks created during document processing.</p><p>The workflow looks like this:</p><pre>University Brochure (PDF)<br>          ↓<br>     Text Extraction<br>          ↓<br>       Text Chunks<br>          ↓<br>    Embedding Model<br>          ↓<br>   Numerical Vectors<br>          ↓<br>    Semantic Search</pre><p>Each stage has a specific responsibility:</p><p><strong>1. Text extraction</strong></p><p>Extract readable text from the PDF so that the content can be processed programmatically.</p><p><strong>2. Chunking</strong></p><p>Split the extracted text into smaller sections. This allows the system to retrieve relevant portions instead of searching through the entire document as one large block.</p><p><strong>3. Embedding generation</strong></p><p>Pass each chunk through the embedding model to obtain its numerical vector representation.</p><p><strong>4. Semantic search</strong></p><p>When a user asks a question, generate an embedding for that question and compare it with the stored chunk embeddings. The system can then rank chunks according to their similarity.</p><p>This is the foundation of the retrieval component of a RAG chatbot.</p><blockquote>Why Do Vectors Matter in RAG?</blockquote><p>Consider a user asking:</p><p><em>“Am I eligible for admission to M.Sc. Informatics?”</em></p><p>A RAG system can convert the question into a vector and compare it with the vectors representing the brochure’s text chunks. The retrieval process then identifies potentially relevant sections, such as those discussing eligibility or admission requirements.</p><p>Those retrieved sections can later be passed to an LLM, which uses the supplied context to formulate an answer. This distinction is important: <strong>embeddings help find relevant information; they do not generate the final answer themselves.</strong></p><p>The quality of the final response also depends on the source document, chunking strategy, retrieval quality, and how the language model uses the retrieved context.</p><blockquote>The Next Challenge</blockquote><p>I have reached the point where the document-processing pipeline is ready for the next stage. My next task is to generate embeddings for the text chunks and inspect the resulting vectors.</p><p>I want to understand not only whether the code runs, but also:</p><ul><li>How many vectors are generated?</li><li>What is the dimensionality of each vector?</li><li>How can I compare a user’s question with the stored chunks?</li><li>Will the most relevant chunk actually appear at the top of the search results?</li></ul><p>These questions will help me move from simply processing documents to building a system that can retrieve information intelligently. I am also learning that a working pipeline is only the beginning. Retrieval needs to be tested with realistic questions, and the results need to be inspected rather than blindly trusted.</p><blockquote>What I Am Learning Along the Way</blockquote><p>Building this project step by step is helping me connect concepts that I previously encountered mainly in theory. I am beginning to see how document processing, machine learning, vector mathematics, and information retrieval fit together in a practical AI application.</p><p>I don’t want to build a chatbot by simply connecting a few libraries and hoping for the best. I want to understand what happens behind the scenes, why each component is needed, and how to identify problems when the results are not satisfactory.</p><p><strong>Day 3’s goal is clear: move from text chunks toward numerical representations that make semantic search possible.</strong></p><p>One milestone at a time. One concept at a time.</p><p>The chatbot is still under construction, and so is my understanding of the technology behind it.</p><h3>Follow the Journey</h3><ul><li><strong>Day 1:</strong> <a href="https://medium.com/@nitisha.aggarwal/day-1-of-building-my-rag-chatbot-8180e82dd3d8">Getting Python, Anaconda, and VS Code to Work Together</a></li><li><strong>Day 2:</strong> <a href="https://medium.com/@nitisha.aggarwal/day-2-of-building-my-rag-chatbot-e5a118e7c8b0">Document Processing: Text Extraction, Chunking, and Embeddings</a></li><li><strong>Day 3:</strong> From Text Chunks to Numerical Vectors — the next step in my RAG chatbot journey.</li></ul><p><em>If you’re also learning RAG, Python, or applied AI, I’d love to hear about the challenges you’re facing and the lessons you’re learning along the way.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=958dd674a8d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hidden Journey of Your Prompt: From Enter Key to the First AI Token]]></title>
            <link>https://medium.com/@nitisha.aggarwal/the-hidden-journey-of-your-prompt-from-enter-key-to-the-first-ai-token-406b79f73e89?source=rss-3ba192adec4------2</link>
            <guid isPermaLink="false">https://medium.com/p/406b79f73e89</guid>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[token]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Nitisha Aggarwal]]></dc:creator>
            <pubDate>Wed, 14 Oct 2026 07:33:00 GMT</pubDate>
            <atom:updated>2026-10-14T07:33:00.415Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine opening ChatGPT and typing: <strong>“Explain artificial intelligence in simple words.”</strong></p><p>You press Enter.</p><p>Within seconds, an answer begins to appear on your screen. It looks effortless. You ask a question, and the AI responds. But have you ever wondered what happens inside the system during the time between pressing Enter and seeing the first word?</p><p>Does the model immediately start writing? Does it understand the sentence first? Does it search for an answer? Or does something much more technical happen?</p><p>Let’s follow our prompt on its journey through a Large Language Model (LLM).</p><p>We will use this simple question throughout the story:</p><p><strong><em>“Explain artificial intelligence in simple words.”</em></strong></p><p>At first glance, the input appears to be an ordinary English sentence. However, the model does not necessarily receive only this sentence. The application may assemble several pieces of information:</p><ul><li>System instructions that define how the assistant should behave.</li><li>Previous messages in the conversation.</li><li>The latest user question.</li><li>Tool instructions or other contextual information, when applicable.</li></ul><p>These pieces are arranged into a structured input that the model can process.</p><p>For example, our conversation might contain:</p><p><strong>System instruction:</strong> Answer clearly and help beginners understand technical concepts.</p><p><strong>User message:</strong> Explain artificial intelligence in simple words.</p><p>The model receives the available context according to the application’s input format. This is important because the answer can depend on more than the latest sentence. Previous messages and instructions can influence what the model generates. At this stage, however, the model has not yet produced the first answer token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_AGRG7b3e5tuayks" /></figure><p>The next step is to convert the input into a representation the neural network can process.<br>Computers work with numerical representations. An LLM therefore needs a way to convert text into a sequence of token IDs. This is the job of the <strong>tokenizer</strong>.</p><p>A token is a unit of text. Depending on the tokenizer, it may represent a complete word, part of a word, punctuation, or another text fragment.</p><p>Consider our prompt:</p><p>“Explain artificial intelligence in simple words.”</p><p>For illustration, imagine that a tokenizer divides it into pieces such as:</p><p>Explain | artificial | intelligence | in | simple | words | .</p><p>This is only an illustrative segmentation. An actual tokenizer might split some words differently. Each token is mapped to an integer ID from the model’s vocabulary.</p><p>For example, we might imagine the following IDs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*FspP-9PcsKYVQpvD9R8W2Q.png" /></figure><p>These numbers are hypothetical; they are not the actual IDs used by a particular LLM.</p><p>The text has now become a sequence of numbers:</p><p>[1842, 7261, 9345, 318, 2417, 5291, 13]</p><p>Why is this necessary?</p><p>Because the neural network operates on numerical data rather than directly manipulating English words.</p><p><strong>Important distinction:</strong> Tokenization does not mean that the model has understood the question. It has converted the text into token IDs. Our prompt has taken its first technical step.</p><p><em>The next step is to map the token ID to an </em><strong><em>embedding vector</em></strong><em>.</em></p><p>Now imagine that the model receives the token ID for the word “artificial.” Is the integer 7261 itself a meaningful representation of that word? Not in the way we need. The ID primarily identifies a vocabulary entry; its numerical magnitude does not represent the word’s meaning. An embedding is a list of learned numerical values representing a token in a high-dimensional space.</p><p>For example, imagine a tiny, simplified embedding:</p><p>artificial → [0.21, -0.43, 0.78, 0.16]</p><p>This four-number vector is purely illustrative. Real models generally use much larger vectors.</p><p>Each token receives its corresponding embedding. Positional information is also incorporated, either through positional embeddings or a mechanism such as Rotary Position Embeddings (RoPE).</p><p>Why does position matter?</p><p>Consider these two sentences:</p><ul><li>“The dog chased the cat.”</li><li>“The cat chased the dog.”</li></ul><p>The words are nearly identical, but their order changes the meaning. The model therefore needs information about where tokens occur in the sequence, not just which tokens are present. After this stage, the input is represented numerically in a form suitable for the Transformer network. But the real computational work is about to begin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_diNwdYVnsi6fTxI" /></figure><p><em>Transformer starts to process the entire context</em></p><p>This is where the core neural network begins processing the prompt. Modern decoder-only LLMs commonly use a Transformer architecture. The input representations pass through multiple layers. These layers typically contain two major computational components:</p><ol><li>Self-attention.</li><li>Feed-forward neural network layers.</li></ol><p>Let’s understand them using our example.</p><p>“Explain artificial intelligence in simple words.”</p><blockquote>What does self-attention do?</blockquote><p>Suppose the model is processing the token associated with “intelligence.” To interpret this token in context, the network can use information from relevant preceding tokens, such as “artificial.” Self-attention allows each position to combine information from other permitted positions in the sequence. In a standard causal language model, the attention mask prevents a position from attending to future positions. During prompt processing, the model can nevertheless compute representations for many positions in parallel.</p><p>Internally, attention uses three learned projections:</p><ul><li><strong>Query (Q):</strong> what information a position is looking for.</li><li><strong>Key (K):</strong> information used to determine how relevant another position is.</li><li><strong>Value (V):</strong> information that can be combined into the resulting representation.</li></ul><p>A simplified attention calculation is from the paper “Attention Is All You Need”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*XPoZwDAybKSU-N3Kg4tPaA.png" /></figure><p>Here, dₖ is the key-vector dimension.</p><p>You do not need to memorize the formula yet. The essential idea is that the model calculates relationships between positions and combines information accordingly. These relationships are not simply a dictionary of word meanings. They are learned numerical interactions that help the network build context-dependent representations.</p><blockquote>What happens after attention?</blockquote><p>The resulting representations pass through feed-forward layers, which apply additional learned transformations. This process repeats across the model’s layers. As information flows through the network, the representation at each position becomes more context-dependent.</p><p>For our prompt, the final representation at the last input position contains information processed from the available preceding context. The model is now ready to calculate what token should come next. However, it still has not generated the first token of the answer.</p><blockquote>To generate an answer, we need more steps, so the next step is prefill.</blockquote><p>The processing of the input context is commonly called the <strong>prefill phase</strong>. This is a critical concept in LLM inference. During prefill, the model processes the input tokens through its Transformer layers. It also generally computes and stores attention keys and values in a structure called the <strong>KV cache</strong>.</p><p>Why store them?</p><p>Imagine generating a 100-word answer.</p><p>When generating the next token, the model needs information from the preceding prompt and generated tokens. Recomputing every previous key and value from scratch at every step would be wasteful. The KV cache lets the model reuse the cached attention information during subsequent decoding steps.</p><p>Think of it as keeping useful intermediate calculations available instead of repeating the same work. There is an important distinction here:</p><p><strong>Prefill processes the input prompt. Decoding generates the answer one token at a time.</strong></p><p>The KV cache helps connect these phases efficiently. Once the prefill computation reaches the final input position, the model can calculate its prediction for the first answer token. Now, we are getting closer to getting the answer. The final hidden representation at the relevant input position is passed through the model’s output projection, often called the language-model head. The result is a vector of numbers called <strong>logits</strong>. Each logit corresponds to a token in the model’s vocabulary.</p><p>Imagine that our model has a tiny vocabulary containing only five possible candidates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*6nRoieu-IxuSXF3doFf1pQ.png" /></figure><p>These scores are fictional and demonstrate the concept only. A real model evaluates a much larger vocabulary.</p><p>A higher logit generally indicates a stronger preference before decoding adjustments are applied. But logits are not probabilities.</p><p>To convert logits into a probability distribution, a system can apply the softmax function:</p><p>P(tokenᵢ) = exp(zᵢ) / Σⱼ exp(zⱼ)</p><p>Here, zᵢ is the logit for a candidate token. Softmax transforms the scores into non-negative values that sum to one. The model has now produced a distribution over possible next tokens.</p><p><strong>Notice something important:</strong> it has not independently searched for a sentence in a database. It has computed next-token scores using its learned parameters and the supplied context. Other systems may perform retrieval or tool calls before or during generation, but those are additional parts of the overall application.</p><p>The model has scores for possible next tokens. The decoding procedure determines which token is selected. Different decoding strategies can produce different results.</p><p><strong>Greedy decoding:</strong> Choose the token with the highest adjusted score.</p><p><strong>Sampling:</strong> Select a token probabilistically from the distribution.</p><p>Some generation configurations also use temperature, top-k, or top-p to adjust or restrict the candidates. For our fictional example, greedy decoding would choose:</p><p>Artificial</p><p>This becomes the first generated token.</p><p>With a different prompt, model, or decoding configuration, the first token might instead be “AI,” “Artificially,” or another token entirely.</p><p>Also, a token is not necessarily a complete word. The first generated token might be only part of a word, a space-prefixed word, or a punctuation mark. The exact token depends on the model’s vocabulary and the decoding process. At last, the system has selected the first token of its answer.</p><p>Suppose the selected token is “Artificial.” The serving system decodes the token into its corresponding text representation and makes it available to the output stream.</p><p>The interface may now display:</p><blockquote>“Artificial”</blockquote><p>But the answer is not complete.</p><p>The model continues its autoregressive generation process:</p><ol><li>Select the first token.</li><li>Add that token to the generated sequence.</li><li>Run the next decoding step using the available context and cached attention information.</li><li>Calculate the next-token distribution.</li><li>Select the next token.</li><li>Repeat until a stopping condition is reached.</li></ol><p>The next token might be “intelligence,” followed by “is,” and then “a.”</p><p>The illustrative answer could gradually become:</p><blockquote>“Artificial intelligence is a field of computer science…”</blockquote><p>Each newly generated token becomes part of the context used to predict the following token. This is the central principle of <strong>autoregressive generation</strong>: the model generates a sequence by repeatedly predicting the next token from the context available so far. In practice, the application may stream output as tokens become available, although token boundaries do not always align with visible words.</p><blockquote>You might now ask: if the model performs so much computation before generating anything, why does the answer often appear so quickly?</blockquote><p>The time between submitting a request and receiving the first output token is commonly called <strong>Time to First Token (TTFT)</strong>.</p><p>It can be influenced by several factors:</p><ul><li><strong>Input length:</strong> A long conversation or document usually requires more prefill work.</li><li><strong>Model size and architecture:</strong> Larger or more computationally demanding models may take longer.</li><li><strong>Hardware:</strong> GPU performance, memory bandwidth, and available resources matter.</li><li><strong>Serving infrastructure:</strong> Request queues, batching, scheduling, and network delays can contribute.</li><li><strong>Model initialization:</strong> Loading weights and initializing the inference runtime may add time, especially during a cold start.</li><li><strong>Output pipeline:</strong> Decoding and transmitting the first token also take time.</li></ul><p>The exact contributions depend on the implementation.</p><p>A system that has already loaded its model can begin processing a new request without repeating all model-initialization work. Similarly, caching or reusing a shared prompt prefix may reduce some of the computation required for certain requests. This is why two prompts submitted to the same application may have different response times.</p><p>The next time you press Enter in an AI chatbot, remember that the first word you see is the visible end of a hidden computational journey.</p><p><strong>The conversation appears to begin with a word. Computationally, that word begins with a forward pass.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=406b79f73e89" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 2 of building my RAG chatbot]]></title>
            <link>https://medium.com/@nitisha.aggarwal/day-2-of-building-my-rag-chatbot-e5a118e7c8b0?source=rss-3ba192adec4------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5a118e7c8b0</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[text-chunking]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Nitisha Aggarwal]]></dc:creator>
            <pubDate>Tue, 13 Oct 2026 07:03:17 GMT</pubDate>
            <atom:updated>2026-10-13T07:03:17.981Z</atom:updated>
            <content:encoded><![CDATA[<h3>Day 2: Building the Foundation of a RAG Chatbot — From PDF Extraction to Text Chunking</h3><p>In my previous post, <em>“A Beginner’s Journey Through Anaconda, VS Code, and Windows Errors,”</em> I shared the challenges I faced while setting up my Python development environment. With the environment finally working, it was time to move from configuration to implementation.</p><p>My objective is to build a web-based <strong>Retrieval-Augmented Generation (RAG) chatbot</strong> that can answer questions using information from documents rather than relying entirely on the knowledge embedded in a language model.</p><p>But how does a RAG system actually prepare a document before answering a question?</p><p>Today, I worked on the first two stages of the pipeline: <strong>document ingestion and text chunking</strong>.</p><h3>1. Understanding the RAG Pipeline</h3><p>A typical RAG system follows this workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OYVRDlAOmX1LA0Yn" /></figure><p><strong>Document → Text Extraction → Chunking → Embeddings → Vector Search → Retrieved Context → LLM Response</strong></p><p>Each component has a specific responsibility.</p><ul><li><strong>Text extraction:</strong> Converts document content into machine-processable text.</li><li><strong>Chunking:</strong> Divides long documents into smaller passages.</li><li><strong>Embeddings:</strong> Represents text as numerical vectors that capture aspects of its meaning.</li><li><strong>Vector search:</strong> Finds passages semantically similar to a user’s question.</li><li><strong>Generation:</strong> Uses the retrieved passages as context for answering the question.</li></ul><p>I decided to implement these components incrementally so that I could understand the process instead of treating RAG as a black box.</p><h3>2. Stage One: PDF Text Extraction Using Python</h3><p>My first task was to extract text from a real PDF document.</p><p>I installed the pypdf library and created a Python script named extract_pdf.py.</p><p>The script reads each page, extracts its text, and writes the results to a text file. I also preserved page markers so that I could trace the extracted content back to its original page.</p><p>The result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*-yCJqm-bZGGGJQvmNBIAYw.png" /></figure><p>This stage is commonly called <em>document ingestion</em> or part of the ingestion pipeline.</p><p>One important observation: successful extraction does not automatically guarantee good-quality text. PDF layouts, tables, multi-column pages, and scanned images can create extraction problems. Scanned documents may require Optical Character Recognition (OCR).</p><p>The quality of the extracted text directly affects the quality of subsequent processing.</p><h3>3. Stage Two: Why Do We Need Text Chunking?</h3><p>Suppose we have a 76-page document containing more than 200,000 characters.</p><p>If we treat the entire document as a single unit, retrieving only the most relevant information becomes difficult. Sending the entire document to a language model can also increase context usage, cost, and irrelevant information.</p><p>Chunking addresses this problem by dividing a document into smaller, searchable units.</p><p>For my first implementation, I created chunk_text.py with the following configuration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*MRnLv6Fv4cjfwZAOd2tUyg.png" /></figure><p>The script reads the extracted text, creates overlapping chunks, and saves them for inspection.</p><h3>How does overlap work?</h3><p>Consider a simplified example:</p><p>Chunk 1: A retrieval system searches the document and identifies relevant passages at the end of a section…</p><p>Chunk 2: …relevant passages at the end of a section, preserving part of the preceding context before continuing with the next topic.</p><p>The shared text helps reduce the chance that a useful sentence or explanation will lose its context at a chunk boundary.</p><p>However, overlap is not a complete solution. Chunk size and overlap must be selected according to the document structure, retrieval task, and embedding model.</p><p>My current implementation uses character-based chunking. In a future iteration, I want to experiment with paragraph-aware and token-aware approaches.</p><h3>4. Stage Three: Preparing for Embeddings</h3><p>After chunking, the next challenge is helping a computer identify passages by meaning.</p><p>This is where <strong>embeddings</strong> come in.</p><p>An embedding model maps a piece of text to a numerical vector. Texts with similar meanings can often be represented by vectors that are close together in the embedding space.</p><p>For example, a user might ask:</p><p>“Which factors affect a student’s academic performance?”</p><p>The relevant passage might discuss “variables influencing learning outcomes,” even though the wording differs.</p><p>A semantic retrieval system can potentially identify that passage because it considers relationships in meaning, not only exact keyword matches.</p><p>I have installed the sentence-transformers library and verified the installation. My next step is to use the all-MiniLM-L6-v2 model to generate embeddings for my 243 chunks.</p><p>I have not generated the vectors yet, so that will be the next milestone.</p><h3>5. What I Learned</h3><p>This exercise helped me understand that RAG is not simply a language model connected to a PDF.</p><p>The ingestion and preparation stages matter because retrieval can only work well when the source content is extracted, segmented, and represented appropriately.</p><p>My progress so far:</p><p>✅ Configured Python, Conda, and VS Code.</p><p>✅ Extracted 206,281 characters from a 76-page PDF.</p><p>✅ Created 243 overlapping text chunks.</p><p>✅ Installed and verified the embedding library.</p><h3>What’s Next?</h3><p>My next goal is to generate embeddings, inspect their dimensions, and build a basic vector search system.</p><p>After that, I will connect retrieval to a language model and eventually build a web interface for the chatbot.</p><p>I’m building this project step by step, learning not only how to make it work, but also why each component is necessary.</p><p><strong>The goal is not just to build a RAG chatbot. It’s to understand the engineering decisions behind one.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5a118e7c8b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>