<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anirban Roy on Medium]]></title>
        <description><![CDATA[Stories by Anirban Roy on Medium]]></description>
        <link>https://medium.com/@anirbanfiem?source=rss-601abdb7348e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*kQZ0rBL77n5nD1IK</url>
            <title>Stories by Anirban Roy on Medium</title>
            <link>https://medium.com/@anirbanfiem?source=rss-601abdb7348e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:52:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anirbanfiem/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Azure Private DNS Zones and the DNS Private Resolver: A Field Guide]]></title>
            <link>https://medium.com/@anirbanfiem/azure-private-dns-zones-and-the-dns-private-resolver-a-field-guide-ddac9e56062d?source=rss-601abdb7348e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddac9e56062d</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[dns]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[enterprise-architecture]]></category>
            <dc:creator><![CDATA[Anirban Roy]]></dc:creator>
            <pubDate>Wed, 12 Aug 2026 10:31:27 GMT</pubDate>
            <atom:updated>2026-08-12T10:31:27.748Z</atom:updated>
            <content:encoded><![CDATA[<h3>Azure Private DNS Zones and the Private Resolver</h3><p><strong>A </strong>private DNS zone has no IP address. Nothing listens on it. You cannot point a resolver at it or query it from anywhere that isn’t explicitly linked to it.</p><p>Every hard problem in hybrid Azure DNS falls out of that one fact — and the DNS Private Resolver exists to solve exactly it.</p><h3>The three-part mental model</h3><ul><li><strong>The zone</strong> — a container of records. Authoritative data, no listener, no IP.</li><li><strong>The link</strong> — the permission boundary. A zone is invisible to any VNet not linked to it.</li><li><strong>The resolver</strong> — the translator between a world that expects an IP to query and a world that expects a network to be entitled.</li></ul><p>Most broken designs come from people reasoning about <em>servers</em> while Azure is reasoning about <em>links</em>.</p><h3>The case study: Company XYZ</h3><p>A mid-size enterprise:</p><ul><li>~400 private endpoints across storage, SQL, Key Vault, private AKS</li><li>Two Azure regions, hub-and-spoke in each, ExpressRoute to two datacentres</li><li>AD-integrated DNS on-premises, authoritative for xyz.internal</li><li>Recently acquired Company ABC, running abc.local, separate forest, no trust</li></ul><p><strong>Four requirements:</strong></p><ol><li>On-premises apps reach Azure PaaS privately, using normal public hostnames</li><li>Azure workloads resolve on-premises AD names</li><li>Company ABC spokes resolve abc.local; original spokes must never see it</li><li>Every DNS query leaving a VNet is logged — the regulator asked</li></ol><h3>Problem 1: On-premises can’t see the private zone</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mtW7Nb4pfUtwFW2fpcjATQ.png" /></figure><h3>What actually happens</h3><ul><li>Public Azure DNS returns a CNAME: sa01.blob.core.windows.net → sa01.privatelink.blob.core.windows.net</li><li>That CNAME is public. Anyone can get it.</li><li>Your private zone contributes <strong>only</strong> the final A record for the privatelink label</li><li>The on-premises server chases the CNAME itself, then re-checks its own forwarder list</li></ul><h3>Steps to solve</h3><ol><li>Deploy a resolver <strong>inbound endpoint</strong> in the hub VNet, in a subnet delegated to the resolver service.</li><li>Link every privatelink.* zone to the hub VNet (resolution link).</li><li>On the corporate DNS servers, add conditional forwarders for privatelink.blob.core.windows.net, privatelink.database.windows.net, etc., pointing at the inbound endpoint IP.</li><li>Repeat in the second region. Add both inbound IPs to every forwarder.</li><li>Validate from on-premises: the answer must be RFC1918.</li></ol><h3>The mistake to avoid</h3><ul><li>Forwarding blob.core.windows.net (the public zone) instead of privatelink.blob.core.windows.net</li><li>It works — until a workload needs a storage account with <strong>no</strong> private endpoint, which you have now blackholed</li><li>Forward narrow. Let the public half resolve normally; only the last hop crosses ExpressRoute</li></ul><h3>Problem 2: Azure workloads can’t resolve on-premises names</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zAg9cBjWcbmkGqPVCFVurw.png" /></figure><h3>Steps to solve</h3><ol><li>Create an <strong>outbound endpoint</strong> in its own delegated subnet in the hub.</li><li>Create a <strong>forwarding ruleset</strong> and attach it to the outbound endpoint.</li><li>Add rules: xyz.internal → corporate domain controller IPs.</li><li><strong>Link the ruleset to the VNets that need it.</strong> A ruleset does nothing until it is linked.</li><li>Validate from a spoke VM — this path fails independently of the inbound path.</li></ol><h3>Subnet rules worth knowing before you design</h3><ul><li>One dedicated subnet per endpoint, delegated to the resolver service</li><li>No other resources in it, not shareable, same region as the VNet</li><li>Retrofitting a delegation into a mature landing zone is a change-control conversation. Plan it early.</li></ul><h3>Problem 3: Two forests, overlapping namespaces</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qyFVHE41eGoBM8Ryg3gjHg.png" /></figure><p>This is the property that makes rulesets more interesting than the DNS forwarder VMs they replace: <strong>rulesets are linked per virtual network, so the same FQDN can resolve differently in different spokes.</strong></p><h3>Steps to solve</h3><ol><li>Build Ruleset-XYZ with rules for xyz.internal. Link it to the XYZ spokes only.</li><li>Build Ruleset-ABC with rules for abc.local. Link it to the ABC spokes only.</li><li>Deliberately omit the other domain from each ruleset.</li><li>Write a <strong>negative test</strong>: from an XYZ spoke, abc.local must return NXDOMAIN.</li></ol><p>One resolver, one hub, two disjoint namespace views — with the boundary drawn at the VNet link rather than a firewall rule. It buys you the years you need to consolidate AD without pretending it’s already done.</p><h3>Problem 4: No DNS visibility</h3><h3>Steps to solve</h3><ol><li>Enable <strong>DNS proxy</strong> on Azure Firewall in the hub.</li><li>Point the firewall’s DNS setting at the resolver inbound endpoint IP.</li><li>Set the VNet DNS servers to the firewall’s private IP.</li><li>Send firewall logs to Log Analytics; query DNS events with source IP attached.</li></ol><h3>Why it earns its hop</h3><ul><li>You get query logs with attribution, not just aggregate counts</li><li>FQDN-based network rules become trustworthy, because the firewall and the workload resolve through the same path</li></ul><h3>When to skip it</h3><ul><li>Small estate, no regulatory obligation. It is a real dependency and a real hop. Decide deliberately rather than by copying a reference diagram.</li></ul><h3>The precedence rule</h3><p>When a VNet is linked to a private DNS zone <strong>and</strong> a ruleset rule matches the same name, <strong>the private DNS zone wins.</strong>Azure answers locally and never forwards.</p><p>That is usually what you want, and occasionally the reason your change didn’t take effect. The dangerous version: link xyz.com as a private zone to hold three internal API records, and you have made Azure authoritative for the <em>entire</em>xyz.com namespace inside every linked VNet. www.xyz.com no longer falls through to public DNS. It returns NXDOMAIN — the zone exists, Azure answers authoritatively, the record doesn&#39;t. Public sites have gone dark this way, and the fix is recreating dozens of public records inside the private zone.</p><blockquote><strong><em>Rule of thumb:</em></strong><em> never link a private zone for a namespace unless you are prepared to own </em>every<em> name in it. If you only need a few overrides in a namespace you don’t fully control, use a ruleset rule pointing at a server that holds the full zone.</em></blockquote><h3>Five traps that survive good design reviews</h3><ul><li><strong>Forwarding loops.</strong> A wildcard (.) rule sending everything on-premises, plus an on-premises conditional forwarder pointing back at the inbound endpoint, is a loop that saturates both sides under load. Wildcards are legitimate for on-premises-first estates — but they demand an audit of every forwarder on the far end.</li><li><strong>Multi-region private endpoints on one name.</strong> The privatelink zone is global and holds one A record per FQDN. Two regional endpoints for the same resource means round-robin, or whichever automation ran last. If regional isolation matters, use separate zone instances linked to separate regional VNet sets.</li><li><strong>Record drift.</strong> Hand-typed A records in privatelink zones will be wrong within a quarter. Bind endpoint creation to a private DNS zone group and enforce it with Azure Policy so nobody can create an endpoint without one.</li><li><strong>Cache during failover.</strong> On-premises servers cache negative answers too. A five-minute negative cache turns a two-minute regional failover into a seven-minute outage. Lower TTLs <em>before</em> the failover test.</li><li><strong>One auto-registration link per VNet.</strong> Every other link is resolution-only. Teams trying to auto-register VM names into several zones find out late, usually from a script that has been silently failing.</li></ul><h3>The validation matrix — run all five after every change</h3><p>The failure modes are asymmetric. Inbound can work perfectly while outbound is dead, and a single smoke test misses half of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x8t4IohFCF4iK1R-xdKdgQ.png" /></figure><h3>Summary</h3><ul><li><strong>A private DNS zone is data, not a server.</strong> The link is the permission boundary.</li><li><strong>The inbound endpoint</strong> gives that zone an IP address so anything outside can query it.</li><li><strong>The outbound endpoint plus a forwarding ruleset</strong> sends Azure queries to on-premises — and does nothing until the ruleset is linked to a VNet.</li><li><strong>Rulesets link per VNet</strong>, which lets one hub serve two conflicting namespaces during an M&amp;A.</li><li><strong>The private zone always beats a ruleset rule.</strong> Link a zone and you own the whole namespace, including the names you forgot about.</li><li><strong>Forward the </strong><strong>privatelink zone, not the public zone.</strong> The CNAME half is public and should stay that way.</li><li><strong>Automate records via zone groups and policy.</strong> Manual records rot.</li><li><strong>Log DNS at the firewall</strong> if you need attribution or reliable FQDN rules.</li><li><strong>Test in five directions, including the negative one.</strong></li></ul><p>Get the first two bullets right and the architecture assembles itself. Get them wrong and you will spend a year chasing intermittent faults that look like connectivity problems and are always, always DNS.</p><p><em>Service limits and feature behavior in Azure move quickly — verify against current Microsoft documentation before committing a design.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddac9e56062d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG Returns Wrong Chunks — And Your LLM Is Too Polite to Tell You]]></title>
            <link>https://medium.com/@anirbanfiem/rag-returns-wrong-chunks-and-your-llm-is-too-polite-to-tell-you-802113fbc2e6?source=rss-601abdb7348e------2</link>
            <guid isPermaLink="false">https://medium.com/p/802113fbc2e6</guid>
            <category><![CDATA[generative-ai-solution]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-inteligence]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Anirban Roy]]></dc:creator>
            <pubDate>Fri, 03 Apr 2026 11:42:57 GMT</pubDate>
            <atom:updated>2026-04-03T11:52:30.368Z</atom:updated>
            <content:encoded><![CDATA[<h3>Your Support Chatbot Sounds Confident — But It’s Answering the Wrong Question</h3><p><em>The bug isn’t in your prompt. It isn’t in your model. It’s in how you’re slicing your documents.</em></p><p>Picture this.</p><p>A user opens your support chat and types: <em>“Can I get a refund if I cancel mid-month?”</em></p><p>Your chatbot replies with complete confidence: <em>“Yes! You can cancel your subscription at any time from the Billing Settings page. Just click the Cancel Subscription button and follow the on-screen steps.”</em></p><p>Factually accurate. Completely useless. The user asked about <strong>refunds</strong>. The bot answered about <strong>cancellation steps</strong>.</p><p>Nobody wrote a bad prompt. Nobody misconfigured the model. The real culprit is something most engineers never look at: <strong>the chunks that got retrieved from your knowledge base were simply the wrong ones.</strong> The retrieval layer fetched a chunk about <em>how to cancel</em>, not <em>what happens financially when you do</em>.</p><p>This happens constantly in RAG-powered support bots. And the fix isn’t what most tutorials tell you.</p><h3>Why Support Bots Fail Silently</h3><p>A standard RAG pipeline looks like this:</p><pre>User Question → Embed Query → Vector Search → Top-K Chunks → LLM → Answer</pre><p>When engineers debug answer quality, they almost always focus on the last piece. They tweak system prompts. They try different models. They add chain-of-thought instructions. They get marginal improvements and remain puzzled.</p><p>The first 70% of the pipeline — how documents are split into chunks, and which chunks actually get retrieved — stays untouched.</p><p>Here’s the uncomfortable reality: <strong>a well-prompted LLM given the wrong context will produce a confident, well-structured, wrong answer every time.</strong> There is no prompt clever enough to overcome retrieval failure. The model can only synthesize what it receives.</p><p>Let’s look at a real support document and watch what happens at each stage of the pipeline.</p><h3>Our Test Document</h3><p>We’ll use a simplified version of the kind of document that lives in every SaaS company’s help center — a combined billing and cancellation policy page:</p><pre>BILLING &amp; CANCELLATION POLICY<br><br>Subscription Plans<br>We offer three plans: Starter ($9/month), Pro ($29/month), and Business ($99/month).<br>All plans are billed on the first of each month. Annual billing is available at a 20%<br>discount. You can upgrade or downgrade your plan at any time from the Billing Settings<br>page in your account dashboard.<br>Refund Policy<br>We offer a full refund within 7 days of your initial purchase or annual renewal. After<br>the 7-day window, refunds are issued only in cases of technical failure on our end that<br>prevented you from using the product. Mid-month cancellations do not qualify for a<br>partial refund - your subscription remains active until the end of the current billing<br>period. To request a refund, email billing@company.com with your account email and<br>the reason for your request.<br>Cancellation Process<br>You can cancel your subscription at any time. Go to Settings → Billing → Cancel<br>Subscription. You will see a confirmation screen summarising what happens next: your<br>account stays active until the billing period ends, after which it moves to our free<br>tier. You will not be charged again. All your data is retained for 90 days after<br>downgrade, giving you time to export anything you need.<br>Payment Methods<br>We accept all major credit and debit cards (Visa, Mastercard, AmEx, Discover) as well<br>as PayPal. Corporate accounts can request invoice-based billing by contacting<br>sales@company.com. We do not currently support cryptocurrency payments.</pre><p>Simple. Four sections. Clean structure. Now watch how three different chunking strategies handle it.</p><h3>Part 1 — The Chunking Problem</h3><h3>Strategy 0: Fixed-Size Chunking (What Everyone Starts With)</h3><p>The most common approach: split every N characters with a small overlap.</p><pre>def fixed_size_chunk(text: str, chunk_size: int = 300, overlap: int = 50) -&gt; list[str]:<br>    chunks = []<br>    start = 0<br>    while start &lt; len(text):<br>        end = start + chunk_size<br>        chunks.append(text[start:end])<br>        start += chunk_size - overlap<br>    return chunks</pre><p>Run this on our policy document with chunk_size=300 and here is what you get for the critical refund section:</p><p><strong>Chunk 3:</strong></p><pre>...You can upgrade or downgrade your plan at any time from the Billing Settings<br>page in your account dashboard.<br><br>Refund Policy<br>We offer a full refund within 7 days of your initial purchase or annual renewal. After<br>the 7-day window, refunds are issued only in cases of</pre><p><strong>Chunk 4:</strong></p><pre>technical failure on our end that prevented you from using the product. Mid-month<br>cancellations do not qualify for a partial refund — your subscription remains active<br>until the end of the current billing period. To request a refund, email billing@</pre><p>The user asks: <em>“Can I get a refund if I cancel mid-month?”</em></p><p>Vector search finds Chunk 4 as the top result — it contains the words “mid-month”, “refund”, and “cancel” — all strong signal. But Chunk 4 starts mid-sentence. The retrieved context opens with <em>“technical failure on our end that prevented you from using the product”</em> — which is the wrong condition. The full refund rule (7-day window) never appears in any single chunk. The model synthesises an answer from fragments, and it sounds plausible.</p><p><strong>The fundamental problem:</strong> Fixed-size chunking treats your document like a byte array. It has no concept of sentences, paragraphs, or the logical boundaries between ideas.</p><h3>Strategy 1: Sentence-Aware Chunking</h3><p>The first meaningful upgrade: always split at sentence boundaries.</p><pre>import nltk<br>from typing import List<br><br>nltk.download(&#39;punkt_tab&#39;, quiet=True)<br>def sentence_chunk(<br>    text: str,<br>    max_tokens: int = 120,<br>    overlap_sentences: int = 1<br>) -&gt; List[str]:<br>    &quot;&quot;&quot;<br>    Split at sentence boundaries. Never cut mid-sentence.<br>    Carry the last N sentences forward for context continuity.<br>    &quot;&quot;&quot;<br>    sentences = nltk.sent_tokenize(text)<br>    chunks: List[str] = []<br>    current: List[str] = []<br>    current_len = 0<br>    for sentence in sentences:<br>        word_count = len(sentence.split())<br>        if current_len + word_count &gt; max_tokens and current:<br>            chunks.append(&quot; &quot;.join(current))<br>            # Carry the last sentence forward as overlap<br>            current = current[-overlap_sentences:]<br>            current_len = sum(len(s.split()) for s in current)<br>        current.append(sentence)<br>        current_len += word_count<br>    if current:<br>        chunks.append(&quot; &quot;.join(current))<br>    return chunks</pre><p>This produces coherent sentences. No more mid-sentence cuts. But the refund policy paragraph still gets split across two chunks because the paragraph runs long, and the sentence grouping doesn’t know that <em>“We offer a full refund within 7 days”</em> and <em>“Mid-month cancellations do not qualify for a partial refund”</em> are part of the same policy answer.</p><p>If someone asks about mid-month refunds, the second half of the refund policy (the relevant half) scores high on vector similarity, but the context explaining the conditions is in the previous chunk. Better. Still incomplete.</p><h3>Strategy 2: Recursive Structural Chunking</h3><p>This is the right default for most support documentation. The core idea: split on the most meaningful boundary available, and only fall back to smaller splits when necessary.</p><pre>from typing import List, Optional<br><br>class RecursiveChunker:<br>    &quot;&quot;&quot;<br>    Prefer large structural breaks. Fall back to smaller ones only when a<br>    chunk still exceeds the size limit.<br>    &quot;&quot;&quot;<br>    SEPARATORS = [<br>        &quot;\n\n&quot;,    # Blank line between paragraphs (highest preference)<br>        &quot;\n&quot;,      # Single line break<br>        &quot;. &quot;,      # Sentence ending<br>        &quot;, &quot;,      # Clause boundary<br>        &quot; &quot;,       # Individual words (last resort)<br>    ]<br>    def __init__(self, chunk_size: int = 500, overlap: int = 60):<br>        self.chunk_size = chunk_size<br>        self.overlap = overlap<br>    def split(self, text: str, separators: Optional[List[str]] = None) -&gt; List[str]:<br>        separators = separators or self.SEPARATORS<br>        sep = separators[0]<br>        remaining = separators[1:]<br>        parts = text.split(sep)<br>        chunks: List[str] = []<br>        current: List[str] = []<br>        current_size = 0<br>        for part in parts:<br>            part_size = len(part)<br>            # Part still too big - recurse with a finer separator<br>            if part_size &gt; self.chunk_size and remaining:<br>                sub_chunks = self.split(part, remaining)<br>                chunks.extend(sub_chunks)<br>                continue<br>            if current_size + part_size &gt; self.chunk_size and current:<br>                chunks.append(sep.join(current))<br>                overlap_text = sep.join(current)[-self.overlap:]<br>                current = [overlap_text] if overlap_text.strip() else []<br>                current_size = len(overlap_text)<br>            current.append(part)<br>            current_size += part_size<br>        if current:<br>            chunks.append(sep.join(current))<br>        return [c for c in chunks if c.strip()]</pre><p>With a chunk_size of 500 characters, our policy document splits cleanly along the paragraph breaks between sections. The entire &quot;Refund Policy&quot; section lands in one chunk. When a user asks about mid-month refunds, the retrieved chunk contains:</p><ul><li>The 7-day full refund rule</li><li>The mid-month cancellation clause</li><li>The email address to contact</li></ul><p>Everything the LLM needs to give a complete, accurate answer.</p><h3>Strategy 3: Semantic Chunking (When Structure Isn’t Enough)</h3><p>Sometimes your support docs aren’t cleanly formatted. Think: a 4,000-word wall of text exported from Confluence, a call transcript, or a product FAQ that someone wrote in a hurry with no consistent spacing.</p><p>When visual structure can’t be trusted, measure the structure instead.</p><p>The idea: embed each sentence individually. When the cosine similarity between two consecutive sentences drops sharply, that’s a topic boundary — even if there’s no blank line to indicate it.</p><pre>import numpy as np<br>from sentence_transformers import SentenceTransformer<br>from typing import List<br>import nltk<br><br><br>class SemanticChunker:<br>    &quot;&quot;&quot;<br>    Detect topic boundaries by measuring similarity drops between<br>    consecutive sentence embeddings. Split where meaning changes.<br>    &quot;&quot;&quot;<br>    def __init__(<br>        self,<br>        model_name: str = &quot;all-MiniLM-L6-v2&quot;,<br>        split_percentile: int = 85,<br>        min_chunk_chars: int = 150<br>    ):<br>        self.model = SentenceTransformer(model_name)<br>        self.split_percentile = split_percentile<br>        self.min_chunk_chars = min_chunk_chars<br>    def _similarity_drops(self, embeddings: np.ndarray) -&gt; np.ndarray:<br>        drops = []<br>        for i in range(len(embeddings) - 1):<br>            sim = np.dot(embeddings[i], embeddings[i + 1])  # Already normalised<br>            drops.append(1.0 - sim)  # Drop = topic distance<br>        return np.array(drops)<br>    def chunk(self, text: str) -&gt; List[str]:<br>        nltk.download(&#39;punkt_tab&#39;, quiet=True)<br>        sentences = nltk.sent_tokenize(text)<br>        if len(sentences) &lt;= 2:<br>            return [text]<br>        embeddings = self.model.encode(sentences, normalize_embeddings=True)<br>        drops = self._similarity_drops(embeddings)<br>        threshold = np.percentile(drops, self.split_percentile)<br>        split_at = set(np.where(drops &gt; threshold)[0] + 1)<br>        chunks: List[str] = []<br>        current: List[str] = []<br>        for i, sentence in enumerate(sentences):<br>            current.append(sentence)<br>            combined = &quot; &quot;.join(current)<br>            if i in split_at and len(combined) &gt;= self.min_chunk_chars:<br>                chunks.append(combined)<br>                current = []<br>        if current:<br>            chunks.append(&quot; &quot;.join(current))<br>        return chunks</pre><p>This works particularly well for content that mixes topics within a paragraph — like a terms-of-service document where billing terms, data retention rules, and liability clauses coexist in the same wall of text.</p><h3>Part 2 — The Re-Ranker</h3><p>Good chunking solves the structure problem. The re-ranker solves the relevance problem.</p><p>Here is the gap: vector similarity measures how close your query is to a chunk in embedding space. Two pieces of text can have very similar embeddings while one answers the question and the other merely talks about the same general topic.</p><p>For a user asking <em>“Can I get a refund for a mid-month cancellation?”</em>, vector search might return these with similar scores:</p><ul><li>Chunk A: The full refund policy paragraph <em>(what you want)</em></li><li>Chunk B: The cancellation process steps <em>(same topic, wrong focus)</em></li><li>Chunk C: The billing cycle explanation <em>(related vocabulary, irrelevant answer)</em></li></ul><p>A re-ranker sees the query and each candidate chunk <em>together as a pair</em> and produces a proper relevance score — not an approximation. It asks: does this chunk actually answer this question? The cross-encoder architecture makes this far more accurate than embedding similarity alone.</p><h3>Cohere Rerank Integration</h3><pre>import cohere<br>from dataclasses import dataclass<br>from typing import List<br><br>@dataclass<br>class ScoredChunk:<br>    text: str<br>    original_rank: int      # Position in vector search results<br>    relevance_score: float  # Cross-encoder score from re-ranker<br>class Reranker:<br>    &quot;&quot;&quot;<br>    Two-stage retrieval: vector search casts a wide net,<br>    re-ranker applies a precision filter.<br>    &quot;&quot;&quot;<br>    def __init__(<br>        self,<br>        api_key: str,<br>        model: str = &quot;rerank-english-v3.0&quot;,<br>        min_score: float = 0.15<br>    ):<br>        self.client = cohere.Client(api_key)<br>        self.model = model<br>        self.min_score = min_score<br>    def rerank(<br>        self,<br>        query: str,<br>        candidates: List[str],<br>        top_n: int = 4<br>    ) -&gt; List[ScoredChunk]:<br>        if not candidates:<br>            return []<br>        response = self.client.rerank(<br>            query=query,<br>            documents=candidates,<br>            top_n=min(top_n, len(candidates)),<br>            model=self.model<br>        )<br>        return [<br>            ScoredChunk(<br>                text=candidates[r.index],<br>                original_rank=r.index,<br>                relevance_score=r.relevance_score<br>            )<br>            for r in response.results<br>            if r.relevance_score &gt;= self.min_score<br>        ]</pre><p>The original_rank field is useful for logging: if a chunk that ranked 8th in vector search ends up ranked 1st after re-ranking, that&#39;s a signal your embeddings aren&#39;t well-tuned for this domain.</p><h3>The Complete Pipeline</h3><p>Now wire everything together into a single class you can drop into any project:</p><pre>import os<br>from typing import List<br>import chromadb<br>from sentence_transformers import SentenceTransformer<br><br>class SupportRAGPipeline:<br>    &quot;&quot;&quot;<br>    Production-ready RAG pipeline for support documentation.<br>    Stage 1 - Ingest: chunk documents with recursive structure-aware splitting<br>    Stage 2 - Retrieve: vector search (wide recall) → re-rank (tight precision)<br>    Stage 3 - Generate: pass clean context to the LLM<br>    &quot;&quot;&quot;<br>    def __init__(self, cohere_api_key: str):<br>        self.embedder = SentenceTransformer(&quot;all-MiniLM-L6-v2&quot;)<br>        self.db = chromadb.Client()<br>        self.collection = self.db.create_collection(&quot;support_docs&quot;)<br>        self.chunker = RecursiveChunker(chunk_size=500, overlap=60)<br>        self.reranker = Reranker(api_key=cohere_api_key)<br>        self._chunks: List[str] = []<br>    # ── Ingestion ──────────────────────────────────────────────────────────<br>    def ingest(self, documents: List[str], source_labels: List[str] = None) -&gt; None:<br>        all_chunks = []<br>        for doc in documents:<br>            all_chunks.extend(self.chunker.split(doc))<br>        self._chunks = all_chunks<br>        embeddings = self.embedder.encode(all_chunks, normalize_embeddings=True)<br>        self.collection.add(<br>            documents=all_chunks,<br>            embeddings=embeddings.tolist(),<br>            ids=[f&quot;chunk_{i}&quot; for i in range(len(all_chunks))]<br>        )<br>        print(f&quot;Ingested {len(documents)} doc(s) → {len(all_chunks)} chunks&quot;)<br>    # ── Retrieval ──────────────────────────────────────────────────────────<br>    def retrieve(self, query: str, initial_k: int = 15, final_k: int = 4) -&gt; List[str]:<br>        &quot;&quot;&quot;<br>        Two-stage retrieval.<br>        initial_k: how many candidates vector search fetches (recall)<br>        final_k:   how many the re-ranker passes to the LLM (precision)<br>        &quot;&quot;&quot;<br>        query_vec = self.embedder.encode([query], normalize_embeddings=True)<br>        results = self.collection.query(<br>            query_embeddings=query_vec.tolist(),<br>            n_results=min(initial_k, len(self._chunks))<br>        )<br>        candidates = results[&quot;documents&quot;][0]<br>        ranked = self.reranker.rerank(query, candidates, top_n=final_k)<br>        # Log ranking shifts for monitoring<br>        for chunk in ranked:<br>            shift = chunk.original_rank - ranked.index(chunk)<br>            if abs(shift) &gt;= 3:<br>                print(f&quot;  Re-ranker moved chunk {shift:+d} positions &quot;<br>                      f&quot;(score: {chunk.relevance_score:.2f})&quot;)<br>        return [c.text for c in ranked]<br>    # ── Generation ─────────────────────────────────────────────────────────<br>    def answer(self, question: str, client) -&gt; str:<br>        context_chunks = self.retrieve(question)<br>        if not context_chunks:<br>            return &quot;I don&#39;t have enough information to answer that question.&quot;<br>        context = &quot;\n\n---\n\n&quot;.join(context_chunks)<br>        response = client.chat.completions.create(<br>            model=&quot;gpt-4o-mini&quot;,<br>            messages=[<br>                {<br>                    &quot;role&quot;: &quot;system&quot;,<br>                    &quot;content&quot;: (<br>                        &quot;You are a helpful support assistant. Answer the user&#39;s question &quot;<br>                        &quot;using only the information provided in the context below.\n\n&quot;<br>                        &quot;If the context does not contain the answer, respond with: &quot;<br>                        &quot;&#39;I don&#39;t have that information - please contact our support team &quot;<br>                        &quot;at help@company.com.&#39;\n\n&quot;<br>                        &quot;Do not make up policies, prices, or procedures not mentioned &quot;<br>                        &quot;in the context.&quot;<br>                    )<br>                },<br>                {<br>                    &quot;role&quot;: &quot;user&quot;,<br>                    &quot;content&quot;: f&quot;Context:\n{context}\n\nQuestion: {question}&quot;<br>                }<br>            ],<br>            temperature=0.2,<br>        )<br>        return response.choices[0].message.content<br><br># ── Run it ─────────────────────────────────────────────────────────────────<br>if __name__ == &quot;__main__&quot;:<br>    from openai import OpenAI<br>    pipeline = SupportRAGPipeline(<br>        cohere_api_key=os.environ[&quot;COHERE_API_KEY&quot;]<br>    )<br>    billing_policy = &quot;&quot;&quot;<br>    BILLING &amp; CANCELLATION POLICY<br>    Subscription Plans<br>    We offer three plans: Starter ($9/month), Pro ($29/month), and Business ($99/month).<br>    All plans are billed on the first of each month. Annual billing is available at a<br>    20% discount. You can upgrade or downgrade at any time from Billing Settings.<br>    Refund Policy<br>    We offer a full refund within 7 days of your initial purchase or annual renewal.<br>    After the 7-day window, refunds are issued only in cases of technical failure on<br>    our end. Mid-month cancellations do not qualify for a partial refund - your<br>    subscription remains active until the end of the current billing period.<br>    To request a refund, email billing@company.com with your account email and reason.<br>    Cancellation Process<br>    Cancel at any time: Settings → Billing → Cancel Subscription.<br>    Your account stays active until the billing period ends, then moves to the free tier.<br>    All data is retained for 90 days after downgrade.<br>    Payment Methods<br>    We accept Visa, Mastercard, AmEx, Discover, and PayPal.<br>    Corporate accounts can request invoice billing via sales@company.com.<br>    &quot;&quot;&quot;<br>    pipeline.ingest([billing_policy])<br>    # Reads OPENAI_API_KEY from environment automatically<br>    client = OpenAI()<br>    questions = [<br>        &quot;Can I get a refund if I cancel mid-month?&quot;,<br>        &quot;What happens to my data after I cancel?&quot;,<br>        &quot;Do you support PayPal?&quot;,<br>    ]<br>    for q in questions:<br>        print(f&quot;\nQ: {q}&quot;)<br>        print(f&quot;A: {pipeline.answer(q, client)}&quot;)</pre><h3>Measuring the Difference</h3><p>Don’t ship chunking changes without measuring their impact. Here’s a minimal eval harness — no external libraries, no embedding calls, just keyword checks that verify the right information made it into the retrieved context.</p><pre>from dataclasses import dataclass<br>from typing import List, Callable<br><br>@dataclass<br>class SupportEvalCase:<br>    question: str<br>    must_include: List[str]    # Keywords that must appear in retrieved context<br>    must_exclude: List[str]    # Keywords that indicate wrong section was retrieved<br>def run_eval(<br>    retrieve_fn: Callable[[str], List[str]],<br>    cases: List[SupportEvalCase]<br>) -&gt; dict:<br>    scores = {&quot;precision&quot;: [], &quot;recall&quot;: []}<br>    for case in cases:<br>        context = &quot; &quot;.join(retrieve_fn(case.question)).lower()<br>        hits = sum(1 for kw in case.must_include if kw.lower() in context)<br>        noise = sum(1 for kw in case.must_exclude if kw.lower() in context)<br>        recall = hits / len(case.must_include) if case.must_include else 0<br>        precision = hits / (hits + noise) if (hits + noise) &gt; 0 else 0<br>        scores[&quot;precision&quot;].append(precision)<br>        scores[&quot;recall&quot;].append(recall)<br>    return {<br>        &quot;precision&quot;: round(sum(scores[&quot;precision&quot;]) / len(scores[&quot;precision&quot;]), 2),<br>        &quot;recall&quot;:    round(sum(scores[&quot;recall&quot;])    / len(scores[&quot;recall&quot;]),    2),<br>    }<br><br>eval_cases = [<br>    SupportEvalCase(<br>        question=&quot;Can I get a refund if I cancel mid-month?&quot;,<br>        must_include=[&quot;7 days&quot;, &quot;mid-month&quot;, &quot;billing period&quot;, &quot;billing@company.com&quot;],<br>        must_exclude=[&quot;cancel subscription&quot;, &quot;settings →&quot;, &quot;free tier&quot;]<br>    ),<br>    SupportEvalCase(<br>        question=&quot;What happens to my data after I cancel?&quot;,<br>        must_include=[&quot;90 days&quot;, &quot;retain&quot;, &quot;export&quot;, &quot;downgrade&quot;],<br>        must_exclude=[&quot;refund&quot;, &quot;7 days&quot;, &quot;paypal&quot;]<br>    ),<br>    SupportEvalCase(<br>        question=&quot;How do I upgrade my plan?&quot;,<br>        must_include=[&quot;billing settings&quot;, &quot;upgrade&quot;, &quot;downgrade&quot;],<br>        must_exclude=[&quot;cancel subscription&quot;, &quot;refund policy&quot;]<br>    ),<br>]<br># Compare naive (fixed-size) vs improved (recursive + reranked) pipeline<br>naive_score    = run_eval(naive_pipeline.retrieve,    eval_cases)<br>improved_score = run_eval(improved_pipeline.retrieve, eval_cases)<br>print(f&quot;Naive    - Precision: {naive_score[&#39;precision&#39;]} | Recall: {naive_score[&#39;recall&#39;]}&quot;)<br>print(f&quot;Improved - Precision: {improved_score[&#39;precision&#39;]} | Recall: {improved_score[&#39;recall&#39;]}&quot;)</pre><p>On a real support knowledge base with 30 documents, switching from fixed-size to recursive chunking typically improves recall by 18–25%. Adding a re-ranker on top lifts precision by another 15–30%, depending on how much topic overlap exists across your documents.</p><h3>The Framework to Carry Forward</h3><p>Think of your retrieval pipeline as a funnel with three distinct jobs:</p><pre>All support documentation<br>         ↓<br>[Recursive chunking]      ← No severed thoughts, paragraphs stay intact<br>         ↓<br>[Vector search  k=15]     ← Cast wide, optimise for recall<br>         ↓<br>[Re-ranker  top=4]        ← Cut noise, optimise for precision<br>         ↓<br>[LLM generation]          ← Synthesise with complete, relevant context<br>         ↓<br>     Correct answer</pre><p>Vector search is a recall tool — get everything that might be relevant. Re-ranking is a precision tool — throw away everything that isn’t. The LLM is a synthesis tool — given the right pieces, it reliably assembles a good answer.</p><p>Most chatbot failures are misdiagnosed as synthesis problems. They’re usually retrieval problems.</p><h3>The Audit You Should Run This Week</h3><p><strong>Step 1 — Pull 20 random chunks from your vector store and read them.</strong></p><p>Just print them out. Are they coherent thoughts? Do any start or end mid-sentence? Does any single chunk mix two unrelated topics? This five-minute check is more informative than hours of prompt debugging.</p><p><strong>Step 2 — Log your retrieval results for a day.</strong></p><p>Before passing chunks to your LLM, log the query and the retrieved chunks to a file. Read through 20 examples. You will immediately spot patterns in what goes wrong.</p><p><strong>Step 3 — Add Cohere Rerank to one endpoint as an A/B test.</strong></p><p>The API costs roughly $1 per 1,000 re-rankings. The latency hit is 100–200ms. Run both versions in parallel for a week and compare user satisfaction scores. The difference is usually obvious.</p><p><strong>Step 4 — Write five eval cases for your most common support questions.</strong></p><p>You don’t need a sophisticated evaluation framework to start. Five keyword-based cases that check whether the right policy information appears in retrieved context will catch the majority of chunking regressions.</p><p>The support chatbot problem is a great lens for RAG because the stakes are clear. A user asking about refunds needs the refund policy — not a related paragraph from the cancellation guide. When retrieval fails, the user either gets a wrong answer or goes to a human agent. Both are expensive.</p><p>The good news is that the fixes are straightforward, well-understood, and measurable. You don’t need to retrain anything, switch models, or rearchitect your system. You need to split your documents more carefully, and add a re-ranker to your retrieval step.</p><p>Fix retrieval first. The rest of the pipeline takes care of itself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=802113fbc2e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>