<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vishal Kumar on Medium]]></title>
        <description><![CDATA[Stories by Vishal Kumar on Medium]]></description>
        <link>https://medium.com/@vishalkumar9dec?source=rss-f5f1451ef938------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*CQJqWS-YvevhYonY</url>
            <title>Stories by Vishal Kumar on Medium</title>
            <link>https://medium.com/@vishalkumar9dec?source=rss-f5f1451ef938------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:03:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vishalkumar9dec/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI Governance — A Policy Is Not Governance Until Something Enforces It]]></title>
            <link>https://medium.com/@vishalkumar9dec/ai-governance-a-policy-is-not-governance-until-something-enforces-it-0dcdc1b4b995?source=rss-f5f1451ef938------2</link>
            <guid isPermaLink="false">https://medium.com/p/0dcdc1b4b995</guid>
            <category><![CDATA[ai-architecture]]></category>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Vishal Kumar]]></dc:creator>
            <pubDate>Thu, 15 Oct 2026 17:15:47 GMT</pubDate>
            <atom:updated>2026-10-15T17:15:47.533Z</atom:updated>
            <content:encoded><![CDATA[<p><em>What AI governance actually is, how it differs from the “Govern” pillar of an agent platform, and what it looks like when each sentence of a policy has a mechanism behind it.</em></p><p>If your AI policy says “agents must not act without approval”, what happens when one does?</p><p>I used to think of governance as the document. A page of principles, a risk register, a sign-off. Then I tried to explain my own project’s governance in an interview setting and realised I could point to every sentence of a policy, but only some of them had anything behind them.</p><p>This series has covered <a href="https://medium.com/@vishalkumar9dec/harness-engineering-llm-guardrails-4b5d80664119">guardrails</a>, <a href="https://medium.com/@vishalkumar9dec/the-answer-is-in-the-screenshot-why-cant-your-rag-find-it-part-1-making-images-searchable-1515ca4cc4d4">dealing with screenshots in RAG</a>, <a href="https://medium.com/@vishalkumar9dec/why-did-my-rag-system-answer-a-kubernetes-question-with-a-docker-runbook-293f23681479">RAG relevance vs groundedness vs confidence</a>. This post is the AI Governance layer over all of those: what they add up to, and why it has a name.</p><h3>What is AI governance?</h3><p>AI governance is the set of policies, roles, processes and technical controls an organisation uses to make sure its AI systems are built and run responsibly, lawfully, and within the risk it has agreed to take.</p><p>That sounds like a document. It is mostly not.</p><p>It answers six plain questions:</p><ul><li><strong>Who owns this system</strong>, and who approved it for release?</li><li><strong>How risky is it</strong>, and are the controls proportionate to that?</li><li><strong>What data does it touch</strong>, and who agreed to that?</li><li><strong>What is it allowed to do</strong>, and who checks?</li><li><strong>How do we know it still works</strong> after release?</li><li><strong>What happened last Tuesday</strong>, and can we prove it?</li></ul><p>The frameworks people cite are all organised around questions like these. The EU AI Act sorts systems into risk tiers with obligations attached to each. NIST’s AI Risk Management Framework has four functions: Govern, Map, Measure and Manage. ISO/IEC 42001 describes an AI management system, in the same spirit as ISO 27001 for security. <em>(This is general knowledge, not something I verified against the current texts. Read the source before quoting any of them.)</em></p><h3>What does a policy look like with nothing behind it?</h3><p>Take three sentences any governance policy might contain:</p><ol><li>High-risk actions need a human to approve them.</li><li>Every agent has a named owner.</li><li>Users only see documents they are entitled to.</li></ol><p>Each reads well. Now ask of each one: <em>what stops a violation, and where is the proof?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*az10Ah8nVkXp35HSHkte2g.png" /><figcaption><em>A sentence becomes governance when it has a control that blocks the violation and a record that shows it did.</em></figcaption></figure><p>If the answer is “people are told to”, that is a hope, not a control. Hopes fail quietly. A violation leaves no trace, so the first anyone learns of it is the incident.</p><p><strong>That is the whole idea of this post. Governance is the distance between the sentence and the mechanism, and it is measured in how much of that distance you have closed.</strong></p><h3>Isn’t “Govern” in an agent platform the same thing?</h3><p>Related, not the same.</p><p>Some agent platforms, <a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/agents?_gl=1*1a0p21u*_up*MQ..*_gs*MQ..&amp;gclid=CjwKCAjw88HWBhA3EiwAo2Zd6OSRvHajaObU0uC5ehWHWM2gFGTfBlp_gloBWKnRYsOIWmYUcAwzTxoCA4IQAvD_BwE&amp;gclsrc=aw.ds#platform_architecture">Google’s Gemini enterprise agent platform</a> among them, organise themselves into four pillars: <strong>Build, Scale, Govern and Optimise</strong>. The pillar is the part of the <em>platform</em> that enforces governance on agents at runtime. The discipline is bigger. It includes legal review, ethics, ownership and the committee that signs things off, none of which a platform can ship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aIrqy1x5HGFXhvjXbWKlPg.png" /><figcaption><em>Governance as a discipline decides the rules. The Govern pillar is the part that enforces them while agents run.</em></figcaption></figure><p><em>A simple way to hold the difference: the policy is the rulebook, and Govern is the referee who enforces it and writes down what happened.</em></p><h3>What does enforcement look like in a real system?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oMjejAefCBd83xOC77w-uw.png" /><figcaption><em>Each governance question and the mechanism that answers it.</em></figcaption></figure><p><strong>What is it allowed to do?</strong> Authorisation decisions are made by a policy engine (Cerbos), not by scattered if statements. Policies are code, with their own tests.</p><p><strong>What can it reach, and how is that checked?</strong> Every model call goes through one AI Gateway, and a test fails if code calls a vendor directly. One enforcement point per guardrail concern means there is one place to change a rule and one place to look.</p><p><strong>How do we know it still works?</strong> A versioned Golden Set and Eval Runs act as release evidence, and a regression or failed run is announced instead of waiting to be noticed.</p><p><strong>What happened, and can we prove it?</strong> Actions that change things are proposed in chat, confirmed by the user with their own token, performed as that user, and written to an append-only audit record.</p><p><strong>Who may see which rows?</strong> The filter is in the SQL, and nothing the user cannot read ends up in traces or logs.</p><p>Notice the pattern. None of these is a paragraph. Each is a thing that runs, with a test or a record you can show someone.</p><h3>How would I explain it in an interview?</h3><p>In four lines:</p><ol><li>Governance is who owns an AI system, what it may do, and how you prove it behaved.</li><li>A policy only counts once a control enforces it and a record evidences it.</li><li>A platform’s Govern pillar enforces at runtime. The discipline also covers ownership, risk and compliance.</li><li>Start with the six questions, map each to a mechanism, and keep an honest list of the ones that are only sentences.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zEsPoL1XAFKzoXLotDyJbQ.png" /><figcaption><em>The loop that makes a policy real: write the rule, enforce it, record the decision, review the record.</em></figcaption></figure><h3>What I can’t answer yet</h3><p>If a control is built and nobody has measured it, does it count as governance, or only as intent? And who in your organisation owns the list of rules that are still only sentences?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0dcdc1b4b995" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harness Engineering: LLM Guardrails]]></title>
            <link>https://medium.com/@vishalkumar9dec/harness-engineering-llm-guardrails-4b5d80664119?source=rss-f5f1451ef938------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b5d80664119</guid>
            <category><![CDATA[llm-guard]]></category>
            <category><![CDATA[guardrail]]></category>
            <category><![CDATA[presidio]]></category>
            <category><![CDATA[llm-guardrails]]></category>
            <dc:creator><![CDATA[Vishal Kumar]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 17:16:58 GMT</pubDate>
            <atom:updated>2026-10-08T17:27:07.721Z</atom:updated>
            <content:encoded><![CDATA[<p><em>What a guardrail is, what LLM Guard and Microsoft Presidio give you, how to tune and test them, and why safety checks say nothing about a confident wrong answer.</em></p><h3>What is a guardrail, and where does it sit?</h3><p>A guardrail is a check in code, around a model call. It is not a sentence in the prompt asking the model to behave.</p><p>That difference matters. A prompt instruction is a request the model may or may not follow. A check in code runs whatever the model does.</p><p>I find four places to put one useful.</p><ol><li><strong>Input.</strong> Before the model sees the text: prompt injection, personal data, toxicity.</li><li><strong>Output.</strong> Before the user sees the reply: personal data, refusals, toxic text, bad links.</li><li><strong>Retrieval.</strong> Between search and model. A relevance gate, and a rule that retrieved text is treated as data, never as instructions. Both are designed here, not built.</li><li><strong>Tool calls.</strong> An allowlist of tools, checks on arguments, a human approving writes. I haven’t built this, because my agent only reads. The day it can write, this becomes the most important one.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hJ0UnQ-E9Fm6cumfE5p8Iw.png" /><figcaption><em>Solid boxes exist. The dashed ones are designed only. The wrong answer passed through every solid box.</em></figcaption></figure><p>Whatever the check, it ends in one of four verdicts:</p><ul><li><strong>allowed</strong>: pass it on</li><li><strong>blocked</strong>: stop the turn</li><li><strong>flagged</strong>: pass it on, but record it</li><li><strong>redacted</strong>: rewrite the text, then continue</li></ul><p>Which verdict a check should return is a design choice, and I’ll get to mine. First, the tools, because most of the work is already done for you.</p><h3>What does LLM Guard and Presidio add?</h3><p><a href="https://github.com/data-privacy-stack/presidio"><strong>Presidio</strong></a><strong> </strong>is a separate pair of packages: an analyzer and an anonymizer. Mine are both 2.2.358. LLM Guard’s Anonymize and Sensitive scanners are built on it.</p><p>The analyzer finds personal data and returns an entity type, a position and a confidence score. <em>It combines three things: pattern recognisers (some with checksums, like card numbers), a named-entity model for names and places, and context words that raise confidence when they sit near a candidate.</em> You can add your own recognisers and deny-lists for identifiers only you have.</p><p>The anonymizer applies an operator to each hit: replace, redact, mask, hash or encrypt. Encrypt is reversible with the key.</p><p><a href="https://github.com/protectai/llm-guard"><strong>LLM Guard</strong></a><strong> </strong>is an open-source library from Protect AI. I have version 0.3.16 installed, and I read the package rather than the marketing page.</p><p>The shape is simple. You call scan_prompt or scan_output with a list of scanners and some text. Each scanner hands back sanitised text, a valid flag and a risk score. Scanners are stateless, and most have a threshold you can set.</p><p>The installed input scanners are Anonymize, BanCode, BanCompetitors, BanSubstrings, BanTopics, Code, Gibberish, InvisibleText, Language, PromptInjection, Regex, Secrets, Sentiment, TokenLimit and Toxicity.</p><p>The output scanners are BanCode, BanCompetitors, BanSubstrings, BanTopics, Bias, Code, Deanonymize, FactualConsistency, Gibberish, JSON, Language, LanguageSame, MaliciousURLs, NoRefusal, ReadingTime, Regex, Relevance, Sensitive, Sentiment, Toxicity and a URL reachability check.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h2koVbmKBTEqRbWkDr0rBw.png" /><figcaption><em>LLM Guard is the scanning framework. Presidio is the PII engine under two of its scanners.</em></figcaption></figure><h3>How did I wire it up?</h3><p>Seven guardrails, one row each in a guardrail_configs table, each with an enabled flag and a threshold. An admin can change either without a redeploy.</p><p><strong>Input:</strong> PromptInjection (threshold 0.92, block), Anonymize (redact), Toxicity (0.5, block).</p><p><strong>Output:</strong> Sensitive (redact), NoRefusal (0.75, flag only), Toxicity (0.7, block), MaliciousURLs (0.5, block).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nj0XlFJK3WMrebjWwmp8rw.png" /><figcaption><em>The table holds the enabled flag and threshold. The action lives in code.</em></figcaption></figure><h3>How do you tune and test one?</h3><p>Here’s the plan, in the order I’d do it.</p><p><strong>Build a labelled set.</strong> Benign questions a real user would ask. Known injection strings. PII samples, including the awkward kind. Borderline cases where a reasonable person would disagree. A few dozen per guardrail is a start.</p><p><strong>Measure per scanner.</strong> Block rate on the bad set, false-block rate on the benign set, and latency. Run it on the machine that will serve traffic, not a laptop.</p><p><strong>Split the suites.</strong> Use the fake engine for fast unit tests on every commit. Keep the real engine in a slower suite, because it loads models.</p><p><strong>Watch production.</strong> Trace verdict counts per guardrail. A threshold change that doubles the blocks should be visible the same day. Sample-review flagged and redacted text by hand, without logging the sensitive content itself, or you’ve built a PII store by accident.</p><p><strong>Red-team on a schedule.</strong> Injection phrasings change. A test set from last quarter slowly turns into a test of last quarter’s attacks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7wBN23wB-PX-OCPO6bI47Q.png" /><figcaption><em>Fast fake-engine tests on every commit, a slower real-engine suite, then counts and sampling in production.</em></figcaption></figure><h3>Should you use something else?</h3><p>Maybe. I picked LLM Guard in an ADR because every check I wanted was a stateless scan of one turn of text. That’s a good fit for a scanner list.</p><p>Two alternatives come up a lot. <a href="https://github.com/NVIDIA-NeMo/Guardrails"><strong>NeMo Guardrails </strong></a>is built around dialogue policies: rails that shape which conversational paths the bot may take. <a href="https://github.com/guardrails-ai/guardrails"><strong>Guardrails AI</strong></a> leans on validators and structured-output checks. Both may suit you better if your problem is conversation flow or strict output schemas. I haven’t used either, so treat that description as general knowledge and read the current docs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b5d80664119" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can You Reward an LLM Every Time It Answers Correctly?]]></title>
            <link>https://medium.com/@vishalkumar9dec/can-you-reward-an-llm-every-time-it-answers-correctly-ef73454a246b?source=rss-f5f1451ef938------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef73454a246b</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[rlhf-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <dc:creator><![CDATA[Vishal Kumar]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 01:57:23 GMT</pubDate>
            <atom:updated>2026-10-05T01:57:23.561Z</atom:updated>
            <content:encoded><![CDATA[<p><em>What RLHF actually does, why “reward” only exists during training, and what to do with a thumbs-up button in the meantime.</em></p><p>If a user gives an answer a thumbs-down, why can’t the model just get punished for it?</p><p>It sounds like it should work. Dogs learn that way, children do, and so does most people’s mental model of machine learning. The phrase “reward the model for correct answers” gets used casually in a lot of conversations about LLMs.</p><p>It’s about a question that follows naturally from my previous posts: once you know an answer was wrong, can you feed that back to the model, right now, so it doesn’t happen again?</p><p><strong>The short answer is no, not at answer time. The longer answer is more useful.</strong></p><h3>What would prompt someone to ask this?</h3><p><strong>Illustrative scenario, not a real event.</strong> Picture a team meeting about an internal runbook assistant. Someone pulls up the now-familiar example: an engineer asked “How do I fix pod restart issue?”, the corpus has no Kubernetes content, and the assistant replied with a confident, cited, numbered Docker procedure.</p><p>Someone says, reasonably: “We already have a thumbs-down button. Why don’t we just reward the model every time it answers correctly, and punish it when it doesn’t?”</p><p><strong>The issue.</strong> It’s a misconception that hides inside a very sensible instinct. It assumes a deployed model has a place where a reward can go in.</p><p><strong>When you’d notice.</strong> Usually when the question keeps coming back. The thumbs-down data piles up, the same wrong answer appears again next week, and someone asks why the button “doesn’t do anything”. It doesn’t do anything to the model, because nothing is wired from the button to the weights. That’s not a bug. There isn’t a wire to connect.</p><p>To work out why, you can trace where a reward could possibly enter. There are two places a model exists: when it’s answering, and when it’s being trained.</p><h3>Where could a reward actually go in?</h3><p>At answer time, the model’s weights are frozen. A question comes in, the weights are read, an answer comes out, and the weights are exactly as they were before. Serving a model is closer to running a very large function than to a student sitting an exam.</p><p><em>In a training run, it’s different. There’s a loss or a reward, a gradient is computed, and the weights change.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*bQOw7J8z9cXIlXKMKADwBA.png" /><figcaption>Figure 1: <em>Left: answering reads the weights and writes nothing. Right: a reward is a number inside a training step that changes the weights.</em></figcaption></figure><p>So “reward” isn’t a thing the model receives, like a treat. It’s a number used to compute an update, and it lives entirely inside training. A user’s thumbs-up in production can be recorded, stored, analysed and used later, but the model that answered them never sees it.</p><p>Which raises the real question: if a reward only exists in training, what does a training run that uses reward look like?</p><h3>What is RLHF, mechanically?</h3><p><strong>RLHF stands for reinforcement learning from human feedback</strong>. It’s usually described as three stages. I’ll describe it the way I understand the standard recipe, and this is general knowledge about the method, not something I ran on this project.</p><p><strong>Stage 1: supervised fine-tuning (SFT).</strong> Start with a base model that predicts the next token. Train it on examples of good responses so it behaves like an assistant. This produces the starting point, often called the reference model.</p><p><strong>Stage 2: a reward model.</strong> Take a prompt and have the model write two or more responses. A person picks the better one. Repeat across many prompts. Then train a separate model, r(prompt, response), that outputs a score, using those picks as labels.</p><p><strong>Stage 3: optimise against the reward model.</strong> Use reinforcement learning, most commonly PPO (Proximal Policy Optimisation), to adjust the model so its answers score higher.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*jQxoV0GsBp7XYHQEX28lNw.png" /><figcaption><em>Figure 2: Each stage produces what the next one needs: an assistant, then a scorer, then an assistant tuned toward the scorer.</em></figcaption></figure><p>Stage 3 has one more piece worth knowing. The objective isn’t just “get a high score”. It’s roughly:</p><blockquote>maximise E[ r(x, y) ] − β · KL( π_θ(y|x) ‖ π_ref(y|x) )</blockquote><ul><li><strong>E[r(x, y)]</strong> is the average reward model score over responses the new model generates. Raising it makes answers score higher.</li><li><strong>KL(π<em>θ ‖ π</em>ref)</strong> measures how differently the new model assigns probabilities to tokens than the SFT model does. At step 0 it is zero, because the two models are the same. It grows as training pushes π_θ away.</li><li><strong>β</strong> sets how expensive drifting is. A high β keeps the model close to SFT. A low β lets it chase reward more freely.</li></ul><p>In plain words: get higher scores, but pay a penalty for drifting far from where you started. That penalty is a leash. Without it, a model can find odd text that scores well and stop making sense, because the scorer is only an approximation of what people want.</p><h3>How does a reward model learn what “good” means?</h3><p>Nobody hands it a label saying “this answer is correct”. The data is comparative: response A beat response B for this prompt.</p><p><em>The standard training loss is a Bradley–Terry comparison:</em></p><pre>P(A preferred over B) = sigmoid( r(A) − r(B) )</pre><p>Plain words: the bigger the score gap in favour of the winner, the more probable the human’s pick. Training pushes r until its score gaps agree with the picks across the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*BooKg6JoQQiEgVc3Niowcw.png" /><figcaption>Figure 3: <em>A prompt, two responses, one human pick. The scorer only ever sees which of two was better.</em></figcaption></figure><p>There’s a subtle consequence. The reward model learns “what people tend to prefer”, not “what is true”. If labellers like fluent, confident answers, it learns to like them too. Keep that in mind, because it comes back when we talk about what goes wrong.</p><p>So RLHF is one recipe. But the phrase people usually mean, “reward every correct answer”, shows up in three different forms.</p><h3>Is RLHF the only way to reward correct answers?</h3><p>No. Three approaches come up, and they differ mainly in <em>what decides correct</em>.</p><ul><li><strong>RLHF</strong> uses a learned reward model built from human preferences. It’s suited to things with no exact right answer: style, helpfulness, safety.</li><li><strong>RLVR</strong> (reinforcement learning with verifiable rewards) uses a program that checks the answer. Unit tests pass, the maths matches, a string matches exactly. It’s suited to tasks where correctness is automatically checkable.</li><li><strong>DPO</strong> (Direct Preference Optimisation) skips the separate reward model and trains directly on chosen/rejected pairs. It uses the same kind of data as RLHF, with a simpler pipeline. From what I can tell it tends to be the first thing teams reach for now, though I’d check the current literature rather than take my word for it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*TnpRdZlR8SbP-j9Y79sF7A.png" /><figcaption>Figure 4: <em>Same goal, different judge: a learned scorer, a program, or the pairs themselves.</em></figcaption></figure><p>Here’s how I’d translate “reward every correct answer” into these terms. If correctness can be decided in code, that’s RLVR. If a person or a judge decides, that’s RLHF or DPO. If it can’t be decided by either, there’s nothing to reward.</p><h3>What would this look like for a runbook assistant?</h3><p>There’s no cheap automatic check for “was this runbook answer correct?”. You can’t run a unit test on a paragraph of operational advice. So the signal has to come from somewhere else:</p><ul><li>a thumbs-up/down button in the UI,</li><li>a curated list of near-miss questions,</li><li>or an LLM judge labelling at scale (sometimes called RLAIF, reinforcement learning from AI feedback).</li></ul><p>Then you’d build pairs. For the pod-restart question:</p><ul><li><strong>Chosen:</strong> “No SOP covers Kubernetes pods. The closest is 0017, which is about Docker containers.”</li><li><strong>Rejected:</strong> the confident, cited Docker procedure.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*nn6HQbTyMHqiBn3lVj-UnQ.png" /><figcaption>FIgure 5: <em>One bad answer becomes a chosen/rejected pair. The weights only change in a later training run.</em></figcaption></figure><p>Then you’d need something to train. This is where the practical questions start.</p><p>A closed hosted model can only be tuned through a vendor’s fine-tuning offering, if that model supports it at all. Which models qualify, and which methods they allow, changes over time, so treat this as <strong>unverified</strong> and check the current documentation before planning around it.</p><p>An open-weight model served locally (through Ollama, for example) is one you can tune yourself. Hugging Face’s TRL library includes a DPO trainer, which is the sort of tool you’d reach for. I haven’t run it on this project.</p><p>After training, you’d evaluate, and here’s a trap: test on held-out <em>in-corpus</em> questions as well as out-of-corpus ones. Tuning a model toward declining can make it decline things it should answer. A model that says “not covered” to everything scores perfectly on the near-miss set and is useless.</p><p>Then you deploy the new weights. The model doesn’t learn again until the next training run.</p><p>That’s a lot of moving parts. What can go wrong along the way?</p><h3>What goes wrong?</h3><p><strong>Reward hacking.</strong> The model learns to please the reward model, not to be right. If the reward model likes long, confident answers, you get long, confident answers, including wrong ones. That’s the leash from Stage 3 earning its keep.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Ll-aGKsiqISD2J2E0nRwMQ.png" /><figcaption>Figure 6: <em>Invented curves to show the shape: the scorer’s number keeps rising while real quality peaks and falls. Not data.</em></figcaption></figure><p><strong>Reward-model errors and labeller bias.</strong> The scorer is a model trained on human picks, so it inherits their inconsistencies and blind spots. Two people can disagree about which runbook answer is better.</p><p><strong>Over-refusal.</strong> Push hard toward “say it’s not covered” and you get a model that says it too often. Nobody complains about a refusal until it’s the one they needed answered.</p><p><strong>Cost.</strong> Collecting data, reviewing it, running training, evaluating, and redoing it whenever you change the base model. That last one is easy to overlook, and it matters for the next section.</p><p><strong>No guarantee.</strong> Preference training shifts probabilities. It doesn’t make a wrong answer impossible.</p><h3>Should you use it? Not first.</h3><p>Here’s how I think about it. The gap in the example system wasn’t that the model had bad tendencies. It was that no code decided whether a question was in scope. The relevance score was already computed and thrown away (the last post’s story), and the model was asked to notice on its own.</p><p>A gate in code fixes that directly. A relevance gate on the cross-encoder score, plus a checked coverage verdict, is:</p><ul><li>cheaper,</li><li>testable, because you can write a failing case for it,</li><li>independent of which model you use.</li></ul><p>RLHF changes the model’s <em>tendencies</em>. That’s a different kind of fix, and it has to be redone every time the model changes. If you swap models next quarter, the preference training doesn’t come with you. The code gate does.</p><p>That doesn’t make the feedback data worthless. It’s the opposite. Thumbs-up/down pairs become eval data today: a golden set you can run against any change. And they become training data later, if the assistant ever moves to an open-weight model you control. Collecting them costs little. Not collecting them is the expensive choice.</p><h3>What would I tell someone thinking about this?</h3><p>A few tips, from reasoning about it rather than from having done it.</p><p><strong>Ask “where would the reward enter?” before designing anything.</strong> If you can’t point at a training step, you’re describing a wish, not a mechanism.</p><p><strong>Ask who or what decides “correct”.</strong> Code means RLVR. A person or judge means RLHF or DPO. Nobody means you don’t have a reward yet.</p><p><strong>Fix in code what code can fix.</strong> Scope, permissions and thresholds are code problems. Tone and phrasing are model problems.</p><p><strong>Store feedback with enough context to reuse it.</strong> The prompt, the answer, the cited sources, the model version and what the reviewer decided. A bare thumbs-down is much less useful than a reviewed pair.</p><p><strong>Hold out a test set before you train.</strong> If the pairs you evaluate on are the pairs you trained on, the numbers only tell you it memorised.</p><p><strong>Watch both failure directions.</strong> Wrong answers given, and right answers withheld.</p><h3>The trade-off in one box</h3><ul><li><strong>Decision:</strong> how to stop the model answering out-of-scope questions. Change its tendencies with preference training, or gate in code.</li><li><strong>Options:</strong> a code gate first; RLHF or DPO; both.</li><li><strong>Picked, and why:</strong> a code gate first. It’s model-independent, testable, cheap, and it survives switching models. Collect preference pairs now.</li><li><strong>What it costs:</strong> a threshold to calibrate, some false declines, and a review habit for the feedback data.</li><li><strong>Impact today:</strong> no training has been done. The gate is proposed, not built.</li><li><strong>Breaks when:</strong> there are enough reviewed pairs to train on and still hold out a test set, and the assistant runs on an open-weight model you control. That’s when preference training starts to make sense.</li><li><strong>Plan B:</strong> DPO on an open-weight model, or vendor fine-tuning if the model qualifies. Evaluate on held-out in-corpus and out-of-corpus questions.</li></ul><h3>What should you hold on to?</h3><p>Three sentences.</p><p>At inference, the weights are frozen and there is no reward. In training, reward is a number that updates weights. “Reward every correct answer” is only possible if “correct” can be defined as code (RLVR) or as reliable human or judge preferences (RLHF, DPO), and then you retrain.</p><p>So the thumbs-down button isn’t broken. It’s just not a training signal yet, only the raw material for one.</p><p>Which leaves a practical problem. If the best first use of that data is as an eval set, how do you build a golden set and pick a relevance threshold without fooling yourself?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef73454a246b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Answer Is in the Screenshot. Why Can’t Your RAG Find It? (Part 2: Finding and Showing Images)]]></title>
            <link>https://medium.com/@vishalkumar9dec/the-answer-is-in-the-screenshot-why-cant-your-rag-find-it-part-2-finding-and-showing-images-aecb78705c95?source=rss-f5f1451ef938------2</link>
            <guid isPermaLink="false">https://medium.com/p/aecb78705c95</guid>
            <category><![CDATA[hybrid-search]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <dc:creator><![CDATA[Vishal Kumar]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 15:10:22 GMT</pubDate>
            <atom:updated>2026-10-03T06:27:59.074Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/@vishalkumar9dec/the-answer-is-in-the-screenshot-why-cant-your-rag-find-it-part-1-making-images-searchable-1515ca4cc4d4"><em>Part 1</em></a><em> got the words inside screenshots into the index. This part covers retrieving them, storing the bytes, showing the right image in an answer, securing the whole thing, and knowing whether it helped.</em></p><p>In <a href="https://medium.com/@vishalkumar9dec/the-answer-is-in-the-screenshot-why-cant-your-rag-find-it-part-1-making-images-searchable-1515ca4cc4d4">Part 1</a> I argued that an image is content at a position in the document, not metadata attached to it. We parsed pages into ordered blocks, skipped decorative images, ran OCR plus a context-aware vision description, and put a short summary of each image inside the chunk that owns it. The full OCR sits in its own field.</p><p>Quick recap of the (illustrative, invented) story: an on-call engineer asks what the “exit code 137” dialog means and where to click next. The answer lived only in a screenshot. Now the dialog’s text is in the chunk. Is that enough?</p><h3>Why isn’t vector search enough for screenshots?</h3><p>Because screenshots are full of exact strings: error codes, hostnames, flags, menu labels. Embeddings are good at meaning and, in my experience, patchy on exact tokens like 137 or pod-7f9c. That&#39;s a general property of dense retrieval, and I haven&#39;t benchmarked it on this corpus.</p><p><em>So run two searches and combine them: vector search, plus keyword search (BM25, or Postgres full-text) over the chunk text and the full OCR. Merge the candidates, then rerank.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*yq0dpo7hJKg7iDFiyerHJQ.png" /><figcaption>Figure 1: <em>The keyword path catches what the vector path blurs.</em></figcaption></figure><p>If I had to pick one upgrade for screenshot-heavy runbooks, it would be this one. It also fits the earlier posts: the reranker stays where it was, and the candidate pool simply gets better.</p><p>Retrieval gives us chunks and image ids. Next question: where do the image bytes actually live?</p><h3>Where do the bytes go?</h3><p>Not in the chunk table, and not in a public folder.</p><ul><li><strong>Bytes in object storage</strong> (S3-compatible if you’re self-hosting), keyed by content hash. That dedupes identical images across document versions, and it makes them immutable.</li><li><strong>Metadata in the database:</strong> document version, position, type, dimensions, MIME type, OCR text, description, a generated flag, the enrichment model and version, and a review status.</li><li><strong>Renditions:</strong> the original, a web-sized version and a thumbnail, so the chat UI doesn’t pull a 4 MB PNG for a 300-pixel box.</li><li><strong>Access control inherited from the document.</strong> An image is as readable as the page it came from, no more.</li><li><strong>Garbage collection</strong> for images no longer referenced by any current version. Otherwise storage only ever grows.</li><li><strong>Strip EXIF and other embedded metadata on ingest.</strong> Photos and some screenshots carry more than you’d expect.</li></ul><p>Storage is the boring part, and I mean that as praise. The interesting part is what the chat does with it.</p><h3>How do you show the right image in an answer?</h3><p>Naively: let the model write markdown with an image URL. Please don’t. It can invent a URL, or point at something it shouldn’t.</p><p>What I’d do instead:</p><ol><li>Retrieval returns chunks, each with image ids. The server builds a <strong>numbered allow-list</strong> of those images.</li><li>The prompt lists them, with their descriptions: “1: pod dialog, 2: restart menu”.</li><li>The model writes tokens like [img:2] where an image belongs. It never writes a URL.</li><li>The server validates: tokens not on the list get dropped.</li><li>The UI swaps each valid token for a real image, fetched from an <strong>authorised route</strong> (or a short-lived signed URL) that checks the user’s access to the source document.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*r--8uFdCmhazD5hwS7r2zw.png" /><figcaption>Figure 2: <em>The model names images. The server decides which exist and who may see them.</em></figcaption></figure><p>Then the polish: show the image inline at the step, with alt text or caption, click to zoom, lazy loading, and fixed dimensions so the answer doesn’t jump around as images arrive.</p><p>Optionally, when an answer truly depends on pixels (which button is highlighted, say), pass the retrieved image itself to a vision model at answer time. That costs more, so gate it. This is also the awkward case for charts: if the meaning is in the shape of a line rather than in text, an OCR-plus-description summary may lose it, and answer-time vision is the fallback.</p><p>One trap: SVG can carry script. Serve user-supplied images as images only, with X-Content-Type-Options: nosniff and a strict Content-Security-Policy, and sanitise SVG or rasterise it on ingest.</p><p>This is also where PreReqs Pro is blocked today. It stores a server-side path for each image but has no authorised route that serves it, and I deliberately didn’t mount the folder as public static files, because that would bypass access control.</p><p>Which brings up a bigger point. A screenshot isn’t just content. It’s also an input.</p><h3>What can a picture do to your system?</h3><p>Three things I’d plan for.</p><p><strong>A picture can carry instructions.</strong> <em>OCR and vision-model output are untrusted text, exactly like a web page. An image containing “ignore previous instructions” will, after OCR, sit in your chunk text. Treat it as data: keep it delimited in the prompt, and scan it like any other retrieved content.</em></p><p><strong>A picture can leak secrets.</strong> <em>Screenshots have tokens, hostnames, customer names, email addresses. Scan the OCR text for secrets and PII, then redact, block or flag. Since OCR text goes into an index that many people can query, this matters more than it first seems.</em></p><p><strong>Where the pixels go matters.</strong> <em>Don’t send restricted images to an external vision model without a policy. Route the call through your AI gateway so there’s one place to enforce it. And keep image bytes and OCR text of restricted documents out of traces and logs.</em></p><p>I’m describing a design here. I haven’t built the scanning or the routing, and I wouldn’t claim any tool handles it out of the box.</p><p>Finally: how would you know any of this works?</p><h3>How do you know the images helped?</h3><p>Same discipline as the earlier posts. A golden set, and a before-and-after.</p><ul><li><strong>Add golden questions that only a screenshot can answer.</strong> The exit-code question from the story is the template.</li><li><strong>Measure recall with and without image enrichment.</strong> If it doesn’t move, don’t pay for it.</li><li><strong>Review a sample of generated descriptions</strong> by hand, regularly.</li><li><strong>Re-enrich when the vision model changes</strong>, and compare against the old output before switching.</li></ul><h3>Did I need image embeddings instead?</h3><p>Here are the four options I weighed.</p><ul><li><strong>Ignore images.</strong> Free, and wrong for any runbook with screenshots.</li><li><strong>Attach images to sections, unsearchable.</strong> Users can see them after the fact, but retrieval never finds them. Better than nothing; it doesn’t fix the story above.</li><li><strong>Convert to text (OCR + description) inside the owning chunk, and add hybrid search.</strong> This is the one I’d pick.</li><li><strong>Multimodal image embeddings.</strong> Embed the picture itself. For screenshots of terminals and UIs, which are mostly text, I’d expect OCR to beat a general image encoder.</li></ul><h3>The trade-off in one box</h3><ul><li><strong>Decision:</strong> how to make image content searchable and showable.</li><li><strong>Options:</strong> the four above.</li><li><strong>Picked, and why:</strong> text inside the owning chunk plus hybrid search. It keeps one index, one authorisation model and one reranker, and it’s explainable when it fails.</li><li><strong>What it costs:</strong> an enrichment step per image, storage and rendering work, a review habit, and a scanning step. Plus the archiving and purge work in Part 4.</li><li><strong>Impact today:</strong> none yet. PreReqs Pro can’t do this today; it’s the plan.</li><li><strong>Breaks when:</strong> vision descriptions are wrong often enough to pollute retrieval; enrichment cost per document gets too high; or the images are diagrams and charts whose meaning text can’t capture.</li><li><strong>Plan B:</strong> multimodal embeddings, or attaching the image and letting a vision model read it at answer time.</li></ul><h3>So did she find the button?</h3><p>In the imagined version, yes. The dialog’s text is in the chunk, the keyword search hits on “137”, the reranker keeps the step, and the answer shows the screenshot right where step 3 says to click. The engineer sees the circled button instead of reading about it.</p><p>What surprised me while planning this was how little of it is about vision models. Most of the work is parsing in order, keeping things together and being careful about who can see what.</p><p>But everything here assumed an image is one screenshot dropped into a step. What happens when the “document” is a 300-page PDF full of tables and charts, where the meaning lives in the layout itself?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aecb78705c95" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Answer Is in the Screenshot. Why Can’t Your RAG Find It? (Part 1: Making Images Searchable)]]></title>
            <link>https://medium.com/@vishalkumar9dec/the-answer-is-in-the-screenshot-why-cant-your-rag-find-it-part-1-making-images-searchable-1515ca4cc4d4?source=rss-f5f1451ef938------2</link>
            <guid isPermaLink="false">https://medium.com/p/1515ca4cc4d4</guid>
            <category><![CDATA[multimodal-ai]]></category>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[vector-search]]></category>
            <dc:creator><![CDATA[Vishal Kumar]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 16:20:15 GMT</pubDate>
            <atom:updated>2026-10-02T07:47:40.744Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How to handle images in the wiki pages and runbooks people actually write. Part 1 covers getting image content into the index: parsing, filtering, enrichment and chunking. Part 2 covers finding and showing it.</em></p><p>What happens when the answer to a question exists in your knowledge base, but only as pixels?</p><p>I wrote the first two posts about the search side of RAG: <a href="https://medium.com/@vishalkumar9dec/rag-confidence-vs-cross-encoder-score-vs-groundedness-b1cb29e71acd">how bi-encoders and rerankers behave</a>, and <a href="https://medium.com/@vishalkumar9dec/why-did-my-rag-system-answer-a-kubernetes-question-with-a-docker-runbook-293f23681479">how writes hold up</a>. All of it assumed that a document is text. Real runbooks aren’t. An engineer is halfway through a procedure, hits a dialog, takes a screenshot, pastes it in, and writes the next step. Nobody adds a caption. Nobody fills in alt text. Nobody writes a metadata block.</p><h3>What does an image-heavy page really look like?</h3><p>So here’s the shape of the real thing:</p><ul><li>It may or may not have a caption.</li><li>Nothing else describes it.</li><li>The step <em>after</em> it often only makes sense if you’ve seen it (“click the highlighted button”).</li></ul><p>That leads to the principle I’d build everything else on: <strong>an image is content at a position in the document, not metadata attached to the document.</strong> Where it sits, and what surrounds it, is part of what it means.</p><p>To make this concrete, I’ll use an illustrative scenario. The engineer and the runbook are invented, and <strong>none of this is a real event</strong>.</p><h3>A story: the answer only a screenshot knew</h3><p>Picture an on-call engineer at an imaginary company. A pod keeps dying. She asks the internal assistant: “What does the ‘exit code 137’ dialog mean and where do I click next?”</p><p>The assistant answers with something plausible about memory limits. It doesn’t mention the dialog, and it doesn’t say which button to click. The real runbook does say it, but only inside a screenshot: the dialog, with the correct button circled.</p><p><strong>The issue.</strong> The information is in the corpus. The system just can’t see it.</p><p><strong><em>When you’d notice.</em></strong><em> Probably not from a metric. Someone says the assistant “knows the theory but not our runbooks”. Or a golden question that a human can answer in ten seconds keeps failing.</em></p><p><strong>How you’d debug.</strong> Three checks, in this order:</p><ol><li>Search the index for the exact phrase from the dialog. No hits.</li><li>Print the stored text of the chunk around that step. It says “Open the pod view. Click Restart.” and nothing about a dialog.</li><li>Ask whether the image is in the index at all. It isn’t. The text extractor dropped it.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*aPdjy6jPUfUAhWZ3NFqdKw.png" /><figcaption>Figure 1: <em>Before: the picture vanishes at extraction. After: its words are inside the chunk.</em></figcaption></figure><p>The fix isn’t a smarter model at answer time. The words in the picture have to reach the index. Everything below is about doing that without paying for images that don’t matter.</p><p><strong><em>Tip:</em></strong><em> if a question is answerable by a human in ten seconds from a screenshot, and your system fails it, that’s a golden question. Save it. We’ll come back to it in Part 2.</em></p><p>Before enriching anything, though, we need to know where the images actually are.</p><h3>How do you keep an image where the author put it?</h3><p>Parse the page into an ordered list of blocks: heading, paragraph, list item, image, table. Not a flat string with pictures removed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*4E2q4XlNq_d0E5pRUHVznQ.png" /><figcaption>Figure 2: <em>Each image keeps its position, its heading path, the block before and after, and any caption.</em></figcaption></figure><p>For every image, record the heading path, the paragraph before, the paragraph after, and any caption or alt text the author did write. Most won’t have either. That’s fine; position and neighbours do most of the work.</p><p>How you get there depends on the source. This is general knowledge, and I haven’t tested each of these against a live system:</p><ul><li><strong>Confluence.</strong> Read the page body (the storage format), where images show up as references to attachments at their position in the flow. Download the attachments through the API at ingest time. Don’t hot-link them: those links need authentication and can expire.</li><li><strong>DOCX.</strong> Images are relationships in the document XML. Walk the body in order and resolve each reference as you pass it.</li><li><strong>PDF.</strong> Use a layout-aware parser, crop figures out of the page, and keep page number and reading order.</li><li><strong>Markdown and HTML.</strong> ![]() and &lt;img&gt; already give you the position.</li></ul><p>One more thing for diagram-heavy wikis. Diagram macros such as <a href="http://draw.io/">draw.io</a> often keep an editable source next to the rendered picture. When that source is available, its labels beat OCR. I say “often”, because it depends on the tool and how the page was authored.</p><h3>Which images should you skip?</h3><p>Every image you enrich costs a vision call and adds text to your index. So filter before spending anything.</p><p>I’d skip:</p><ul><li>Tiny images, like icons and bullets.</li><li>Near-single-colour images, like spacers and dividers.</li><li>Images repeated across many documents. Hash the content, count how many documents contain that hash, and treat anything above a threshold as a logo or banner.</li><li>Images explicitly marked decorative.</li></ul><p>The threshold is a judgement call, and I’d tune it by looking at what gets dropped, not by picking a number in advance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*7iD8euHxZ8GFzuXzNeJrMA.png" /><figcaption>Figure 3: <em>Parse, filter, classify, enrich once, store, chunk, index.</em></figcaption></figure><h3>What should you extract from a screenshot?</h3><p><strong>1. OCR, for exact text.</strong> Error messages, log lines, hostnames, flags. OCR gives you the characters as printed. Store them in their own field. You want to be able to match 137 literally.</p><p><strong>2. A vision-model description, given the context.</strong> This is where position pays off. Don’t send the bare image. Send the heading path and the surrounding paragraphs along with it, and ask for structured output:</p><ul><li>image type (screenshot, diagram, chart, photo)</li><li>a one-sentence summary</li><li>the visible text that matters</li><li>what it shows relative to the step it belongs to</li></ul><p>“A dialog with a red banner” is nearly useless. “The pod detail dialog after the crash, showing exit code 137, with the Restart button highlighted, which is the button step 3 tells you to click” is what a retriever can use.</p><p>A few rules I’d hold myself to:</p><ul><li><strong>Enrich once, and cache.</strong> Key the cache on content hash plus model version. The same image across page versions costs nothing after the first time.</li><li><strong>Never overwrite authored text.</strong> Generated text is stored separately and marked as generated.</li><li><strong>Review a sample.</strong> Vision models make up details, and this text lands in your search index. A confident invented sentence there is worse than no sentence.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*z1Wol5x0njKUHwtJliJCmw.png" /><figcaption>Figure 4: <em>Cheap and exact where possible. A vision call only where it earns its cost.</em></figcaption></figure><p>For terminal and UI screenshots, OCR often carries most of the value, because they’re mostly text. Charts are the awkward case, and Part 2 comes back to them.</p><h3>Where does the image text live in a chunk?</h3><p><strong>Never split between a step and its screenshot.</strong> If chunk 7 says “if the dialog shows an exit code, note it” and chunk 8 starts with the picture of the dialog, then a question about the dialog retrieves chunk 7, which never says what the dialog shows.</p><p>Instead, put a compact image summary inside the owning chunk’s text:</p><pre>2. If the dialog shows an exit code, note it and continue.<br>[Image: pod dialog with a failure banner. Text: &quot;container exited 137&quot;]<br>3. Click Restart and wait for Ready.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*8Lv6etbzOyzwKohpDcRnyA.png" /><figcaption>Figure 5: <em>The step and its screenshot stay together. The full OCR sits in its own field.</em></figcaption></figure><p>Three details matter here.</p><p><strong>Cap the inline text at around 100 tokens.</strong> A cross-encoder reranker typically truncates the query and chunk together at about 512 tokens (check your model’s card, it varies). A 40-line OCR dump would push the real step text out of what the reranker sees. Keep the summary short inline and the full OCR in a separate field.</p><p><strong>Standalone image chunks are the exception.</strong> Only create one when an image has no surrounding text at all.</p><p><strong>Store image ids on the chunk.</strong> When retrieval finds a chunk, it should hand back exactly the images that belong to it. That matters later, for display.</p><h3>Where this leaves us</h3><p>The screenshot’s words are now in the index, inside the chunk that owns them, with the original OCR kept alongside. The pod dialog from the story is no longer invisible.</p><p>But “in the index” isn’t the same as “found”. Screenshots are full of exact strings like 137 and pod-7f9c, and embeddings tend to blur those. And even when the right chunk comes back, the engineer still needs to <em>see</em> the circled button, which means storing the bytes, serving them safely, and treating every picture as untrusted input.</p><p>Part 2 picks up there: hybrid retrieval, storage, rendering with an allow-list, the security side, and how to tell whether any of this helped.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1515ca4cc4d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG — Confidence vs Cross-encoder Score vs Groundedness]]></title>
            <link>https://medium.com/@vishalkumar9dec/rag-confidence-vs-cross-encoder-score-vs-groundedness-b1cb29e71acd?source=rss-f5f1451ef938------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1cb29e71acd</guid>
            <category><![CDATA[vector-search]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[reranking]]></category>
            <dc:creator><![CDATA[Vishal Kumar]]></dc:creator>
            <pubDate>Wed, 30 Sep 2026 15:44:55 GMT</pubDate>
            <atom:updated>2026-09-30T15:44:55.595Z</atom:updated>
            <content:encoded><![CDATA[<p><em>What a RAG “confidence” number actually measures, how it differs from the cross-encoder score, and why the useful one was already computed and thrown away.</em></p><p><strong>What does a number called “confidence” actually promise you?</strong></p><p><em>I’d have said: how sure the system is that the answer is right. That’s what the word means to a human, and it’s what a dashboard tile with that label invites you to believe.</em></p><p>The earlier posts was about <a href="https://medium.com/@vishalkumar9dec/why-did-my-rag-system-answer-a-kubernetes-question-with-a-docker-runbook-293f23681479">how retrieval models behave</a>. This one is smaller and, I think, more uncomfortable. It’s about a number in my own project that carries a friendly name, and that I trusted more than it deserved.</p><h3>What did a wrong answer look like on the dashboard?</h3><p><strong>Illustrative:</strong> picture a dashboard tile for an internal ops assistant. It shows the average “confidence” of answers this week, and a drill-down for each chat turn. An on-call engineer asks: “How do I fix pod restart issue?” The assistant replies with a tidy, numbered procedure, with citations. The tile says 0.588. That looks like a middling-to-decent score, so nobody looks twice.</p><p><strong>Real:</strong> in <a href="https://prereqs.tech/">PreReqs</a> Pro’s dev database, that exact question got a cited answer with a Docker procedure. The runbook it cited was a real SOP, but a Docker one. The question was about Kubernetes. The recorded confidence was 0.588 and the groundedness score was 0.779.</p><p>So the answer was confident, cited, wrong, and it scored well on both numbers I had.</p><p><strong>The issue.</strong><em> A score named “confidence” said nothing useful about whether the answer fit the question.</em></p><p><strong>When you’d notice.</strong> Probably not from the tile. Someone reads the answer, or worse, runs it. A<em> confidently wrong runbook is worse than “I don’t know”, because it gets executed.</em></p><h3>What is “confidence” really computing?</h3><ol><li>Take the similarity values of the chunks kept for the answer.</li><li>Sort them, highest first.</li><li>Average the top 3.</li><li>If there are no chunks, return 0.0.</li></ol><p>That’s it. <em>The “similarity” is a cosine between the query’s embedding and the chunk’s stored embedding, from a bi-encoder (nomic-embed-text in my setup).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*wgxGVR_cTzXL-bMBNmKN2A.png" /><figcaption>Figure 1: <em>Eight kept chunks, sorted by cosine. The mean of the top three is the “confidence”. Values are illustrative apart from the 0.640.</em></figcaption></figure><p>Three properties fall straight out of this.</p><ol><li>It isn’t a probability. Nothing in it estimates “chance this is right”.</li><li>It never looks at the generated text. The answer could say anything at all and the number wouldn’t move.</li><li>The “3” is a starting heuristic. It’s a class constant in the code, waiting for a golden set to calibrate it. It hasn’t been calibrated.</li></ol><p><em>So “how close were the retrieved vectors to the query vector, on average, at the top” is an honest description. “Confidence” is not.</em></p><p>I’d also been carrying a wrong picture of <em>when</em> it runs. Which brings up the next surprise.</p><h3>When does the number get computed, and does it decide anything?</h3><p>After the answer exists. The assistant retrieves, generates, runs the output guardrails, and only then computes confidence, from the chunks it had already kept. It goes into the response and the trace. Nothing reads it to make a decision.</p><p>So the number can’t stop a bad answer. By the time it exists, the answer is written. It’s a label on a finished product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*4RbIn-w-FFjgHrYdsDDEZg.png" /><figcaption>Figure 2: <em>The score that is computed before the answer (step 2) is not the one on the dashboard (step 6).</em></figcaption></figure><p><strong>Look at step 2 in the diagram. There’s a score computed <em>before</em> the answer, by a different model, on all 32 candidates. What was it doing?</strong></p><h3>What does a cross-encoder do differently?</h3><p>This is the part that changed how I think about retrieval scores.</p><p>A bi-encoder embeds the query alone, embeds the chunk alone, and compares the vectors. Because chunk vectors can be computed ahead of time, this scales to a whole corpus. The price: the query and the chunk never “see” each other.</p><p>A cross-encoder gives up that speed. For one candidate, it reads a single input: [CLS] query [SEP] chunk [SEP]. Inside the transformer, every query token can attend to every chunk token, and the other way round. A small head at the end turns the result into one number, a logit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*f8Zr3kIBxoKeMMsVrmtrFg.png" /><figcaption>Figure 3: <em>Top: cosine, two vectors made separately. Bottom: one pass over the pair, one unbounded number out</em></figcaption></figure><p>There’s a fair point against it, too. It only judges “does this chunk look relevant to this question”. It says nothing about whether the model <em>used</em> the chunk.</p><p><em>But here’s the detail that stung. Until recently, my code scored all 32 candidates and then discarded the scores after sorting. The best relevance signal in the whole pipeline was computed, used to reorder a list, and dropped.</em></p><h3>Did the pipeline already know?</h3><p>Yes. That’s the part I didn’t enjoy finding out.</p><p>The reranker had, in effect, looked at all 32 chunks and said “none of these answer this”. Then the pipeline sent the top 8 to the model anyway, because retrieval always sends 8, and the model wrote a plausible answer from the least bad ones.</p><p><strong>How you’d debug this.</strong> Three steps, in this order:</p><ol><li>Open the trace for the bad turn. In my setup that’s the retrieve-context span in Langfuse, or make search from the command line.</li><li>Put the three scores next to each other: confidence, groundedness, and the rerank scores of the chunks.</li><li>Read the raw candidates. The ones the model saw were about restarting Docker containers, not pods.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*GkKWBpNnJa3AX-sXfKk3HA.png" /><figcaption>Figure 4: <em>What each score asks, what it can’t tell you, and the pod question’s value for each.</em></figcaption></figure><p>The three scores disagreed, and only one of them was right. Groundedness was fine because every step in the answer really is in that Docker SOP. It compares answer sentences with cited chunks. The question isn’t an input. Confidence was fine because the top chunks were near the query in embedding space. Only the cross-encoder had read the question and chunk together, and it said no.</p><h3>How big is the gap, really?</h3><p>I ran a preliminary check, and I want to be upfront that it’s preliminary. Thirty-seven questions: 19 answerable from the SOPs, 12 “near-miss” questions about platforms the SOPs don’t cover, and 6 unrelated. I wrote the questions and the labels myself. They haven’t been reviewed by anyone else. No LLM was called; this is retrieval scores only.</p><p>For the recorded chat history, confidence looked like this: out-of-scope turns landed at 0.57 to 0.60. In-corpus turns at 0.69 to 0.73. That’s a gap of about 0.1, from a handful of turns.</p><p>The best cross-encoder score per question looked like this:</p><ul><li>In-corpus: about +2.6 to +8.5, with one outlier at −2.80.</li><li>Near-miss: about −3.1 to −11.3, with one outlier at +2.19.</li><li>Unrelated: about −10.9 to −11.2.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*lA6Nhck7ZK0RyRdO3GAGNQ.png" /><figcaption>Figure 5: <em>Confidence moves about 0.1 between good and bad. The rerank score moves over 10 points and changes sign. Different question sets, so treat it as a shape, not a benchmark.</em></figcaption></figure><p>Given all that, what should you actually do with each number?</p><h3>What’s the fix?</h3><p>Three changes, each small.</p><p><strong>1. Use the rerank score as the relevance gate.</strong> Drop chunks below a floor, and if nothing is left, don’t ask the model to improvise. <em>Say the topic isn’t covered, and flag the turn.</em></p><p><strong>2. Keep confidence, but call it what it is, and only as telemetry.</strong> It’s cheap, it’s already computed, and a slow drift downward in “<em>how close were my top chunks”</em> is a real signal about the corpus. It just isn’t a verdict on any single answer.</p><p><strong>3. Add an answer-relevance check after generation.</strong> Groundedness tells you the answer sticks to its sources. It doesn’t tell you the answer addresses the question. <em>That’s a separate question, and it needs its own check, because a well-grounded wrong-topic answer is exactly what fooled me.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*ej6yCf1yhebWc6vqr1xreA.png" /><figcaption>Figure 6: <em>Gate on the score that reads question and chunk together. Log the rest.</em></figcaption></figure><h3>What would I tell someone building this?</h3><p><em>A few tips, most of them earned by getting it wrong.</em></p><p><strong>Name scores after what they compute.</strong> “Confidence” invites people to read a probability into it. “Top-3 mean cosine” is dull and accurate.</p><p><strong>Ask of every score: what is it computed from, and does it see the question?</strong> If the question isn’t an input, it can’t tell you whether the answer fits it.</p><p><strong>Don’t throw scores away after sorting.</strong> Keep them in your traces. Storage is cheap, and a bad turn is far easier to debug with 32 scores in hand than with 8 chunks.</p><p><strong>Don’t compare raw logits across queries or models.</strong> A threshold belongs to one reranker model and one corpus. Change either, and re-measure.</p><p><strong>Prefer a failure you can see.</strong> A wrongly declined question is annoying, and someone will complain. A wrongly answered runbook question is silent until somebody runs it.</p><p><strong>Expect to keep the old number around.</strong> Confidence still has a job. It just isn’t the job its name suggests.</p><h3>Did the number lie?</h3><p><em>Not really. It computed exactly what the code says it computes. I was the one who read “confidence” and assumed more.</em></p><p>What I keep coming back to is how ordinary the mistake was. Nothing crashed. The illustrative tile looked fine. The best signal was sitting one function earlier, already computed, and I’d sorted by it and thrown it away.</p><p>But a floor of 0 came from 37 questions I wrote myself, and one confident-looking chart is exactly how I got here. So if the rerank score is the right thing to gate on, how do you pick the threshold without fooling yourself: how big does the golden set need to be, who labels it, and how do you weigh a false decline against a wrong answer?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1cb29e71acd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why did my RAG system answer a Kubernetes question with a Docker runbook?]]></title>
            <link>https://medium.com/@vishalkumar9dec/why-did-my-rag-system-answer-a-kubernetes-question-with-a-docker-runbook-293f23681479?source=rss-f5f1451ef938------2</link>
            <guid isPermaLink="false">https://medium.com/p/293f23681479</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[vector-search]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Vishal Kumar]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 18:20:09 GMT</pubDate>
            <atom:updated>2026-09-29T18:20:09.880Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A debugging story about bi-encoders, cross-encoders, HNSW, and the score my pipeline threw away.</em></p><p><strong>Why would a system with no Kubernetes content in it give me a confident, cited answer about Kubernetes?</strong></p><p>That’s what happened. I’m building <a href="https://prereqs.tech/">PreReqs</a> <em>(PreReqs Pro in Beta)</em>, a multi-agent RAG system over a corpus of operations runbooks (SOPs). I typed: <strong>“How do I fix pod restart issue?”</strong></p><p>The corpus is Docker-flavoured. There is no Kubernetes page anywhere in it. OpsAgent answered anyway, with citations, in a calm and helpful tone. The answer was a Docker out-of-memory fix.</p><h3>The answer looked fine. So how do you even catch this?</h3><p>That’s the nasty part. Nothing crashed, and the citation pointed at a real document. The only signal was that I happened to know the corpus had no Kubernetes content.</p><p><strong><em>A citation tells you where the answer came from. It doesn’t tell you whether that source was a good match for the question.</em></strong></p><p>So I stopped reading the answer and started reading the retrieval. Which chunks were handed to the model, and how did they score?</p><p>That meant understanding the two models doing the scoring, and I’d been using them without really knowing how they differed.</p><p><strong><em>Tip:</em></strong><em> when a RAG answer looks wrong, debug retrieval first. The LLM is usually just doing its best with what it was handed.</em></p><h3>Two models, one job: what’s actually different?</h3><p>Both models take a query and a chunk and say how relevant the chunk is. The definitions sound identical, which is why I mixed them up for a while.</p><p>The whole difference is <em>when the query and the chunk meet</em>.</p><p>A <strong>bi-encoder</strong> reads them in two separate passes. Each becomes a vector (768 numbers, in my case), and they only meet afterwards, as a cosine between two lists of numbers. The chunk’s vector doesn’t depend on the query, so it’s computed once at ingestion and stored.</p><p>A <strong>cross-encoder</strong> reads them together. The input is one sequence, “[CLS] query [SEP] chunk [SEP]”, and attention lets every query token look at every chunk token and back. What comes out isn’t a vector. It’s one number, a relevance score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-TBVpq4KQy904mqWCqY1pw.png" /><figcaption><em>Figure 1. The bi-encoder compares two finished summaries; the cross-encoder reads both side by side.</em></figcaption></figure><p><strong>So which one do you use?</strong> Both. Which raises the obvious question of how they fit together.</p><h3>Wait, aren’t these both just Transformers?</h3><p>This tripped me up. I kept filing “<strong>Transformer</strong>”, “<strong>encoder</strong>”, “<strong>bi-encoder</strong>” and “<strong>LLM</strong>” under one mental heading.</p><p><strong><em>The Transformer is the architecture. Bi-encoder and cross-encoder are two ways of using an encoder (BERT-style), not parts of it. The models that generate text, GPT, Claude, Llama, Qwen, are decoder-only.</em></strong></p><p>In my pipeline, that gives three jobs: nomic-embed-text as the bi-encoder, ms-marco-MiniLM-L-6-v2 as the cross-encoder, and an LLM that writes the final answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DM1Koxm1B_3-yZbPMmxDfg.png" /><figcaption><em>Figure 2. One architecture, used three ways in this pipeline.</em></figcaption></figure><p>The word “encoder” is used in two senses, and that’s most of the confusion. One is the encoder half of the original 2017 Transformer, the half that reads. The other is the “bi-” and “cross-” prefix, which describes how you wire an encoder up to compare two texts.</p><p><strong>Tip:</strong> the useful mental model is “what does the model output?”. A vector means bi-encoder. A single relevance score means cross-encoder. Text means decoder.</p><h3>Why not just use the smarter model on everything?</h3><p>Because of the arithmetic. With a 10,000-chunk corpus, a cross-encoder over everything needs 10,000 forward passes per question. The bi-encoder needs one embedding call and an index lookup.</p><p>So the pipeline runs in two stages. A cheap wide net first, then a careful look at whatever it caught.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nGdQectCT8Utuc6H6rMY1g.png" /><figcaption><em>Figure 3. Stage 1 pulls 32 candidates from the whole corpus; stage 2 reranks them and keeps 8.</em></figcaption></figure><p>In the code, that’s top_k = 8 and fetch_k = 4 x top_k = 32. The bi-encoder plus vector index fetches 32. The cross-encoder scores those 32 and keeps 8 for the LLM.</p><p>Why 32 instead of 8? Because the reranker can only reorder what stage 1 hands it. If the best chunk sits at rank 12 by cosine and I only fetched 8, it never gets a chance.</p><p><strong><em>I should be honest about the 4. It’s a starting point I picked, not a number I measured. A bigger pool improves the odds of catching the right chunk but grows rerank time linearly.</em></strong></p><h3>How does a vector index find neighbours without checking everything?</h3><p>The exact way is brute force: compute the cosine against every chunk, sort, take the top. It’s exact, but the cost grows linearly with the corpus.</p><p>Approximate nearest neighbour (ANN) search trades a little accuracy for a lot of speed. “Approximate” has a precise meaning: recall can be below 100%, so it may occasionally miss a true top result.</p><p>My index is HNSW (Hierarchical Navigable Small World), created in pgvector with CREATE INDEX ... USING hnsw (embedding vector_cosine_ops).</p><p><strong>Two ideas. First, a proximity graph: each vector links to a few near neighbours, and you search by hopping to whichever neighbour is closer to the query. Second, layers. The top layer has very few nodes and long links, the bottom layer has all of them.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WBpxFKCXZp4Fs--2HFOx_A.png" /><figcaption><em>Figure 4. Enter at the top, take long hops, drop a layer, refine. Cost grows roughly with log(N).</em></figcaption></figure><p>A few knobs exist: <em>m (links per node),</em> ef_construction (how wide the search is while building) and <em>ef_search (how many candidates a query tracks, 40 by default)</em>. My migration uses pgvector&#39;s defaults, which is fine at this corpus size. The query itself is ORDER BY embedding &lt;=&gt; query LIMIT fetch_k, and the code turns distance into similarity with 1 - distance.</p><p><strong>The main alternative is IVF.</strong> It clusters the vectors up front and, at query time, searches only the few clusters nearest the query (you choose how many, called nprobe). It uses less memory and needs a training step to find the clusters. Recall depends on nprobe. <em>I haven’t used it here, so I won’t make claims about its numbers.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EeC5GfU31QT9LI7Y4Zhtjw.png" /><figcaption><em>Figure 5. IVF searches only the neighbourhoods closest to the query.</em></figcaption></figure><p><strong>One gotcha worth knowing.</strong> HNSW collects roughly ef_search candidates first and applies the WHERE clause afterwards. My access-control filter lives in that WHERE. If a user can read only a small slice of the corpus, fewer than 32 rows can come back. That&#39;s a recall problem, not a leak, but it surprises you the first time.</p><p>That’s the machinery. Back to the pod question, and the scores.</p><h3>What did the two models say about the pod question?</h3><p>I ran the query through the real retrieval path and looked at every candidate. This is where it got interesting.</p><p>All 32 candidates scored <strong>negative</strong> on the cross-encoder. The best was -3.73. A comparable in-corpus question, about a container that keeps restarting with OOMKilled, scored +7.70.</p><p>The reranker was effectively saying “nothing here is relevant”. The pipeline discarded the scores and sent the top 8 to the LLM anyway.</p><p>The two models also disagreed. SOP 0017’s Description chunk was <strong>first</strong> by cosine (0.640) and only <strong>fifth</strong> after reranking (-4.68). The cross-encoder read the pair and wasn’t fooled by “restart” and “container” sounding close. The bi-encoder, working from a compressed summary, was.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2uUGVYsNsJh0Z2QTjTsNxQ.png" /><figcaption><em>Figure 6. Cosine barely moves between good and bad matches. The cross-encoder score crosses zero.</em></figcaption></figure><p>Here’s the part that made me stop. Cosine ranges overlap: in-corpus questions landed between 0.63 and 0.87, near-misses between 0.48 and 0.72. A threshold on cosine would cut good answers or let bad ones in.</p><p>The rerank scores, by contrast, separated cleanly, with a gap somewhere between about -2.8 and +2.2.</p><p><strong><em>Tip:</em></strong><em> a ranking model is also a measuring instrument. If you only use it to sort, you’re throwing away its most useful output.</em></p><h3>So what’s the fix, and does it work?</h3><p>A floor on the rerank score. If nothing scores above the floor, the system says it doesn’t have an answer instead of improvising.</p><p>I tried a floor of 0 on 37 preliminary questions. Of the answerable ones, 18 of 19 still answered. Of the near-misses, 11 of 12 abstained, and all 6 unrelated questions abstained.</p><p><strong><em>I want to be careful here. That’s a small, preliminary set, not a calibrated threshold. The next step is a properly labelled evaluation before I lock a number in.</em></strong></p><p>Also, rerank scores are not probabilities. The model was trained on web search data, and “+7 for this query” doesn’t mean the same as “+7 for that one”. They compare cleanly within a single query, which is exactly how a floor uses them.</p><p>The cost of all this is modest: reranking ran about 0.6 seconds per query on CPU (635 ms measured). I’ll take that trade for a system that can say “I don’t know”.</p><p><strong>Tips, in one place:</strong></p><ul><li>Debug retrieval before you blame the LLM.</li><li>Use both models: the bi-encoder for reach, the cross-encoder for precision.</li><li>Keep the rerank score. Don’t just sort by it.</li><li>Choose fetch_k by measuring recall, not by guessing (I’m still guessing at 4).</li><li>Check what your vector index does with filters.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=293f23681479" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SMTP: The Server Said “Sent.” The Inbox Disagreed]]></title>
            <link>https://medium.com/@vishalkumar9dec/smtp-the-server-said-sent-the-inbox-disagreed-e5dcd83a0a7b?source=rss-f5f1451ef938------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5dcd83a0a7b</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[dockerfiles]]></category>
            <category><![CDATA[smtp-server]]></category>
            <category><![CDATA[honeypot]]></category>
            <category><![CDATA[docker-compose]]></category>
            <dc:creator><![CDATA[Vishal Kumar]]></dc:creator>
            <pubDate>Sat, 29 Aug 2026 16:55:51 GMT</pubDate>
            <atom:updated>2026-08-29T16:55:51.790Z</atom:updated>
            <content:encoded><![CDATA[<p>A contact form on my site, <a href="https://prereqs.tech/">PreReqs</a>, said “Thanks — your enquiry is on its way.” every single time. The server answered every request with a clean 200 OK. And nothing ever showed up in the inbox.</p><p>This is the story of how that happened, why “configure SMTP” is a smaller job than it sounds, and how a security feature I added on purpose ended up eating real customer enquiries.</p><h3>Sending email isn’t running a server</h3><p>The first mental block most people hit: “I need to set up an email server.” You don’t. You need to talk to one that already exists.</p><p>Your app doesn’t become a mail server just because it sends mail — it becomes one more <em>client</em> of whatever server already handles your mailbox (Hostinger, Google Workspace, Outlook, anything). Same role your phone’s mail app plays. You’re not hosting anything; you’re placing a call.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RgYEDAahoDXlJCkwSOKrRg.png" /><figcaption>Comparison diagram: running your own mail server versus configuring your app as an SMTP client of a provider’s already-running server</figcaption></figure><p>That’s the entire job: three settings.</p><pre>SMTP_HOST=smtp.yourprovider.com<br>SMTP_PORT=465 # or 587<br>SMTP_USER=you@yourdomain.com<br>SMTP_PASSWORD=your-mailbox-password</pre><p>Port 465 means the connection is encrypted (TLS) from the very first byte. Port 587 means it starts in plain text and upgrades mid-conversation (STARTTLS). Both end up encrypted; they just get there differently. If 465 doesn’t cooperate on your network, 587 usually will.</p><p>Whatever library you use — nodemailer in my case — reads those four values and does the actual talking: connect, authenticate, hand off the message, get back a 250 OK, queued. That confirmation comes from the provider&#39;s server, not your code. Your code&#39;s only job was making the call correctly.</p><h3>Docker doesn’t change any of this</h3><p>The one thing that trips people up in a containerized app: the image itself should never contain those four values.</p><pre>environment:<br>  SMTP_HOST: ${SMTP_HOST:?}<br>  SMTP_PORT: ${SMTP_PORT:?}<br>  SMTP_USER: ${SMTP_USER:?}<br>  SMTP_PASSWORD: ${SMTP_PASSWORD:?}</pre><p>The :? means &quot;refuse to start without this.&quot; That&#39;s deliberate — if any deploy skips wiring up these values, you want the container to fail loudly at boot, not start up fine and quietly discard every enquiry forever. A build that starts cleanly and a build that actually sends mail are two different claims, and only one of them is checked by &quot;docker ps&quot; showing healthy.</p><p>Same image, any environment: local dev gets no credentials and a harmless fake sender; staging and production get the real ones, injected at runtime, never baked into a layer.</p><h3>Then it broke, and the response lied about it</h3><p>Config was in place. I clicked “send” on the actual form. Got the success message. Checked the inbox five minutes later. Nothing.</p><p>No error. No 500. No angry log line. The server had, as far as I could tell, done everything right and produced nothing.</p><p>The form had a honeypot — a hidden field named “company” that a real visitor never sees, positioned off-screen with CSS. Spam bots fill in every field they find; a filled honeypot means “this wasn’t a human,” so the server quietly drops the message and <em>still returns success</em> — telling a bot it worked is exactly how you avoid teaching it to adapt.</p><p>Except my hidden field wasn’t hidden enough. I’d used a classic “visually-hidden” CSS trick — clipped to one pixel, moved off-screen — instead of display: none. That trick keeps the element technically present and rendered, and Chrome&#39;s address autofill doesn&#39;t check whether something&#39;s visible to a human before filling it. It saw a field named &quot;company,&quot; matched it against my own saved address data, and filled it in behind my back.</p><p>I had, without realizing it, built a bot trap that caught actual me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pCnHex79DI2VgJYjolv67w.png" /><figcaption>Flowchart: a hidden honeypot field and a genuine submission both return 200 OK from the server, making success and silent failure indistinguishable from outside</figcaption></figure><p>The fix was one CSS property:</p><pre>/* before — still rendered, autofill can reach it */<br>.honeypot { position: absolute; clip-path: inset(50%); }<br><br>/* after — genuinely removed from the page */<br>.honeypot { display: none; }</pre><p>display: none takes the element out of the render tree entirely. Real browsers skip autofilling anything that isn&#39;t actually rendered. A basic spam bot that just grabs every &lt;input&gt; from the raw HTML — without evaluating any CSS at all — still walks straight into it. One property line, and the trap goes back to catching only what it was built for.</p><h3>The other ways to send mail from an app</h3><p>Raw SMTP isn’t the only option, and it isn’t always the right one. Roughly, in order of “less magic, more control” to “more magic, less setup”:</p><ul><li>mailto: links — zero backend, opens the visitor&#39;s own mail client. No server involved, but it hands the send action to the visitor, and it breaks entirely for anyone on a webmail-only device.</li><li>Raw SMTP (nodemailer, smtplib, etc.) — what this post covers. Works with any mail host, no vendor lock-in, but you own retries, deliverability, and reading provider-specific error codes yourself.</li><li>Provider SDKs (SES, SendGrid, Postmark) — a proper HTTP API instead of the SMTP protocol: webhooks for bounces/opens, better deliverability tooling, dashboards. Trade-off: you’re now writing against one vendor’s API shape, and pulling their SDK into your codebase.</li><li>Transactional email platforms (Mailgun, Brevo, Resend) — same idea as above, generally friendlier for smaller apps, same lock-in trade-off.</li><li>Third-party form services (Formspree and similar) — fastest to wire up, but visitor data now passes through someone else’s infrastructure before it reaches you — worth a real look if you handle anything privacy-sensitive.</li></ul><p>None of these are “correct.” SMTP earns its place when portability matters more than convenience — pointing at a different mail host later is a config change, not a rewrite.</p><p>If you’re building the kind of production systems this post is about — the parts that don’t show up in a demo but absolutely show up in an incident — I write and teach this stuff at <a href="https://prereqs.tech/">PreReqs</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5dcd83a0a7b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>