<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Saibal Ghosh on Medium]]></title>
        <description><![CDATA[Stories by Saibal Ghosh on Medium]]></description>
        <link>https://medium.com/@saibalghosh?source=rss-40f5089506fc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*u5NDWY5D63lams9P.</url>
            <title>Stories by Saibal Ghosh on Medium</title>
            <link>https://medium.com/@saibalghosh?source=rss-40f5089506fc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 13:47:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@saibalghosh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your AI Prompt Has a Price Tag. Here’s How to Read It — For Free.]]></title>
            <link>https://medium.com/@saibalghosh/your-ai-prompt-has-a-price-tag-heres-how-to-read-it-for-free-eac152c6f1d4?source=rss-40f5089506fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/eac152c6f1d4</guid>
            <dc:creator><![CDATA[Saibal Ghosh]]></dc:creator>
            <pubDate>Tue, 02 Jun 2026 06:28:51 GMT</pubDate>
            <atom:updated>2026-06-02T06:28:51.365Z</atom:updated>
            <content:encoded><![CDATA[<p>One thing I learned very early in my career as an Oracle DBA is that every system has a meter somewhere. Sometimes that meter is CPU utilization. Sometimes it is memory consumption, disk I/O, network traffic, or storage growth. Most of the time, nobody thinks about the meter while the system is running. People focus on features, functionality, and user experience. Then one day a performance report arrives, a cloud bill lands in someone’s inbox, or a capacity dashboard turns red, and suddenly everyone discovers that the meter has been running quietly in the background all along.</p><p>Generative AI is no different.</p><p>The meter simply has a different name: tokens.</p><p>The challenge is that most people never look at it. Teams spend weeks comparing models, tuning prompts, evaluating RAG architectures, and discussing hallucinations. Yet surprisingly few pay attention to the thing that is directly tied to cost. As a result, many applications end up spending far more money than necessary while producing exactly the same answers</p><p><strong>First, the only idea you really need</strong></p><p>When you interact with a Large Language Model, you are not paying for the question. You are paying for tokens.</p><p>A token is simply a chunk of text. Sometimes it is a complete word. Sometimes it is part of a word. Sometimes it is punctuation. You do not need to understand the underlying tokenization algorithms to understand the economics. For practical purposes, you can think of a token as the basic unit of currency in the GenAI world.</p><p>If you come from a database background, think of tokens the way you think about logical reads. A single logical read is inexpensive. Nobody worries about one. But millions of unnecessary logical reads executed repeatedly across a busy system eventually become a performance and cost problem. Tokens behave in much the same way. A few extra tokens seem insignificant.</p><p>There is another idea that is even more important, because it explains why prompt costs often surprise people.</p><p><strong>The model has no memory. None.</strong></p><p>Every single time you send it something, it’s meeting you for the first time. It forgot you the instant the last answer finished. So to keep a conversation going, <em>you</em> have to hand it everything again — your instructions, the documents, the entire chat history — on every turn. Imagine explaining your problem to a brilliant consultant who forgets you completely the moment you leave the room. Next visit, you re-explain from scratch. And they charge you by the word, every time you do.<br> <em>This is the part that is often the gray area: People may know academically that the model has no memory, but how that impacts you is something which may not be crystal clear to everyone. Between one message and the next, the model forgets you completely — and it isn’t even the thing you’re typing into. The window on your screen belongs to an app, and the app is the one keeping score. Every time you hit send, it quietly copies the entire conversation so far — your instructions, every earlier question, every earlier answer — staples your new line to the bottom, and ships the whole bundle to the model as if for the first time. The model reads all of it, replies, and forgets again. So, a “short follow-up” isn’t short. You typed say, eight words; the model re-read everything — and you paid for every token.</em></p><p>Keep that picture in your head. We’re going to watch it cost real money in three different scenarios.</p><p><strong>The free gadget: the Bedrock Tokenizer</strong></p><p>Inside Amazon Bedrock, AWS provides a utility called the <em>Tokenizer</em>. You can find it under the Test section of the Bedrock console. What makes this tool interesting is that it does not execute a model. It simply counts tokens.</p><p>Think of it as checking the taxi meter before starting the journey.</p><p>No model execution.</p><p>No inference charges.</p><p>No surprise bill.</p><p>You paste your text into the Tokenizer, select the model you intend to use, and click Count Tokens. The tool tells you exactly how many input tokens would be sent if you actually executed that prompt. Because the model never runs, the exercise is completely free. Yet the token count you receive is the same count that would ultimately contribute to your bill.</p><p>That gives us a safe environment in which to experiment.</p><p>So let us perform a few experiments and see how seemingly innocent prompt-design decisions can have a surprisingly large impact on cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*uPwDlQ7RbnFvJPO6KWHwZg.png" /></figure><p>Let’s put it to work.</p><p><strong>Experiment A: The same answer, two completely different prices</strong></p><p>Picture a customer-support bot. A customer asks one simple thing:</p><p><em>“What’s your refund window?”</em></p><p>There are two ways to set this up. Both give the customer the exact same correct answer. But they cost wildly different amounts.</p><p><strong>The lean way</strong></p><p>You give the model only what it needs:</p><p><em>You are a support assistant. Use this policy to answer.<br> Policy: Our refund window is 30 days from the delivery date.<br> <br> Question: What is the refund window?</em></p><p>Paste that into the Tokenizer and count it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*cKbxAjy3eefHCRHD1rO1wA.png" /></figure><h4>The lazy way</h4><p>Now the way it actually happens in the real world. Someone, somewhere, decided it was “safer” to just hand the model the <em>entire</em> returns-and-shipping policy — all eight pages of it — and let it find the answer. Same question. Same answer comes back. But look at what you just handed over:</p><p><em>You are a support assistant. Use this policy to answer.<br> Policy: [the full 8-page returns, refunds, exchanges, and<br> shipping policy — see the Lab Kit at the end of this article<br> for the exact text to paste]<br> <br> Question: What is the refund window?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*gXtF01S8WeeCVKsOeXgVWQ.png" /></figure><p>Count <em>that</em></p><p>This is the librarian moment. Asking “what’s your refund window?” while dumping the whole policy binder is like walking up to a librarian, asking what time they close, and handing them all 900 pages of the library’s rulebook “just in case.” They’ll find your answer. But you just paid them to hold all 900 pages — for a one-line question.</p><p><strong>Same answer. Same customer. 25–30 times the input cost.</strong> And nobody noticed, because the answer looked fine.</p><p><strong>Now watch it repeat — the part nobody tells you</strong></p><p>Remember the consultant with no memory? Here’s why that one fact turns a small mistake into a big bill.</p><p>That bloated prompt isn’t paid for once. It’s paid for <strong>every time the bot answers a question.</strong> A thousand customers ask things today? You handed over those eight pages a thousand times. Tomorrow, another thousand. The waste doesn’t add up — it <em>multiplies</em></p><p>I will turn this into dollars in a minute. First, two more places this exact pattern hides.</p><p><strong>Experiment B: The fat system prompt that bills you forever</strong></p><p>Every AI app has a <strong>system prompt</strong> — the standing instructions that tell the model who it is and how to behave. The customer never sees it. And because it’s invisible, it tends to grow fat. People keep adding “just one more rule,” and nobody ever trims it.</p><p>Here’s the trap: <strong>the system prompt is re-sent on every single call.</strong> It’s the most-repeated text in your entire application. Every wasted word in it gets paid for again, and again, and again.</p><p><strong>A tight system prompt</strong></p><p><em>You are a concise support assistant. Answer in one sentence.<br> If unsure, say so.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*ayUmTGp15uKmKd06EeNdcg.png" /></figure><p>Count it</p><p>A bloated one (the kind that actually ships)</p><p><em>You are an extremely helpful, friendly, professional, and<br> courteous customer support assistant working on behalf of the<br> company. Always greet the customer warmly. Always be polite,<br> empathetic, and understanding. Remember to never be rude. Try<br> your best to fully understand the customer’s needs before<br> answering. Always provide accurate information. If you are not<br> sure about something, please do your best and try to be helpful<br> anyway. Remember the company values: integrity, excellence,<br> customer-first thinking, and innovation. Always thank the<br> customer at the end. Never share internal information. Be<br> concise but also thorough and complete in your answers…</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*HLSc5pko6AHqc0w1_iB5qw.png" /></figure><p>Count <em>that</em></p><p>Both prompts produce a perfectly good assistant. But one of them quietly taxes you on every reply, forever. A fat system prompt is the slow leak under the sink — small, constant, and you only notice it on the bill.</p><p><strong>Experiment C: The conversation that gets pricier every turn</strong></p><p>Last one, and it’s the sneakiest. Because the model has no memory, a chat conversation works by re-sending the <strong>entire history</strong> every turn. Turn one is cheap. By turn five, you’re re-handing over everything that came before — five times the paperwork for the same single new question.</p><p>Let’s see it.</p><h4>Turn 1 — just the opening</h4><p><em>You are a travel assistant.<br>User: I want to plan a 3-day trip to Goa.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*VS2LOO8HVQcsqAU9TMgECg.png" /></figure><p>Count it.</p><p><strong>Turn 5 — the whole conversation so far</strong></p><p><em>You are a travel assistant.<br> User: I want to plan a 3-day trip to Goa.<br> Assistant: Great! Here’s a 3-day Goa itinerary… [long reply]<br> User: Can you add some beach options?<br> Assistant: Sure, here are the best beaches… [long reply]<br> User: What about budget hotels?<br> Assistant: Here are some budget-friendly stays… [long reply]<br> User: And how do I get around?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*WalDUerrg-7hHe0qen8oCg.png" /></figure><p>Count <em>that</em></p><p>Notice what happened. The customer’s actual new question — “how do I get around?” — is tiny. But the <em>bill</em> for that turn is huge, because you paid to re-send everything that led up to it. A long conversation isn’t a little more expensive than a short one. It’s expensive in a way that compounds with every turn.</p><p><em>(There’s a fix for this called “prompt caching,” where you tell the model “this part doesn’t change, don’t re-charge me full price for it.” It’s a fantastic lever — but it’s a live feature, not something the free Tokenizer can show. Let us park it. It deserves its own article.)</em></p><p><strong>Turning tokens into money — napkin math</strong></p><p>Here’s the arithmetic, and it’s the kind you can do on a napkin:</p><p>Cost = number of tokens × price per token</p><p>On Claude Sonnet, input tokens cost <strong>$3 per million.</strong> That’s it. Let’s run the refund-bot numbers from Experiment A.</p><p>Say the bloated prompt was about <strong>665 tokens</strong> and the lean one about <strong>44</strong>. The waste is roughly <strong>621 tokens per question</strong></p><p>• One question: 621 × ($3 ÷ 1,000,000) = <strong>about $0.0018.</strong> Less than a fifth of a cent. Who cares, right?</p><p>• Now do it <strong>5,000 times a day</strong> (a small, ordinary product): 621 × 5,000 = 3,105,000 wasted tokens a day × $3/million = <strong>about $9.35 a day.</strong></p><p>• That’s <strong>roughly ~$280 a month</strong> — burned — for <em>zero</em> extra value. Same answers. Same customers. Just paper you paid to hand over and over.</p><p>That’s the whole trick of token costs. One call looks like nothing. The model’s no-memory habit means you do that “nothing” thousands of times a day, and “nothing” turns into a salary.</p><h3>Quick rate reference (Disclaimer: rates are subject to change)</h3><p><strong>Model</strong></p><p><strong>Input ($/million tokens)</strong></p><p><em>Claude Sonnet ~$3</em></p><p><em>Claude Haiku ~$0.80</em></p><p>Output costs more — usually 3 to 5 times the input rate — but output isn’t something we measured here, and that’s the honest line to draw. More on that in the summary.</p><p><strong>So, what did we actually do here? (Read this and you’re done.)</strong></p><p>Let me close the loop plainly, so you walk away knowing exactly what was proven and what wasn’t — no handwaving.</p><p><strong>The tool.</strong> We used one free gadget in the Amazon Bedrock console: the <strong>Tokenizer</strong>. It counts tokens without running the model, so it costs nothing, and the count it returns is the exact number you’d be billed for.</p><p><strong>What we measured.</strong> We measured <strong>input</strong> cost — the size of what you <em>send</em> the model. We did it three ways, and each one showed the same lesson from a different angle:</p><p>1. <strong>Experiment A — Context dumping.</strong> Handing the model a whole document to answer a one-line question costs 25–30× more than handing it just the relevant sentence. Same answer, far bigger bill.</p><p>2. <strong>Experiment B — Fat system prompts.</strong> The invisible standing instructions get re-sent on every call, so every wasted word in them is paid for again and again, forever.</p><p>3. <strong>Experiment C — Long conversations.</strong> Because the model has no memory, every chat turn re-sends the entire history, so cost compounds turn over turn.</p><p><strong>The one idea underneath all three.</strong> The model forgets you between calls, so you re-hand it everything every time — which means <em>input is paid repeatedly, not once.</em> A small per-call waste becomes a real monthly number once you multiply by how often you call.</p><p><strong>The math.</strong> Cost = tokens × price per token. On Claude Sonnet, that’s $3 per million input tokens. Tiny per call; a salary at scale.</p><p><strong>What we did NOT do — and why that’s fine.</strong> We did not measure <strong>output</strong> tokens (what the model <em>generates</em>), because the free Tokenizer can’t — output requires actually running the model, which costs money. So you won’t find claims here about “output runaway” or “retry costs.” Those are real, they’re often even bigger, and they’re the natural Part 2 of this story — a live experiment for another day. Everything in <em>this</em> article was proven for $<strong>0</strong>, which is rather the point: you can see, today, for free, that bad prompts are costing you money — before you spend a single rupee finding out.</p><p>If you take one thing away: <strong>before you ship a prompt, paste it into the Tokenizer and read the meter.</strong> It’s free, it’s honest, and it’ll change how you write prompts forever.</p><p><strong>So What Did We Actually Learn?</strong></p><p>If you’ve read this far, I hope the biggest takeaway isn’t the token counts themselves. Token prices will change. Models will change. Context windows will get larger. New optimization techniques will appear.</p><p>The underlying principle, however, is unlikely to change.</p><p>Every system has a meter.</p><p>In my Oracle DBA days, we spent countless hours tuning SQL queries. Two queries could return exactly the same answer, yet one would consume dramatically more CPU, memory, and I/O than the other. The user saw the same result. The system saw two very different costs.</p><p>Prompt engineering is often the same story wearing different clothes.</p><p>Two prompts can produce the same answer. One simply requires the model to do far more work than necessary.</p><p>That is ultimately what the Bedrock Tokenizer helps us see. It allows us to look behind the curtain and understand the size of what we are sending before we spend money sending it. More importantly, it helps us develop a habit that every experienced engineer eventually learns: measure first, optimize second, and assume nothing.</p><p>The three experiments in this article all pointed to the same underlying truth. Whether we are dumping entire documents into context, building bloated system prompts, or carrying around long conversation histories, the model must process everything we send it. Because the model has no memory between calls, that cost is often paid repeatedly rather than once.</p><p>The surprising part is not that tokens cost money. The surprising part is how quickly small inefficiencies become meaningful at scale. A few hundred unnecessary tokens seem harmless. A few hundred unnecessary tokens multiplied across thousands or millions of requests become a line item on someone’s budget.</p><p>The good news is that you do not need a sophisticated observability platform, a consulting engagement, or a complex cost-analysis exercise to begin improving. You can start with a free tool and five minutes of curiosity.</p><p>Paste your prompt into the Tokenizer.</p><p>Read the meter.</p><p>Ask yourself whether every piece of context truly belongs there.</p><p>You may be surprised by what you find.</p><p>In the Oracle era, we learned to tune queries before they reached production. In the cloud era, we learned to optimize infrastructure before the bill arrived. In the GenAI era, we should learn to do the same with prompts.</p><p>Because the most expensive token is usually the one you never needed to send.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eac152c6f1d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You Deployed an LLM. You Have No Idea What It’s Actually Doing.]]></title>
            <link>https://medium.com/@saibalghosh/you-deployed-an-llm-you-have-no-idea-what-its-actually-doing-8c740723cbcc?source=rss-40f5089506fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c740723cbcc</guid>
            <dc:creator><![CDATA[Saibal Ghosh]]></dc:creator>
            <pubDate>Thu, 28 May 2026 04:33:08 GMT</pubDate>
            <atom:updated>2026-05-28T04:33:08.901Z</atom:updated>
            <content:encoded><![CDATA[<p>I have spent the last two years embedded in enterprise AI deployments — building RAG pipelines, agentic systems, and LLM-powered workflows on AWS Bedrock. What I keep seeing, consistently, is teams that can tell you their model’s MMLU score to three decimal places but cannot answer a single question about what is happening in production. They don’t know whether responses are grounded in source documents or hallucinated. They don’t know how often the model refuses to answer a legitimate question. They have no idea whether their agentic tool calls are succeeding or silently failing mid-chain.</p><p>This is not an AI problem. This is an observability problem. And it has a solution — if you know which metrics to instrument.</p><h3>Why LLM Observability Is Not APM With a Different Name</h3><p>If you’re coming from a traditional APM background — Prometheus, Splunk, CloudWatch, Datadog — your instinct will be to instrument latency, error rate, and throughput. That’s the right instinct. But it’s incomplete for LLMs in a way that will bite you.</p><p>A microservice either returns a 200 or it doesn’t. An LLM returns a 200 containing a confident-sounding paragraph that may be completely fabricated. The infrastructure call succeeded. The <em>answer</em> failed. Classical APM has no category for this.</p><p>LLM observability requires two parallel layers: <strong>operational metrics</strong> (speed, cost, throughput — what your infrastructure is doing) and <strong>quality metrics</strong> (faithfulness, relevance, refusal behaviour — what your model is actually saying). Both layers matter. Skipping quality metrics because they’re harder to instrument is how you end up with a production RAG system that confidently hallucinates contract terms to a legal team.</p><h3>Layer One: Operational Metrics</h3><h3>Time to First Token (TTFT)</h3><p>TTFT measures the elapsed time from when the user sends a request to when the first token of the response begins streaming. It is your primary perception metric. Humans experience latency as the gap before <em>anything</em> appears — not the total generation time.</p><p>In streaming applications, a TTFT under 500ms feels responsive. Above 1.5 seconds, users begin to disengage. The drivers of high TTFT are almost always upstream: model loading cold-starts, inference queue depth, network egress to the model endpoint, or — in RAG systems — slow retrieval before the prompt is even assembled.</p><p>If your TTFT is high, check your retrieval layer before blaming the model. In my experience, 60–70% of TTFT problems in enterprise RAG are retrieval problems masquerading as inference problems.</p><h3>Time Per Output Token (TPOT)</h3><p>Where TTFT is a perception metric, TPOT is a throughput metric. It measures the average time taken to generate each successive token after the first one. The inverse — tokens per second — is its more intuitive expression.</p><p>TPOT is driven by model size, quantisation, hardware (GPU memory bandwidth is the binding constraint for autoregressive generation), and batch size. For a 7B parameter model on a well-provisioned GPU, you should expect 80–120 tokens/sec. For a 70B model, expect 15–30 tokens/sec without optimisation.</p><p>The relationship between TTFT and TPOT matters operationally. A system with fast TTFT but slow TPOT streams quickly then stalls — which users experience as the model “thinking” mid-sentence. This is often worse UX than uniform latency. Instrument both independently.</p><h3>Token Velocity and Throughput</h3><p>Token velocity — tokens per second at the system level, not just per-request — is your capacity planning metric. It tells you whether your inference infrastructure can sustain current load, and where the ceiling is before you need to scale horizontally or switch to a faster serving stack.</p><p>Track it across three dimensions: total system throughput (tokens/sec across all concurrent requests), per-request generation speed (TPOT inverse), and prompt-to-completion token ratio. That last one matters more than people realise — a system where users send 50-token prompts and receive 2,000-token completions has a very different compute cost profile than one with symmetric exchanges.</p><h3>End-to-End Latency and Queue Time</h3><p>Beyond TTFT and TPOT, track total request latency (wall-clock time from request receipt to final token delivered) and — critically — <strong>queue wait time</strong>: how long requests sat waiting before inference began. In high-concurrency deployments, queue time is often the dominant latency component and the earliest signal of capacity exhaustion.</p><h3>Cost Per Token and Cost Per Session</h3><p>If your LLM runs on a consumption-priced API (Bedrock, OpenAI, Anthropic), cost per 1,000 tokens is a first-class operational metric, not a finance concern. Break it down: input token cost, output token cost, and — if you’re using prompt caching — cache hit rate. A cache hit rate above 60% on a RAG system with stable context windows can halve your inference bill. Instrument it. Optimise it.</p><h3>Layer Two: Quality Metrics</h3><p>This is where most enterprise observability programmes fall short. These metrics are harder to compute — they often require a judge model or an evaluation framework — but they are the only ones that tell you whether your LLM is actually doing its job.</p><h3>Faithfulness</h3><p>Faithfulness measures whether the model’s response is <em>grounded</em> in the retrieved context. In a RAG system, this is your primary anti-hallucination metric. A faithfulness score of 1.0 means every claim in the response can be traced to a source document. A score of 0.6 means 40% of the response is, at minimum, unsupported — and potentially fabricated.</p><p>Faithfulness is evaluated by decomposing the response into atomic claims, then verifying each claim against the retrieved context using an evaluator LLM. Frameworks like RAGAS, DeepEval, and TruLens implement this natively. It is not cheap to compute at scale — which is why most teams sample (5–10% of production traffic is a reasonable starting point).</p><p>Do not skip this metric. A RAG system without faithfulness monitoring is a hallucination machine with a confidence problem.</p><h3>Answer Relevance</h3><p>Faithfulness tells you whether the answer is grounded. Answer relevance tells you whether it actually answers the question asked. These are not the same thing. A model can produce a perfectly faithful, beautifully sourced response that is entirely off-topic.</p><p>Measured by generating candidate questions from the response and checking their semantic similarity to the original query. Low answer relevance scores typically indicate retrieval failure (the wrong chunks were retrieved) or a prompt engineering problem (the model isn’t being directed to answer the question directly).</p><h3>Context Precision and Context Recall</h3><p>These are retrieval-layer quality metrics that belong in your observability stack even though they’re measuring the vector search, not the LLM.</p><p><strong>Context precision:</strong> of the chunks retrieved, what fraction were actually relevant to the query? Low precision means your retrieval is noisy — the model is getting distracted by irrelevant context.</p><p><strong>Context recall:</strong> of the relevant information that exists in your knowledge base, what fraction was actually retrieved? Low recall means your chunking strategy, embedding model, or index configuration is leaving critical information on the floor.</p><p>These two metrics together tell you <em>why</em> your faithfulness and relevance scores are what they are. Without them, you’re optimising blind.</p><h3>Refusal Rate</h3><p>This one is underinstrumented and underappreciated. Refusal rate measures how often your LLM declines to answer — and it has two failure modes, not one.</p><p><strong>Over-refusal</strong> is when the model rejects legitimate queries because of overly conservative guardrails or misclassification. In enterprise deployments, over-refusal destroys user trust faster than occasional hallucination — users experience it as the system being broken, not cautious.</p><p><strong>Under-refusal</strong> is when the model answers queries it should reject — out-of-scope questions, policy violations, adversarial inputs.</p><p>Track refusal rate overall, then segment it: refusals by guardrail category, refusals by user cohort, refusals by query type. A spike in refusals after a prompt template change tells you the change broke something. A sustained high refusal rate on a specific topic cluster tells you your guardrail calibration is wrong.</p><h3>Hallucination Rate</h3><p>Distinct from faithfulness (a per-response score), hallucination rate is a population-level metric: what percentage of your responses contain at least one factually incorrect or unsupported claim. In production RAG systems, a hallucination rate above 5% is operationally dangerous. Above 10% is a product crisis.</p><h3>Layer Three: Agentic and Tool Call Metrics</h3><p>If you’re running agentic systems — multi-step reasoning chains, tool-using agents, multi-agent workflows — you need an additional metric layer that most observability platforms are only beginning to support.</p><h3>Tool Call Success Rate</h3><p>The fraction of tool invocations that complete without error. Sounds basic. Most teams aren’t measuring it. In complex agentic pipelines, tool failures cascade — a failed API call in step 2 produces degraded context in step 3 and a hallucinated synthesis in step 5. Without per-tool success rate instrumentation, you see the output failure but can’t trace it to its source.</p><h3>Tool Selection Accuracy</h3><p>Did the agent call the <em>right</em> tool? A reasoning agent with access to five tools might consistently choose the wrong one for a given query class — and still return a 200. This requires evaluation against a labelled dataset of expected tool selections, but even a sampled evaluation reveals systematic routing failures that are invisible to infrastructure monitoring.</p><h3>Agent Loop Iterations and Termination Behaviour</h3><p>How many reasoning steps did the agent take before producing a final answer? A well-calibrated agent should solve most tasks in 2–4 iterations. Consistently high iteration counts (8+) signal either an under-specified system prompt, a tool schema problem, or a task that is genuinely outside the agent’s capability envelope. Also track <strong>abnormal termination rate</strong>: how often does the agent hit a maximum iteration limit, timeout, or enter a reasoning loop?</p><h3>Task Completion Rate</h3><p>The end metric that all the above feed into: what fraction of agentic sessions result in a successfully completed task? This is your SLA metric for agentic systems — equivalent to conversion rate in a commerce context.</p><h3>The Tooling Landscape: Who Provides What</h3><p>Here is the honest picture of which platforms natively support which metrics, versus which require custom instrumentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*y07BUlP9g_hr2DOk6ySllw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*OTDnHPQvCm97WHbFTWfKiQ.png" /></figure><p>The pattern is clear: <strong>Splunk and Prometheus are infrastructure-layer tools</strong> being asked to do application-layer AI work. They can do it — but you will build a significant custom instrumentation layer to get there. For teams already standardised on these platforms, OpenTelemetry semantic conventions for LLM spans (the gen_ai.* attribute namespace) provide a path to integration without abandoning existing infrastructure.</p><p><strong>Langfuse and LangSmith</strong> are the strongest open-source-aligned options for quality metrics, particularly for RAG evaluation. Langfuse’s RAGAS integration is production-ready. LangSmith’s tracing is best-in-class for LangChain-native deployments.</p><p><strong>Arize AI</strong> is the most complete platform if quality metrics are your primary concern — tool selection accuracy and context precision/recall support puts it ahead for agentic use cases.</p><p><strong>Datadog’s LLM Observability</strong> module is maturing rapidly and is the right choice if you’re already on Datadog and want a single-pane-of-glass across infrastructure and LLM metrics.</p><h3>The Metric Nobody Talks About</h3><p>I want to close with something that doesn’t appear on any vendor feature list.</p><p><strong>Session abandonment rate correlated with quality metric degradation.</strong></p><p>It’s a composite metric — you derive it by joining your application’s session telemetry (did the user stop engaging mid-session?) with your LLM quality scores (what were faithfulness and relevance doing at that point?). When you build this correlation, you stop arguing about abstract quality thresholds and start having concrete conversations: “every time faithfulness drops below 0.75, session abandonment triples.”</p><p>That is the conversation that moves AI governance from a compliance exercise to a business imperative. Most teams will never build this metric because it requires joining data across systems that don’t naturally talk to each other. Build it anyway. It’s the one metric that turns an LLM observability programme into a product optimisation programme — and that’s the only version of this work that sustains executive support.</p><h3>Where Do You Start?</h3><p>If you’re instrumenting from scratch, this is the sequence I’d follow:</p><ol><li><strong>TTFT and end-to-end latency first</strong> — fix perception before fixing quality. Users who experience a broken-feeling system will never trust the quality improvements you ship.</li><li><strong>Faithfulness on a 10% sample</strong> — set a threshold. Alert on it. Start the conversation with your team about what “acceptable” means.</li><li><strong>Refusal rate by category</strong> — you will find surprises here within the first week.</li><li><strong>Tool call success rate</strong> — if you’re running agents. Non-negotiable.</li><li><strong>Everything else</strong> — once the above four are stable and trusted.</li></ol><p>The goal is not an observability dashboard. The goal is to be the team that can answer, with data, when a business stakeholder asks: “Is our AI actually working?” Most teams today cannot answer that question. The ones that can are the ones that will be trusted to scale it.</p><p><em>Saibal Ghosh is a Senior GenAI Architect and Enterprise AI Transformation Leader based in Hyderabad, India. He builds production-grade RAG and agentic AI systems on AWS Bedrock and writes about the gap between AI hype and AI that actually works in production. He is the author of Docker Demystified and holds the NVIDIA Certified Agentic AI Professional (NCP-AAI) credential, the Databricks Machine Learning Professional certification, and eight AWS certifications.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c740723cbcc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Oracle 26ai Just Quietly Became One of the Most Interesting Vector Databases on the Market]]></title>
            <link>https://medium.com/@saibalghosh/why-oracle-26ai-just-quietly-became-one-of-the-most-interesting-vector-databases-on-the-market-87158ced9691?source=rss-40f5089506fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/87158ced9691</guid>
            <dc:creator><![CDATA[Saibal Ghosh]]></dc:creator>
            <pubDate>Sat, 16 May 2026 10:51:14 GMT</pubDate>
            <atom:updated>2026-05-16T10:51:14.474Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Why Oracle 26ai Just Quietly Became One of the Most Interesting Vector Databases on the Market</strong></p><p>When people debate “the best vector database,” the names that come up are usually the same three or four — OpenSearch, Pinecone, Weaviate, Milvus, sometimes Chroma. Oracle rarely makes the shortlist. And that, in my view, is a mistake worth examining.</p><p>I’ve been spending time with Oracle Database 26ai over the last few weeks — partly because of a real project, partly because I wanted to see whether the “vectors are now a first-class data type” pitch holds up under load. The short answer: it holds up better than I expected, and the architectural argument behind it is more interesting than the marketing suggests.</p><p>Let me walk through what’s actually there.</p><p><strong>The basic move</strong></p><p>The fundamental design choice in 26ai is simple to state but surprisingly hard to fully appreciate. Oracle decided that vectors are not a separate concern. They are not a sidecar service, not an external add-on, not a JSON blob to be parsed by application code. They are a column type. You can CREATE TABLE with a VECTOR column the same way you would with NUMBER or VARCHAR2.</p><p>CREATE TABLE product_kb (</p><p>id NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,</p><p>product VARCHAR2(200),</p><p>description CLOB,</p><p>embedding VECTOR(1536, FLOAT32)</p><p>);</p><p>That’s it. No external vector store. No sync pipeline. No dual-write problem. The vector lives in the same row as the relational data that describes it, under the same transactional guarantees, queryable from the same SQL session.</p><p>If you’ve ever built a production RAG system, you’ll already know why this matters. The standard architecture — embeddings in Pinecone, metadata in Postgres, application code stitching them together — is a maintenance burden. Every dual-write is a potential consistency bug. Every cross-system join is a network hop. Every refresh of the source data requires orchestration across two stores. The pain doesn’t show up in the demo. It shows up six months in, when somebody’s vector is pointing at a row that no longer exists.</p><p><strong>What 26ai actually gives you</strong></p><p>The headline features, condensed:</p><p>A native <strong>VECTOR</strong> data type with flexible dimensionality — you can leave it unconstrained, or pin it to specific dimensions and formats (FLOAT32, FLOAT64, INT8, BINARY). Different rows can hold different embedding models if you want, though I’d argue you usually don’t want to.</p><p><strong>Multiple distance metrics</strong> baked into SQL — cosine, Euclidean, dot product, Manhattan, Hamming for binary vectors. And, new in 26ai, <strong>user-defined distance functions</strong> written in JavaScript. That last one sounds niche, and mostly it is, but for genomics, financial signals, or any domain with proprietary similarity scoring, it’s a genuine capability.</p><p><strong>Two index types.</strong> HNSW (Hierarchical Navigable Small World) is the in-memory graph index — fastest, but memory-hungry. IVF (Inverted File Index) is partition-based and handles datasets too large to keep in RAM. You pick based on your data size and latency budget.</p><p><strong>Hybrid vector indexes.</strong> This is the one most platforms don’t have — a single index that combines full-text keyword search with semantic vector search. You can write one query that says “find me documents matching ‘charging error’ that are also semantically close to ‘won’t connect to fast charger,’ filtered to 2024 models, in the support knowledge base.” Keyword precision plus semantic recall, in one SQL statement.</p><p><strong>ACID-consistent vector search.</strong> This was the missing piece in 23ai. You couldn’t write to a table while it had an HNSW index built on it. In 26ai you can — and your search results stay transactionally consistent with your read snapshot, even on Oracle RAC across multiple instances. That’s the line between “demo” and “production.”</p><p>Here’s the conceptual architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*UPi_cweO8CFp7nWOoSuC-w.png" /><figcaption>Architecture — Oracle 26ai</figcaption></figure><p>Notice what’s missing from that diagram: a separate vector database tier, a separate embedding service, an external orchestration layer. Everything lives inside the database engine. Whether that’s a feature or a constraint depends on your perspective — but for enterprise teams already running Oracle, the architectural simplification is real.</p><p><strong>The in-database embedding story</strong></p><p>One feature that doesn’t get enough attention: 26ai can generate embeddings inside the database. You import an ONNX-format model — Oracle ships small open-source ones like all-MiniLM-L12-v2, but you can bring your own — and then DBMS_VECTOR.UTL_TO_EMBEDDING converts text to vectors directly in SQL.</p><p>sql</p><p>— Generate embedding from text, inline in a query</p><p>SELECT id, product,</p><p>VECTOR_DISTANCE(</p><p>embedding,</p><p>VECTOR_EMBEDDING(my_model USING ‘wireless headphones with ANC’),</p><p>COSINE</p><p>) AS similarity</p><p>FROM product_kb</p><p>ORDER BY similarity</p><p>FETCH FIRST 5 ROWS ONLY;</p><p>No external API call. No OpenAI bill. No data leaving the database server. For organizations with data residency requirements — banks, healthcare, government — this is a meaningful change. You’re not sending your customer notes to a third-party embedding endpoint anymore.</p><p>For embeddings that need a stronger model, Oracle also offers a Private AI Services Container — a separately deployable container running large models, callable from the database via REST. The database server doesn’t have to host the heavy ML workload, but the data still never leaves your infrastructure.</p><p><strong>Hybrid search — the feature that actually changes things</strong></p><p>Most demos focus on pure semantic similarity. Real production search is rarely pure semantic. You almost always need to combine “meaning” with “filters” — find the documents semantically close to this query, but only from this customer, in this date range, with this status code.</p><p>In a typical multi-store architecture, this is painful. You either fetch a large vector candidate set and filter in application code (slow), or you push filters down to the vector store (limited capability), or you maintain parallel indexes (complex).</p><p>In 26ai, it’s just SQL:</p><p>sql</p><p>SELECT id, error_code, description</p><p>FROM dba_error_kb</p><p>WHERE severity = ‘CRITICAL’</p><p>AND created_at &gt; SYSDATE — 30</p><p>ORDER BY VECTOR_DISTANCE(</p><p>embedding,</p><p>:query_vector,</p><p>COSINE</p><p>)</p><p>FETCH APPROX FIRST 10 ROWS ONLY;</p><p>The optimizer plans this as a single execution. Filters and vector ranking happen together, not in sequence across two systems. And because 26ai exposes search accuracy as a percentage — FETCH APPROX FIRST 10 ROWS ONLY WITH TARGET ACCURACY 90 PERCENT — you can tune the recall/latency trade-off without learning HNSW hyperparameters.</p><p><strong>Real-world: a couple of case studies</strong></p><p>Two examples worth knowing about, both publicly cited by Oracle.</p><p><strong>Disease identification in healthcare.</strong> An Oracle customer in the medical diagnostics space migrated their disease identification process onto AI Vector Search and Autonomous Database. The reported outcome: reduced diagnosis time, improved accuracy, better patient care. The architectural value here is specific — diagnostic workflows blend structured patient data (relational), free-text clinical notes (vector embeddings), and historical case records. Having all three in one database, with consistent security and access controls, removes a category of integration risk that’s especially acute in healthcare.</p><p><strong>Analytics services in Australia.</strong> An Australia-based data analytics company adopted Oracle AI Database 26ai with AI Vector Search and reported a 30% revenue increase, attributing it to faster, more reliable data and analytics services. The interesting detail here isn’t the revenue number — those numbers always have selection bias — but the architectural pattern. Analytics workloads benefit disproportionately from converged storage because the data that needs to be vectorized (customer interactions, support tickets, product descriptions) usually already lives in the relational warehouse. Eliminating the ETL pipeline to a separate vector store removes weeks of integration work.</p><p>Beyond the named cases, I’ve heard variations of the same use case from DBAs I trust: searchable error log knowledge bases, where the goal is to find semantically similar incidents and filter by service and severity simultaneously. This is exactly the hybrid pattern 26ai is built for, and the early reports are that it works.</p><p><strong>A simple end-to-end RAG flow</strong></p><p>For teams thinking about retrieval-augmented generation, here’s how a 26ai-based RAG pipeline actually flows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OXXoyB7ihl_XcBRuD1DnBw.png" /></figure><p>The thing to notice is what happens — and what doesn’t — inside the database. Embedding generation, hybrid retrieval, filtering, ranking, joining: all in one engine, one SQL session, one transactional boundary. The only external dependency is the LLM call to generate the final answer.</p><p><strong>What I’d be cautious about</strong></p><p>I want to be honest about where 26ai is <em>not</em> obviously the right answer.</p><p>If you’re a startup with no existing Oracle footprint, the licensing economics are not friendly. Pinecone or Weaviate on commodity infrastructure is dramatically cheaper to get started with, and you don’t need the converged-database story until you have real data integration pain. 26ai is the right answer when you already have an Oracle estate, not when you’re building greenfield from a blank slate.</p><p>If you’re operating at multi-billion-row vector scale, the in-memory HNSW story has memory limits. A million rows at 1536 dimensions needs about 6 GB just for the vectors, plus the index graph. You can use IVF for larger datasets, but at that scale you’ll want to benchmark against specialized vector stores carefully.</p><p>And if your team has no Oracle DBA depth, the operational learning curve is real. The architectural elegance assumes you know how to operate the underlying database.</p><p><strong>The bigger picture</strong></p><p>What 26ai represents — and what I find genuinely interesting about it — is the consolidation thesis playing out in the AI infrastructure layer. The early years of vector databases were specialist tools: standalone systems doing one job extremely well. We’re now entering the phase where general-purpose data platforms are absorbing the specialist capabilities. Postgres did it with pgvector. SQL Server did it. MongoDB did it. Oracle did it with 26ai, and arguably did it most deeply because Oracle’s converged-database philosophy was already there to absorb the new data type cleanly.</p><p>This doesn’t kill specialist vector databases. There will always be use cases — very high cardinality, extreme scale, very specific access patterns — where a dedicated system wins. But for the broad enterprise middle, where the data already lives in a relational database and the vector capability is one piece of a larger application, the case for keeping vectors in the same store as everything else gets stronger every release.</p><p>If you’re an architect making this decision in 2026, the question worth asking is not “which vector database is fastest.” It’s “where does the rest of my data already live, and what does it cost me to add a new tier?” For Oracle shops, the answer just got considerably simpler.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87158ced9691" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a RAG Pipeline Using Amazon Bedrock]]></title>
            <link>https://medium.com/@saibalghosh/creating-a-rag-pipeline-using-amazon-bedrock-7a717e9af853?source=rss-40f5089506fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a717e9af853</guid>
            <dc:creator><![CDATA[Saibal Ghosh]]></dc:creator>
            <pubDate>Sun, 08 Mar 2026 09:42:58 GMT</pubDate>
            <atom:updated>2026-03-08T09:42:58.639Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Step 1: Create an S3 Bucket</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*pfnSgodfficGGIuERJ4Vog.png" /></figure><p>We start by creating an S3 bucket that will serve as the document repository for our RAG pipeline. The bucket name must be globally unique — in this example, we name it rag-demo-saibal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*kHEgN8ZCLzuHkFgHSQLVWA.png" /></figure><p>The bucket is now created successfully.</p><p><strong>Step 2: Upload Documents to S3</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*d9NsYMlqgunkkZzicDLi5A.png" /></figure><p>We upload two documents into the rag-demo-saibal bucket that will form the knowledge base for our RAG pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*cjo7A1i3czg_Dlc7OvPkPQ.png" /></figure><p>Both documents uploaded successfully to S3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*tUB9fXoR-gRURj-7tGVrew.png" /></figure><p><strong>Step 3: Create a Bedrock Knowledge Base</strong></p><p>We navigate to Amazon Bedrock → Knowledge Bases and begin creating a new Knowledge Base.</p><p>We select ‘Create Knowledge Base with Vector Store’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*Zqjoc-nmD5Srp5dm3VwUyw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*-nu2OTsbJtA2pWC2mJDTMA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*J012Qf6YO2FYoqkKoyC_QA.png" /></figure><p>We name the Knowledge Base rag-demo-knowledge-base and provide a short description of its purpose. For the data source, we select S3 — since that is where our documents are stored. We then click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*5plSYOZ9_x-hy4kmdHBH4g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*ULUhCIBokBhrvETAZn9OOQ.png" /></figure><p>The data source name is auto-populated — we leave it as is. We carefully enter the S3 URI in the format s3://&lt;bucket-name&gt;. For the chunking strategy, we use the default setting and leave all other options at their defaults before clicking Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*WzfWuXPUxVEjlKkHmixn1w.png" /></figure><p>On the next page, we choose an embedding model. We go with the Titan Text Embeddings v2 model.</p><p>Then, we need to fill in information about our vector store, In our case we will use OpenSearch as our vector store in preference to S3 or pgvector and other options available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*J2EKR_ZTbHLUkOXeDk0Mrw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*84M08hnwYg-YXKrd7FE8gw.png" /></figure><p>On the next page we review our final configuration and click on create Knowledge Base. This process may take several minutes to complete, but eventually you ought to get a screen like the one above, and this will give you a confirmation that your Knowledge Base indeed got created. However, before we start using the Knowledge Base, we have a preparatory step to complete . And that is the syncing part. Think of sync as an ingestion of your source documents, chunking them, generating embeddings, and updating the vector database so that the Knowledge Base can retrieve the latest information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*nOzycn8W1a4AdD1Jf5eFcA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*mJwyZAe7Eck4ZoiyLyunBw.png" /></figure><p>The sync operation may take a few minutes as well, but will eventually complete, and you should get a screen like the one above.</p><p><strong>Step 4: Test the Knowledge Base</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*XZCgpmXzpi-CXyScaTjceA.png" /></figure><p>The Knowledge Base is now created and synced. We click on ‘Test Knowledge Base’.</p><p>This takes us to the testing page, where we are asked to select a model for retrieval. Scrolling down in the Configurations pane, we can also select a separate model for re-ranking. We configure both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*Qdv7dtkzDCjnkBH2kBpOnQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*eARPFB9bRiB2_PrhN8qiYA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*9EXrvJ19BZ79oF-I3mOWoA.png" /></figure><p><strong>For retrieval, we select Claude Sonnet 4.6 v1.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*T1GgklJNgaGX_Yxkg5XJzQ.png" /></figure><p>For re-ranking, we select the only currently available re-ranking model.</p><p><strong>Step 5: Run Test Queries</strong><br> <br> When selecting the reranking model, we may be prompted to update the IAM service role — no problem, just click Update and it completes in about 30 seconds. We are now ready to test.<br> <br> There are two documents in our S3 repository. We will ask five questions: three of which have answers in the documents, and one that does not. Here are the five questions:<br> <br> 1) Is IVFFLAT better than HNSW?</p><p>2) Can I migrate to Aurora PostgreSQL using an MCP server?</p><p>3) Does IVFFLAT use an inverted index?</p><p>4) Where would HNSW perform better than IVFFLAT?</p><p>5) How can I install RAC on Oracle?</p><p>Below are the answers I got:</p><p><em>Ans 1) It depends on your use case, but in most production scenarios, HNSW tends to outperform IVFFlat. Here’s a comparison:</em></p><p><em>**Performance:** HNSW is significantly faster at query time. In a real benchmark with 5M 768-dimensional vectors, HNSW achieved 96% recall in ~8ms, while IVFFlat needed ~85ms to reach 97% recall. That’s roughly a 5–10x speed advantage for HNSW.</em></p><p><em>**Tuning complexity:** IVFFlat requires constant tuning of parameters like `nprobe` and the number of clusters, and performance can vary across different query patterns. HNSW is simpler — you set `m` and `ef_construction` once and generally don’t need to revisit them.</em></p><p><em>**Scalability:** IVFFlat’s clustering becomes less effective as datasets grow, requiring more cluster probes to maintain accuracy. HNSW scales more consistently.</em></p><p><em>**Memory:** IVFFlat wins here — it uses significantly less memory. For example, IVFFlat used ~3GB for 5M vectors compared to ~8GB for HNSW (a ~2.6x difference).</em></p><p><em>**Index build time:** IVFFlat builds indexes much faster (~10 minutes for 5M vectors vs. ~45 minutes for HNSW).</em></p><p><em>**When IVFFlat might still be the better choice:**</em></p><p><em>- You are severely memory-constrained (e.g., edge devices or billions of vectors with limited RAM)</em></p><p><em>- Your queries are predictable and IVFFlat is already well-tuned</em></p><p><em>- You need fast real-time indexing of streaming data</em></p><p><em>- Your system doesn’t support HNSW yet</em></p><p><em>**Bottom line:** For most production systems where query latency matters, HNSW is the better choice. IVFFlat remains useful in memory-constrained or specific niche scenarios.</em></p><p><em>Ans 2) Here’s an overview of how to migrate from Oracle to Aurora PostgreSQL using an MCP (Model Context Protocol) server with Kiro IDE:</em></p><p><em>## Architecture Overview</em></p><p><em>The solution uses:</em></p><p><em>- **Kiro IDE** — AI-powered development environment</em></p><p><em>- **MCP (Model Context Protocol)** — standard for AI tool integration</em></p><p><em>- **Oracle 19c (simulated schema)** — source database</em></p><p><em>- **Amazon Aurora PostgreSQL** — target database</em></p><p><em>- **Python + FastMCP SDK** — MCP server implementation</em></p><p><em>— -</em></p><p><em>## Key Components</em></p><p><em>### MCP Server</em></p><p><em>- Located at `oracle-to-pg-migration-mcp/server.py`</em></p><p><em>- Built using Python with the FastMCP SDK</em></p><p><em>- Defines **7 tools** using the `@mcp.tool()` decorator</em></p><p><em>- Reads the Oracle schema from a JSON file and performs specific migration tasks</em></p><p><em>### Oracle Schema (Simulated)</em></p><p><em>- 5 tables, ~2 million rows</em></p><p><em>- PL/SQL procedures, views, triggers, and synonyms</em></p><p><em>- Oracle-specific types: `NUMBER`, `VARCHAR2`, `CLOB`, `BLOB`, `TIMESTAMP(6)`</em></p><p><em>— -</em></p><p><em>## Migration Steps</em></p><p><em>### Phase 1: Setup</em></p><p><em>1. Install Python 3.12 with MCP SDK: `pip install mcp`</em></p><p><em>2. Set up Kiro IDE</em></p><p><em>3. Configure MCP in `.kiro/settings/mcp.json`</em></p><p><em>4. No live Oracle or Aurora PostgreSQL database needed — schema is simulated via JSON</em></p><p><em>### Phase 2: Schema Discovery</em></p><p><em>- Ask the MCP tool to display Oracle schema details (tables, column types, constraints, indexes, foreign keys)</em></p><p><em>### Phase 3: Compatibility Analysis</em></p><p><em>The tool identifies critical issues and warnings, such as:</em></p><p><em>- **Critical**: PL/SQL procedures with `COMMIT`/`ROLLBACK` inside them (PostgreSQL functions cannot manage transactions)</em></p><p><em>- Oracle-specific syntax like `DECODE`, `NVL`, `SYS_REFCURSOR`, `ROWNUM`</em></p><p><em>- `BLOB`/`CLOB` handling</em></p><p><em>- `NUMBER(1)` to `BOOLEAN` conversion</em></p><p><em>- Synonyms (no PostgreSQL equivalent)</em></p><p><em>### Phase 4: DDL &amp; PL/SQL Conversion</em></p><p><em>Key Oracle-to-PostgreSQL conversions include:</em></p><p><em>| Oracle | PostgreSQL |</em></p><p><em>| — -| — -|</em></p><p><em>| `NUMBER(1)` | `BOOLEAN` |</em></p><p><em>| `VARCHAR2` | `VARCHAR` |</em></p><p><em>| `CLOB`/`BLOB` | `TEXT`/`BYTEA` |</em></p><p><em>| `DECODE()` | `CASE WHEN` |</em></p><p><em>| `NVL()` | `COALESCE()` |</em></p><p><em>| `ROWNUM` | `LIMIT` |</em></p><p><em>| `SYS_REFCURSOR` | `REFCURSOR` |</em></p><p><em>| `CREATE OR REPLACE VIEW … AS SELECT *` | Separate `FUNCTION + CREATE TRIGGER` |</em></p><p><em>| Sequences + triggers | `SERIAL`/`BIGSERIAL` or `CREATE SEQUENCE` |</em></p><p><em>| `OUT` parameters | `RETURNS type` or `RETURNS TABLE` |</em></p><p><em>### Phase 5: Generate Migration Script</em></p><p><em>The MCP server generates a complete, runnable migration script covering all schema objects.</em></p><p><em>— -</em></p><p><em>## Ad-hoc SQL Conversion Examples</em></p><p><em>You can ask Kiro to convert Oracle SQL on the fly:</em></p><p><em>- `SELECT NVL(SUM(amount), 0) FROM orders WHERE ROWNUM &lt;= 10`</em></p><p><em>- `BEGIN DBMS_OUTPUT.PUT_LINE(‘Hello’); END;`</em></p><p><em>- `DECODE(status, ‘A’, 1, ‘B’, 2, 0)`</em></p><p>— -</p><p><em>## Actual Data Migration</em></p><p><em>For moving the actual data, use **AWS Database Migration Service (DMS)** alongside the generated DDL scripts.</em></p><p><em>Ans 3) The search results do not mention whether IVFFlat uses an inverted index. They describe IVFFlat as a clustering-based approach that assigns vectors to clusters and uses a parameter called `nprobe` to determine how many clusters to search at query time, but the underlying index structure (such as whether it uses an inverted index) is not discussed.</em></p><p><em>Ans 4) Based on real-world benchmarks with 5 million 768-dimensional vectors, HNSW significantly outperforms IVFFlat in query speed and recall:</em></p><p><em>- HNSW at ef=40 achieved ~8ms latency with 96% recall</em></p><p><em>- HNSW at ef=80 achieved ~15ms latency with 98% recall</em></p><p><em>- IVFFlat at nprobe=50 achieved ~45ms with 94% recall</em></p><p><em>- IVFFlat at nprobe=100 achieved ~85ms with 97% recall</em></p><p><em>That’s roughly 5–10x faster query performance while maintaining equal or better recall.</em></p><p><em>Here are the key scenarios where HNSW outperforms IVFFlat:</em></p><p><em>1. **Large and growing datasets**: IVFFlat’s clustering becomes less effective as data scales (e.g., beyond millions of vectors), requiring more cluster probes to maintain accuracy, which degrades performance. HNSW handles scaling more gracefully.</em></p><p><em>2. **Low-latency requirements**: When query speed is critical, HNSW consistently delivers much faster results without needing constant tuning.</em></p><p><em>3. **Varied or unpredictable query patterns**: IVFFlat’s performance can vary depending on query patterns, while HNSW delivers consistent recall regardless of the query type.</em></p><p><em>4. **Simpler operational overhead**: HNSW requires setting only two parameters (`m` and `ef_construction`) once at index creation, whereas IVFFlat requires ongoing tuning of `nprobe` at query time to balance speed vs. recall.</em></p><p><em>5. **Debugging and reliability**: Because HNSW is more consistent, it’s easier to isolate issues (e.g., problems in the embedding model vs. the index itself), making production systems easier to maintain.</em></p><p>Production Enhancements (from the RAG answer):</p><p><em>- Connect to a live Oracle database using `cx_Oracle`/`oracledb` instead of JSON</em></p><p><em>- Add a tool to estimate AWS DMS task configuration</em></p><p><em>- Add a tool to generate DMS table mappings</em></p><p><em>- Add data validation queries (row counts, checksums)</em></p><p><em>- Integrate with **AWS Schema Conversion Tool (SCT)** for comparison</em></p><p><em>Ans 5) The search results provided do not contain any information about installing Oracle RAC (Real Application Clusters). The results only cover topics related to migrating from Oracle to Aurora PostgreSQL using AWS DMS and an MCP server demo. I cannot provide an accurate answer to your question based on the available search results.</em></p><p><em>For guidance on installing Oracle RAC, I recommend consulting the official Oracle documentation at docs.oracle.com or Oracle’s official installation guides for your specific version.</em></p><p>Cool! I got the answers for anything that was in the document; for answers that were not in the document, it let us know in very clear terms that the search results do not contain the required information. Additionally, we understand the fact that the generator is just not copying retrieved chunks, it derives a coherent answer from them — it is a structured response synthesized from the retrieved knowledge and shaped to directly answer our question. In other words, it read my documents but prepared it’s answers in its own way.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a717e9af853" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Will AI Take Away our Jobs? A Chess Analogy]]></title>
            <link>https://medium.com/@saibalghosh/will-ai-take-away-our-jobs-a-chess-analogy-428b3e6bdd90?source=rss-40f5089506fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/428b3e6bdd90</guid>
            <dc:creator><![CDATA[Saibal Ghosh]]></dc:creator>
            <pubDate>Thu, 26 Feb 2026 13:27:53 GMT</pubDate>
            <atom:updated>2026-02-26T13:27:53.533Z</atom:updated>
            <content:encoded><![CDATA[<p>In the early days of computer chess, engines were helpful, but pretty limited. They made tactical suggestions, sometimes good ones as well, but they also made a lot errors, positional or otherwise, and were clearly inferior to elite human players.<br>Then in 1997, Garri Kasparov lost to Deep Blue, and life was not the same again. It was a symbolic moment, the first time a machine defeated the reigning world champion under classical tournament conditions.<br>Today engines like Stockfish operate at a level far beyond even the strongest human grandmasters. Magnus Carlsen, arguably the strongest chess player of all times, would have to literally play out of his skin to even draw a single game against Stockfish. For those who understand chess, Magnus has a 2800+ chess rating, while Stockfish is 3000+, so the gulf is enormous, and it goes without saying that chess softwares today operate at a level far beyond what human beings can ever hope to scale.</p><p>Yet chess did not disappear.<br>Grandmasters did not become obsolete. What changed was the standard. Young players today grow up analyzing every game with engines. They test variations instantly. They validate ideas objectively. They learn patterns at a speed that previous generations could not imagine.<br><em>The engine did not replace chess players. It simply raised the level of chess.</em></p><p>I see something similar happening with AI. Models can write code, generate documentation, analyze data and draft stratefies with remarkable fluency. In many cases, they will outperform average execution.<br>But writing code was not the entire profession.</p><p>Solving customer problems. Understanding constraints. Making architectural trade-offs. Designing systems for reliability, governance and return on investment, as well as connecting technology decisions to business outcomes.</p><p>These require context, accountability and judgement.<br>AI will almost certainly change the nature of work. Routine tasks will compress. Certain skills will decline in value. Others will compound.</p><p>The professionals who thrive will not be those who resist AI — nor those who rely on it blindly — but those who integrate it into their thinking.</p><p>In my opinion, just as chess engines elevated the game, AI will elevate the standard of professional work.</p><p>The board remains.<br>The expectations rise.</p><p><em>The way I see it is not whether AI will take away our jobs, rather whether we are willing to grow alongside it.</em><br>Every technological shift has forced professionals to redefine their value. The printing press did not eliminate thinking. Calculators did not eliminate math, chess engines did not eliminate chess.</p><p>AI may well automate parts of what we do — in fact, it is already doing it. It may compress routine work and expose shallow expertise, but it will also expand what is possible for those willing to adapt.</p><p>The future will not belong to those who compete with AI on speed or recall. It will belong to those who combine judgement, context, and responsibility with the leverage AI provides.</p><p>The board is still there — t<em>he moves are just deeper now.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=428b3e6bdd90" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding how to make the best vector indexing choices]]></title>
            <link>https://medium.com/@saibalghosh/understanding-how-to-make-the-best-vector-indexing-choices-77611aa278ac?source=rss-40f5089506fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/77611aa278ac</guid>
            <dc:creator><![CDATA[Saibal Ghosh]]></dc:creator>
            <pubDate>Sun, 22 Feb 2026 07:11:56 GMT</pubDate>
            <atom:updated>2026-02-22T07:11:56.664Z</atom:updated>
            <content:encoded><![CDATA[<p>Most GenAI systems fail not because of the LLM, but because of poor vector indexing choices.</p><p>Last time I wrote about IVFFlat and how it partitions your vector space into clusters using Voronoi cells. It is fast to build, light on memory, and works well when your dataset is moderate and you can afford to scan a few extra clusters to get good recall. But here is the thing — as your data grows into millions of vectors and your users expect sub-second responses, IVFFlat starts showing its seams. You end up increasing the number of probes to maintain recall, and suddenly your latency is climbing in a direction you did not plan for.</p><p>That is where HNSW changes the game.</p><p>HNSW stands for Hierarchical Navigable Small World, and the core idea is beautifully intuitive. Instead of partitioning your data into flat clusters, it builds a multi-layered graph where each vector is a node connected to its nearest neighbors. The top layers are sparse — think of them as express highways that let you jump across large distances in the vector space quickly. As you descend through the layers, the graph gets denser and the connections become more local, allowing you to zero in on the true nearest neighbors with high precision.</p><p>When a query comes in, the search starts at the top layer, greedily hopping to the closest node it can find, then drops down a layer and repeats. By the time it reaches the bottom layer, it has already narrowed the search space dramatically. The result is that HNSW consistently delivers higher recall at lower latency compared to IVFFlat, especially as dataset size grows. In many production benchmarks, HNSW achieves 95 percent or higher recall while keeping query times in the low single-digit milliseconds, even on datasets with tens of millions of vectors.</p><p>The tradeoff is real though. HNSW indexes are more expensive to build — both in time and memory. The graph structure needs to be maintained in memory for optimal performance, and index creation can take significantly longer than IVFFlat. If you are working with a smaller dataset or your use case tolerates slightly lower recall, IVFFlat remains a perfectly valid and more resource-efficient choice.</p><p>But if you are building RAG pipelines at scale — where retrieval quality directly impacts the answers your LLM generates — understanding this difference is not optional. A retrieval miss at the vector search layer propagates all the way through your pipeline. Your LLM never even gets a chance to reason over the right context. No amount of prompt engineering fixes a retrieval problem.</p><p>The choice between IVFFlat and HNSW is not about which one is better in absolute terms. It is about understanding your data scale, your latency budget, your memory constraints, and your recall requirements, and then making a deliberate engineering decision rather than a default one.</p><p>If you are serious about production GenAI, the indexing layer deserves as much attention as the model layer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77611aa278ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Vector Indexing: HNSW — Why I Ditched IVFFlat (And Why You Might Too)]]></title>
            <link>https://medium.com/@saibalghosh/demystifying-vector-indexing-hnsw-why-i-ditched-ivfflat-and-why-you-might-too-a92c481d25ac?source=rss-40f5089506fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a92c481d25ac</guid>
            <dc:creator><![CDATA[Saibal Ghosh]]></dc:creator>
            <pubDate>Tue, 10 Feb 2026 07:27:52 GMT</pubDate>
            <atom:updated>2026-02-10T07:27:52.921Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Demystifying Vector Indexing: HNSW — Why I Ditched IVFFlat (And Why You Might Too)</strong></p><p>Last week I wrote about IVFFlat, and honestly, I got a bunch of DMs asking “but what about HNSW?” Fair question. I’ve spent the last few months wrestling with both in production, so let me share what I’ve actually learned instead of just the textbook stuff.</p><p>The IVFFlat Problem I Hit</p><p>So here’s what happened. We had this e-commerce search system running on IVFFlat with about 2 million product embeddings. Everything was fine until we hit 5 million. Suddenly, queries that used to take 20ms were taking 80ms. We tuned nprobe, added more clusters, threw more hardware at it. Nothing really helped.</p><p>The issue? As your dataset grows, IVFFlat’s clustering becomes less effective. You end up probing more clusters to maintain accuracy, which defeats the whole purpose. It’s like having a phone book with 10,000 pages — clustering by last name works great until you realize half your users have the last name “Singh” or “Smith” or whatever</p><p>HNSW: The Graph Thing Everyone’s Talking About</p><p>HNSW is different. Instead of asking “which cluster is this vector in?”, it asks “which vectors are nearby?” and builds a graph to answer that question efficiently.</p><p>Here’s the thing that clicked for me: imagine you’re trying to find a specific person in a massive crowd. IVFFlat’s approach is like dividing the crowd into sections and searching each section. HNSW’s approach is like having a network of people who know each other — you ask someone, they point you to someone closer to who you’re looking for, and you keep following the chain until you find them.</p><p>The “hierarchical” part means you have multiple layers of this network. Top layer is sparse (long jumps), bottom layer is dense (fine-grained search). You start at the top and work your way down.</p><p>Let Me Show You Some Code</p><p>Here’s how you’d set up HNSW with pgvector (which I’ve been using):</p><p>— Create a table with embeddings</p><p>CREATE TABLE products (</p><p>id BIGSERIAL PRIMARY KEY,</p><p>name TEXT,</p><p>embedding vector(768)</p><p>);</p><p>— Create HNSW index</p><p>CREATE INDEX ON products USING hnsw (embedding vector_cosine_ops)</p><p>WITH (m = 16, ef_construction = 64);</p><p>That’s it. Two parameters: m and ef_construction. Compare that to IVFFlat:</p><p>— IVFFlat setup</p><p>CREATE INDEX ON products USING ivfflat (embedding vector_cosine_ops)</p><p>WITH (lists = 100);</p><p>— Then you have to tune nprobe at query time</p><p>SET ivfflat.probes = 50;</p><p>With IVFFlat, you’re constantly tweaking nprobe based on your recall requirements. With HNSW, you set it once and forget it.</p><p>The Query Performance Thing</p><p>Here’s a real benchmark from our system (5M products, 768-dim embeddings):</p><p>IVFFlat (nprobe=50): ~45ms, 94% recall</p><p>IVFFlat (nprobe=100): ~85ms, 97% recall</p><p>HNSW (ef=40): ~8ms, 96% recall</p><p>HNSW (ef=80): ~15ms, 98% recall</p><p>That’s not a typo. HNSW was 5–10x faster while maintaining better recall. We switched immediately.</p><p>But Here’s the Catch</p><p>HNSW uses more memory. A lot more. With IVFFlat, you’re storing cluster assignments. With HNSW, you’re storing a graph — each vector has connections to other vectors.</p><p>In our case, IVFFlat used about 3GB for 5M vectors. HNSW used about 8GB. That’s a 2.6x increase. For us, it was worth it because query latency matters more than storage, but if you’re running on a tight budget, this is real.</p><p>Also, building the index takes longer. IVFFlat can index 5M vectors in maybe 10 minutes. HNSW took about 45 minutes. Not a dealbreaker for batch jobs, but something to know.</p><p>When I’d Still Use IVFFlat</p><p>I’m not saying HNSW is always better. There are situations where IVFFlat makes sense:</p><p>You’re memory-constrained and have billions of vectors. Seriously, if you’re running on edge devices or have massive scale with limited RAM, IVFFlat is still your friend.</p><p>Your queries are super predictable and you’ve already tuned IVFFlat perfectly. Don’t fix what works.</p><p>You’re using an older system that doesn’t support HNSW yet. Some databases still don’t have it.</p><p>You need the fastest possible index building and can tolerate slower queries. If you’re doing real-time indexing of streaming data, IVFFlat’s faster build time might matter.</p><p>The Tuning Headache I Avoided</p><p>Here’s something nobody talks about: the mental load of tuning IVFFlat. You’re constantly asking yourself:</p><p>Is my recall dropping because I have too few clusters?</p><p>Should I increase nprobe or add more clusters?</p><p>Why does this query pattern perform differently than that one?</p><p>With HNSW, I set m=16 and ef_construction=64 based on some guidelines, and honestly, I haven’t touched it since. The recall is consistent across different query patterns. It just works.</p><p>A Practical Example: The Debugging Session</p><p>Last month, a customer complained that search results were inconsistent. Sometimes they’d get relevant products, sometimes they wouldn’t. With IVFFlat, this would’ve been a nightmare to debug — is it the clustering? The nprobe setting? The query pattern?</p><p>With HNSW, I ran a quick test and realized the issue was actually in our embedding model, not the index. The index was working perfectly. That’s the kind of clarity you get with a simpler system.</p><p>The Real Talk</p><p>If I’m being honest, HNSW has become my default choice for new projects. The performance is better, the tuning is simpler, and the consistency is worth the extra memory. IVFFlat is still great for specific use cases, but HNSW is the safer bet for most production systems.</p><p>The only reason I’d start with IVFFlat now is if I knew upfront that memory was the bottleneck. Otherwise, HNSW saves me time and headaches.</p><p>What About Your Use Case?</p><p>Are you dealing with vector search? What’s your current setup? I’m curious whether you’ve hit the same IVFFlat scaling issues or if you’re running something completely different. Drop a comment — I read them all.</p><p>Next week, I’m diving into how these indexing strategies actually fit into the broader vector database architecture. There’s some interesting stuff happening with learned indices that I’ve been experimenting with.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a92c481d25ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying IVFFlat: The Vector Index for Database People]]></title>
            <link>https://medium.com/@saibalghosh/demystifying-ivfflat-the-vector-index-for-database-people-4a74124ef66d?source=rss-40f5089506fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a74124ef66d</guid>
            <dc:creator><![CDATA[Saibal Ghosh]]></dc:creator>
            <pubDate>Wed, 04 Feb 2026 05:07:20 GMT</pubDate>
            <atom:updated>2026-02-04T05:07:20.709Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>If You Know B-Trees, You Already Get Half of This</strong></p><p>Remember B-tree indexes from your Oracle days? You’d create an index on a column, and suddenly your WHERE customer_id = 12345 queries went from table scans to lightning-fast lookups. The B-tree organized your data so the database could jump straight to what you needed.</p><p>IVFFlat is doing something similar, but for a completely different problem: finding similar vectors instead of exact matches.</p><p><strong>The Problem IVFFlat Solves</strong></p><p>Here’s where things get interesting. In the AI/ML world, we’re not searching for exact values anymore. We’re searching for <em>similarity</em>.</p><p>Think about it:</p><ul><li>“Find documents similar to this one”</li><li>“Show me products like what this customer bought”</li><li>“Which support tickets are related to this issue?”</li></ul><p>These aren’t WHERE clauses. These are “find me things that are <em>close</em> to this thing” queries. And “close” is measured in high-dimensional space (often 768 or 1536 dimensions for embeddings).</p><p>Without an index, you’d have to compare your query vector against every single vector in your database. Got a million embeddings? That’s a million distance calculations. Every. Single. Time.</p><p>That’s where IVFFlat comes in.</p><p><strong>What IVFFlat Actually Does</strong></p><p>IVFFlat stands for <strong>Inverted File with Flat compression</strong>. Let’s break that down:</p><p><strong>The “Inverted File” Part (IVF)</strong></p><p>Think of this like creating clusters of similar vectors — kind of like partitioning your data, but based on similarity instead of ranges.</p><p>Here’s the analogy: Imagine you’re organizing a massive library. Instead of checking every single book to find ones about “machine learning,” you first go to the “Computer Science” section, then the “AI” subsection. You’ve narrowed your search space dramatically.</p><p>IVFFlat does the same thing:</p><ol><li>It clusters your vectors into groups (called “lists” or “cells”)</li><li>Each cluster has a centroid (the center point)</li><li>When you search, it first finds the closest centroids</li><li>Then it only searches within those clusters</li></ol><p><strong>The “Flat” Part</strong></p><p>This just means the vectors within each cluster are stored as-is, without further compression. It’s a trade-off: you get exact distance calculations within clusters, but you use more memory than more compressed approaches.</p><p><strong>The B-Tree Parallel (And Where It Breaks Down)</strong></p><p><strong>What’s Similar:</strong></p><ul><li>Both avoid full scans</li><li>Both organize data for faster lookups</li><li>Both have a “navigate then search” pattern</li></ul><p><strong>What’s Different:</strong></p><ul><li>B-trees find <em>exact</em> matches; IVFFlat finds <em>approximate</em> neighbors</li><li>B-trees use comparison operators (&lt;, &gt;, =); IVFFlat uses distance metrics (cosine, L2)</li><li>B-trees guarantee finding the value if it exists; IVFFlat trades accuracy for speed</li></ul><p>That last point is crucial: <strong>IVFFlat is an approximate index</strong>. It might miss the absolute closest vector if it’s in a cluster you didn’t search. But in practice, it finds “close enough” results 95%+ of the time, and it does it 10–100x faster than brute force.</p><p><strong>How It Works (The Simple Version)</strong></p><p>Let’s say you’re building a document search system:</p><p><strong>Index Creation:</strong></p><p>— In pgvector (PostgreSQL extension)</p><p>CREATE INDEX ON documents</p><p>USING ivfflat (embedding vector_cosine_ops)</p><p>WITH (lists = 100);</p><p>What just happened?</p><ul><li>You told PostgreSQL to create 100 clusters</li><li>It ran k-means clustering on your embeddings</li><li>Each vector got assigned to its nearest cluster</li><li>The index stores: cluster centroids + vectors in each cluster</li></ul><p><strong>Query Time:</strong></p><p>SELECT * FROM documents</p><p>ORDER BY embedding &lt;=&gt; query_vector</p><p>LIMIT 10;</p><p>Behind the scenes:</p><ol><li>Calculate distance from query_vector to all 100 centroids (fast)</li><li>Pick the closest 10 centroids (configurable with probes)</li><li>Search only within those 10 clusters</li><li>Return the top 10 closest vectors</li></ol><p>Instead of comparing against 1 million vectors, you compared against 100 centroids + maybe 100,000 vectors in the selected clusters. Huge speedup.</p><p><strong>The Tuning Knobs</strong></p><p>IVFFlat has two main parameters you’ll care about:</p><p><strong>1. Lists (at index creation)</strong></p><ul><li>More lists = more clusters = faster search but slower index build</li><li>Rule of thumb: sqrt(number_of_rows) for lists</li><li>1M vectors? Try 1000 lists</li></ul><p><strong>2. Probes (at query time)</strong></p><ul><li>How many clusters to search</li><li>More probes = better accuracy but slower queries</li><li>Start with 10, tune based on your accuracy needs</li></ul><p>It’s the classic speed vs. accuracy trade-off.</p><p><strong>When to Use IVFFlat</strong></p><p><strong>Use it when:</strong></p><ul><li>You have 10K+ vectors (below that, brute force is fine)</li><li>You can tolerate ~95% recall (missing a few results occasionally)</li><li>You need sub-second query times</li><li>You’re working with pgvector, FAISS, or similar libraries</li></ul><p><strong>Skip it when:</strong></p><ul><li>You need 100% accuracy (use brute force or HNSW)</li><li>You have &lt; 10K vectors (overhead isn’t worth it)</li><li>Your vectors change constantly (index rebuilds are expensive)</li></ul><p><strong>Real-World Example</strong></p><p>Let’s say you’re building a support ticket system with semantic search:</p><p># Using pgvector in Python</p><p>import psycopg2</p><p>from sentence_transformers import SentenceTransformer</p><p># Generate embedding for new ticket</p><p>model = SentenceTransformer(‘all-MiniLM-L6-v2’)</p><p>query_embedding = model.encode(“My payment failed”)</p><p># Search for similar tickets</p><p>cursor.execute(“””</p><p>SELECT ticket_id, description,</p><p>embedding &lt;=&gt; %s AS distance</p><p>FROM support_tickets</p><p>ORDER BY embedding &lt;=&gt; %s</p><p>LIMIT 5</p><p>“””, (query_embedding, query_embedding))</p><p># Returns 5 most similar tickets in milliseconds</p><p>Without IVFFlat, this query might take seconds on a large database. With IVFFlat, it’s sub-100ms.</p><p><strong>The Bottom Line</strong></p><p>If B-trees are about finding the needle in the haystack, IVFFlat is about finding needles that <em>look like</em> your needle — and doing it fast enough to be useful in production.</p><p>It’s not perfect (that’s why HNSW and other indexes exist), but it’s simple, effective, and available in tools you’re already using like PostgreSQL with pgvector.</p><p>For most vector search use cases, IVFFlat is the “good enough, fast enough, simple enough” choice. And in engineering, that’s often exactly what you need.</p><p><strong>Next in the series</strong>: Demystifying HNSW (and when to use it instead of IVFFlat)</p><p><strong>What topics should I cover next?</strong> Drop a comment and let me know what you want demystified!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a74124ef66d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Strands Demystified]]></title>
            <link>https://medium.com/@saibalghosh/strands-demystified-4c96f51ce437?source=rss-40f5089506fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c96f51ce437</guid>
            <dc:creator><![CDATA[Saibal Ghosh]]></dc:creator>
            <pubDate>Mon, 27 Oct 2025 08:30:08 GMT</pubDate>
            <atom:updated>2025-10-27T08:30:08.584Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s take a step back and first understand what an agent does for us. Think of it as a software<br>entity that can understand our instructions in natural language, reason about how to achieve that<br>goal and act by using available tools like functions, APIs, databases. It then observes its actions,<br>maybe iterates a few times before producing the final outcome.<br>Now that we understand agents, anyone serious about building agentic systems need to<br>understand what Strands is. If I talk from an AWS perspective, Strands is an open-source SDK<br>that lets us build real production-ready AI agents with minimal orchestration.<br>So, we can think of it as a mechanism whereby instead of engineering complex workflows, we<br>give the model a prompt, equip it with tools, and let the intelligence emerge.<br>Lets break it down a bit more: It is an intelligent component that uses a LLM to reason about a<br>task and may use external tools or other data sources to complete its task.<br>This means less manual tool calls and more focus on intent and value. Instead of building<br>workflows we are building agents. Instead of coding at every step we are defining intent.<br>There are three components of strands: The model (which is the brain; it interprets, reasons and<br>generates — examples could be Claude, Titan, Llama etc.), the tools (the hands — external<br>functions, APIs or programs that agents can call to fetch data, compute or perform real world<br>actions), and the prompt (the personality + memory — defines what kind of agent it is — its rules,<br>goals and style of operation).<br>So, what emerges, is that in strands when you define an agent, you are literally wiring these three<br>together.<br>Before I go ahead any further, let us understand the strands ecosystem a bit. When you build an<br>agent using strands, you provide the model, the prompt and the tools, while strands does the<br>heavy lifting deciding when to reason, when to call a tool, when to return a result. Additionally,<br>it does a host of other things like calling the tools, handling the workflow memory between steps,<br>helping agents talk to each other as well helping us scale and deploy the agents.<br>Bottom line: Strands creates the agent instance for us, but more precisely it builds, runs<br>and manages the lifecycle of that agent.<br>It is incredibly easy:</p><p>From strands inport Agent<br># Define the agent<br>agent = Agent (<br>model = “anthropic.claude-3-sonnet”,<br>tools = [calculator, web_search]<br>prompt = “You are a Data Analyst that answers using precise calculations.”<br>#Run the agent</p><p>That’s it. I hope you find the article interesting and worth a read.<br>result = agent (“What is the average of 45, 60 and 90<br>print (result)</p><p>Here:<br>- you didn’t manually code any LLM reasoning or tool loop.<br>-Strands automatically built and ran the agent loop using your model + tools</p><p>So far, so good. When I was trying to understand strands, I had a basic question that needed a lot<br>of research and mining of the documentation to understand. The question was: Its all pretty<br>simple talking about models, but what about the tools, where do they live; how do I know which<br>tool would be the most optimal for the work I am doing.<br>As of today, there should be, but there is no discovery layer for tools. Frameworks like Strands,<br>LangChain, or MCP don’t have one unified global “tool store”.<br>What I gleaned from various sources are that tools come from three distinct sources:<br>1) Built-in tools: packaged inside SDKs like Strands, LangChain or MCP. We need to check the<br>official SDK docs or GitHub Readme for finding these tools.<br>2) External API connectors: Hosted as REST or MCP servers. And the way we can know about it<br>is to browse the public registries like Model Control Protocol Registry or GitHub<br>3) Custom tools: Defined locally in our own systems. We know about them because we create or<br>register them manually.</p><p>In practice, I can find the tools:<br>1) For Strands or LangChain by running a simple inspection command like:</p><p>from strands import registry<br>print (registry.list_tools ())</p><p>And if supported, we can list all the tools, and that is it.<br>2) For MCP, each MCP server declares its capabilities via a capabilities manifest. You can<br>simply query it.<br>3) Else, we can do a documentation search by going to:<br><a href="https://docs.strands.ai/">https://docs.strands.ai</a><br><a href="https://modelcontextprotocol.io/registry">https://modelcontextprotocol.io/registry</a><br>and search for “tools” or “endpoints”</p><p>That’s it. I hope you find the article interesting and worth</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c96f51ce437" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Model Context Protocol]]></title>
            <link>https://medium.com/@saibalghosh/model-context-protocol-411e84d19de0?source=rss-40f5089506fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/411e84d19de0</guid>
            <dc:creator><![CDATA[Saibal Ghosh]]></dc:creator>
            <pubDate>Thu, 23 Oct 2025 07:39:29 GMT</pubDate>
            <atom:updated>2025-10-23T07:39:29.429Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the things that initially intrigued me is why Model Context Protocol (MCP) is becoming such a game changer in GenAI applications.<br> The way I understand it is while LLMs powering our chatbots and code assistants have become more sophisticated in generating coherent text, summarizing content, and even reasoning, we always had a problem to reliably connect these models to real-world systems, live data, business workflows and enterprise tools.<br> Somewhere I read that MCP is analogous to the USB-C connector that we have today. For someone who learns a lot through analogies and visualization, this made a lot of sense. I guess we all remember the days before the USB-C came into existence; how every device required its own custom cable and driver — printers, phones, cameras, hard drives — all had incompatible connectors. Integrating or replacing them was messy and inconsistent.<br> <br> Now, with the USB-C, one universal connector handles power, data, video and peripherals. You plug it in, and it just works — whether it is laptop, phone or monitor.</p><p>Visualize MCP as doing something very similar, it introduces a kind of standard connector layer between the AI models and the outside world. Just as USB-C defines how devices exchange data and power, MCP defines how models exchange context and capabilities.<br> So, putting things in perspective, any LLM or agent that ‘speaks MCP’ can plug into any MCP — compliant server, be it a database, a filesystem, analytics engine…anything without needing custom/new adapters everytime.<br> So far, so good; the next thing that I had difficulty in conceptualizing is how is MCP different from an API and what makes an MCP an MCP. So, the way I now see it is:<br> Unlike APIs which are unidirectional in the sense that the client calls the server, MCP enables bi-directional awareness. This two-way connection is not just a connector but an ecosystem or a language. Basically, what is happening is that we now have a universal grammar for AI to tool communication — using JSON RPC, self-describing schemas, and capability manifests. We are no longer coding a custom adapter for every system — we just implement the protocol and everything that speaks MCP can talk to everything else!<br> But, again this begs the question: how does the external world understand MCP. Let us say I have an Oracle database that speaks SQL over JDBC/ODBC — this system knows nothing about LLMs or MCP. Well, the way it works is we deploy a small MCP server process alongside it, typically on a host that already has network access to the DB. This server speaks two languages — to the outside world it talks JSON-RPC over HTTP/stdio, so that an LLM or agent can talk to it, and with Oracle it talks SQL over JDBC/ODBC. Making sense?<br> Last, but not the least, why is it called MCP Server? Where is the server?<br> There is no server actually, what is really meant is that it is a program that implements the MCP protocol in the server role. So, the word server is shorthand, not a literal description of infrastructure. It distinguishes the role from the MCP client, which is the model host or the agent framework that initiates calls. Finally, there are two common places an MCP server can run: on the same machine as the LLM or IDE, or on a separate VM, container or Cloud function.<br> In conclusion, MCP is a full-fledged shift from fragmented integrations to a shared language between intelligence and infrastructure. As GenAI continues to evolve, standards like MCP will define how seamlessly models can reason, act and connect with the real world. In my humble opinion the true power of AI will no longer come from what a model remembers, but from how intelligently it connects.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=411e84d19de0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>