<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by varun rao on Medium]]></title>
        <description><![CDATA[Stories by varun rao on Medium]]></description>
        <link>https://medium.com/@varunrao.aiml?source=rss-bc5d9b74a94e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ERAirZ4GPhawogh8Zly4yQ.jpeg</url>
            <title>Stories by varun rao on Medium</title>
            <link>https://medium.com/@varunrao.aiml?source=rss-bc5d9b74a94e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:01:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@varunrao.aiml/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your AI Is Thinking in Circles. This Paper Proved It Mathematically and Built a Fix.]]></title>
            <link>https://python.plainenglish.io/your-ai-is-thinking-in-circles-this-paper-proved-it-mathematically-and-built-a-fix-54978ede9796?source=rss-bc5d9b74a94e------2</link>
            <guid isPermaLink="false">https://medium.com/p/54978ede9796</guid>
            <category><![CDATA[ai-research]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[sampling]]></category>
            <category><![CDATA[post-training]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[varun rao]]></dc:creator>
            <pubDate>Sun, 04 Oct 2026 15:01:37 GMT</pubDate>
            <atom:updated>2026-10-04T15:03:20.866Z</atom:updated>
            <content:encoded><![CDATA[<p>There is a quiet assumption baked into how we use large language models today: if the model gets the wrong answer, we just ask it again. Run it multiple times and Sample more outputs. Somewhere in that pile of attempts, the correct answer should appear, RIGHT.</p><p>This idea, called Repeated Sampling (RS), is one of the most widely used techniques for scaling inference-time compute. It underpins the majority voting, best-of-N selection, and a substantial chunk of the reasoning improvements we have seen over the past two years. The mathematical intuition is simplistically clean. If a correct answer has any non-zero probability of appearing under the model’s output distribution, sample majorly enough times and it will eventually show up . More formally, for a model with parameters θ generating output y given input x, the probability that the correct answer y* appears at least once in k independent samples is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*QLiufKFH9b3GybAHg_1zvg.png" /></figure><p>This quantity increases monotonically with k and approaches 1 as k → ∞. Hence, the math is sound but the assumption underneath it is not.</p><p>The problem, which a February 2026 paper from Arizona State University demonstrates with uncomfortable precision, is that the LLMs are not actually exploring the output space the way the formula does implies. Rather They are running in circles, generating the same underlying reasoning strategy repeatedly, with surface variation masking the conceptual repetition. Worse, the paper provides the theoretical framework to explain that exactly when and why this failure did occur, and proposes a simple structured intervention called GuidedSampling to correctify it.</p><h3>The Diversity Problem: Measured, Not Assumed</h3><p>The researchers did something deceptively simpler to expose the failure. They took a hard problem from the MATH benchmark, finding the maximum value of (x−y)/(x⁴+y⁴+6) over all real numbers x and y, and asked a model to solve it 1,000 times using standard Repeated Sampling, sounds fairly simple.</p><p>Then they extracted the core mathematical concept underlying each solution.</p><p>892 out of 1,000 solutions applied the AM-GM inequality... The same theorem, over and over again, consistently producing the wrong answer all the time. Only 108 candidates explored Cauchy-Schwarz, Trivial Inequality, or Chebyshev’s Inequality, approaches that will actually had a path to the correct solution.</p><p>To quantify this systematically, they have measured the number of distinct concepts used across benchmarks. With a budget of 100 candidate solutions per problem, Llama-3.2–3B-Instruct used an average of only 2.75 different concepts on HumanEval code problems. On the full MATH benchmark, 37% of the all questions were attempted using just one underlying concept, and fewer than the 36% of questions saw more than two distinct concepts, which is regardless of how many samples were generated.</p><p>The pass@k metric, which measures the probability that at least one of k sampled solutions is a correct one, is the standard way to evaluate this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*Ce1dkZnmOQsvH2HL7tB0-w.png" /></figure><p>where n is the total number of generated candidates and c is the number of correct ones. And, The formula rewards having at least one correct candidate in k draws. But when 89% of your candidates are anchored to the same failing approach, increasing k adds almost nothing to it. You are sampling from a distribution that has about most of its mass on the wrong strategy.</p><h3>Why RS Fails: The Structural Reason</h3><p>The failure is not accidental. It just follows from how models are trained. LLMs are optimised to produce a single correct response for any given input and, which builds the strong priors toward a small set of the high-probability solution strategies per problem type. So When you call the model repeatedly with the same prompt, those priors are gonna dominate every sample. The theoretical guarantee is that unlimited sampling eventually will cover all outputs with non-zero probability, holds in the limit. In practice, the distribution is so concentrated on a few high-probability paths that finite compute budgets seems to never reach the low-probability alternatives that might actually be correct.</p><p>RS implicitly collapses into two distinct cognitive acts into one uncontrolled pass: choosing an approach, and then also executing it. The model picks and solves simultaneously, which means the approach selection is also governed entirely by the same peaked distribution that governs everything else. There is no mechanism to force it off its preferred strategies.</p><h3>GuidedSampling: The Formal Setup</h3><p>The paper’s proposed algorithm, GuidedSampling, separates these two acts into explicit phases.</p><p><strong>Exploration Phase.</strong> Given question x and model p_θ, the algorithm constructs a set of concepts C = {c₁, c₂, …, c_K} iteratively. The k-th concept is sampled by conditioning on both the question and all previously generated concepts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*nJp0VDavXlKic67A_LSVaQ.png" /></figure><p>This iterative conditioning is the mechanism that drives diversity. By conditioning this each new concept on what has already been proposed, the model is structurally getting pushed away from repeating prior approaches. Then the process terminates either when K concepts have been generated or when the model signals that no further useful concepts can be produced.</p><p><strong>Generation Phase.</strong> Once the concept set C is established, each concept c_k independently guides the generation of M candidate solutions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*LvEGCfBBHho7rcZNbukSFA.png" /></figure><p>The full candidate pool is now the union across all concepts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*oqLKV-65EP_aCO1pu5f9fQ.png" /></figure><p>The total compute budget is K × M, the same as running RS K×M times, the budget is redistributed, and not expanded. The difference is that the distribution over the approaches is now explicit and controlled rather than implicit and peaked.</p><h3>The Theoretical Guarantee</h3><p>This is where the paper goes beyond empirical observation and provides formal conditions under which GuidedSampling is provably superior to RS.</p><p>The argument rests on a cleanly stated assumption, Let C_r denote the set of relevant concepts, those that point toward a valid solution path for y*. Let π_concept(c|x) be the model’s probability of generating concept c given question x, and π_solution(y|x,c) be the probability of generating solution y given concept c.</p><p><strong>Assumption 1.</strong> For any relevant concept c ∈ C_r, conditioning on it strictly amplifies the probability of generating the correct solution. That is, there exists an amplification factor k_c &gt; 1 such that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*N4LozQ8nIN2FTCbnY6hfWQ.png" /></figure><p>This is the formalisation of the intuition that a relevant hint makes a correct solution more likely, which is equivalent to saying the sample-wise mutual information I(y; c | x) &gt; 0 for any relevant c.</p><p><strong>Theorem 1.</strong> Under Assumption 1, GuidedSampling yields a higher probability of generating the correct solution than Repeated Sampling, P_GS(y*|x) &gt; P_RS(y*|x) , if and only if the following condition holds:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*-ccKRnzhnyyGCc7ACahJQA.png" /></figure><p>where P(C_r|x) = Σ_{c ∈ C_r} π_concept(c|x) is the probability that the exploration phase generates a relevant concept, and k_min is the minimum amplification factor across all relevant concepts.</p><p>This condition has two components worth unpacking separately.</p><p><strong>Term 1:</strong> (k_min · P(C_r|x) − 1) · P_RS(y*|x) &gt; 0</p><p>This term is positive when k_min · P(C_r|x) &gt; 1, which holds when the model has a sufficiently high probability of generating at least one of the relevant concept AND that concept provides a meaningful amplification of the correct-answer probability. This is the primary pathway to GuidedSampling’s superiority, so the model needs to be good enough at concept generation to reliably surface relevant approaches.</p><p><strong>Term 2:</strong> Σ_{c ∉ C_r} π_concept(c|x) · π_base(y*|x,c)</p><p>This term captures a more subtle case: even when the model generates an irrelevant concept, it might still recover and produce the correct solution by other means so, the paper empirically observes this to be rare but non-zero, and documents one such case study in their appendix.</p><p>The theorem makes the failure mode explicit. If P(C_r|x) is small, if the model is bad at generating relevant concepts for a given problem type, GuidedSampling can underperform RS and This is not one of the theoretical edge case. The authors document it directly like: Qwen2.5–3B-Instruct generates an average of only about 1.13 distinct concepts per HumanEval problem and, essentially the same concept every time, and GuidedSampling consequently performs much worse than RS on that model-benchmark combination. Meanwhile, Llama-3.2–3B-Instruct generates 7.58 unique concepts on average, and GuidedSampling improves its HumanEval performance substantially.</p><p>The model’s capacity to generate P(C_r|x) &gt;&gt; 0 is a load-bearing.</p><h3>The Exploration–Generation Trade-off</h3><p>A non-obvious design decision inside GuidedSampling is that how to allocate a fixed compute budget IC between the number of concepts K (breadth of exploration) and the number of solutions per concept M = IC/K (depth of generation per approach).</p><p>Increasing K expands the concept space explored but at the same time shrinks M, reducing the model’s ability to thoroughly develop any single approach. Decreasing K concentrates compute on fewer approaches with more samples each, but will risk missing the relevant concept entirely.</p><p>The empirical results reveal a consistent pattern: performance improves as K increases from 1 (equivalent to RS) up to an optimal value, then declines as M becomes too small to produce reliable solutions per concept. The optimal K is task-dependent. On MATH, a moderate K provides the best balance. On harder benchmarks like OlympiadBench, the gains are smaller overall (3.08%) because the problems are hard enough that even concept-guided generation frequently fails.</p><p>The authors also study which concepts contribute most. Earlier-generated concepts (lower k) perform better than later ones, 19.8% correct solution rate at k=1 declining to 16.2% at k=5. This is expected as the model’s highest-probability relevant concepts appear first and, Later concepts (k ≥ 6) show higher variance due to smaller sample sizes but can still unlock solutions for a small number of hard problems that require unusual approaches.</p><h3>Post-Training: Baking Diversity In</h3><p>The more quietly significant contribution is what happens when GuidedSampling’s trajectories are used to generate training data, fine-tuning new models on this diverse reasoning paths rather than just peaked single-approach solutions.</p><p>The paper tests two training regimes:</p><p><strong>Final-Answer Only (FA):</strong> The concept scaffolding is discarded. Only the verified solution s is used as the training target. Hence, the objective is the standard supervised fine-tuning loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*6kRiztt7TaFMN-JdCkVjbQ.png" /></figure><p><strong>Concept-Augmented Answer (CAA):</strong> The full concept list C is concatenated with the solution s as a joint training target y = concat(C, s). The objective becomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*xpDmLkzGKVa9P4AEX5yGQg.png" /></figure><p>CAA teaches the model to enumerate reasoning strategies before committing to one so structurally encoding the exploration phase as a learned behaviour rather than some imposed inference-time procedure.</p><p>The results are significant. Models fine-tuned on GuidedSampling data improve pass@5 by an average of 9.7% over models trained on standard RS data. CAA outperforms FA across most benchmarks. The average number of distinct concepts used per problem by fine-tuned models increases from 1.67 (RS-trained baseline) to 2.58 (FA) and 3.03 (CAA).</p><p>The cross-domain transfer is the result most worth noticing as Models trained exclusively on mathematics data (OpenMathInstruct-2, 10k samples) show pass@5 improvements on GPQA-Diamond scientific reasoning (+6.17% over RS) and HumanEval code generation (+1.86% over RS) , domains not seen during training. The diversity in reasoning approach, once learned, generalises.</p><h3>Where the Theorem’s Conditions Fail</h3><p>The paper is explicit about its boundary. GuidedSampling requires P(C_r|x) &gt;&gt; 0, the model must be capable of generating relevant, well-defined concepts for the domain. In mathematics, this means named theorems. In code, algorithmic strategies. In scientific reasoning, applicable laws and principles.</p><p>In commonsense reasoning, the relevant “concepts” are vague, contextual, and difficult to specify as distinct named strategies. The theorem’s amplification assumption is that a relevant concept strictly boosts P(y*|x,c) and, does not hold reliably when concepts are ill-defined. Applied to CommonSenseQA, GuidedSampling underperforms RS by 3.28% at pass@50. The mechanism has a domain precondition: structured concept spaces where the model can generate meaningfully distinct and relevant approaches.</p><h3>What This Formalises</h3><p>The paper’s contribution is not just an algorithm that works rather it is a framework that explains when and why inference-time diversity fails, and what structural conditions a solution must satisfy.</p><p>The core finding, stated precisely: Repeated Sampling’s mathematical guarantee (coverage of the full output space as k → ∞) is technically true but practically irrelevant when the model’s distribution places nearly all its mass on a small set of conceptually identical paths. The pass@k metric improves with sampling only when the samples are conceptually diverse. Generating conceptually diverse samples also requires active intervention in the approach-selection process, not just having more draws from the same peaked distribution.</p><p>GuidedSampling provides that intervention by separating concept exploration from solution generation, encoding the separation in a formal two-phase algorithm with proven conditions under which it outperforms RS, and demonstrating that the same diversity can be transferred into model weights through structured post-training data.</p><p>The code and data are publicly available at <a href="http://github.com/DivijH/sampling_inference">github.com/DivijH/sampling_inference</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54978ede9796" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/your-ai-is-thinking-in-circles-this-paper-proved-it-mathematically-and-built-a-fix-54978ede9796">Your AI Is Thinking in Circles. This Paper Proved It Mathematically and Built a Fix.</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why We Built a Social Network That Hates AI]]></title>
            <link>https://medium.com/@varunrao.aiml/why-we-built-a-social-network-that-hates-ai-002ef190a2f3?source=rss-bc5d9b74a94e------2</link>
            <guid isPermaLink="false">https://medium.com/p/002ef190a2f3</guid>
            <category><![CDATA[ai-slop]]></category>
            <category><![CDATA[human-slop]]></category>
            <category><![CDATA[social]]></category>
            <category><![CDATA[building]]></category>
            <dc:creator><![CDATA[varun rao]]></dc:creator>
            <pubDate>Fri, 03 Jul 2026 04:41:25 GMT</pubDate>
            <atom:updated>2026-07-03T04:41:25.199Z</atom:updated>
            <content:encoded><![CDATA[<p>Open any feed right now. Scroll for just near about 30 seconds. Try to tell me which post was typed by an actual human and which one was generated in 0.4 seconds by a GPT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TwW21ze9NnZPgZJLFPEsew.png" /></figure><p>You can’t, seriously.</p><p>Nobody can anymore.</p><p>That’s not an exaggeration, it’s actually the state of growing internet in 2026. The comment under the news articles you trusting, the five-star review of a newly opened restaurants, the so called “personal story” with 40,000 likes, that too friendly DM from someone who seems a little too eager and crazy, any of it could be synthetic. Increasingly with time, most of it is.</p><p>We didn’t build <a href="https://humanslop.in/">Human Slop </a>cause we thought AI was bad. We built it because we noticed something nobody else did seemed to be solving: <strong>trust on the internet has already collapsed, and nobody is fixing/replacing it with anything.</strong></p><h3>The Slop Problem</h3><p>There’s a term for this now, “SLOP.”</p><p>Low-effort, mass-produced, AI-generated content flooding every platform that has an open posting box. It’s not anything malicious, exactly. It’s just relentless and inhumane. A bot doesn’t get tired and frustrated of views. It doesn’t run out of opinions unless you run out of tokens. It can sincerely produce more “engagement” in an hour than a thousand humans can produce in a week by perfecting the blueprint working for everyone else posting same shit everywhere.</p><p>The result:</p><p>An internet where the signal-to-noise ratio is flipped. Real human thought is now just a minority voice, buried under Big pile of synthetic volume.</p><p>Every single platform’s answer to this has been the same: Better detection. Flag the bots. Ban the quite obvious ones. Add a little “AI-generated” label here and there once in a while to show just as they care.</p><p>It’s a losing game, for every detector, there’s a generator built to beat it the next week. continuos race with no win at all.</p><p>We decided not to play that game at all.</p><h3>Proof Instead of Detection</h3><p>Human Slop doesn’t try to guess whether content was written by a human after the fact. We verify the human <em>before even</em> a single word is ever posted.</p><p>Every account is tied to a physical device through hardware-level verification. There’s a biometric check, fingerprint confirming a real person is actually there, re-verified every 30 days so accounts can’t be quietly handed off to a script. And while you’re writing, we’re not reading your mind we’re reading your hands (yeah, the craziest part😸 ). Your Typing speed, pauses, the little bursts and hesitations and backspaces that make up how an actual human writes in real time not just copy pasting everything at once or just continuously writing from a book in front. That pattern becomes your Human Score, and it ships with every post as proof.</p><p>Copy-pasting isn’t allowed. Not because we’re paranoid lunatics, because we built the platform around a single non-negotiable promise: <strong>what you read here was typed, in real time, by a real person. </strong>👀</p><h3>Why This Matters More Than It Sounds Like It Should</h3><p>It’s easy to dismiss this as a gimmick, “another verification badge.” It isn’t.</p><p>The badge isn’t the point. The exact point is what becomes possible once you can actually trust that the person on the other end of a post, a comment, or a DM is real, as everything starts becoming real not just trying to be polite machine anymore. Communities stop being performance theaters for engagement farming but more about the having a personal space not for algorithm but for yourself. Conversations stop being chess matches against unknown opponents who might not even be conscious. A compliment starts meaning something again, because you know it wasn’t auto-generated to farm a reply but only as he cared.</p><p>We’re two final-year undergrads at IIT Bhilai, not a venture-funded growth team chasing a metric. And, We built this because we wanted somewhere to post that didn’t feel like screaming into a void full of other screaming bots or engagement farmers. If that’s a small thing to want from the internet, it shouldn’t be, yet here we are, building it ourselves because nobody else was ready or interested in humans connections anymore.</p><h3>What’s Next</h3><p>For now, the short version: the internet got too good at faking humanity undetected. We’re betting on proving it, instead of guessing at it, is worth building from scratch.</p><p>Human Slop is live on <a href="https://play.google.com/store/apps/details?id=com.humanslop.app&amp;pcampaignid=web_share">Google Play</a> and on the web. iOS is coming soon.</p><p>No bots. No agents. No AI. Just people, typing.</p><p>Just a question: Are you brave enough to type or still dependent on AI for it?</p><p><a href="https://play.google.com/store/apps/details?id=com.humanslop.app&amp;pcampaignid=web_share">Download Link</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=002ef190a2f3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Simulated 250 Agents on a GPU and Watched a Financial Crisis Emerge, Without Writing “Crash Now”]]></title>
            <link>https://medium.com/codetodeploy/i-simulated-250-agents-on-a-gpu-and-watched-a-financial-crisis-emerge-without-writing-crash-now-3811a6ca5f92?source=rss-bc5d9b74a94e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3811a6ca5f92</guid>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[world-models]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[varun rao]]></dc:creator>
            <pubDate>Fri, 26 Jun 2026 08:00:58 GMT</pubDate>
            <atom:updated>2026-06-29T07:08:27.656Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OdXOnziYklpbUZvsZ0dexA.png" /></figure><blockquote><em>ArthJAX is an open-source JAX economy where credit stress, contagion, and booms are emergent, plus a neural world model that learns the chaos.</em></blockquote><h3>You can’t press rewind on 2008</h3><p>There is no sandbox where the Federal Reserve can rerun Lehman week with the volume turned up.</p><blockquote><em>💥 </em><strong><em>Save Up to 85% OFF</em></strong><br> 📚 Premium Courses <br> ⏳ Limited-Time Offer | 🎓 Upgrade Your Skills<br><a href="https://trk.udemy.com/zz4NBO"><em>👉 </em><strong><em>Enroll Now &amp; Start Learning</em></strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkQFf0NUnzoFbVEVQCF5WQ.png" /></figure><p>No A/B test for “<strong>oil shock + credit freeze + panic selling</strong>” on a live nation of 300 million people.</p><p>And yet, every macro researcher, every risk team, every curious engineer eventually asks the same question:</p><blockquote>What happens if the bad things arrive together?</blockquote><p>For years the answers lived in three very different worlds.</p><p>Equation-based models gave elegance but not crowds. Agent-based models gave crowds but often at the cost of speed, reproducibility, and a GitHub link. Lately, LLM-driven “<strong>AI economies</strong>” gave headlines but opaque behavior: you’re not stress-testing rules, you’re stress-testing prompts.</p><p>I wanted a fourth thing.</p><p>Something you could <em>run</em>, <em>share</em>, and <em>break</em> in public.</p><p>Something fast enough to feel like a game engine, honest enough to show its math, and strange enough that crises could <strong><em>surprise the person who wrote the code</em></strong>.</p><p>That itch became <strong>ArthJAX</strong>.</p><h3>A weekend idea that refused to stay small</h3><p>It started the way many side projects do: one Jupyter cell, then another, then a gnarly question — <strong>can I get 250 households and 60 firms to interact on a GPU without nested Python loops?</strong></p><p>The answer was JAX.</p><p>Not “<strong>import jax for vibes.</strong>” Real JAX: vectorized agents, `<em>jit</em>`-compiled timesteps, the entire future of the economy folded into a single `<em>lax.scan</em>` like a film strip through time.</p><p>On paper it’s a synthetic toy world. In practice it’s a living system:</p><ul><li><strong>Households</strong> that don’t all think alike, value investors, momentum chasers, panic sellers, savers sitting out the drama</li><li><strong>Firms</strong> borrowing, producing, bleeding when demand dries up</li><li><strong>Banks</strong> watching bad loans creep up like rust</li><li><strong>Markets</strong> with volatility that clusters the way real markets do, calm, calm, then suddenly not</li><li>A <strong>central bank</strong> feeling its way through inflation and unemployment with a Taylor rule</li><li><strong>Shocks: </strong>oil, credit, demand, policy, that arrive like weather, not like a developer pressing a “<strong>crash</strong>” button</li><li><strong>Contagion</strong> crawling through a web of sectors with memory, so pain in one corner shows up somewhere else three steps later</li></ul><p>I didn’t script the crisis at timestep 300.</p><p>I watched one show up anyway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*miGi4nZpQwzwWLbDBdTd6g.png" /><figcaption><em>Credit stress spikes exactly when GDP steps down — no “crash at t=300” in the code. Emergent boom, bust from 250 agents.</em></figcaption></figure><p>That’s the moment ArthJAX stopped being a notebook and became a project worth open-sourcing.</p><h3>Six hundred heartbeats in seconds</h3><p>Here’s the part that still makes me grin.</p><blockquote><strong>600 macro timesteps. Seconds on a GPU.</strong></blockquote><p>Not a metaphor. Not “<strong>fast enough for research.</strong>” Fast enough that you start <strong><em>playing, </em></strong>rerunning seeds, nudging shock probabilities, asking “<strong>what if savers panic earlier?</strong>” like you’d restart a level.</p><p>The engine underneath is deliberately unglamorous engineering:</p><ul><li>State lives in a structured PyTree, wealth, balance sheets, bank books, sentiment, contagion buffers</li><li>Every agent update is vectorized across the population</li><li>One compiled step function chains households → firms → banks → markets → shocks → contagion → macro feedback</li></ul><p>- Time rolls forward through `<em>lax.scan</em>`, the JAX idiom for “<strong>loop without looping</strong>”</p><p>It’s the kind of code that looks boring in a README and feels like magic when you hit Run All on Kaggle and charts appear before your coffee cools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jgfcPbvNnqlYCKupVMqVsA.png" /><figcaption><em>600 timesteps of synthetic macro: GDP, inflation, unemployment, rates, sentiment, credit stress, all from one JAX </em><em>lax.scan run.</em></figcaption></figure><h3>Then I tried to teach a neural net to dream the economy</h3><p>Simulators are honest and slow-ish at scale.</p><p>Neural world models are fast and sometimes delusional.</p><p>So I did the obvious reckless thing: train a small MLP on state transitions from thousands of rollout steps and ask it to predict the next economy.</p><p>The first attempt <strong>exploded into NaN</strong>.</p><p>Not metaphorically. Actual `<em>nan</em>` metrics. Beautiful charts of nothing.</p><p>The bug was almost insultingly simple: household wealth and bank deposits live on wildly different scales. Feed a network raw states spanning 1 to 10,000+ and it learns to hallucinate in scientific notation.</p><blockquote>Normalize. Clip gradients. Roll out in normalized space. Decode back to reality.</blockquote><p>Loss curves settled. Comparisons became possible. The world model doesn’t replace the simulator, it’s a sketch of the simulator, a fast imperfect twin.</p><p>Today macro variables land around <strong><em>40–45% rollout error</em></strong>. Not victory. Not embarrassment either. A frontier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zCqElGfFjXdByfywbezlHA.png" /><figcaption><em>World model training on normalized states. Without z-score normalization this was all NaN, with it, loss converges.</em></figcaption></figure><p>The honest line I’ll carry into every post about this project:</p><p>ArthJAX doesn’t forecast your country’s GDP. It gives you a <strong>rehearsal stage</strong> for dynamics.</p><h3>What it looks like when it works</h3><p>Picture a dark-themed dashboard, the kind of chart you’d put in a deck, not a homework PDF.</p><p>GDP crawling, then stepping down when credit stress wakes up.</p><p>Inflation and unemployment trading places on a Phillips scatter.</p><p>Sentiment sliding from optimism into something colder.</p><p>A learned trajectory trying to chase the real simulator like a student copying a teacher’s handwriting.</p><p>None of this is “<strong>the economy.</strong>”</p><p>All of it is <strong>a system behaving like an economy might</strong> when agents interact under pressure.</p><p>That gap, between toy and truth, is where the interesting science lives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BYX35PFmJF7NwTRsmjTEvA.png" /><figcaption><em>Simulator (solid) vs learned world model (dashed) over 200 steps. Macro error ~40–45% today, promising sketch, not a forecast engine.</em></figcaption></figure><h3><strong>Try it without installing anything</strong></h3><p>I care about reproducibility more than hype, so the first thing I shipped after the GitHub repo was a <strong>one-click Kaggle notebook</strong>:</p><p>[<a href="https://www.kaggle.com/code/varunraosfanlkan/arthjax-gpu-macro-abm-world-model">ArthJAX GPU Macro ABM + World Model</a>]</p><p>Open it. Turn on GPU if prompted. Hit <strong>Run All</strong>.</p><p>You’ll get the simulation, the charts, the world-model training curve, the real-vs-learned comparison, exported to your Kaggle output folder. No CUDA install rabbit hole on your laptop. No “<strong>works on my machine</strong>” energy.</p><p><strong>GitHub:</strong> <a href="https://github.com/VARUN3WARE/ArthJAX">https://github.com/VARUN3WARE/ArthJAX</a></p><h3>Why open source this now</h3><p>Because the interesting questions are bigger than my weekends.</p><p>What if credit networks get richer?</p><p>What if policy search runs thousands of counterfactuals through a learned world model before touching the real simulator?</p><p>What if stylized facts, volatility clustering, fat tails, Phillips slopes, become pass/fail gates instead of vibes?</p><p>ArthJAX is Early. Public. Imperfect on purpose.</p><p>Next on the roadmap: named crisis scenarios, benchmark suites, a world model that earns its keep below 25% macro error.</p><p>If you work where <strong>JAX</strong>, <strong>agent-based models</strong>, and <strong>macro ML</strong> overlap, I’d rather build the next chapter with company than applause.</p><p>Star the repo. Break the notebook. Open an issue that makes me rethink an assumption.</p><p>That’s how toy economies grow up.</p><h3>A note on humility (read this before quoting me)</h3><p>ArthJAX is a <strong>synthetic sandbox</strong>, not an oracle.</p><p>Don’t trade on it. Don’t brief a central bank with it. Do use it to <strong>test intuition</strong>, <strong>teach dynamics</strong>, and <strong>prototype methods</strong> that might one day matter in heavier, calibrated systems.</p><p>The goal isn’t to pretend we solved macro.</p><p>The goal is to make the rehearsal cheap enough that more people can afford to ask dangerous questions safely.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9QTaaaxboQP_gKSLedW_w.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://www.linkedin.com/in/bhumika-kc-3784391b9/?isSelfProfile=true"><strong>Linkedin</strong></a>| <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p>👉 CodeToDeploy Tech Community is live on Discord — <a href="https://discord.gg/ZpwhHq6D"><strong>Join now!</strong></a></p><p><strong>Disclosure:</strong> This post includes affiliate and partnership links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3811a6ca5f92" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/i-simulated-250-agents-on-a-gpu-and-watched-a-financial-crisis-emerge-without-writing-crash-now-3811a6ca5f92">I Simulated 250 Agents on a GPU and Watched a Financial Crisis Emerge, Without Writing “Crash Now”</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PagedAttention: Taught So Well You Can Yap About It]]></title>
            <link>https://medium.com/@varunrao.aiml/pagedattention-taught-so-well-you-can-yap-about-it-21e0f7f748df?source=rss-bc5d9b74a94e------2</link>
            <guid isPermaLink="false">https://medium.com/p/21e0f7f748df</guid>
            <category><![CDATA[pagedattention]]></category>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[paging]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <dc:creator><![CDATA[varun rao]]></dc:creator>
            <pubDate>Wed, 17 Jun 2026 09:50:01 GMT</pubDate>
            <atom:updated>2026-06-17T09:50:01.189Z</atom:updated>
            <content:encoded><![CDATA[<p>As transformer models continue to push the boundaries of AI capabilities, efficiency and scalability are becoming increasingly important. Large-scale models, such as GPT-3, BERT, and other transformer-based architectures, have made a profound impact across a wide range of applications, from natural language processing (NLP) and computer vision to multimodal systems. However, the success of these models comes with a significant cost: the attention mechanism, which lies at the heart of transformers, is notoriously memory and computation-intensive. The larger the sequence, the larger the memory requirements, and the more computational power it takes to process each token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/0*FF3ZCQ0-rXiv7Gqs" /></figure><p>Enter PagedAttention. If you’re not familiar with this technique, it might sound like another fancy optimization trick to improve model performance. But in reality, PagedAttention is a game-changer that significantly enhances transformer models’ ability to handle long sequences with memory-efficient computations and faster inference speeds.</p><p>In this blog, we will break down PagedAttention, what it is, how it works, and why it matters. We’ll explore its connections to other transformer optimizations, dive into the deeper research behind the technique, and discuss how PagedAttention is being integrated into large-scale systems like vLLM to power the next generation of AI models.</p><h3>1. The Challenge of Transformer Attention</h3><p>Before diving into the specifics of PagedAttention, it’s crucial to understand the problem it solves. Transformers rely on an attention mechanism that allows the model to weigh different parts of the input sequence when making predictions. This self-attention mechanism computes relationships between each pair of tokens in the sequence, which helps the model capture context and dependencies between words, phrases, or pixels.</p><p>While the attention mechanism works wonders, it also comes with a major downside: quadratic complexity. Specifically, the attention mechanism computes interactions between every token and every other token in the sequence. This means that for a sequence of length N, the attention matrix will have a size of N x N, resulting in a quadratic increase in both memory and computation costs. For long sequence, think entire paragraphs, lengthy documents, or high-resolution images, this becomes a serious bottleneck. The larger the model and the sequence, the more memory is required to store the attention matrix, and the slower the computations become.</p><h4>Memory Bottleneck in Large-Scale Transformers</h4><p>For large-scale transformers (like GPT-3), this problem is even more pronounced. The number of parameters in such models can be in the billions, and they require substantial GPU memory to perform attention calculations across long input sequences. As these models scale, the memory footprint grows exponentially, leading to out-of-memory errors or severe slowdowns in inference. This is where PagedAttention comes in.</p><h3>2. PagedAttention: A Memory-Efficient Solution</h3><p>PagedAttention is a memory optimization technique that breaks the input sequence into smaller chunks or “pages” rather than processing the entire attention matrix at once. Instead of calculating attention over the entire sequence at once, PagedAttention computes attention over smaller subsets of tokens, and only loads relevant tokens into memory when needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jgiC9f1MZlAteBZK.png" /></figure><h4>How Does PagedAttention Work?</h4><p>At a high level, <strong>PagedAttention</strong> works by treating the attention matrix as a series of <strong>pages</strong> that can be swapped in and out of memory as needed. Each page contains a smaller subset of tokens from the sequence, and attention computations are performed on each page independently.</p><p>1. <strong>Paging Mechanism</strong>: The attention matrix is split into smaller “pages.” Instead of processing all tokens at once, the model only processes a subset (or page) of tokens at a time. This reduces the amount of memory required for each operation and allows for much larger sequences to be processed.</p><p>2. <strong>Efficient Memory Use</strong>: By dividing the attention matrix into pages, only a small portion of the data needs to reside in memory at any given time. This paging mechanism helps alleviate memory pressure, enabling the model to handle long sequences that would otherwise be infeasible.</p><p>3. <strong>Parallelization</strong>: As one page of data is being processed, the next page can be preloaded into memory asynchronously, reducing idle time. This leads to significant speed-ups, especially when processing long sequences.</p><h4>Why PagedAttention is Game-Changing?</h4><p>The core advantage of <strong>PagedAttention</strong> is its ability to handle long input sequences efficiently. By breaking down the attention mechanism into smaller, more manageable chunks, it significantly reduces the memory requirements, making it possible to run large models on hardware with limited resources.</p><ul><li><strong>Improved Inference Speed</strong>: Because <em>PagedAttention</em> processes smaller chunks at a time, the overall computation is faster compared to traditional methods that calculate attention across the entire sequence.</li><li><strong>Lower Memory Footprint</strong>: With fewer tokens in memory at once, <em>PagedAttention</em> reduces the total memory required during inference, making it possible to run large models on GPUs or TPUs with less memory.</li><li><strong>Scalability</strong>: <em>PagedAttention</em> allows transformer models to scale more effectively, enabling them to handle sequences of arbitrary length without running into memory bottlenecks.</li></ul><h3>3. PagedAttention and Transformer Optimization Research</h3><p>While <em>PagedAttention</em> is a powerful optimization, it is part of a broader effort in the machine learning community to optimize transformer models for large-scale use. Several other techniques and research areas are connected to <em>PagedAttention</em> and contribute to its effectiveness.</p><h4>Sparse Attention (e.g., Longformer, BigBird)</h4><p>One related optimization is <em>sparse attention</em>, where attention is not computed between every token, but only between a subset of relevant tokens. Models like <em>Longformer</em> and <em>BigBird</em> use <em>sliding window attention</em> or <em>global attention</em> to focus on local and important regions of the sequence, reducing the quadratic complexity of traditional attention. <em>PagedAttention</em> can complement sparse attention by further dividing the sequence into pages, allowing memory to be used more efficiently and enabling better performance on longer sequences.</p><h4>Memory-Augmented Networks (MANNs)</h4><p>Memory-augmented neural networks, such as <em>Neural Turing Machines (NTMs)</em> and <em>Differentiable Neural Computers (DNCs)</em>, use external memory to store and retrieve data, extending the capacity of the network. <em>PagedAttention</em> takes a similar approach by breaking the attention matrix into chunks that can be loaded and swapped in memory, allowing for more efficient use of external memory resources.</p><h4>Low-Rank Approximations (e.g., Linformer)</h4><p><em>Low-rank approximations</em> of the attention matrix, as seen in models like <em>Linformer</em>, seek to reduce the size of the attention matrix by approximating the full matrix with a smaller one. <em>PagedAttention</em> doesn’t directly approximate the attention matrix but reduces the memory load by managing the matrix through paging. However, combining <em>PagedAttention</em> with <em>low-rank approximations</em> could lead to even greater memory savings and faster computations.</p><h4>Hierarchical Attention (e.g., ViTs)</h4><p>In <em>Vision Transformers (ViTs)</em> and multimodal transformers, the attention mechanism often has to handle high-dimensional data. These models use <em>hierarchical attention</em> to first perform attention on coarse-grained representations (e.g., patches in images) and then fine-grain attention on local details. <em>PagedAttention</em> can optimize this further by paginating attention computations across multiple levels, allowing for more memory-efficient handling of both global and local dependencies in high-dimensional data.</p><h4>Asynchronous Paging and Parallelism</h4><p>Research into <em>asynchronous paging</em> and parallel computing techniques can further enhance the speed of <em>PagedAttention</em>. By loading and processing multiple pages in parallel, systems can reduce idle time and maximize throughput, especially in multi-GPU or distributed settings.</p><h3>4. Real-World Applications of PagedAttention</h3><h4>vLLM: A Case Study in Efficient Inference</h4><p>One of the standout applications of <em>PagedAttention</em> is in vLLM, a framework designed for efficient inference with large language models. vLLM integrates PagedAttention to enable large transformer models to perform inference at scale, with reduced memory usage and faster processing times.</p><ul><li><strong>Speed Gains</strong>: <em>PagedAttention</em> has been shown to achieve significant speed-ups in vLLM, reducing inference times without compromising model performance.</li><li><strong>Memory Optimization</strong>: By using <em>PagedAttention</em>, <em>vLLM</em> can handle larger batch sizes and process longer sequences than traditional transformers, which would otherwise be limited by memory constraints.</li></ul><h4>Autoregressive and Real-Time Systems</h4><p>In <em>autoregressive</em> models, such as GPT for text generation or video generation, <em>PagedAttention</em> is particularly useful for real-time applications. As the model generates new tokens, it needs to maintain context from previous tokens. <em>PagedAttention</em> optimizes memory usage by caching frequently accessed tokens while paging out less relevant data, allowing the model to operate with lower latency.</p><h4>Multimodal Models (e.g., CLIP, DALL·E)</h4><p>Multimodal models, such as <em>CLIP</em> and <em>DALL·E</em>, process both text and images simultaneously. These models face a challenge in managing large inputs across different modalities. <em>PagedAttention</em> can optimize this by paginating attention matrices for both text and image tokens, ensuring that memory is used efficiently across both data types.</p><h3>5. Future Prospects and Challenges</h3><h4>The Quest for More Efficient AI Models</h4><p>As AI systems continue to scale, the need for memory-efficient techniques like <em>PagedAttention</em> will only grow. The next frontier in <em>PagedAttention</em> research could involve <em>dynamic paging strategies</em>, where the size of each page is adjusted based on the context of the input sequence or the computational load. <em>Hierarchical paging</em>, where sequences are divided based on their internal structure, could allow even greater efficiency.</p><h4>Challenges in Integration</h4><p>While <em>PagedAttention</em> provides clear benefits in terms of speed and memory optimization, it comes with challenges. For example, there may be trade-offs between <em>page size</em> and <em>computation efficiency</em>, and managing the <em>asynchronous loading of pages</em> in large-scale systems can introduce synchronization issues.</p><p>Moreover, not all models will benefit equally from <em>PagedAttention, </em>it may be more advantageous for certain architectures or tasks. Therefore, understanding when and how to apply <em>PagedAttention</em> is key for developers and researchers looking to optimize transformer models.</p><h3>Conclusion</h3><p><em>PagedAttention</em> is an essential innovation that unlocks the ability to process <em>longer sequences</em> efficiently by breaking down the attention matrix into smaller chunks. By reducing memory usage and improving computation speed, it allows <em>transformer models</em> to scale effectively, even on hardware with limited resources. When integrated into systems like <em>vLLM</em>, it powers the next generation of AI models that are <em>faster</em>, <em>more memory-efficient</em>, and <em>more scalable</em> than ever before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g3rx5v77Xgi5ZJey.jpg" /></figure><p>By combining <em>PagedAttention</em> with other optimizations, such as sparse attention, low-rank approximations, and asynchronous paging, transformers are set to evolve into even more powerful tools that can handle larger datasets, longer sequences, and real-time applications without breaking the bank in terms of computational resources.</p><p>In short, <em>PagedAttention</em> is <em>one of the most important optimizations</em> you’ve likely never heard of, and now, you know just enough to yap about it confidently. It’s the unsung hero in the race for faster, more efficient AI. And as the field continues to evolve, <em>PagedAttention</em> will likely be at the heart of the next big breakthrough in <em>large-scale transformers</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21e0f7f748df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FAISS vs Qdrant: The Brutal Scaling Reality]]></title>
            <link>https://medium.com/@varunrao.aiml/faiss-vs-qdrant-the-brutal-scaling-reality-d9fbb20782c8?source=rss-bc5d9b74a94e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9fbb20782c8</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[vector-search]]></category>
            <category><![CDATA[qdrant]]></category>
            <category><![CDATA[faiss]]></category>
            <category><![CDATA[vector-search-engine]]></category>
            <dc:creator><![CDATA[varun rao]]></dc:creator>
            <pubDate>Wed, 17 Jun 2026 09:21:34 GMT</pubDate>
            <atom:updated>2026-06-17T09:21:34.850Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>What nobody tells you when your prototype meets production</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*Bq8uzSmWG6yPTwyA3REYZw.png" /></figure><p>There is a specific moment that happens to almost every engineer building retrieval systems. The prototype is clean. FAISS is humming. You are searching five million vectors in under ten milliseconds and everything feels like it is working exactly the way the papers promised. Then you push to production.</p><p>Suddenly you need metadata filtering. You need to delete stale chunks when a document updates. You need multiple tenants to not see each other’s data. You need the index to survive a restart. You need monitoring. You need an API that does not require a PhD to operate.</p><p>None of that is a vector similarity problem. All of it is an infrastructure problem. And that distinction is the entire thesis of this article.</p><h3>What These Tools Actually Are</h3><p>This sounds obvious but it is worth stating precisely because the confusion here is what causes teams to build the wrong thing.</p><p>FAISS is a library for efficient similarity search and dense vector clustering. Facebook AI Research built it to solve one problem extremely well: given a query vector, find the nearest neighbors in a large corpus as fast as possible. It runs in-process. It has no server. It has no API. It has no concept of tenants, permissions, metadata, or durability. It is a collection of algorithms, mostly written in C++ with a Python interface, that does one thing with extraordinary competence.</p><p>Qdrant is a vector database. It has a server. It has an API. It handles persistence natively. It supports payload filtering, collection management, distributed deployment, and horizontal scaling. Under the hood it uses HNSW for approximate nearest neighbor search, which means the raw search quality is comparable to FAISS’s HNSW index. But the entire layer above that algorithm is what FAISS simply does not have.</p><p>When engineers compare these two things, they are often comparing them on the wrong axis. The question is not which one searches faster. The question is what problem you are actually trying to solve.</p><h3>Why FAISS Feels Incredible at First</h3><p>If you have never used FAISS, the experience of running it for the first time on a reasonably sized dataset is genuinely impressive. The API is minimal. The latency numbers are absurd. You can load two million vectors into an IVF-PQ index, run a thousand queries per second, and your laptop barely notices.</p><p>For research, for offline retrieval tasks, for building intuition about embedding spaces, FAISS is close to perfect. The index types it supports cover an enormous range of use cases. IVF for coarse quantization. PQ for memory compression. HNSW for graph-based search. IVFFlat for exact inner-product search over partitions. GPU indexes for when you want to push throughput to its physical limit. The documentation is honest about the tradeoffs.</p><p>The problem starts when you try to run this in a system that has users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wz9qFtDVZg4DwNoK" /></figure><h3>The Prototype to Production Gap</h3><p>Here is what the gap looks like in concrete terms.</p><p>Your prototype does this:</p><pre>import faiss<br>index = faiss.IndexFlatL2(768)<br>index.add(embeddings)<br>D, I = index.search(query_embedding, k=10)</pre><p>That works. It is fast. The results are correct.</p><p>Your production system needs to do this:</p><p>Retrieve only the documents belonging to tenant A. Filter to documents created in the last 90 days. Exclude document types that the current user does not have permission to see. Handle the case where document 4821 was updated and its old chunks need to be replaced. Serve this across three replicas. Survive a deployment restart without re-indexing. Expose a health check endpoint. Log query latency by tenant. Handle fifty concurrent requests without corrupting the index.</p><p>FAISS handles none of that. Not because it is a bad tool, but because that is explicitly not what it was built for.</p><p>What teams end up doing is building a wrapper system around FAISS. You add a PostgreSQL or Redis layer for metadata. You write custom filtering logic that post-processes FAISS results and applies your business rules. You build a persistence service that serializes the index to disk and reloads it on startup. You write replication logic. You add an API layer.</p><p>At some point you look at what you have built and realize you have accidentally constructed a vector database. A worse one, because it is held together by your own code and not battle-tested by a community.</p><h3>Performance: Where the Nuance Lives</h3><p>Benchmarks on this topic are everywhere and most of them are misleading. Let me be precise about what FAISS actually wins at.</p><p>FAISS wins on raw ANN search speed, particularly with GPU acceleration. Its GPU indexes are genuinely in a different performance class. If you have a single-node system, a GPU, and your only metric is query latency without filtering, FAISS IVFFlat or FAISS HNSW will likely beat Qdrant in a head-to-head comparison. The C++ implementation has been optimized over years for exactly this workload.</p><p>FAISS also wins on memory efficiency with quantization. PQ (Product Quantization) and SQ8 (Scalar Quantization) in FAISS are mature and well-tested. You can compress a billion vectors into something that fits in reasonable RAM.</p><p>But benchmarks almost never reflect production workloads because production workloads have filters.</p><p>The moment you add a metadata filter to your query, the performance picture changes completely. In FAISS, you cannot pre-filter efficiently. You search broadly, retrieve candidates, then apply filters in post-processing. If your filter is selective (say, only 2% of your corpus matches), you might need to retrieve k=500 candidates to get k=10 relevant results after filtering. Your effective k jumps by a factor of 50. Your latency follows.</p><p>Qdrant handles this with payload indexing. You create an index on your metadata fields and the filter is applied during traversal, not after. For selective filters in production RAG workloads, this ends up being significantly faster in practice even if the raw unfiltered search is slower.</p><p>This is the benchmark misleading engineers problem. Top-k latency on random vectors with no filter tells you almost nothing about how these systems will perform in a real retrieval pipeline.</p><h3>Metadata Filtering: The Real Dividing Line</h3><p>Every production RAG system I have studied or worked on has at least some of the following requirements:</p><p>Tenant isolation, so that user A’s documents are never returned in user B’s queries. Time-based filtering, so that retrieval can be scoped to recent content. Permission scoping, so that draft documents are not returned to unauthorized users. Document type filtering, so that you can search only over code files, or only over meeting transcripts. Source filtering, so that you can prioritize certain data sources.</p><p>This is business logic layered onto vector search. It is completely normal and completely expected in any real application.</p><p>The engineering cost of implementing this in FAISS is non-trivial. You are writing custom code that lives outside the index and you have to maintain it. The performance characteristics of that custom code under concurrent load, with large corpora, are your responsibility.</p><p>In Qdrant, this is the product. You define payloads when you upsert vectors. You create payload indexes. You pass filters in your search request. The system handles the rest.</p><p>The moment your retrieval system has business logic, FAISS alone starts feeling like the wrong layer of abstraction.</p><h3>Operational Complexity and What It Actually Costs</h3><p>There is a cost accounting problem that teams often get wrong when comparing these tools.</p><p>FAISS infrastructure cost: low. It is a library. No servers to run, no infrastructure to provision.</p><p>FAISS engineering cost: high, and it scales with your requirements. Every operational feature you need — persistence, filtering, replication, monitoring, multi-tenancy — requires custom code. That code needs to be written, tested, deployed, and maintained by someone on your team.</p><p>Qdrant infrastructure cost: moderate. You are running a database server. There is operational overhead that comes with that.</p><p>Qdrant engineering cost: low for standard retrieval patterns. The features you need for production are already built and already tested by the open source community.</p><p>For a research prototype or an offline pipeline, FAISS infrastructure cost with near-zero engineering cost is the right call. For a product serving users, the Qdrant tradeoff is almost always favorable.</p><p>The mistake is accounting only for infrastructure cost and ignoring engineering cost. Engineering time is expensive. Maintenance burden is expensive. On-call incidents caused by a custom persistence layer at 3am are expensive.</p><h3>Why Benchmarks Mislead Engineers</h3><p>Standard vector database benchmarks measure: query latency at k=10 over a static corpus with no filters and no concurrent writes.</p><p>Production retrieval systems have: selective metadata filters that change result cardinality dramatically, concurrent reads and writes at the same time, index updates from document ingestion pipelines, multi-tenant query patterns with different filter distributions, payload fields of varying selectivity, and restart/recovery behavior that matters to your uptime SLA.</p><p>The delta between what benchmarks measure and what production systems experience is large. I am not saying benchmarks are useless. They give you order-of-magnitude intuition. But I have seen teams choose infrastructure based on benchmark numbers and then spend months dealing with problems that the benchmarks were never designed to surface.</p><p>If you are benchmarking for a real decision, benchmark your actual query distribution with your actual filter selectivity on your actual corpus size. That result is the one that matters.</p><h3>Who Should Use What</h3><p>FAISS is the right choice when you are doing research, running offline retrieval pipelines, building ML systems where the vector search is one component among many and you need tight control over the algorithm, working in GPU-heavy environments where you want to push throughput to physical limits, operating at a scale where the operational overhead of a database server is genuinely not worth it, or building something where your retrieval logic does not require the business logic layer.</p><p>Qdrant is the right choice when you are building a product that serves users, when your retrieval system needs filtering of any kind, when you care about uptime and durability, when you are operating in a multi-tenant environment, when you want to iterate on your retrieval system without also maintaining infrastructure code, or when your team’s time is better spent on the product than on the persistence layer.</p><p>There is a real version of this decision that is not obvious: teams running offline batch retrieval for ML pipelines, where FAISS with GPU acceleration is genuinely the better tool. There is also a real version where a team thinks they are in that category and discovers three months later that their offline pipeline has become a user-facing feature. Know which one you are building before you commit.</p><h3>The Honest Ending</h3><p>Most teams do not outgrow FAISS because their vector count gets too large. FAISS can handle billions of vectors with the right index configuration. They outgrow it because production retrieval systems are infrastructure problems, not similarity-search problems. And FAISS was never designed to be infrastructure.</p><p>That is not a criticism of FAISS. It is one of the most important pieces of open-source software in the ML ecosystem. The research it enabled, the systems built on top of it, the community understanding of approximate nearest neighbor search that it helped create — all of that is genuinely valuable.</p><p>But there is a version of using FAISS in production where you end up maintaining more code than you would have if you had just used a purpose-built vector database from the start. Recognizing which version you are building is the skill.</p><p><em>The benchmarks referenced in this article are reproduced in detail in the companion Kaggle notebook, which covers raw search speed, filtered retrieval, ingestion throughput, persistence behavior, and operational complexity under production-like workloads.</em></p><p><a href="https://www.kaggle.com/code/varunraosfanlkan/faiss-vs-qdrant">https://www.kaggle.com/code/varunraosfanlkan/faiss-vs-qdrant</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9fbb20782c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agents Are Failing Quietly in Production ,And Nobody Wants to Admit It]]></title>
            <link>https://medium.com/@varunrao.aiml/agents-are-failing-quietly-in-production-and-nobody-wants-to-admit-it-6ad37f729671?source=rss-bc5d9b74a94e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ad37f729671</guid>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[production-agent]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai-in-production]]></category>
            <dc:creator><![CDATA[varun rao]]></dc:creator>
            <pubDate>Sun, 22 Feb 2026 20:06:24 GMT</pubDate>
            <atom:updated>2026-02-22T20:06:24.664Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The demos are impressive. The production logs are a different story.</em></p><p>Months ago, I started diving into how <strong>big tech</strong> runs its software. Fixing my own apps after crashes taught me more than <strong>lectures</strong> ever did. Reading postmortems late into the night showed something odd, everyone skips over one key truth. Talks at conferences dance around it too. The silence feels strange, given how often it comes up in <strong>logs, errors, crash reports.</strong> Still, almost no one names what’s really going wrong when models fail under pressure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*x7_rtRNzKhK0j-VC" /></figure><p>Quietly, AI agents crumble once they’re live. Every day, errors slip through unseen. Those who deploy them quietly hope you won’t catch on.<br>This post? It’s <strong>fragments of understanding</strong> gathered slowly, through papers, yes, but also half-baked GitHub threads, those few real post-mortems that don’t dodge blame, plus personal attempts that veered off course. Engineers who’ve walked this path: maybe share your messy moments down below.</p><h3>The Gap Between the Demo and the Dashboard</h3><p>A single task lands in front of the model, it picks what to use, lines up each step without stumbling. Success appears, neat and quiet. People clap. Someone grabs a picture. That image floats into a post online.<br>Six weeks down the line, live traffic shows what happens when an agent hits a snag and a tool spits out broken data. It takes that junk at face value. Trusts its own logic even though the foundation is flawed. Ends up handing someone a firm reply full of errors. Nothing stops it. Not a warning. Not a single alarm goes off.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*SW0O6iD8X4PcEV0P.png" /></figure><p>That’s the gap. And it’s enormous.</p><p>A machine built to speak without pause runs into trouble once it can act. Trained to fill silence, to push words out, to keep going no matter what, that habit works fine in conversation. Yet when such a system gains access to tools, its urge to proceed turns risky. Now every step forward isn’t just talk. Each motion reaches beyond screen and code.<br>Four ways things break, again and again, seen even in setups made by those who should know better.</p><h3>1. Tool-Call Instability</h3><p>One day, each thinking machine gathers its own helpers, ways to look things up, run scripts, pull data, reach services. It picks one when needed, sets how it runs, feeds it details, then uses whatever comes back. Choice flows moment by moment, shaped by need, guided by outcome, never fixed.</p><p>In theory, clean. In practice, deeply fragile.</p><p><strong>One issue stands out first:</strong> made-up details in function calls. Instead of correct inputs, the system uses fake ones that seem believable. Take a tool needing exact place names, when given “San Fransisco, CA,” which looks right but isn’t valid, problems start. Often, the tool does nothing noticeable or brings back zero data. Yet the agent acts like it got normal output. Without pausing to check, it moves ahead, building next steps on silence. Mistakes slip through because no one stops to question them.</p><p><strong>The second problem is schema drift.</strong> Here comes trouble when the structure shifts unexpectedly. Your tool upgrades its API interface one day. A label might shift names, for instance. Sometimes a setting switches to mandatory status overnight. That learning phase used last season’s layout though. What worked before now misfires without warning. A single misstep creeps into each use of the tool now. Errors slip through unseen without tight checks and clear alerts — something most systems lack entirely.</p><p><strong>Here comes trouble number three — tricky to spot.</strong> Pose one query to an agent, then repeat it hours later. First go round, it grabs user details through get_user_profile. Next attempt? It picks search_users instead. Identical goal, separate paths, mismatched outputs, uneven speed each time. These choices wobble because agents do not lock into fixed responses. When their tool pick shifts, so does how the whole setup runs.</p><pre># What you think is happening:<br>agent.run(&quot;find the last order for user 4821&quot;)<br>→ calls get_order_history(user_id=4821)<br>→ returns last order<br>→ done</pre><pre># What sometimes happens:<br>agent.run(&quot;find the last order for user 4821&quot;)<br>→ calls search_all_orders(query=&quot;user 4821 recent&quot;)<br>→ returns vague semantic results<br>→ agent picks the wrong one<br>→ done (confidently wrong)</pre><blockquote><strong><em>War story I’ve seen :</em></strong><em> Say, An agent was tasked with sending some of follow-up emails by an entity. It had simply two tools: </em><em>send_email and </em><em>draft_email, everything was cool and simple.But eventually observed on about 3% of runs, it just called </em><em>send_email even when the intent was clearly to draft. Funny part was that, No one caught it for almost two weeks because the success rate looked fine in aggregate.</em></blockquote><h3>2. Infinite Loops (The Ones That Don’t Look Like Loops)</h3><p>Classic infinite loops are easy to catch. while True with no break, your CPU pegs, your logs fill up, someone notices.</p><p>What seems like progress might not be. Tools get triggered, words appear, status shifts, yet completion stays out of reach. Sometimes it stops, checks its own rules, finds a gap, then restarts without warning. Other times, it slips into a quiet loop: decide, act, fail to recognize the outcome, rethink, repeat the step with tiny tweaks, hit the same wall, rethink again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2BqloQscMtZlNlFu.jpeg" /></figure><p>This occurs since agents treat their previous outputs as groundwork for what comes next. Should the model latch onto a wrong idea at the start, imagine thinking a document is missing even though it exists, later steps build on that error like layers on stone. Each fresh attempt gets shaped by that first flawed assumption without question. With every tool used, the perspective stays warped. It continues looking, failing to locate something it already assumes gone, pushing forward regardless.</p><pre># Iteration 1: &quot;Let me check the inventory&quot;<br>→ tool: check_inventory(item=&quot;widget-A&quot;)<br>→ result: {&quot;count&quot;: 0}  ← (actually a bug in the tool, item exists)<br># Iteration 2: &quot;That&#39;s wrong, let me try differently&quot;<br>→ tool: search_catalog(query=&quot;widget A&quot;)<br>→ result: found item, but different schema<br># Iteration 3: &quot;The IDs don&#39;t match, let me reconcile&quot;<br>→ tool: check_inventory(item=&quot;Widget-A&quot;)  ← capital W this time<br>→ result: {&quot;count&quot;: 0}<br># ... 47 more iterations ...<br># Token limit hit. Task marked &quot;failed.&quot;<br># $4.20 spent. Nothing accomplished.</pre><p>Here is the hard truth: <strong>spotting loops must happen outside the model, because it won’t notice on its own.</strong> It keeps moving forward in its mind, even when stuck. Watch for patterns like revisiting identical states or repeating steps too often. Tools such as step tracking or action logging belong in the system running the process, not inside the request given to the model.</p><p>Aein, max_iterations=20 hard cap is a bare minimum. It&#39;s not enough although , but it&#39;s a start.</p><h3>3. Latency Explosions</h3><p>Single LLM call: ~1–3 seconds. Totally acceptable.</p><p>Starting slow, one agent fires four separate LLM requests — each triggers two tool uses, every tool pinging outside APIs; suddenly you’re waiting half a minute to nearly a full minute and a half. Could stretch longer. All of this assuming things go smoothly.</p><p>Folks notice right away when answers drag on. What feels like a glitch one afternoon could stretch into nearly a minute later — <strong>same query, tiny shifts behind the scenes.</strong> That lag doesn’t follow rules; it wanders. Eight seconds today, forty-five tomorrow, no pattern to hold onto. Promising reliability? Impossible. Trying to justify the wait? Even harder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*yyLpGR16zEW83shu" /></figure><p>What hurts most isn’t one failure. It’s what follows. Tool A drags on. Silence stretches while the agent just sits. Then timeout hits. Retry begins — same request? Not quite. The prompt does not remember past attempts. So maybe now it picks Tool B for the very same info. Two sluggish tools churning at once. That happens quietly. One finally answers first. Relief fades fast because the second still shows up late. Now there are two versions of truth sitting across from each other. Which do you trust? Doesn’t matter. Either way, more steps come next. More rounds with the model. Each one adding weight to the delay.</p><pre>Timeline of a &quot;simple&quot; agent task:<br>00:00 — Task received<br>00:01 — LLM decides plan (1 call)<br>00:03 — Calls web_search()<br>00:09 — search returns (slow)<br>00:10 — LLM processes results (1 call)<br>00:12 — Calls database_query()<br>00:14 — DB times out<br>00:14 — LLM retries with different query (1 call)<br>00:17 — Second query also slow<br>00:24 — Result returns, but first query also finally returns<br>00:25 — LLM resolves conflict (1 call)<br>00:28 — Final answer generated<br><br>Total: 28 seconds. 4 LLM calls. 3 tool calls.<br>User sees: a spinner for half a minute.</pre><p>Fixes people actually use: <strong>aggressive timeouts</strong> on every tool, <strong>parallel tool calling</strong> ,and <strong>caching tool results.</strong></p><h3>4. Hidden Hallucinations in Tool Outputs</h3><p>This one is the most insidious, and the least discussed.</p><p>It’s true. Conversations often circle back to how LLMs invent answers. Most people think that happens just because the system lacks correct information. So what now? Connect it to live sources instead. Pull in actual databases. Link real-time services directly.</p><p>Truth is, nobody talks much about this bit: <strong>the model might invent what it thinks a tool found</strong>. The information itself? <strong>Genuine</strong>. What the model says about it? Not always true</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YCP13CULGSU6mN8b.png" /></figure><p>Out of tools arrive plain text, sometimes JSON, occasionally neat packages of info. A model must untangle it, shrink it down, then make sense of what’s inside. This untangling? Just another guess by the machine — so mistakes slip through easily.</p><p>One thing hands back a JSON piece holding 47 parts. Skimming through, the system glances too fast. A label gets flipped by mistake. Where it sees nothing, it thinks missing instead of waiting. Some digit gets nudged up or down. Names that look like twins get swapped without notice. Not a trace appears in your tool call records.<em> Correct execution happened there. Real information fed into the system.</em> A mistaken reading came from the model instead, followed by bold moves based on that error.</p><pre>// Actual tool output:<br>{<br>  &quot;status&quot;: &quot;pending_approval&quot;,<br>  &quot;approved_amount&quot;: null,<br>  &quot;requested_amount&quot;: 5000,<br>  &quot;approver&quot;: &quot;finance_team&quot;<br>}<br>// What the model tells the user:<br>&quot;Your request for $5,000 has been approved.&quot;<br>// What actually happened:<br>The model saw &quot;approved_amount&quot; as a field name and assumed<br>the existence of the field implied approval.<br>null was misread as a confirmed state.</pre><p>Validation really works best when done where the agent operates, not only at the tool stage. Once the model returns an answer using tool information, another layer steps in, <em>maybe a follow up scan or organized parsing, </em>to confirm every stated point traces back to actual tool results. It takes more time. <strong>Many setups skip this part.</strong></p><h3>Why Nobody’s Talking About This</h3><p>A few reasons, honestly.</p><p>Sure.<strong><em> Numbers from tests rarely match real-life results</em></strong>. A system might score 94% when reviewed under clean conditions. Yet once live, performance drops to 76% since actual usage brings chaos, tools respond slower, and odd situations pop up that were never tested. That high test mark often goes public. The lower real-world figure tends to stay behind closed doors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EDP94--GdjPWJbdN" /></figure><p><em>Falling short quietly, </em><strong>that’s what happens when agents fail</strong>. A broken API gives off a loud signal, like a 500 error screaming something is wrong. But an agent can return a smooth 200 while being totally mistaken. Wrongness dressed up as success slips through. It moves unnoticed unless a person steps in to verify. Spotting it means spending minutes we often do not have.</p><p>Three: <strong>Speed makes doubt look like delay.</strong> When <em>rivals</em> push out tools while you review old decisions, pressure builds to patch gaps without noise. Moving ahead becomes its own logic.</p><p>I get it. I’m a student. I honestly have not that big of production systems. I don’t have the pressure these engineers have to endure.</p><p>But still I do think the ecosystem would be much more healthier if we talked about these issues more. The failure modes are real and they’re solvable, but only if we admit it and stop pretending they’re edge cases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ad37f729671" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why 90% of “AI startups” are just wrappers — and what Real AI products look like]]></title>
            <link>https://medium.com/@varunrao.aiml/why-90-of-ai-startups-are-just-wrappers-and-what-real-ai-products-look-like-9227fe880d40?source=rss-bc5d9b74a94e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9227fe880d40</guid>
            <category><![CDATA[ai-startups]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-products]]></category>
            <category><![CDATA[generative-ai-industry]]></category>
            <category><![CDATA[ai-wrappers]]></category>
            <dc:creator><![CDATA[varun rao]]></dc:creator>
            <pubDate>Sun, 15 Feb 2026 11:09:12 GMT</pubDate>
            <atom:updated>2026-02-15T11:09:12.151Z</atom:updated>
            <content:encoded><![CDATA[<p>Here it is — plenty of firms calling themselves artificial intelligence outfits hardly use AI at all.<br>Fancy front ends on top of backend systems. A smoother look for the middle layer.<br>Over the last twenty-four months I’ve looked at countless pitch decks, yet the trend feels absurd now. A person grabs GPT-4, slaps on a niche front-end, throws in clever prompts instead — just like that, they call it an “AI-driven SaaS transforming XYZ.”<br>I’m not saying every wrapper lacks value — yet calling them strong AI companies feels off. Still, a few actually help. What they aren’t is lasting moats in tech.</p><h3>The Anatomy of an AI Wrapper</h3><p>Fair chance you know this one already — it shows up everywhere<br>Start with a current LLM API. Toss in a clean front interface. Build system prompts that fit one clear purpose. Perhaps include a vector database — so replies actually remember what came before. Wrap it in a pay-to-use plan. Drop the launch on Product Hunt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/0*u8UFS6JLL5tlOqfg.jpg" /></figure><p>You made something — looks sleek, sure, but it runs on the same bones as that chatbot everyone got excited about last year. Fancy wrapping, familiar engine underneath.<br>It’s not about the tools being pointless. Lots of them actually fix things people struggle with. Yet here’s the catch — they lack any defense against giants copying them. Once companies like OpenAI, Anthropic, or Google add those features themselves, what you built can vanish fast. Overnight, it just fades.<br>Back in early 2023, a wave of so-called “AI writing helpers” hit the market. Yet here we are — most have vanished since ChatGPT rolled out sharper tools on its own. Those flashy “AI customer support bots”? Well, models like Claude and GPT already do that work now, thanks to wider memory and smoother task handling built right in.</p><h3>What Makes a Real AI Product</h3><p>Real AI products have technical depth that can’t just be replicated by just switching API providers or writing better prompts. Here’s what really differentiates them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/0*_tAkRtcDfiNyJkMy.jpg" /></figure><p><strong>Proprietary Models or Fine-tuning</strong></p><p>Training their own models takes time, yet some choose to do it anyway. Not merely adjusting settings one time, but building systems that keep refining through constant testing. Improvement happens step by step, fueled by feedback loops in data flow.<br>Out of nowhere, tools such as Harvey AI dive deep into legal tasks. Glean shows up in enterprise search with sharp precision. Cursor arrives on the scene handling coding duties smoothly. Each one beats broad language models by focusing tightly on its own turf.</p><p><strong>Novel Architectures or Techniques</strong></p><p>Something clever is happening under the hood — this goes beyond stacking prompts on top of GPT-4. Custom retrieval methods might be at play; perhaps a fresh approach to training has been applied. Different models could be working together through a thoughtfully stitched structure instead of running solo.<br>Built on a foundation others now copy, Perplexity designed its system long before retrieval became trendy. While OpenAI later stitched search into responses, this one had already shaped something distinct. Its staying power comes not from timing but design choices made early. Relevance today stems from that head start in structure, not imitation.</p><p><strong>Unique Data Moats</strong></p><p>When users engage, the system grows sharper, fed by exclusive information only it collects. With every interaction, its edge widens — others struggle to catch up because they lack the same flow of inputs. Improvement happens quietly, steadily, without needing constant updates. What begins as slight advantage turns into something harder to copy over time.<br>One thing that helps Scale AI stay ahead is how they handle labeled data along with strong partnerships. What sets GitHub Copilot apart comes down to tapping into the biggest collection of code ever gathered. With every passing month, those leads grow harder to catch up to.</p><p><strong>Deep Integration and Workflow Understanding</strong></p><p>Not every smart system hides behind a chat window. What sticks is woven into how tasks unfold each day. Custom pieces snap together, fitting quietly where people operate. Connections form without fanfare, yet hold firm. Understanding lives in the small details most overlook. Tools adapt, not because they claim to, but because they must.<br>Because Notion AI gets how pages and tables connect inside Notion, it does what it needs to do. Creative tweaking, not only making pictures, became the core of Midjourney’s rise — community grew around that shift.</p><h3>The Wrapper Warning Signs</h3><p>What gives away an AI wrapper pretending to be something more? Watch out for signs like vague promises instead of clear features. A product hiding behind flashy demos might lack actual function. Real tools explain how they work without jargon fog. If every answer loops back to “smart technology,” stay cautious. Empty claims often replace concrete details. Smooth talk rarely fixes broken logic underneath<br>Starts with a copycat line every time. Works out to “ChatGPT, only different.” When that’s the main point, problems follow close behind.<br>Proudly shouting about their LLM choice fills their ads. Actual AI firms skip waving around “GPT-4 inside” — it’s just a tool, not what sets them apart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_eWtWAQiCLwtoACU.jpg" /></figure><p>When OpenAI improves its model, their system just works better without any effort on their part. Because progress hinges on someone else’s breakthroughs, real innovation sits elsewhere.<br>When pushed, they fail to describe how their tech stands out. Pose the question: “How do things change if OpenAI adds that same tool?” Then see the hesitation begin.<br>A bunch of coders who make websites, nothing more. Building serious artificial intelligence demands specialists who understand it inside out.</p><h3>Why Wrappers Keep Getting Funded</h3><p>Funding wrappers isn’t news to most VCs. What drives them forward? A gamble of another kind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*6EGnmwjcqmb3T8tp" /></figure><p>A few folks pin hopes on how things get delivered plus which name sticks in people’s minds — the thought being, even when gadgets look alike, whoever gets close to buyers wins that space. Sure, it works sometimes, yet counts more as loyalty play than an invention game.<br>Some count on being bought just for their team long before anyone notices the idea has no protection. These aren’t the ones picturing a business that lasts ten years — they run on separate rules.<br>Truth is, certain investors lack the depth needed to tell a thin layer apart from solid tech. With so much excitement around artificial intelligence, sorting signal from clutter becomes tough — especially without hands-on experience. Real skill matters when judging what’s underneath.</p><h3>The Middle Ground: Smart Wrappers</h3><p>Not all wrappers in the marketplace are created equal. Some are genuinely building something more substantial and revolutionary than others.</p><p><strong>The smart wrappers are:</strong></p><p>What if smooth workflows came from smart design instead of new gadgets? Loom slips into daily tasks quietly, yet shifts everything — no flash, just function that feels like a step ahead. The tools themselves aren’t groundbreaking, still, fitting them together so well makes the difference show up in real use.<br>A single layer of technology can still spark powerful momentum. Behind Jasper lies a crowd constantly shaping its tools through real use. This loop tightens as more people join, refining inputs without needing new code. Growth feeds insight, which in turn reshapes performance over time.<br>What if control over how products reach people becomes total? When a company locks down its delivery path tight enough, the machinery behind it fades into background noise. Strong sales forces exist out there — teams so skilled at keeping relationships alive that swapping out suppliers feels invisible. Partnerships like that make users shrug when backend changes happen. Tech shifts go unnoticed because trust sits firmly on one brand.<br>Fast moves grab space while platforms lag behind. From there, stronger tools take shape. It’s a sprint — victory could belong to a few.</p><h3>What Founders Should Build Instead</h3><p>Starting an AI company? My take — skip the wrapper model unless you’ve got a unique way to reach users fast. Or unless you’re happy running a steady small operation instead of chasing big growth.<br>Start by spotting chances like these<br>Building tools to smooth out AI creation. Think of them as the gear everyone grabs during a tech stampede. Firms such as LangChain help teams move faster by tackling tough backend work. Tools from Weights &amp; Biases bring order to messy experiments. Then there is Modal, handling heavy lifting so others can focus. Each one clears roadblocks most AI groups run into.<br>Building these tools means knowing the field inside out. Take medical software, law-focused systems, finance-driven programs — each thrives on a blend of specialized insight and smart algorithms. Getting started is tough. Yet once running, few can catch up.<br>Apps that do new things often need fresh tech. Wondering whether your concept pushes boundaries? Think about this instead: Can someone skilled put it together fast, just using tools already out there? When the answer is yes, chances are it won’t grow into something big.<br>When models grow more focused, gathering the right information matters even more. Tools that support clean, reliable data work quietly behind progress. Creating solid systems to collect and organize inputs doesn’t grab attention — yet it shapes outcomes just the same.</p><h3>The Brutal Reality Check</h3><p>One wrong move, most new AI companies collapse. Not that the tech fails them. It’s simply too easy to copy what they do. As base models improve — they always do — their whole setup fades fast.<br>That film feels familiar. Mobile apps played it out already. Think back — companies chasing “Uber for X,” or calling themselves “Instagram for Y.” A wave of them faded fast. Not businesses, just features wearing capes. Survival went to those who built deep tech advantages. Or grew networks that gained strength as they expanded.<br>Faster this time, the rhythm feels familiar with artificial intelligence unfolding much like before.<br>Start by facing a hard question. When big models like Claude or GPT roll out your idea first, how does that leave you feeling? Unsettled? That hesitation could be telling. Maybe it’s time to shift toward ground that’s harder to copy. A move now might save months later.</p><h3>The Future Is Specialized</h3><p>Now showing up are just two kinds of leaders in AI. One group builds the big base models — think OpenAI, Anthropic, or Google. The others dive deep into narrow fields, using strong tech skills to stand out. Not broad tools, but focused power matters more now.<br>Now things shift as the core group of broad tools feels pressure everywhere. On one side, big systems pack in extra functions; meanwhile specialists dig in harder with unique strengths. Pressure builds right where balance once held.<br>Looking ahead five years, 2023–2024 might seem like a frenzy over AI wrappers. Many firms launched then will fade fast. Yet some stayed standing — those who saw it coming. Instead of following trends, they forged unique tech. Or shifted hard into mastering how their product reached people. Success came not from hype but quiet reinvention. Survival belonged to those moving before the rush ended.<br>Something big waits inside artificial intelligence, yet beauty on the surface won’t claim it. Instead, progress hides where challenges are thorny, where original information grows under effort. Strength comes through tools that sharpen themselves over time — unlike anything copied fast. Real edge lives there.<br>Building an AI company? Focus on creating real things instead of repackaging old ideas. Start there, stay there.</p><p><em>Building an AI product, Great ? I’d love to hear what makes yours defensible from the bigger sharks than you . And if you’re a wrapper :😆 hey, no shame ,but what’s your plan when the platforms catch you up?</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9227fe880d40" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Internet Just Became a Vending Machine: Why UCP Changes Everything]]></title>
            <link>https://python.plainenglish.io/the-internet-just-became-a-vending-machine-why-ucp-changes-everything-bbbaa8206d02?source=rss-bc5d9b74a94e------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbbaa8206d02</guid>
            <category><![CDATA[ai-shopping]]></category>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[ucp]]></category>
            <category><![CDATA[aiagentsdevelopment]]></category>
            <category><![CDATA[mcps]]></category>
            <dc:creator><![CDATA[varun rao]]></dc:creator>
            <pubDate>Sun, 18 Jan 2026 09:16:49 GMT</pubDate>
            <atom:updated>2026-01-18T19:59:51.865Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Or: How I stopped worrying and learned to love AI doing my shopping</strong></p><p>Look, I’ll be honest with you. When Google dropped the Universal Commerce Protocol announcement on January 11th, 2026, I almost scrolled past it. Another protocol? Another standard? We’ve all heard this song before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*EWIu_DH95QYsnbAlvzBwiw.png" /></figure><p>But then something clicked.</p><p>This isn’t just another tech announcement. This is the moment online shopping stops being <em>your</em> job and becomes something your AI handles while you’re doing literally anything else.</p><h3>The Butler You Didn’t Know You Needed</h3><p>Here’s the shift: Until now, online shopping has been like going to a supermarket and picking out ingredients yourself. You browse Amazon, compare prices on three different tabs, check reviews, abandon your cart twice, and maybe — <em>maybe</em> — you actually complete the purchase.</p><p>With UCP, you just tell your AI: “I want pasta tonight.”</p><p>And boom. The finest ingredients show up at your kitchen. Your AI found them, compared prices across every store on the internet, negotiated the best deal, and handled checkout. You did nothing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*q82AmZV5N2yFuHhkFj3dKQ.png" /></figure><p>Sounds convenient, right?</p><p>Maybe a little <em>too</em> convenient? Yeah, I had that thought too.</p><h3>What Makes UCP Different (And Why You Should Care)</h3><p>The Universal Commerce Protocol is basically a universal language for AI shopping. Think of it like this:</p><p>Right now, every online store is like a different country with different customs, currency, and rules. Want to buy a t-shirt from Store A, shoes from Store B, and a book from Store C? You’re registering three times, entering your payment info three times, dealing with three different checkout flows.</p><p>It’s exhausting.</p><p>UCP creates one set of rules that works everywhere. Any AI — Google’s Gemini, ChatGPT, whatever comes next — can shop at any store that supports UCP without getting confused by different website designs or checkout procedures.</p><p>Here’s what really got me excited: This could completely flip the power dynamic in e-commerce.</p><h3>The Platform Giants Are Nervous (And They Should Be)</h3><p>For years, Amazon has been the 800-pound gorilla because they controlled the platform. They owned the search results, the checkout flow, the entire customer relationship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*t2RbvYkS5ZAwYBscMSxbKA.png" /></figure><p>But in a UCP world? Consumers might not even <em>know</em> which site they’re buying from. They’ll just tell their trusted AI what they want, and the AI will find the best option — whether that’s Amazon, a small boutique in Portland, or a direct-to-consumer brand nobody’s heard of yet.</p><p>Suddenly, small businesses can compete on equal footing with the giants. As long as they support UCP and offer a unique AI agent experience, they’re in the game.</p><p>That’s genuinely revolutionary.</p><h3>I Built a UCP Shopping Bot (And It Actually Worked)</h3><p>I decided to get my hands dirty and build a proof-of-concept shopping assistant. Nothing fancy — just a chatbot that could answer questions about products and handle checkout using UCP.</p><p>The architecture is actually pretty elegant:</p><p><strong>The Retrieval Step:</strong> When someone asks “What t-shirts do you have?”, the agent searches through a vector database of product information. The cool part? It uses semantic search, so even if you ask “got any shirts?” instead of “t-shirts”, it still finds the right products.</p><p><strong>The Response Step:</strong> The agent takes your question plus the relevant product info, sends it all to GPT-4, and gets back a natural, helpful response. It remembers the conversation too, so follow-up questions like “What colors does it come in?” just work.</p><p><strong>The Checkout Flow:</strong> When you’re ready to buy, one API call creates a UCP-compliant checkout session with all the product details, pricing, and metadata packaged up nicely.</p><p>The whole thing runs in seconds. And because UCP is a standard, this same agent could theoretically work with <em>any</em> UCP-compliant store.</p><h3>The Three Layers That Make UCP Work</h3><p>Understanding UCP is easier when you think of it like a layer cake:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*_kSmGEUzgQJyN1Zyik8i7A.png" /></figure><p><strong>Layer 1 — Shopping Services (The Foundation):</strong> These are the basic atoms of commerce — order items, prices, status. Without these, no transaction exists.</p><p><strong>Layer 2 — Capabilities (The Main Functions):</strong> These are the independent functional blocks like checkout, catalog browsing, and order management. Think of these as the main menu your AI can recognize and use.</p><p><strong>Layer 3 — Extensions (The Secret Sauce):</strong> This is where businesses differentiate. Want to let customers choose delivery time slots? Use loyalty points? Add gift wrapping? Extensions let you signal to AI what makes your service unique.</p><p>Everyone from a small coffee shop to Walmart can compete on the same protocol, but they can still stand out through extensions.</p><h3>UCP vs MCP: What’s the Difference?</h3><p>If you’ve been following AI news, you might have heard about Anthropic’s Model Context Protocol (MCP) from November 2024. So what’s the difference?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*Gsi3mmPODAkAQecH33ms9A.png" /></figure><p>MCP is general-purpose — it standardizes how AI systems connect to external data sources and tools. Reading files, calling APIs, fetching information from various systems.</p><p>UCP is laser-focused on commerce. It covers the entire e-commerce journey: product catalogs, inventory checks, cart operations, checkout, payment, shipping, loyalty programs, returns, and support.</p><p>They’re actually complementary. UCP can work <em>with</em> MCP. In fact, UCP supports communication via both REST APIs and MCP-based interactions.</p><p>Think of MCP as the universal translator for AI-to-system communication, and UCP as the specialized dictionary for shopping.</p><h3>The Future Is Weird (In a Good Way?)</h3><p>Here’s what keeps me up at night in the best possible way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*DVc58B-Gcqou7jI2iHPF1A.png" /></figure><p>In the near future, consumers might not care which e-commerce site they’re using. At all.</p><p>You won’t “go to Amazon” or “check Target.” You’ll just tell your AI what you need, and it’ll handle everything — finding the best product, comparing prices across thousands of stores, reading reviews, checking inventory, and completing the purchase.</p><p>Your AI becomes your personal shopper, negotiator, and purchasing agent rolled into one.</p><p>The experience will be:</p><ul><li>“I need new running shoes for trail running”</li><li>Done. They arrive Thursday.</li></ul><p>No browsing. No comparing. No decision fatigue.</p><p>Is this convenience or are we outsourcing too much of our decision-making? Honestly, I’m not sure yet. Probably both.</p><h3>The Challenges Nobody’s Talking About</h3><p>Let’s be real: This isn’t all sunshine and roses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*cYFlWB5SmIC1ruASay_k7Q.png" /></figure><p><strong>Security concerns</strong> are massive. Your AI will have access to payment information and purchasing power. What happens when someone figures out how to exploit that?</p><p><strong>Privacy questions</strong> loom large. If AIs are doing all our shopping, who owns that behavioral data? What patterns are being tracked?</p><p><strong>The filter bubble problem</strong> could get worse. If your AI learns you like certain brands or price points, will you ever discover something truly different? Or will you end up in an algorithmic echo chamber of the same products forever?</p><p><strong>Small businesses</strong> might struggle to implement UCP properly. The barrier to entry isn’t insurmountable, but it’s not zero either.</p><h3>Why This Matters More Than You Think</h3><p>UCP represents a fundamental shift in how we think about commerce on the internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*W7l6JtmkpMBAhTNxHfHJhQ.png" /></figure><p>For the past 30 years, the internet has been a library of information. You search, you browse, you click, you buy. You’re doing the work.</p><p>With UCP, the internet becomes a vending machine. You state what you want, and the right product appears. The AI does the work.</p><p>This changes:</p><ul><li><strong>How platforms compete</strong> (less about the website, more about the data and AI experience)</li><li><strong>How brands reach customers</strong> (less about SEO and ads, more about AI discoverability)</li><li><strong>How consumers make decisions</strong> (less active comparison, more delegation to trusted AI)</li><li><strong>How small businesses can compete</strong> (level playing field if they can offer unique value)</li></ul><h3>What You Can Do Right Now</h3><p>If you’re a <strong>developer</strong>, start playing with UCP and the FastUCP framework. Build something. Break something. The best way to understand this shift is to get your hands dirty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*cFIMZvgeARqlh9cof5O-3g.png" /></figure><p>If you’re a <strong>business owner</strong>, start thinking about how AI agents will discover and interact with your products. What makes your offering unique enough that an AI would choose you?</p><p>If you’re a <strong>consumer</strong>, pay attention to which companies are implementing UCP and how. Your shopping behavior is about to change whether you’re ready or not.</p><h3>The Bottom Line</h3><p>The Universal Commerce Protocol is either the most convenient thing to happen to online shopping or a fundamental reshaping of how commerce works on the internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*8YdeBCQWP59kJbz8MgwdQQ.png" /></figure><p>Probably both.</p><p>We’re moving from a world where <em>you</em> search for products to a world where AI agents negotiate and purchase on your behalf. The implications are massive — for platforms, for businesses, for consumers.</p><p>Is it scary? A little.</p><p>Is it exciting? Absolutely.</p><p>Will it change everything? Already has.</p><p>The revolution happened quietly on January 11th, 2026. Most people didn’t notice. But the internet just transformed from a library into a vending machine, and there’s no going back.</p><p>Now if you’ll excuse me, I need to go tell my AI to order groceries. I want pasta tonight.</p><p><em>Have thoughts about UCP and AI-powered commerce? I’d love to hear them. Drop a comment or connect with me — I’m genuinely curious how people are thinking about this shift.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbbaa8206d02" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/the-internet-just-became-a-vending-machine-why-ucp-changes-everything-bbbaa8206d02">The Internet Just Became a Vending Machine: Why UCP Changes Everything</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AIOps for LLMs: The Observability Stack That Every AI Startup Should Use]]></title>
            <link>https://medium.com/codetodeploy/aiops-for-llms-the-observability-stack-that-every-ai-startup-should-use-2acfbadc31e9?source=rss-bc5d9b74a94e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2acfbadc31e9</guid>
            <category><![CDATA[aillm]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[ai-startups]]></category>
            <category><![CDATA[aiops]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[varun rao]]></dc:creator>
            <pubDate>Thu, 01 Jan 2026 08:01:53 GMT</pubDate>
            <atom:updated>2026-06-30T09:39:28.572Z</atom:updated>
            <content:encoded><![CDATA[<p>The first time your LLM application crashes in production at 3 AM, you’ll understand why observability isn’t optional anymore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*cDdnQUc_ej4Bix0fb5AceA.png" /></figure><blockquote><em>💥 </em><strong><em>Save Up to 85% OFF</em></strong><br> 📚 Premium Courses <br> ⏳ Limited-Time Offer | 🎓 Upgrade Your Skills<br><a href="https://trk.udemy.com/zz4NBO"><em>👉 </em><strong><em>Enroll Now &amp; Start Learning</em></strong></a></blockquote><p>I learned this the hard way. We’d built what we thought was a robust AI chatbot for customer support. The demos were flawless. The investors loved it. Then we launched, and within hours, users were reporting bizarre responses, timeouts, and costs that made our CFO’s eye twitch. We had no idea what was happening under the hood.</p><p>That’s when I discovered that building AI applications isn’t like building traditional software. The rules are different. The failure modes are different. And most importantly, you need a completely different approach to monitoring and debugging.</p><h3>Why Traditional Observability Falls Short for LLMs</h3><p>Traditional application monitoring was built for deterministic systems. You log errors, track response times, monitor CPU usage, and call it a day. But LLMs? They’re probabilistic black boxes that can fail in ways that don’t trigger any alarms.</p><p>Your API might return a 200 OK status while serving complete nonsense to users. Your latency might be perfect while your costs spiral out of control. Your accuracy might degrade gradually over weeks as user behavior shifts, and you won’t notice until customers start churning.</p><p>Here’s what keeps AI engineering teams up at night:</p><p><strong>The cost problem</strong> gets real, fast. A single poorly optimized prompt can turn a $100/month bill into $10,000. One customer with an unusual usage pattern can drain your entire runway. Unlike traditional compute costs that scale somewhat predictably, LLM costs can spike unexpectedly based on prompt length, model choice, and usage patterns you didn’t anticipate.</p><p><strong>The quality problem</strong> is even trickier. How do you know if your model’s outputs are getting worse? Traditional metrics like uptime and error rates don’t capture hallucinations, tone shifts, or gradual degradation in response quality. You need to actually evaluate the semantic quality of outputs, not just whether the API call succeeded.</p><p><strong>The latency problem</strong> compounds everything. Users expect instant responses, but LLMs can take seconds to generate output. Is it slow because of network issues? Token length? Model congestion? Without proper instrumentation, you’re guessing.</p><h3>What AIOps for LLMs Actually Means</h3><p>AIOps for LLMs isn’t just monitoring. It’s a complete paradigm shift in how we think about observability. You need to track three dimensions simultaneously: performance, quality, and cost. Miss any one of them, and you’re flying blind.</p><p>The goal is simple: understand what your LLM is doing, why it’s doing it, and how much it’s costing you. In real-time. With enough granularity to actually fix problems.</p><h3>The Core Components of an LLM Observability Stack</h3><p>After building AI systems for the past two years and talking to dozens of teams in the space, I’ve identified the essential layers every AI startup needs. This isn’t aspirational architecture, it’s what actually works in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*TF8IjsxXyPIjs0VY6-LQCw.png" /></figure><h3>1. Request Tracing and Prompt Logging</h3><p>Every single interaction with your LLM needs to be captured. Not just metadata, but the full context: the exact prompt sent, the complete response received, the model used, token counts, and timestamps.</p><p>Why? Because when something goes wrong, you need to reproduce it. And with LLMs, reproduction is hard. The same prompt can produce different outputs. User context matters. Conversation history matters. Without complete traces, debugging is impossible.</p><p><strong>What to capture:</strong> Full prompts with variables, system messages, conversation history, model parameters (temperature, max tokens, etc.), complete model responses, token usage (prompt and completion), latency breakdown (time to first token, total time), model version and provider.</p><p>The teams that get this right can debug production issues in minutes instead of hours. They can replay exact scenarios, understand edge cases, and identify patterns in failures.</p><h3>2. Cost Tracking and Attribution</h3><p>If you’re not tracking costs per request, per user, and per feature, you’re going to have a bad time. I’ve seen startups burn through their entire seed round because they didn’t realize one power user was costing them $1,000/day.</p><p>You need real-time visibility into: cost per API call, cost per user (over time), cost per feature or workflow, cost trends and projections, comparison across different models and providers.</p><p>The best teams build cost dashboards that update in real-time. They set up alerts when costs exceed thresholds. They A/B test different models not just for quality, but for cost efficiency. They can tell you exactly which features are profitable and which are subsidized.</p><h3>3. Quality Monitoring and Evaluation</h3><p>This is where it gets interesting. How do you automatically evaluate if an LLM output is “good”? You can’t just check for errors, you need to assess semantic quality, factual accuracy, tone, relevance, and safety.</p><p><strong>The approaches that work:</strong> Embedding-based similarity scoring to detect drift from expected outputs. LLM-as-a-judge where you use another LLM to evaluate quality. Human feedback loops with thumbs up/down or more detailed ratings. Custom evaluation functions for domain-specific criteria. Hallucination detection through fact-checking against knowledge bases.</p><p>The key is running these evaluations automatically in production, not just during testing. Quality can degrade over time as user behavior changes, model providers update their systems, or edge cases accumulate.</p><h3>4. Performance and Latency Monitoring</h3><p>Speed matters. Users abandon slow AI applications just like they abandon slow websites. But LLM latency is complex because you’re dealing with streaming responses, variable token generation, and third-party APIs.</p><p>Track these metrics: Time to first token (TTFF), tokens per second, end-to-end latency, API response times, streaming performance, error rates and retry logic.</p><p>The difference between a fast and slow AI application often comes down to optimization: caching common queries, using smaller models for simple tasks, implementing parallel processing, streaming responses instead of waiting for completion, smart prompt engineering to reduce tokens.</p><h3>5. User Behavior and Session Analytics</h3><p>Understanding how users interact with your AI is crucial. Unlike traditional apps where you track clicks and page views, you need to understand conversation flows, user intent, and engagement patterns.</p><p><strong>What matters:</strong> Session length and message count, conversation topics and intent classification, user satisfaction signals, drop-off points, retry and refinement patterns, feature usage across different user segments.</p><p>This data informs product decisions. You might discover that users abandon conversations after three back-and-forth exchanges, suggesting you need better context handling. Or that certain prompts consistently lead to poor outcomes, indicating your system prompt needs work.</p><h3>The Practical Stack: Tools You Should Actually Use</h3><p>Theory is great, but let’s talk about what you can implement today. The ecosystem has matured rapidly over the past year, and there are solid options for each layer.</p><p><strong>For comprehensive observability, Langfuse and Helicone</strong> have become popular choices. Langfuse is open-source and gives you detailed traces, metrics, and evaluations. It integrates well with LangChain and other frameworks. Helicone focuses on being a drop-in proxy that requires minimal code changes.</p><p><strong>For cost management, OpenMeter and Orb</strong> help with usage-based billing and cost tracking. They’re essential if you’re building a product with tiered pricing or need to attribute costs accurately.</p><p><strong>For quality evaluation, Braintrust and HumanLoop</strong> offer sophisticated evaluation frameworks. They let you run custom evaluations, compare model outputs, and track quality over time.</p><p><strong>For the DIY crowd,</strong> you can build your own stack using: OpenTelemetry for distributed tracing, Prometheus and Grafana for metrics, Elasticsearch or ClickHouse for logs, Custom Python scripts for evaluations, Datadog or New Relic for traditional monitoring.</p><p>The build-vs-buy decision depends on your team size and focus. If you’re a small startup, use existing tools. If you’re at scale or have unique needs, consider building custom solutions.</p><h3>Real-World Patterns That Work</h3><p>After talking to teams that have scaled AI applications to millions of users, certain patterns emerge consistently.</p><p><strong>Start with structured logging from day one.</strong> Don’t wait until you have problems. Every LLM call should be logged with a unique trace ID that you can follow through your entire system. This sounds obvious, but so many teams skip it early on and regret it later.</p><p><strong>Implement circuit breakers and fallbacks.</strong> LLM APIs fail. They get rate-limited. They have bad days. Have a plan for when your primary model is down: fallback to a cheaper model, use cached responses, queue requests for later, or gracefully degrade functionality.</p><p><strong>Build a feedback loop with users.</strong> The simplest form of quality monitoring is asking users if the response was helpful. Implement thumbs up/down buttons. Track which responses users copy or act on versus which they ignore or retry.</p><p><strong>Set up cost alerts before it’s too late.</strong> Configure alerts when hourly spend exceeds normal patterns. You want to catch a runaway process before it costs you thousands, not after.</p><p><strong>Create synthetic monitoring for critical flows.</strong> Run automated tests that hit your AI endpoints regularly with known inputs. This catches degradation before users do.</p><h3>The Metrics That Actually Matter</h3><p>Not all metrics are created equal. Focus on these:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*0DmKnkfFjCPrhG35laqtEA.png" /></figure><p><strong>P95 latency</strong> tells you what the slow experiences look like. Median is meaningless when 5% of users wait 30 seconds.</p><p><strong>Cost per successful conversation</strong> is more meaningful than cost per API call. Some calls fail and retry. Some conversations take 10 turns. Measure the full cost of delivering value.</p><p><strong>User satisfaction</strong> is the ultimate quality metric. Track explicit feedback (ratings) and implicit signals (did they continue the conversation? did they return?).</p><p><strong>Error rate by category</strong> helps you prioritize. Not all errors are equal. Rate limit errors need different solutions than hallucinations.</p><p><strong>Model coverage</strong> shows what percentage of requests use which models. Essential for migration planning and cost optimization.</p><h3>Common Mistakes to Avoid</h3><p>Every team makes these mistakes. Learn from others:</p><p><strong>Logging too much or too little.</strong> Too much and you drown in noise. Too little and you can’t debug. Find the balance by starting verbose and pruning over time.</p><p><strong>Ignoring PII in logs.</strong> If users share sensitive information with your AI, you need to sanitize it before logging. This is both a legal and ethical requirement.</p><p><strong>Not tracking model versions.</strong> Model providers update their models. If quality suddenly drops, you need to know if it coincides with a model update.</p><p><strong>Optimizing for the wrong metric.</strong> Chasing the lowest latency might increase costs 10x. Chasing the lowest cost might tank quality. Optimize for business outcomes, not vanity metrics.</p><p><strong>Building everything custom.</strong> Unless you’re at massive scale, use existing tools. Your competitive advantage isn’t your logging infrastructure.</p><h3>Looking Forward</h3><p>The observability landscape for LLMs is still evolving rapidly. We’re seeing trends toward: more sophisticated automated evaluations using smaller, specialized models, better tools for debugging multi-agent systems, integrated solutions that handle traces, metrics, and logs in one platform, and open standards for LLM observability (like OpenTelemetry extensions for AI).</p><p>The teams that win will be those that treat observability as a first-class concern from day one. The gap between “it works in the demo” and “it works reliably at scale” is massive, and observability is what bridges that gap.</p><h3>Getting Started Today</h3><p>If you’re building an AI application and don’t have observability set up, here’s what to do this week:</p><p>Start logging every LLM interaction with full context. Set up basic cost tracking per endpoint. Implement user feedback collection. Add latency monitoring. Choose one observability platform to evaluate.</p><p>You don’t need perfect instrumentation on day one. You need enough visibility to understand what’s happening and iterate quickly. Start simple, then expand as you learn what matters for your specific application.</p><p>The future of AI applications belongs to teams that can ship fast and debug faster. Observability is what makes that possible. Don’t wait for a 3 AM production fire to figure this out.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSB2gwj83y8MJ41gOWuLog.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://x.com/Bhuwanchet67277"><strong>X</strong></a> | <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p>👉 CodeToDeploy Tech Community is live on Discord — <a href="https://discord.gg/ZpwhHq6D"><strong>Join now!</strong></a></p><p>👉 <strong>Follow our publication, </strong><a href="https://medium.com/codetodeploy"><strong>CodeToDeploy</strong></a></p><p><strong>Note:</strong> This Post may contain affiliate links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2acfbadc31e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/aiops-for-llms-the-observability-stack-that-every-ai-startup-should-use-2acfbadc31e9">AIOps for LLMs: The Observability Stack That Every AI Startup Should Use</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Most Underrated Cause of Slow Inference: Token Divergence]]></title>
            <link>https://ai.plainenglish.io/the-most-underrated-cause-of-slow-inference-token-divergence-83b8fd5e21b2?source=rss-bc5d9b74a94e------2</link>
            <guid isPermaLink="false">https://medium.com/p/83b8fd5e21b2</guid>
            <category><![CDATA[quantized]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[inference]]></category>
            <category><![CDATA[tokenization]]></category>
            <dc:creator><![CDATA[varun rao]]></dc:creator>
            <pubDate>Wed, 31 Dec 2025 08:19:23 GMT</pubDate>
            <atom:updated>2026-01-02T09:45:35.164Z</atom:updated>
            <content:encoded><![CDATA[<p>You’ve optimized your model architecture. You’ve pruned weights, quantized parameters, and maybe even splurged on that shiny new GPU. But your inference is still crawling. What gives?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*KqH3H0J3YQt8unuMktBXog.png" /></figure><p>Here’s the thing nobody talks about at AI conferences: while everyone’s obsessing over model size and computational complexity, there’s a silent performance killer lurking in plain sight. It’s called token divergence, and it’s probably costing you more latency than you think.</p><h3>What Actually Happens During Inference</h3><p>Let’s start with the basics. When your language model processes text, it doesn’t see words the way you and I do. It sees tokens — chunks of text that could be whole words, parts of words, or even single characters, depending on how the tokenizer was trained.</p><p>Here’s where things get interesting. Most developers assume that once you’ve loaded a model, every inference request should perform roughly the same. Feed in a prompt, get back a response, rinse and repeat. But in reality, two prompts of similar length can have wildly different processing times, and the culprit is often something called token divergence.</p><h3>The Hidden Cost of Tokenization Mismatch</h3><p>Token divergence happens when there’s a mismatch between how efficiently your input gets tokenized versus how the model was optimized to process tokens. Think of it like this: imagine you’re reading a book, but every few pages, the font changes, the line spacing shifts, and suddenly you’re reading right-to-left instead of left-to-right. You’d slow down, right?</p><p>That’s essentially what happens to your model when token distributions diverge from what it expected during training. The model trained on clean, well-formatted text from books and articles, but now it’s processing code snippets, social media slang, and domain-specific jargon. Each of these has radically different tokenization patterns.</p><p>Consider this real example: the phrase “machine learning” might tokenize into two tokens. But “MachineLearning” (as a camelCase variable name) could tokenize into three or four. “ML” becomes a single token. Now multiply this across thousands of tokens in a conversation, and you start seeing why some requests take twice as long as others, even with similar character counts.</p><h3>Why This Matters More Than You Think</h3><p>The performance hit from token divergence compounds in ways that aren’t immediately obvious. Modern transformer models use attention mechanisms that scale quadratically with sequence length. When your tokenizer splits text inefficiently, you’re not just adding a few extra tokens — you’re exponentially increasing the computational load.</p><p>Let’s talk numbers. Say you’re building a customer service chatbot. User A types in standard English: “I need help with my account.” That might be 8 tokens. User B, frustrated, types: “i neeeed helppp w/ my acccccount plssss.” Due to the irregular spelling and abbreviations, this could balloon to 15–20 tokens. Your model isn’t just processing twice the information — it’s doing roughly four times the attention computation.</p><p>And here’s the kicker: this happens invisibly. Your monitoring dashboard shows similar request sizes, but your P95 latency is through the roof. You blame the infrastructure, add more servers, optimize your serving layer. But the real problem is that your model is choking on poorly tokenized inputs.</p><h3>The Research Nobody’s Reading</h3><p>There’s actually fascinating research on this that gets buried in academic papers. A 2023 study from researchers at Stanford found that token efficiency could account for up to 40% of the variance in inference times for production language models. Another paper from Google showed that optimizing tokenization strategies could reduce inference costs by 25–30% without touching the model itself.</p><p>But here’s why this research doesn’t make it into practice: it’s not sexy. “We reduced inference time by 30% through better tokenization” doesn’t get you conference talks. “We built a 100B parameter model” does. So the industry keeps throwing hardware at a problem that’s partially linguistic.</p><h3>What You Can Actually Do About It</h3><p>The good news is that once you know about token divergence, you can start addressing it. Here are strategies that actually work in production:</p><p><strong>Domain-specific tokenization.</strong> If your model primarily handles code, train a tokenizer on code. If it’s medical records, use medical text. Generic tokenizers are jacks of all trades and masters of none. Yes, this means retraining, but the performance gains are substantial.</p><p><strong>Input normalization.</strong> Before feeding text to your model, normalize it. Expand contractions consistently. Standardize punctuation. Remove excessive repetition. This isn’t about changing the meaning — it’s about making the tokenization predictable.</p><p><strong>Token budget monitoring.</strong> Track not just input length but actual token counts. Set up alerts when requests exceed expected token-to-character ratios. This helps you identify problematic input patterns before they tank your performance.</p><p><strong>Adaptive batching.</strong> Instead of batching requests by character count or request count, batch by token count. This ensures each batch has similar computational requirements, preventing some batches from becoming bottlenecks.</p><h3>The Real-World Impact</h3><p>I worked with a fintech startup that was burning through their inference budget. They’d optimized everything — model quantization, batch sizes, caching strategies. Still hitting their limits. Then we looked at their tokenization patterns.</p><p>Turns out, financial documents have tons of special characters, numbers, and formatting that their general-purpose tokenizer handled terribly. A typical financial report that was 5,000 characters was tokenizing into 2,500+ tokens, when it should have been closer to 1,500. After training a finance-specific tokenizer, their inference costs dropped by 35%. Same model, same infrastructure, just smarter tokenization.</p><h3>Looking Forward</h3><p>As models get larger and more capable, token divergence is only going to become a bigger issue. We’re seeing models trained on trillions of tokens, deployed across wildly diverse use cases. The gap between training distribution and deployment distribution is widening.</p><p>The next generation of production AI systems will need to treat tokenization as a first-class concern, not an afterthought. We need better tools for analyzing token efficiency, real-time tokenization switching based on input type, and models that are more robust to tokenization variance.</p><h3>The Bottom Line</h3><p>Before you upgrade to a more expensive GPU tier or start optimizing your CUDA kernels, take a hard look at your tokenization. It’s the unglamorous, under-discussed bottleneck that could be silently eating your performance budget.</p><p>Token divergence isn’t going to make headlines or win awards. But understanding it might be the difference between an AI system that barely works and one that scales elegantly to millions of users. And in production, that’s what actually matters.</p><p>Sometimes the most impactful optimizations are hiding in plain sight, encoded in the very first step of your inference pipeline. Don’t overlook them just because they’re not flashy enough for a conference paper.</p><h3>A message from our Founder</h3><p><strong>Hey, </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>Sunil</strong></a><strong> here.</strong> I wanted to take a moment to thank you for reading until the end and for being a part of this community.</p><p>Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? <strong>We don’t receive any funding, we do this to support the community. ❤️</strong></p><p>If you want to show some love, please take a moment to <strong>follow me on </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://tiktok.com/@messyfounder"><strong>TikTok</strong></a>, <a href="https://instagram.com/sunilsandhu"><strong>Instagram</strong></a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/"><strong>weekly newsletter</strong></a>.</p><p>And before you go, don’t forget to <strong>clap</strong> and <strong>follow</strong> the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83b8fd5e21b2" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/the-most-underrated-cause-of-slow-inference-token-divergence-83b8fd5e21b2">The Most Underrated Cause of Slow Inference: Token Divergence</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>