<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sulbha Jain on Medium]]></title>
        <description><![CDATA[Stories by Sulbha Jain on Medium]]></description>
        <link>https://medium.com/@sulbhajain?source=rss-82b008267a9d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fWj8-y31Iu04NaLRIO_W4A@2x.jpeg</url>
            <title>Stories by Sulbha Jain on Medium</title>
            <link>https://medium.com/@sulbhajain?source=rss-82b008267a9d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:22:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sulbhajain/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Breaking the Sanskrit Code: How Ancient Grammar Meets Modern Logic!]]></title>
            <link>https://sulbhajain.medium.com/breaking-the-sanskrit-code-how-ancient-grammar-meets-modern-logic-342e0331ffd6?source=rss-82b008267a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/342e0331ffd6</guid>
            <dc:creator><![CDATA[Sulbha Jain]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 07:15:36 GMT</pubDate>
            <atom:updated>2026-11-03T07:15:36.231Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Su2IolZcJwcYlynmJbVeig.png" /><figcaption><a href="https://sanskrit.uohyd.ac.in/scl/#">https://sanskrit.uohyd.ac.in/scl/#</a></figcaption></figure><p>Have you ever tried reading an old Sanskrit text and felt like all the words were crammed together into one giant, confusing puzzle? That’s pretty much what happens the moment you try to read traditional Sanskrit!</p><p>Unlike English, where every word has nice, neat spaces around it, traditional Sanskrit sentences string together like train cars with no breaks in between. But what if I told you that a 2,500-year-old Indian grammarian named <strong>Panini</strong> wrote rules so logical that today’s computers can actually run them like modern software code?</p><p>I recently attended fascinating talk by <strong>Prof. Amba Kulkarni</strong>, head researcher at the Department of Sanskrit Studies at the University of Hyderabad. She shared how her team spent decades combining computational linguistics with ancient wisdom to build a digital toolbox called <a href="https://sanskrit.uohyd.ac.in/scl/">Samsadhani</a>.</p><p><a href="https://www.youtube.com/watch?v=8L2yD6qLeHc&amp;t=316s">Video link Part1</a></p><p><a href="https://www.youtube.com/watch?v=G-lUuzJKE5w">Video link Part 2</a></p><p>Here is a friendly, easy-to-digest look at how ancient grammar meets computer logic!</p><h3>1. A “Living Museum” for Language</h3><p>One of the coolest phrases Prof. Kulkarni used in her talk was calling the living tradition of Sanskrit grammar a <strong>“living museum.”</strong></p><p>Unlike dead ancient languages where we can only guess how grammar worked, India still has living scholars who have memorized the entire <em>Astadhyayi</em> (all 4,000 rules!). They can instantly tell you which exact rule applies in any context.</p><blockquote>“This tradition of learning Sanskrit grammar still exists… It is a living museum for us where we can go, live, and experience it.”</blockquote><blockquote><em>— </em><strong><em>Prof. Amba Kulkarni</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZjWtiYYmydMwXlBPgpDyAQ.png" /><figcaption><a href="https://sanskrit.uohyd.ac.in/scl/#">https://sanskrit.uohyd.ac.in/scl/#</a> — Sandhih tool</figcaption></figure><h3>2. Global Teamwork: French &amp; Indian Computer Scientists</h3><p>Building software that understands ancient grammar wasn’t done in isolation. Prof. Kulkarni highlighted key contributions from international experts, especially <strong>Prof. Gérard Huet</strong>, an emeritus professor and computer scientist from INRIA in Paris, France.</p><p>Prof. Huet built the foundational word-segmentation platform (<em>Sanskrit Heritage</em>) using <strong>Finite State Automata</strong> — a method that uses interconnected software nodes (scaling up to 40 complex linked automata) to track how prefixes, root verbs, and noun endings connect. By linking Prof. Huet’s French platform with Samsadhani, the team created a super-engine that breaks apart complex sentences.</p><h3>3. Panini: The World’s First Coder?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*_Woybd6lsocBAwya" /><figcaption><a href="https://www.pinterest.com/pin/475692779375158108/">https://www.pinterest.com/pin/475692779375158108/</a></figcaption></figure><p>Here’s the really impressive part: <strong>Panini didn’t just write grammar rules — he wrote an algorithm.</strong></p><p>Way back in the 4th century BCE, Panini created the <em>Astadhyayi</em>, a master collection of around 4,000 short rules (<em>sutras</em>). Prof. Kulkarni pointed out three remarkable dimensions of his work:</p><ol><li><strong>Human-Envisioned &amp; Comprehensive:</strong> It validates over <strong>95%</strong> of classical Sanskrit usage, an extraordinary feat for a single human mind.</li><li><strong>Written in Programming Style:</strong> It functions like modern code with modular rules, conditional logic loops, and meta-rules (<em>Paribhasha sutras</em>) to resolve rule conflicts.</li><li><strong>Universal Framework:</strong> His <strong>Karaka system</strong> (how actions connect to subjects and objects) applies universally across languages, including English and Hindi.</li></ol><h3>4. How Samsadhani Reads a Sentence (Shabdabodha)</h3><p>When you plug a tricky verse into the <a href="https://sanskrit.uohyd.ac.in/scl/">Samsadhani</a> website, it doesn’t just guess what it means. It generates a visual dependency graph called a <strong>Shabdabodha</strong> diagram, mapping out the subject (<em>Karta</em>), object (<em>Karma</em>), and action (<em>Kriya</em>).</p><h3>Tackling Tricky Classical Verses</h3><p>In her talk, Prof. Kulkarni shared vivid examples showing how computers solve human reading confusion:</p><ul><li><strong>The <em>Ramayana</em> Example:</strong> In her talk, Prof. Kulkarni showed a famous verse describing Ravana carrying away Sita. Human readers often get confused because Ravana isn’t explicitly named in that specific line! The verse uses indirect pronouns (<em>tena mayavina</em>) and past action participle suffixes (<em>apavahya</em>, <em>hatva</em>). Human readers often mislink who killed Jatayu versus who carried Sita away. The computer applies strict mathematical logic constraints (such as preventing overlapping relationship lines) to correctly map out who caused what without human bias or guesswork!</li><li><strong>The <em>Bhagavad Gita</em> Example:</strong> When analyzing the famous <em>Bhagavad Gita</em> line (<em>Krodhat bhavati sammohah…</em>), human readers must rearrange the poetic word order (<em>shloka</em> order) into prose order (<em>anvaya</em> order). Samsadhani automatically parses compound words (<em>Shasthi Tatpurusha</em> samasa) and reorganizes the chain reaction, showing how anger leads to delusion, loss of memory, and ultimate ruin.</li></ul><h3>5. Try It Out Yourself!</h3><p>You don’t need a PhD in computer science or ancient languages to explore this. Prof. Kulkarni and her team have made these tools free and open for anyone:</p><ul><li><strong>Sandhi Splitter (<em>Chedika</em>):</strong> Paste a glued-together sentence and watch the site break it down into clean words.</li><li><strong>Noun &amp; Verb Generators:</strong> Type in a root word to see all its variations generated step-by-step.</li><li><strong>E-Readers (<em>E-Grantha</em>):</strong> Read texts like the <em>Bhagavad Gita</em>, <em>Ramayana</em>, or <em>Raghuvamsham</em> with interactive relationship graphs and word-by-word meanings in English and Hindi.</li></ul><h3>Acknowledgments &amp; Courtesy</h3><p>All credit for these groundbreaking computational linguistics tools goes to <strong>Prof. Amba Kulkarni</strong>, her PhD students, collaborators like <strong>Prof. Gérard Huet</strong>, and the Department of Sanskrit Studies at the University of Hyderabad.</p><ul><li><strong>Explore the platform:</strong> <a href="https://sanskrit.uohyd.ac.in/scl/">Samsadhani Official Website</a></li><li><strong>Watch the Original Lectures:</strong></li><li><a href="https://www.youtube.com/watch?v=8L2yD6qLeHc">Prof. Amba Kulkarni — Samskrita-Tools Part 1</a> (Source: <em>Vitasta Avasiyashibiram</em> YouTube Channel)</li><li><a href="https://www.youtube.com/watch?v=G-lUuzJKE5w">Prof. Amba Kulkarni — Samskrita-Tools Part 2</a> (Source: <em>Vitasta Avasiyashibiram</em> YouTube Channel)</li><li>Session organized by <a href="https://samskritabharatiusa.org/">https://samskritabharatiusa.org/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=342e0331ffd6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You’re Probably Using Cosine Similarity Wrong ; And It’s Quietly Breaking Your RAG Pipeline]]></title>
            <link>https://sulbhajain.medium.com/youre-probably-using-cosine-similarity-wrong-and-it-s-quietly-breaking-your-rag-pipeline-f285262fdeb9?source=rss-82b008267a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f285262fdeb9</guid>
            <category><![CDATA[llm-similarity]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <dc:creator><![CDATA[Sulbha Jain]]></dc:creator>
            <pubDate>Sun, 08 Mar 2026 06:34:39 GMT</pubDate>
            <atom:updated>2026-03-08T06:34:39.342Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A look at a 2024 Netflix/ACM paper that shook my assumptions, and how I rebuilt retrieval from scratch using a multi-embedding approach.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ERQIsmB1S9HXi0k6" /><figcaption>Photo by <a href="https://unsplash.com/@nadine3?utm_source=medium&amp;utm_medium=referral">Nadine Marfurt</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There’s a line from a 2024 ACM WWW paper that stopped me mid-scroll:</p><blockquote><em>“Cosine-similarity can yield arbitrary and therefore meaningless ‘similarities.’”</em></blockquote><p>It came from Netflix researchers. They weren’t talking about edge cases or pathological inputs. They were talking about the everyday embeddings you get from training a model with standard regularization — the kind almost every recommender system, search index, and RAG pipeline uses.</p><p>That sent me down a rabbit hole. I ended up building a <a href="https://github.com/sulbhajain/embedding-explore/blob/main/README.md">Python package</a> to understand it properly, combining the paper’s mathematical insights with a multi-embedding RAG architecture. Here’s what I learned.</p><h3>The Problem: A Degree of Freedom You Didn’t Know Existed</h3><p>When a model learns embeddings through matrix factorization, it optimizes something like:</p><pre>min ||X - AB^T||² + λ||AB^T||²</pre><p>The second term is <em>product regularization</em> — it penalizes the product of the matrices, not the matrices individually.</p><p>Here’s the catch. This objective is invariant under diagonal rescaling. For <em>any</em> diagonal matrix D, you can substitute:</p><pre>A → A @ D<br>B → B @ D⁻¹</pre><p>The product AB^T = (AD)(BD⁻¹)^T stays identical. Dot products don&#39;t change. The learned representations are mathematically equivalent.</p><p>But cosine similarity? It normalizes by the vector <em>norms</em>. When you scale A by D, the norms of A change. The cosine similarities change with them — arbitrarily, as a function of whatever D happens to satisfy the optimizer.</p><p>The paper proves this formally. I built a function to show it empirically:</p><pre># Scale A by D and B by D⁻¹<br># dot products: UNCHANGED<br># cosine similarities: VARY WILDLY</pre><pre>result = CosineSimilarity.demonstrate_regularization_arbitrariness(A, B,<br>    scale_factors=[0.01, 0.1, 1.0, 10.0, 100.0, 1000.0]<br>)</pre><pre># result[&quot;dot_products&quot;]        → [0.3821, 0.3821, 0.3821, 0.3821, 0.3821, 0.3821]<br># result[&quot;cosine_similarities&quot;] → [0.9910, 0.8743, 0.3512, 0.0421, 0.0042, 0.0004]</pre><p>The same model, the same mathematical equivalence class of solutions, can produce cosine similarities ranging from 0.99 (very similar) to 0.0004 (essentially unrelated). Not because the embeddings encode different information — but because of an arbitrary scaling your optimizer chose.</p><p>This isn’t a fringe issue. Product-form regularization (||AB^T||) appears in collaborative filtering, many autoencoder objectives, and anywhere you regularize a two-factor model jointly. Deep networks stack these layers, compounding the effect.</p><h3>What Should You Use Instead?</h3><p>The paper has three concrete recommendations:</p><p><strong>1. Train with cosine loss explicitly.</strong> If cosine similarity is what you care about at inference, optimize for it directly. Many sentence-transformers models do this via contrastive learning — which is why they tend to work better than raw transformer embeddings for retrieval.</p><p><strong>2. Switch to individual regularization.</strong> Penalizing ||A||² + ||B||² separately (not the product) makes cosine similarities unique. The diagonal degree of freedom disappears.</p><p><strong>3. Normalize during training, not just at inference.</strong> L2-normalizing embeddings as part of the forward pass — not as a post-hoc step — removes the degree of freedom before it can propagate.</p><p>In practice, for most teams: <strong>use </strong><strong>all-MiniLM-L6-v2 or </strong><strong>all-mpnet-base-v2 from sentence-transformers with </strong><strong>normalize_embeddings=True.</strong> These are trained with cosine objectives and are safe to use as-is.</p><p>What you should <em>stop</em> doing: taking raw embeddings from a model trained on a cross-entropy or MSE objective and computing cosine similarity without checking whether the training accounted for this.</p><h3>Part 2: The Retrieval Problem Cosine Similarity Can’t Solve Alone</h3><p>Even with correctly-trained embeddings, retrieval has a deeper architectural problem: <strong>a single embedding per document loses information.</strong></p><p>Consider a product catalog. A product like “Sony WH-1000XM5 over-ear noise-cancelling wireless headphones” has at least two distinct semantic components:</p><ul><li><strong>Identity</strong> (Sony, WH-1000XM5) — brand recognition, model matching</li><li><strong>Functionality</strong> (noise-cancelling, wireless, over-ear) — feature matching</li></ul><p>Compress both into a single 384-dimensional vector and you’re asking one representation to win two very different retrieval games simultaneously.</p><p>The query “Sony headphones” cares mostly about identity. The query “best headphones for long flights” cares entirely about functionality. The same embedding has to serve both.</p><p>The insight that kicked off the second part of this project: <strong>don’t compress everything into one vector. Store three.</strong></p><h3>The Multi-Embedding Architecture</h3><p>Here’s the architecture I implemented:</p><pre>Document<br>  ├── title_embedding      → ChromaDB collection 1  (30% weight, default)<br>  ├── desc_embedding       → ChromaDB collection 2  (40% weight, default)<br>  └── combined_embedding   → ChromaDB collection 3  (30% weight, default)</pre><p>At query time, you first classify the query’s intent:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*VNSVdOsVoL9UqN0xMM0FyQ.png" /></figure><p>Then you search all three collections and compute a weighted fused score:</p><pre>fused_score = w_title * sim_title + w_desc * sim_desc + w_combined * sim_combined</pre><p>The classifier is simple — keyword heuristics for now, a fine-tuned intent model in production. But even the heuristic version meaningfully changes which documents surface for brand vs. feature queries.</p><h3>Hybrid Search: Adding BM25 to the Mix</h3><p>Semantic search is great at understanding <em>meaning</em>. But it’s terrible at exact-term matching. Ask for “GPT-4o” and a pure semantic search might return results about language models in general, missing the specific document that mentions the model by name.</p><p>BM25 — the classic TF-IDF-based keyword scorer — excels here. The combination is almost always better than either alone.</p><p>To merge semantic and keyword results, I used <strong>Reciprocal Rank Fusion (RRF)</strong>:</p><pre>rrf_score = (<br>    semantic_weight / (k + semantic_rank)<br>  + bm25_weight    / (k + bm25_rank)<br>)</pre><p>Where k=60 is the standard smoothing constant. RRF is elegant: it doesn&#39;t require score normalization across systems (semantic scores and BM25 scores live on completely different scales), and it&#39;s been shown empirically to outperform linear score combination in most settings.</p><h3>The Stack: Entirely Free and Local</h3><p>One constraint I set for myself: <strong>no API keys, no paid services, everything runs locally.</strong></p><p>ComponentChoiceCostEmbeddingssentence-transformers/all-MiniLM-L6-v2FreeVector DBChromaDB (persistent, local)FreeBM25rank-bm25FreeDataHuggingFace datasets (Wikipedia, AG News, SciFact, SQuAD)FreeLLM (optional)Ollama local / HuggingFace pipelineFree</p><p>ChromaDB is particularly good for this use case — it stores embeddings to disk, persists across restarts, and natively supports cosine distance with HNSW indexing. Three collections for three embedding types, and they all live in the same local directory.</p><h3>Running It</h3><p><a href="https://github.com/sulbhajain/embedding-explore/blob/main/README.md">https://github.com/sulbhajain/embedding-explore/blob/main/README.md</a></p><pre>git clone https://github.com/yourusername/embedding-explore<br>cd embedding-explore<br>pip install -e .</pre><pre># Show the cosine similarity / paper demo<br>python examples/cosine_demo.py</pre><pre># Build and query a RAG pipeline over AG News (200 articles)<br>python examples/rag_demo.py --source ag_news --max-docs 200</pre><pre># Same, but with a local Ollama LLM for generation<br>python examples/rag_demo.py --source wikipedia --max-docs 300 --backend ollama --model llama3.2</pre><p>Or in Python directly:</p><pre>from embedding_explore import RAGPipeline<br>from embedding_explore.data.ingest import DataIngester</pre><pre>docs = DataIngester().load(source=&quot;wikipedia&quot;, max_docs=300)</pre><pre>pipeline = RAGPipeline(persist_dir=&quot;./my_db&quot;)<br>pipeline.ingest(docs)   # pre-computes 3 embeddings/doc, stores in ChromaDB</pre><pre>response = pipeline.query(&quot;What is machine learning?&quot;)<br>response.pretty_print()  # shows answer + ranked sources with scores</pre><h3>What This Changed For Me</h3><p>Before this project, I treated cosine similarity as a given — something you apply after you have embeddings and before you look at results. The paper broke that assumption cleanly. The <em>how you train</em> matters as much as <em>which metric you use at inference</em>.</p><p>And the multi-embedding architecture is one of those ideas that feels obvious in retrospect but isn’t how most tutorial RAG pipelines are built. Tutorials give you one vector per chunk. Production search systems — the ones that actually work at scale — segment semantic information into separate channels and let the retrieval layer decide how to weight them.</p><p>The full code is on GitHub. It’s a package, not a notebook — importable, modular, with clean interfaces between the similarity layer, the retrieval layer, and the data ingestion layer. Build on it.</p><p><em>References:</em></p><ul><li><em>Is Cosine-Similarity of Embeddings Really About Similarity?</em> — Steck et al., ACM WWW 2024. <a href="https://arxiv.org/abs/2403.05440">arxiv.org/abs/2403.05440</a></li><li><em>Building a Search System That Actually Understands User Intent</em> — Token by Token, Hashnode</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f285262fdeb9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Hotel Booking AI Agent with GRPO and Live Database Tools]]></title>
            <link>https://sulbhajain.medium.com/building-a-hotel-booking-ai-agent-with-grpo-and-live-database-tools-c3533636af66?source=rss-82b008267a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3533636af66</guid>
            <category><![CDATA[llm-finetuning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[grpo]]></category>
            <dc:creator><![CDATA[Sulbha Jain]]></dc:creator>
            <pubDate>Sun, 08 Mar 2026 05:50:44 GMT</pubDate>
            <atom:updated>2026-03-08T05:50:44.963Z</atom:updated>
            <content:encoded><![CDATA[<h4>How I used Reinforcement Fine-Tuning to teach Qwen3–0.6B to call real PostgreSQL functions — accurately</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rNh-sMu3rPejP--3" /><figcaption>Photo by <a href="https://unsplash.com/@bugaiova_valeriya?utm_source=medium&amp;utm_medium=referral">Valeriia Bugaiova</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>What if you could take a tiny 600M parameter language model and teach it to manage hotel reservations against a live database — searching by location, booking rooms, updating check-in dates, and handling cancellations — all through natural conversation?</p><p>That’s exactly what this project does. And the secret ingredient isn’t a bigger model or thousands of labeled examples. It’s <strong>GRPO</strong> — Group Relative Policy Optimization — a reinforcement learning algorithm that lets you train a model against a live reward signal instead of frozen labels.</p><p>Let me walk you through the full implementation, from spinning up a PostgreSQL server to watching the model learn to call the right tools.</p><h3>The Setup: A Real Database, Not a Mock</h3><p>Most LLM tool-calling tutorials cheat a little — they mock the database or hardcode responses. This project doesn’t. It spins up an actual <strong>PostgreSQL server inside Google Colab</strong>, creates a real hotels table, and populates it with 10 hotels across Swiss cities.</p><pre># Install and start PostgreSQL<br>sudo apt-get install postgresql<br>sudo service postgresql start</pre><pre>-- Create a dedicated database and user<br>CREATE USER hotel_user WITH PASSWORD &#39;my-password&#39;;<br>CREATE DATABASE hotel_db;<br>GRANT ALL PRIVILEGES ON DATABASE hotel_db TO hotel_user;</pre><p>The hotels table schema is clean and practical:</p><pre>CREATE TABLE hotels(<br>   id            INTEGER NOT NULL PRIMARY KEY,<br>   name          VARCHAR NOT NULL,<br>   location      VARCHAR NOT NULL,<br>   price_tier    VARCHAR NOT NULL,  -- &#39;Luxury&#39;, &#39;Upscale&#39;, &#39;Upper Midscale&#39;, etc.<br>   checkin_date  DATE    NOT NULL,<br>   checkout_date DATE    NOT NULL,<br>   booked        BIT     NOT NULL<br>);</pre><p>Hotels seeded into the database include real-sounding properties: Hilton Basel, Marriott Zurich, InterContinental Geneva, Radisson Blu Lucerne — spread across Basel, Zurich, Bern, Lucerne, and Geneva, at various price tiers.</p><h3>The Tools: Python Functions as the Agent’s Hands</h3><p>The agent doesn’t write raw SQL. Instead, it calls five Python functions that are exposed as tools. Each function connects to PostgreSQL via psycopg2, performs the operation, and returns structured results.</p><pre>def search_hotels_by_name(name: str) -&gt; List[tuple]:<br>    &quot;&quot;&quot;Search for hotels based on name.&quot;&quot;&quot;<br>    conn = get_connection()<br>    with conn.cursor() as cur:<br>        cur.execute(&quot;SELECT * FROM hotels WHERE name ILIKE %s&quot;, (&#39;%&#39; + name + &#39;%&#39;,))<br>        return cur.fetchall()</pre><pre>def search_hotels_by_location(location: str) -&gt; List[tuple]:<br>    &quot;&quot;&quot;Search for hotels based on location.&quot;&quot;&quot;<br>    conn = get_connection()<br>    with conn.cursor() as cur:<br>        cur.execute(&quot;SELECT * FROM hotels WHERE location ILIKE %s&quot;, (&#39;%&#39; + location + &#39;%&#39;,))<br>        return cur.fetchall()</pre><pre>def book_hotel(hotel_id: str):<br>    &quot;&quot;&quot;Book a hotel by its ID.&quot;&quot;&quot;<br>    conn = get_connection()<br>    with conn.cursor() as cur:<br>        cur.execute(&quot;UPDATE hotels SET booked = B&#39;1&#39; WHERE id = %s&quot;, (hotel_id,))<br>    conn.commit()</pre><pre>def update_hotel(hotel_id: str, checkin_date: str, checkout_date: str) -&gt; str:<br>    &quot;&quot;&quot;Update a hotel&#39;s check-in and check-out dates.&quot;&quot;&quot;<br>    ...</pre><pre>def cancel_hotel(hotel_id: str):<br>    &quot;&quot;&quot;Cancel a hotel by its ID.&quot;&quot;&quot;<br>    conn = get_connection()<br>    with conn.cursor() as cur:<br>        cur.execute(&quot;UPDATE hotels SET booked = B&#39;0&#39; WHERE id = %s&quot;, (hotel_id,))<br>    conn.commit()</pre><p>These functions have proper docstrings and typed signatures — which matters, because the GRPOTrainer uses these to auto-generate tool schemas that get injected into the model&#39;s context. The model learns to output structured &lt;tool_call&gt; JSON that maps to these exact function signatures.</p><h3>The Dataset: 4 Examples Is All You Need</h3><p>Here’s the part that will raise eyebrows: the entire training dataset is <strong>4 examples</strong>. That’s it.</p><pre>dataset = Dataset.from_list([<br>    {<br>        &quot;prompt&quot;: [<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: prompt},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Find hotels in Basel with Basel in it&#39;s name.&quot;},<br>        ],<br>        &quot;answer&quot;: &quot;Hilton Basel&quot;<br>    },<br>    {<br>        &quot;prompt&quot;: [<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: prompt},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Can you book the Hilton Basel for me?&quot;},<br>        ],<br>        &quot;answer&quot;: &quot;booked&quot;<br>    },<br>    {<br>        &quot;prompt&quot;: [..., {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Oh wait, this is too expensive. Please cancel it and book the Hyatt Regency instead.&quot;}],<br>        &quot;answer&quot;: &quot;Hyatt Regency Basel&quot;<br>    },<br>    {<br>        &quot;prompt&quot;: [..., {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;My check in dates would be from April 10, 2024 to April 19, 2024.&quot;}],<br>        &quot;answer&quot;: &quot;updated&quot;<br>    },<br>])</pre><p>Each example pairs a user request with a simple expected keyword in the response — a hotel name, “booked”, “updated”. The model doesn’t need to match exact phrasing; it just needs to get the <em>right outcome</em>.</p><p>This is the power of GRPO: you don’t need to label <em>how</em> the model should respond, only <em>whether</em> the response was correct.</p><h3>The Reward Function: One Clean Signal</h3><p>The reward function is elegant in its simplicity:</p><pre>def accuracy(completions, answer, **kwargs):<br>    predictions = [completion[-1][&quot;content&quot;] for completion in completions]<br>    rewards = [<br>        float(str(ans).lower() in pred.lower())<br>        for pred, ans in zip(predictions, answer)<br>    ]<br>    return rewards</pre><p>For each generated completion, it checks: does the expected answer string appear anywhere in the model’s response? If yes → reward of 1.0. If no → reward of 0.0.</p><p>Binary. Unambiguous. Surprisingly effective.</p><p>The function receives completions — a list of the model&#39;s full tool-calling conversation turns — and answer — the expected keyword from the dataset. It scores each of the num_generations=4 parallel outputs and returns a list of rewards. GRPO then uses these to compute relative advantages across the group.</p><h3>The Training: Qwen3–0.6B + LoRA + GRPO</h3><p>The model is <strong>Qwen3–0.6B</strong> — Alibaba’s newest 600M parameter model, known for strong instruction-following and built-in tool-calling support via its chat template.</p><pre>checkpoint = &quot;Qwen/Qwen3-0.6B&quot;<br>tokenizer = AutoTokenizer.from_pretrained(checkpoint, padding_side=&quot;left&quot;)<br>model = AutoModelForCausalLM.from_pretrained(checkpoint, dtype=&quot;auto&quot;, device_map=&quot;auto&quot;)</pre><p>To keep training memory-efficient, <strong>LoRA</strong> (Low-Rank Adaptation) is applied across all linear layers:</p><pre>lora_config = LoraConfig(<br>    task_type=&quot;CAUSAL_LM&quot;,<br>    r=8,<br>    lora_alpha=16,<br>    target_modules=&quot;all-linear&quot;,<br>)<br>model = get_peft_model(model, lora_config)</pre><p>With r=8, only a tiny fraction of the model&#39;s parameters are trained — the rest are frozen. This makes GRPO on a T4 GPU (free Colab tier!) totally feasible.</p><p>The training config:</p><pre>training_args = GRPOConfig(<br>    output_dir=&quot;hotel-agent&quot;,<br>    learning_rate=1e-4,<br>    per_device_train_batch_size=2,<br>    gradient_accumulation_steps=2,<br>    num_generations=4,    # 4 rollouts per prompt<br>    num_train_epochs=16,<br>    beta=0.05,            # KL penalty coefficient<br>    report_to=&quot;wandb&quot;,<br>)</pre><p>A few things worth noting:</p><ul><li><strong>num_generations=4</strong> — For each prompt, the model generates 4 different responses. These form the &quot;group&quot; in Group Relative Policy Optimization. Rewards are normalized within this group.</li><li><strong>beta=0.05</strong> — This is the KL-divergence penalty that prevents the policy from drifting too far from the base model. A small value like 0.05 keeps training stable without over-constraining learning.</li><li><strong>16 epochs</strong> on 4 examples = 64 total training steps, which completed in about 47 minutes on a T4.</li></ul><p>The GRPOTrainer is initialized with the <strong>actual Python tool functions</strong> passed directly:</p><pre>trainer = GRPOTrainer(<br>    model=model,<br>    args=training_args,<br>    processing_class=tokenizer,<br>    train_dataset=dataset,<br>    tools=[<br>        search_hotels_by_location,<br>        search_hotels_by_name,<br>        book_hotel,<br>        update_hotel,<br>        cancel_hotel,<br>    ],<br>    reward_funcs=accuracy,<br>)</pre><p>This is where the magic happens — TRL’s GRPOTrainer uses these function signatures and docstrings to build tool schemas, inject them into the system prompt, and evaluate the model&#39;s tool-calling completions against the live reward function.</p><h3>Training Results: Loss Converges Quickly</h3><p>The training loss progression tells an interesting story:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*6MTZXBK9bS5bbE77U3so_Q.png" /></figure><p>Loss hits near-zero very early (step 10), spikes briefly at step 20 as the model explores diverse strategies, then settles to near-zero again. The final training loss was <strong>0.005606</strong> — indicating the model has strongly converged on the rewarded behavior.</p><p>This pattern is characteristic of GRPO on small, focused datasets: fast initial learning followed by refinement.</p><h3>Inference: The Model Calls the Right Tools</h3><p>After training, the model is tested on new queries using Qwen3’s built-in chat template with enable_thinking=False (direct tool-calling mode, no chain-of-thought):</p><p><strong>Query: “Find a hotel in Bern”</strong></p><pre>&lt;tool_call&gt;<br>{&quot;name&quot;: &quot;search_hotels_by_location&quot;, &quot;arguments&quot;: {&quot;location&quot;: &quot;Bern&quot;}}<br>&lt;/tool_call&gt;</pre><p>✅ Correct — picks search_hotels_by_location, not search_hotels_by_name.</p><p><strong>Query: “Book a hotel in Bern for April 4, 2024 thru April 16”</strong></p><pre>&lt;tool_call&gt;<br>{&quot;name&quot;: &quot;book_hotel&quot;, &quot;arguments&quot;: {&quot;hotel_id&quot;: &quot;HOTEL_123&quot;, &quot;checkin_date&quot;: &quot;2024-04-04&quot;, &quot;checkout_date&quot;: &quot;2024-04-16&quot;}}<br>&lt;/tool_call&gt;</pre><p>✅ Correct tool, correct date format (ISO 8601), correct parameter names.</p><p>The model has learned to:</p><ul><li>Distinguish between name-based and location-based search</li><li>Recognize booking vs. cancellation intent</li><li>Parse natural language dates into structured formats</li><li>Output valid JSON inside &lt;tool_call&gt; tags</li></ul><p>All from 4 training examples and a binary reward function.</p><h3>Why This Works: The GRPO Intuition</h3><p>Standard fine-tuning would require you to label <em>exactly</em> what the model should output — every token of the tool call, the JSON structure, the parameter values. That’s expensive and brittle.</p><p>GRPO sidesteps this entirely. Here’s the loop:</p><ol><li><strong>Sample 4 responses</strong> to each prompt in the dataset</li><li><strong>Score each response</strong> — does the expected keyword appear? 1.0 or 0.0</li><li><strong>Normalize scores within the group</strong>: responses better than the group average get positive advantage; worse ones get negative</li><li><strong>Update the policy</strong> to increase probability of high-advantage responses while the KL term prevents collapse</li></ol><pre>Advantage(i) = (reward(i) - mean(group_rewards)) / std(group_rewards)</pre><p>The model doesn’t need to be told “output search_hotels_by_location when asked about a city.&quot; It discovers that calling the right tool with the right arguments is what produces rewards — and reinforces that behavior.</p><p>Over 64 steps, what starts as occasionally correct tool calls becomes reliably correct ones.</p><h3>What Makes This Architecture Special</h3><p>A few design choices make this notebook particularly well-engineered:</p><p><strong>Real infrastructure, not mocks.</strong> Running actual PostgreSQL means the tool functions genuinely execute SQL against a live database. There’s no gap between training environment and deployment — you can literally query the hotels table and see booked = 1 after the agent books a room.</p><p><strong>Tools-as-Python-functions.</strong> By passing raw Python functions with docstrings to GRPOTrainer, the schema generation is automatic. Change a function signature and the tool schema updates automatically. This is much more maintainable than hand-crafting JSON schemas.</p><p><strong>Minimal reward complexity.</strong> The accuracy function is 3 lines. It doesn&#39;t try to parse the tool call structure, validate JSON, or check argument types — it just looks for the expected outcome string. This simplicity avoids reward hacking: there&#39;s no clever way for the model to get a reward of 1.0 without actually producing a useful response.</p><p><strong>Qwen3’s native tool calling.</strong> The model’s built-in chat template handles the &lt;tool_call&gt; format natively, meaning the model has strong priors about tool-calling syntax from pretraining. GRPO refines <em>which</em> tool to call and <em>when</em>, rather than teaching tool-call syntax from scratch.</p><h3>Extending This Pattern</h3><p>The hotel booking agent is a proof of concept, but the pattern generalizes to any domain where you have:</p><ul><li>A set of callable functions (database operations, API calls, internal tools)</li><li>A way to verify whether the model’s response was correct (keyword match, execution result, regex check)</li><li>A small handful of representative example queries</li></ul><p>Some directions to explore from here:</p><p><strong>Richer reward functions</strong> — instead of binary keyword matching, execute the tool call and check if the database state matches expectations after the operation.</p><p><strong>Multi-turn conversations</strong> — the current dataset covers single-turn requests. Real booking scenarios involve clarification (“which hotel did you mean?”), confirmation, and follow-up actions.</p><p><strong>Larger models</strong> — Qwen3–0.6B is impressively capable for its size, but running the same setup with Qwen3–7B or Qwen3–14B would likely produce more reliable multi-step reasoning with complex requests.</p><p><strong>More tools</strong> — add functions like get_available_rooms, apply_discount, or send_confirmation_email to handle richer booking workflows.</p><h3>Reproducing It Yourself</h3><p>The full notebook runs end-to-end on a <strong>free Colab T4 GPU</strong> in under an hour. Dependencies are minimal:</p><pre>pip install git+https://github.com/huggingface/transformers.git<br>pip install git+https://github.com/huggingface/trl.git<br>pip install psycopg2-binary peft wandb</pre><p>PostgreSQL is installed directly in the Colab environment — no external database service required.</p><p>The full code is available here: 👉 <a href="https://github.com/sulbhajain/Gen-AI/blob/main/GRPO/GRPO_custom_DB_funtions_fixed.ipynb">GRPO_custom_DB_functions_fixed.ipynb</a></p><h3>Takeaway</h3><p>This project shows something important: you don’t need a massive dataset, a complex reward model, or an enormous base model to get meaningful results with reinforcement fine-tuning.</p><p>Four examples. One binary reward function. Sixty-four training steps. A 600M parameter model learns to reliably call the right database tools in response to natural language hotel requests.</p><p>GRPO’s real promise isn’t that it replaces supervised fine-tuning — it’s that it makes fine-tuning accessible for the long tail of domains where labeled data is scarce but <em>correct behavior is recognizable</em>. If you can write a function that returns 1 when the model does the right thing and 0 when it doesn’t, you have everything you need to train a specialized agent.</p><p>That’s a remarkably low bar to clear — and a remarkably high ceiling for what you can build on top of it.</p><p><em>The notebook was built and tested on Google Colab with a T4 GPU. Training took approximately 47 minutes for 64 steps across 16 epochs.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3533636af66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Agentic RL Breaks (and How rStar2-Agent Fixes It) — Paper Review]]></title>
            <link>https://sulbhajain.medium.com/why-agentic-rl-breaks-and-how-rstar2-agent-fixes-it-paper-review-59e6f3fb9e01?source=rss-82b008267a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/59e6f3fb9e01</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[grpo]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[rl4llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Sulbha Jain]]></dc:creator>
            <pubDate>Fri, 06 Mar 2026 19:24:44 GMT</pubDate>
            <atom:updated>2026-03-06T19:24:44.252Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wwkGk7nBum-7BKPQ9fLKDA.png" /><figcaption><a href="https://arxiv.org/abs/2508.20722">https://arxiv.org/abs/2508.20722</a></figcaption></figure><p>If you’ve ever watched an LLM use tools during reasoning, you’ve seen the magic:</p><ul><li>It writes Python to verify steps.</li><li>It catches arithmetic mistakes.</li><li>It iterates toward better answers.</li></ul><p>But there’s a hidden problem most people miss: <strong>tool environments are noisy</strong>.</p><p>A code call might fail with a syntax error, timeout, or malformed response — yet the model can still get the final answer right by luck. If your reward is only based on final correctness, training will reward these messy trajectories too.</p><p>That is the core insight behind <strong>rStar2-Agent</strong>: don’t complicate reward design — <strong>clean the rollout data instead</strong>.</p><h4>The paper in one line</h4><p>The Microsoft Research report introduces <strong>GRPO-RoC (Resample-on-Correct)</strong>, a trajectory selection strategy for agentic RL that filters noisy positive trajectories while preserving diverse negative ones.</p><p>🔗 Paper: <strong>rStar2-Agent: Agentic Reasoning Technical Report</strong><br><a href="https://arxiv.org/abs/2508.20722">https://arxiv.org/abs/2508.20722</a></p><h4>The failure mode of vanilla GRPO</h4><p>In a standard GRPO setup:</p><ol><li>Sample a group of rollouts.</li><li>Compute group-normalized advantages.</li><li>Update using all trajectories.</li></ol><p>The issue in agentic settings is subtle but severe:</p><ul><li>Two trajectories can both receive reward = 1.</li><li>One is clean and tool-reliable.</li><li>The other is noisy (tool errors, formatting issues), but still ends with a lucky-correct answer.</li></ul><p>Vanilla GRPO treats them similarly. Over time, this can reinforce brittle tool-use behavior and lead to plateaus.The failure mode of vanilla GRPO</p><p>In a standard GRPO setup:</p><ol><li>Sample a group of rollouts.</li><li>Compute group-normalized advantages.</li><li>Update using all trajectories.</li></ol><p>The issue in agentic settings is subtle but severe:</p><ul><li>Two trajectories can both receive reward = 1.</li><li>One is clean and tool-reliable.</li><li>The other is noisy (tool errors, formatting issues), but still ends with a lucky-correct answer.</li></ul><p>Vanilla GRPO treats them similarly. Over time, this can reinforce brittle tool-use behavior and lead to plateaus.</p><h4>The GRPO-RoC idea (simple and clever)</h4><p>GRPO-RoC changes sampling, not the reward function:</p><ul><li><strong>Oversample</strong> 2G rollouts.</li><li>Split into positive (reward=1) and negative (reward=0) sets.</li><li>For positives, score quality with a penalty:</li></ul><p>p_err = tool error rate</p><p>p_format = formatting violation score</p><p>p_total = p_err + p_format</p><ul><li>Keep cleaner positives (lower p_total) with higher probability.</li><li>Downsample negatives uniformly to preserve failure diversity.</li></ul><p>So the algorithm still learns from success/failure, but it is more selective about <strong>which successes deserve reinforcement</strong>.</p><p>rStar2-Agent boosts a pre-trained 14B model to state-of-the-art levels in only 510 RL steps within one week, achieving 80.6% and 69.8% average pass@1 on AIME24 and AIME25, surpassing DeepSeek-R1 (671B) with shorter responses.</p><p>Read that again. A 14-billion parameter model, trained for about a week, beating a 671-billion parameter model. And doing it with <em>shorter</em> outputs.</p><p>How? Three innovations working together:</p><p><strong>1. GRPO-RoC (Resample-on-Correct):</strong> A modified reinforcement learning training algorithm that changes <em>how</em> rollouts are selected for gradient updates. Instead of treating all rollouts equally, it oversamples correct trajectories, then filters them by quality — keeping only the ones with the fewest tool call errors and the cleanest answer formatting. Negative (incorrect) trajectories are uniformly downsampled to a fixed count.</p><p><strong>2. Reliable Code Infrastructure:</strong> The model doesn’t just reason in text. It writes Python code, executes it in a sandboxed interpreter, reads the result, and adjusts its reasoning based on what the code returns. This is multi-turn agentic tool use, and building infrastructure that makes this fast and reliable at training scale was a major contribution.</p><p><strong>3. A Multi-Stage Training Recipe:</strong> Start with non-reasoning SFT (teaching the model to follow tool-use formatting), then run multiple RL stages with increasing problem difficulty and decreasing max response length. Force the model to get more efficient, not just more accurate.</p><p>The paper’s central hypothesis is this: <strong>in a noisy tool environment, the quality of the training signal matters more than its quantity.</strong> Filtering out the “lucky correct” trajectories that happened to work despite poor tool use is the key move. RoC isn’t about getting more data — it’s about getting cleaner data.</p><h4>I built a toy notebook to make this intuitive</h4><p>The paper is <strong>rStar2-Agent</strong> (arXiv:2508.20722), out of Microsoft Research. The toy is rstar2_agent_toy.ipynb, a single Jupyter notebook using only numpy and matplotlib. No GPUs. No real language model. No Python code interpreter. Just enough scaffolding to understand the algorithm&#39;s logic.</p><p>Here’s what I learned — and, importantly, where the toy diverges from what the paper actually claims.</p><p>I created a runnable notebook that mirrors the paper’s core mechanics with a simplified environment:</p><ul><li>Real LLM → mock policy with controllable skill</li><li>Real math benchmark → synthetic integer math tasks</li><li>Real tool execution → noisy simulator (errors/format issues)</li><li>Full RL weight updates → trajectory sampling + advantage dynamics</li></ul><p>🔗 GitHub repo:<br><a href="https://github.com/sulbhajain/agentic-reasoning-toolcalling">https://github.com/sulbhajain/agentic-reasoning-toolcalling</a></p><p>Notebook file: rstar2_agent_toy.ipynb</p><p>The notebook (rstar2_agent_toy.ipynb) is a deliberately simplified simulation. It demonstrates the <em>structural logic</em> of GRPO vs. GRPO-RoC without any of the actual ML machinery. Here&#39;s what it models:</p><p><strong>Synthetic math dataset:</strong> Problems are generated with known difficulty levels. Each “trajectory” is a simulated multi-step reasoning path — not from a real LM, but sampled with tunable noise parameters.</p><p><strong>Tool error simulation:</strong> Each rollout has a random chance of triggering a tool error (wrong format, execution failure) drawn from a Bernoulli distribution with rate env_error_rate. This creates the &quot;noisy tool environment&quot; that makes RoC theoretically valuable.</p><p><strong>Quality penalty:</strong> p_total = p_err + p_format — a combined penalty term that penalizes format violations and tool call errors.</p><p><strong>Reward function:</strong> Outcome-only reward — 1 if the final answer is correct, 0 otherwise. No step-level rewards.</p><p><strong>Vanilla GRPO loop:</strong> Standard group relative policy optimization — sample G rollouts per problem, normalize rewards within the group, compute policy gradient.</p><p><strong>GRPO-RoC loop:</strong> Oversample 2G rollouts per problem, filter the <em>positive</em> rollouts by quality score (keeping the cleanest), downsample <em>negative</em> rollouts uniformly to a fixed count, then compute the policy gradient on this curated batch.</p><p><strong>Comparison metrics:</strong> Final-step accuracy, response length, tool error rate across training steps — plotted side by side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tm4FH3eKneYGM1f2" /><figcaption><a href="https://github.com/sulbhajain/agentic-reasoning-toolcalling/blob/main/rstar2_agent_toy.ipynb">https://github.com/sulbhajain/agentic-reasoning-toolcalling/blob/main/rstar2_agent_toy.ipynb</a></figcaption></figure><h4>What the toy implementation shows</h4><p>The notebook includes:</p><ul><li>End-to-end rollout generation</li><li>Vanilla GRPO baseline</li><li>GRPO-RoC training loop</li><li>Comparative metrics: Accuracy, Tool error rate in positive trajectories, Response length</li><li>Visual diagnostics (rstar2_results.png)</li><li>A run-based summary cell that reports observed outcomes without overclaiming replication</li></ul><p>The notebook does something genuinely useful: it makes the <em>structure</em> of RoC legible. Before you can understand why RoC might matter at scale, you need to understand the mechanical difference between it and vanilla GRPO.</p><p>Running the notebook, you can clearly see:</p><ul><li>The oversample-then-filter loop in action</li><li>How the quality score gates which positives enter the training batch</li><li>That the training loop does produce shorter trajectories under RoC (the notebook reproduces this finding)</li><li>The tension between outcome-only rewards and quality-adjusted sampling</li></ul><p>The two-cell walkthrough of a single trajectory trace is particularly good pedagogy — it shows step by step how a rollout is evaluated, scored, and either kept or discarded under each algorithm. If you’ve read the paper and want to develop intuition for the mechanism, running this notebook is 30 minutes well spent.</p><p>An important takeaway from building this: <strong>run-level behavior in toy environments is highly sensitive to dynamics and hyperparameters</strong>. That’s why the notebook now explicitly frames results as run-specific simulation outcomes, not benchmark claims.</p><h4>Toy implementation diverges</h4><p>Here’s where it gets interesting.</p><blockquote>“GRPO had higher final-step accuracy than GRPO-RoC in the latest run captured in the notebook output.”</blockquote><p>That’s the opposite of what the paper claims. And it’s not a bug — it’s a lesson. Let me explain why it happens.</p><ul><li><strong>Gap 1: Random noise vs. structured failure: </strong>The paper’s RoC is valuable because tool errors in a <em>real code interpreter</em> are <strong>informationally structured</strong>. When a 14B model generates syntactically broken Python, that’s not random — it correlates with confused reasoning, wrong intermediate conclusions, and likely wrong final answers. Filtering these out of the positive training set removes trajectories that “got lucky” despite poor reasoning quality. In the toy, tool errors are drawn from env_error_rate as i.i.d. Bernoulli flips. The errors are structurally independent of reasoning quality. RoC&#39;s quality filter then has nothing meaningful to select on — it&#39;s culling random noise, not correlated failure modes. The asymmetric sampling reduces your effective batch diversity without improving signal quality, which is why vanilla GRPO can win. <strong>The paper’s hypothesis only holds when errors are causally linked to reasoning quality.</strong> That’s an emergent property of real LM behavior in a real code environment. It cannot be simulated with random noise.</li><li><strong>Gap 2: No policy that actually learns: </strong>In the paper, the policy — Qwen3–14B-Base — generates rollouts and then <em>its weights are updated</em> based on the reward signal. Over 510 steps, the model learns to avoid tool errors because cleaner tool use gets rewarded through RoC’s selection mechanism. The compounding effect is critical: as the model improves, the gap between RoC-selected positives and average positives widens, making the selection increasingly meaningful. The toy has no policy that learns. The trajectory generator is fixed — it keeps producing rollouts from the same distribution regardless of what the “training loop” computes. Without a real adaptive policy, you lose the compounding feedback that is the entire point of RL training.</li><li><strong>Gap 3: The penalty term conflicts with the paper’s design philosophy: </strong>This is subtle but important. The paper explicitly argues that <strong>direct error penalties during early training cause reward hacking and hurt exploration.</strong> The answer-only outcome reward (plus RoC’s implicit quality filtering) is the design choice that avoids this trap. The toy applies p_total = p_err + p_format as an explicit penalty — <em>on top of</em> RoC sampling. This conflates two mechanisms the paper deliberately separates. The penalty signal may dominate the RoC filtering effect, which could explain some of the divergence in results. You&#39;re testing a hybrid that the paper&#39;s design specifically argues against.</li><li><strong>Gap 4: Hyperparameters calibrated to nothing: </strong>The official rStar2-Agent training config uses specific, empirically tuned values:</li></ul><pre>down_sample_to_n=16<br>min_zero_reward_trace_num=2<br>min_non_zero_reward_trace_num=2<br>roc_error_ratio=True<br>roc_answer_format=True</pre><p>These numbers were calibrated against real rollout distributions from a 14B language model trained on competition math. The toy’s analogous parameters — oversample ratio, downsample target, penalty weighting — are set without any such calibration. In a synthetic environment with i.i.d. noise, the “correct” hyperparameters are entirely different. Small changes flip which algorithm wins.</p><h4>Why this matters beyond one paper</h4><p>This pattern generalizes to a lot of agent systems:</p><ul><li>Browser agents</li><li>Code agents</li><li>Retrieval-augmented planning loops</li><li>Multi-tool orchestrators</li></ul><p>Whenever environment feedback is noisy, outcome-only reward can hide poor intermediate behavior. Data/trajectory curation at training time can be a cleaner fix than piling complexity into reward shaping.</p><p>In short: <strong>keep rewards simple, improve what you reinforce</strong>.</p><h4>If you want to extend this</h4><p>A few high-impact next experiments:</p><ol><li>Increase number of seeds and report mean ± std.</li><li>Compare area-under-curve, not only final step metrics.</li><li>Sweep environment noise rate to find where RoC helps most.</li><li>Replace the mock policy with a tiny trainable model.</li><li>Connect selected trajectories to an actual PPO/GRPO gradient update.</li></ol><h4>Final thought</h4><p>Agentic RL’s biggest bottleneck is often not just model intelligence — it’s <strong>credit assignment under noisy interaction traces</strong>.</p><p>rStar2-Agent’s core insight — that <strong>signal quality beats signal quantity</strong> — is a thesis about what makes reinforcement learning for reasoning work. The paper is arguing that the problem with scaling RL for LLMs isn’t compute or data volume; it’s <em>gradient pollution</em> from low-quality trajectories that happen to produce correct answers through bad reasoning.</p><p>This is a much more interesting claim than “we tried harder.” It’s a claim about the <em>mechanism</em> by which RL improves reasoning, and it has implications for how we should think about training reward models, filtering training data, and designing evaluation benchmarks.</p><p>rStar2-Agent’s contribution is elegant because it respects that reality: don’t ask rewards to do everything. Use better rollout selection so your model learns from the right signals.</p><p>If you’re building tool-using agents, this is one idea worth stealing early.</p><p>🔗 GitHub repo:<br><a href="https://github.com/sulbhajain/agentic-reasoning-toolcalling">https://github.com/sulbhajain/agentic-reasoning-toolcalling</a></p><p>🔗 Paper: <strong>rStar2-Agent: Agentic Reasoning Technical Report</strong><br><a href="https://arxiv.org/abs/2508.20722">https://arxiv.org/abs/2508.20722</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59e6f3fb9e01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your RAG System Is Paying a Tax It Doesn’t Owe: REFRAG — Paper Review]]></title>
            <link>https://sulbhajain.medium.com/your-rag-system-is-paying-a-tax-it-doesnt-owe-refrag-paper-review-2909cf24ab82?source=rss-82b008267a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2909cf24ab82</guid>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-retrieval]]></category>
            <category><![CDATA[llm-applications]]></category>
            <dc:creator><![CDATA[Sulbha Jain]]></dc:creator>
            <pubDate>Thu, 05 Mar 2026 07:04:04 GMT</pubDate>
            <atom:updated>2026-03-05T07:04:04.962Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FUBbxbFWlfYedZsc" /><figcaption>Photo by <a href="https://unsplash.com/@duforddigital?utm_source=medium&amp;utm_medium=referral">MJ Duford</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Picture this: your AI assistant gets a question. It retrieves ten relevant documents, stuffs them into the context window, and sends the whole thing to the language model. The model reads every word of every passage — carefully, expensively — and generates an answer.</p><p>Sounds reasonable, right? It’s how nearly every production RAG system works today.</p><p>But there’s a problem buried in that flow, and once you see it, you can’t unsee it. When researchers at Meta Superintelligence Labs, NUS, and Rice University visualized the attention patterns of a LLaMA model processing RAG contexts, they found something striking: the passages form near <strong>block-diagonal</strong> attention structures. That is, tokens within a passage pay strong attention to each other — but tokens <em>across</em> passages barely interact at all.</p><p>The model is doing the quadratic attention math on all of it. And most of that math is unnecessary.</p><p>This observation is the seed of <strong>REFRAG</strong> (arXiv:2509.01092), a paper I recently implemented end-to-end in a toy Python notebook — and one of the most practically important ideas I’ve encountered in applied LLM research. It asks a deceptively simple question: <em>what if RAG decoding was designed specifically for RAG, instead of borrowed from generic long-context inference?</em></p><h4>The Problem with How We Do RAG Today</h4><p>Every token you add to a context window has a cost. Transformer attention scales quadratically with sequence length: double the context, quadruple the compute. This means that in a typical RAG setup with ten retrieved passages of 200 tokens each, you’re paying for 2,000 tokens of input processing — even if only one of those passages actually contains the answer.</p><p>The other nine? The model still reads them fully. It maintains their keys and values in the KV cache. It computes cross-attention between every pair of tokens across all passages. And it makes you wait.</p><p>This latency hit is measured as <strong>TTFT</strong> — time to first token — the interval between submitting your prompt and receiving the first word of a response. In production RAG systems, TTFT is often the primary bottleneck, the metric your users feel most acutely, and the one most stubbornly resistant to optimization.</p><p>The problem isn’t that RAG is slow by nature. It’s that we’ve been treating RAG inference as a generic long-context problem. REFRAG argues it isn’t — and proves it.</p><h4>Compress, Sense, Expand: The Core Idea</h4><p>REFRAG’s framework has three elegant steps, and once you understand them, everything clicks.</p><p><strong>Compress.</strong> Instead of feeding raw tokens from each retrieved passage into the decoder, REFRAG first runs each passage chunk (say, 16 tokens) through a lightweight encoder — something like a small RoBERTa — and compresses it into a single dense embedding. Now those 16 tokens become 1 embedding. Your 2,000-token RAG context shrinks to a fraction of its original length. The decoder’s input length plummets, and with it, the quadratic attention cost.</p><p><strong>Sense.</strong> Not all passage chunks are equally important. An RL-trained policy network looks at the compressed embeddings and asks: which of these are actually relevant to the query? This is the “selective” part — the system senses which chunks are worth expanding back to full resolution.</p><p><strong>Expand.</strong> For the high-signal chunks identified by the policy, REFRAG “expands” them: instead of the compressed embedding, the decoder gets the full original tokens. This preserves precision for the passages that matter while keeping everything else compressed.</p><p>The decoder is then trained (via continual pretraining with curriculum learning) to handle both compressed embeddings and full tokens in the same forward pass. It learns that compressed embeddings carry approximate information, and that expanded chunks carry precise information — and it reasons accordingly.</p><h4>The Numbers Are Remarkable</h4><p>The full paper reports results that, when I first read them, felt almost implausible:</p><p><strong>30.85× TTFT acceleration</strong> compared to the standard LLaMA baseline. That’s not a 30% improvement — that’s the model responding thirty times faster before generating the first token.</p><p><strong>3.75× improvement</strong> over the previous state-of-the-art method (CEPE), meaning REFRAG doesn’t just beat the baseline — it dominates the existing competition.</p><p><strong>16× context extension</strong> — by compressing the retrieval context so aggressively, REFRAG effectively allows LLaMA to handle documents and conversations far longer than its nominal context window, without any architectural changes.</p><p>And critically: these speedups come <strong>with no loss in perplexity</strong>. In some settings, accuracy actually improves, because REFRAG can now afford to include more retrieved passages under the same latency budget — particularly valuable when the retriever is noisy and you need breadth to catch the right answer.</p><h4>What I Built: A CPU-Friendly Toy Reproduction</h4><p>When a paper’s claims are this strong, the best thing you can do is try to break them — or at least understand them well enough to build a toy reproduction that shows the algorithm works from first principles.</p><p>My implementation at <a href="https://github.com/sulbhajain/refrag-rag-decoding">github.com/sulbhajain/refrag-rag-decoding</a> is deliberately lightweight. It runs on CPU with no GPU required, uses a toy encoder and decoder in place of LLaMA and RoBERTa, and includes a built-in HotpotQA-style dataset so you can run experiments without downloading anything. Think of it as a working sketch of the algorithm — every step present, all the heavy machinery replaced with interpretable stand-ins.</p><p>The implementation covers all three REFRAG phases plus the critical supporting components: a chunk encoder with projection layer, a toy decoder with perplexity utilities, an RL policy network (REINFORCE) for selective expansion, a vector store for retrieval, and curriculum learning for pretraining.</p><p><strong>From the notebook runs</strong>, the results paint a clear picture:</p><p>The baseline (full token, no compression) sits at a log-perplexity of roughly 3.91 with a TTFT acceleration of 1.0× — the reference point. REFRAG with increasing compression ratios (k=4, 8, 16, 32) delivers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*dG8yEyxw8zO7n9AzYPYIow.png" /></figure><p>The compressed variants actually achieve <em>slightly lower</em> perplexity than the uncompressed baseline — consistent with the paper’s finding that selective compression can act as a kind of noise filter, de-emphasizing irrelevant passages.</p><p>One honest note: the REFRAG+RL (selective expansion) variant didn’t outperform the uniform compression baseline in my toy run (log-ppl ≈ 4.10 with 2.7× acceleration). This is expected — the RL policy is sensitive to hyperparameters and training capacity. The toy policy network simply doesn’t have the learning capacity that a full-scale production policy would. The uniform compression result, though, holds up cleanly and already tells you something important about the underlying mechanism.</p><h4>Why the Block-Diagonal Insight Changes Everything</h4><p>The thing I find most intellectually satisfying about REFRAG is that it doesn’t just engineer a speedup — it derives one from first principles.</p><p>Think about what RAG actually retrieves. When you query a vector store for “What were the economic causes of the 2008 financial crisis?” and get back ten passages, those passages are diverse by design. Your retrieval pipeline works hard to return varied, non-redundant content. Each passage is internally coherent — the tokens within it are semantically dense — but across passages, there’s low overlap by construction.</p><p>This is structurally different from a long document or a multi-turn conversation. In those settings, context has temporal flow and narrative continuity — the beginning genuinely informs the end. In RAG, the passages are more like parallel witnesses. They don’t talk to each other. They each report to the question independently.</p><p>Transformers paying full cross-passage attention in this setting are solving the wrong problem. REFRAG is the first framework I’ve seen that takes this seriously at the architecture level — not as a heuristic shortcut, but as a principled rethinking of the decoding process.</p><h4>The Modular Architecture: Built to Extend</h4><p>Part of what makes this implementation useful as a learning artifact is how it’s organized. The refrag_toy/ directory breaks the algorithm into clean, composable modules:</p><p>toy_encoder.py handles the chunk encoding and projection — converting raw token sequences into dense embeddings that the decoder can consume. toy_decoder.py implements the decoder and perplexity measurement utilities. rl_policy.py contains the reinforcement learning selective expansion policy. vector_store.py handles retrieval. refrag_system.py wires everything together into an end-to-end pipeline.</p><p>Running the full experiment takes just two commands:</p><pre>uv sync<br>uv run jupyter notebook REFRAG_Toy_Implementation.ipynb</pre><p>The notebook walks through all six experiments: block-diagonal attention simulation, the compression pipeline, baseline vs. REFRAG comparisons across compression rates, RL selective expansion, TTFT acceleration analysis, and curriculum learning ablations.</p><h4>What This Means for Production RAG</h4><p>The practical implications cascade outward pretty quickly once you internalize what REFRAG can do.</p><p>The obvious one is latency. If your users are waiting 3–5 seconds for a first token from your RAG system, REFRAG-style compression could shrink that to under a second — not by faster hardware, but by giving the decoder less to process upfront. For real-time applications, voice assistants, or multi-turn conversations where latency accumulates across turns, this matters enormously.</p><p>The less obvious one is retrieval breadth. Today, most production RAG systems cap retrieval at 5–10 passages because more passages mean more tokens mean more latency. REFRAG decouples retrieval breadth from latency cost. If each passage compresses to 1/16th of its token length, you can retrieve 160 passages for the price you previously paid for 10. In low-recall retrieval scenarios — where the right answer might not be in the top-5 results — this changes the game entirely.</p><p>And then there’s the context extension story. Extending a model’s context window normally requires fine-tuning with new positional encodings and careful distribution shift management. REFRAG achieves 16× context extension as a byproduct of compression — no architectural surgery required.</p><h4>An Honest Accounting</h4><p>I want to be clear about what the toy implementation is and isn’t. It’s not a reproduction of the full-scale REFRAG system. The encoder isn’t RoBERTa. The decoder isn’t LLaMA. The RL policy doesn’t have the capacity of a properly trained production system. The TTFT numbers I’m measuring are simulated, not real GPU wall-clock times.</p><p>What the toy <em>does</em> validate is the algorithmic structure — that the compress-sense-expand flow preserves quality while reducing the decoder’s effective input length, and that increasing compression ratios deliver the expected acceleration scaling. The core mechanism works, and you can verify it on a laptop in 20 minutes.</p><p>The full paper’s claims — 30.85× TTFT, 16× context extension, no perplexity loss at scale — require the full-scale implementation with real models. That’s the natural next step, and one I’m actively exploring.</p><h4>The Bigger Pattern</h4><p>This is the second paper in a row I’ve implemented where the central insight is about <em>structure</em> — the inherent structure of the data flowing through an LLM system, and how little of our current infrastructure is designed to exploit it.</p><p>In the SLM routing work, the insight was that agentic task distributions are narrow and repetitive. In REFRAG, the insight is that RAG attention patterns are sparse and block-diagonal. In both cases, the default approach (send everything to the most powerful general model, process everything as if it were equally important) turns out to be wasteful — not at the margins, but dramatically so.</p><p>I suspect this is a pattern we’ll keep finding. As LLM infrastructure matures, the next round of efficiency gains won’t come primarily from faster hardware or bigger models. They’ll come from systems that are actually designed for the specific structure of the problems they solve.</p><p>REFRAG is a crisp example of what that looks like in practice. And now you can run it on your laptop.</p><p><em>The full REFRAG toy implementation is available at </em><a href="https://github.com/sulbhajain/refrag-rag-decoding"><em>github.com/sulbhajain/refrag-rag-decoding</em></a><em>.</em></p><p><em>The original paper is by Lin, Ghosh, Low, Shrivastava, and Mohan (Meta Superintelligence Labs / NUS / Rice), arXiv:2509.01092.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2909cf24ab82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why I Stopped Using GPT-4 for 70% of My AI Agent Calls? — Paper Review]]></title>
            <link>https://sulbhajain.medium.com/why-i-stopped-using-gpt-4-for-70-of-my-ai-agent-calls-paper-review-139af566d3b1?source=rss-82b008267a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/139af566d3b1</guid>
            <category><![CDATA[small-language-model]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[llm-vs-slm]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-applications]]></category>
            <dc:creator><![CDATA[Sulbha Jain]]></dc:creator>
            <pubDate>Tue, 03 Mar 2026 11:59:04 GMT</pubDate>
            <atom:updated>2026-03-03T11:59:04.783Z</atom:updated>
            <content:encoded><![CDATA[<h4>And Why You Should Too</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tYaU9wXg-Ls56az6" /><figcaption>Photo by <a href="https://unsplash.com/@danielkcheung?utm_source=medium&amp;utm_medium=referral">Daniel Cheung</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There’s a quiet revolution happening in AI infrastructure, and most people are missing it because they’re too busy chasing the biggest, most powerful models. I was one of them — until I implemented a research paper from NVIDIA that fundamentally changed how I think about building agentic AI systems.</p><p>The paper is called <em>“Small Language Models are the Future of Agentic AI”</em> (Belcak et al., NVIDIA Research, arXiv:2506.02153). The title alone felt like a provocation. We’ve spent years assuming that bigger = better. So when I decided to implement it end-to-end in a Jupyter notebook, I honestly expected to disprove it. Instead, I became a convert.</p><p>Here’s what I found — and why it matters for every developer building AI agents today.</p><h4>The Hidden Cost Nobody Talks About</h4><p>When you build an AI agent, you don’t make one API call. You make <em>dozens</em>. A single user interaction can trigger a cascade of subtasks: classify the intent, extract structured data from a document, generate a code snippet, call the right tool, summarize the result. Every one of those subtasks hits a frontier model — and every one costs you tokens.</p><p>At scale, this becomes brutal. Not just financially, but in terms of latency. Your users are waiting while GPT-4 or Claude deliberates over a task that could be: <em>“Is this a refund request or a password reset?”</em></p><p>That’s the insight at the heart of the NVIDIA paper, and it’s deceptively simple: <strong>most tasks inside an agentic pipeline don’t need a genius. They need a specialist.</strong></p><h4>The LLM-to-SLM Conversion Algorithm</h4><p>The paper proposes a six-step algorithm for systematically replacing large cloud-based LLMs with small local models (3B–7B parameters) wherever appropriate. I implemented all six steps using Ollama for local inference with models like Phi-3, Qwen2.5:3b, and Llama3.2:3b.</p><p>Here’s the gist of each step:</p><p><strong>S1 — Data Collection.</strong> You start by logging your agent’s production calls — every prompt, every response. If you don’t have production data yet, you can synthesize realistic conversation logs, which is what my notebook demonstrates. Think of this as instrumenting your pipeline to understand what it actually does all day.</p><p><strong>S2 — Data Curation.</strong> Raw logs are messy and often sensitive. This step involves redacting PII and PHI, then filtering for high-quality examples — responses where the LLM performed well and confidently. You’re building a dataset of “gold standard” inputs and outputs for each task type.</p><p><strong>S3 — Task Clustering.</strong> This is where it gets interesting. Using TF-IDF vectorization and k-means clustering, you group your curated logs into recurring task patterns. What you find is almost always surprising: your agent, despite seeming general-purpose, actually does a small number of things repeatedly. In my experiments, six clusters emerged cleanly — code generation, JSON extraction, intent classification, summarization, tool routing, and complex reasoning.</p><p><strong>S4 — SLM Selection.</strong> For each task cluster, you identify which small local model performs best. Not every SLM is equal — Phi-3 shines on structured output tasks, while a slightly larger Qwen2.5 handles summarization better. This step is essentially matchmaking: pairing the right small model to each narrow job.</p><p><strong>S5 — Distillation.</strong> Here, you improve each SLM’s performance on its assigned tasks through distillation-style prompting — essentially using the LLM’s gold responses as few-shot examples to lift the SLM’s quality. No fine-tuning required (though the paper notes that LoRA fine-tuning would push results even further).</p><p><strong>S6 — Routing Analysis.</strong> Finally, you build a router — a lightweight classifier that looks at an incoming task and decides: SLM or LLM? This step also measures how much routing error your system can tolerate before quality degrades meaningfully.</p><h4>The Numbers That Surprised Me</h4><p>My notebook outputs two key visualizations, and both tell a consistent story.</p><p><strong>The cost-quality chart</strong> shows the tradeoff across all six task types. For five of them — code generation, JSON extraction, intent classification, summarization, and tool routing — the SLM-routed approach delivers quality within 3–5% of the LLM baseline while slashing cost by roughly 70%. The only category that genuinely needs the big model is complex reasoning: architecture decisions, multi-step problem solving, novel domains.</p><p><strong>The routing error analysis</strong> shows something equally important: the system is robust to imperfect routing. Even when the router misdirects 15–20% of calls, overall quality stays high, because the occasional escalation to the LLM catches the edge cases. You don’t need a perfect router. You need a good-enough one.</p><p>The headline result: <strong>60–80% of agent calls can be handled by small local models.</strong> The quality hit is minimal. The cost savings are enormous.</p><h4>Why This Works (The Counterintuitive Part)</h4><p>The instinct is to think that general intelligence is always better. If GPT-4 can do everything, why would you ever use something smaller?</p><p>Because agentic subtasks aren’t general. They’re narrow, repetitive, and format-constrained.</p><p>Consider what “extract the line items from this invoice as JSON” actually requires. It doesn’t require world knowledge or creative reasoning. It requires pattern recognition on a predictable input format. A 3B model that has seen thousands of invoice extraction examples — either through fine-tuning or distillation prompting — will nail this task reliably. And it’ll do it locally, in milliseconds, for a fraction of a cent.</p><p>The same logic applies to intent classification, code templating, and structured transformations. These tasks are the <em>backbone</em> of most production agents. They’re boring. They’re repetitive. And that’s exactly why small models are perfect for them.</p><h4>What I Built and How You Can Try It</h4><p>My full implementation is on GitHub at <a href="https://github.com/sulbhajain/slm_agentic_future">sulbhajain/slm_agentic_future</a>. It’s a single Jupyter notebook — slm_agents_ollama.ipynb — that walks through all six algorithm steps end-to-end.</p><p>You’ll need Ollama installed locally and at least one small model pulled (phi3, qwen2.5:3b, or llama3.2:3b). If you don&#39;t have Ollama set up, the notebook runs in <strong>MOCK_MODE</strong> automatically, simulating responses so you can explore the full algorithm structure without any local models.</p><p>The setup is genuinely lightweight:</p><pre>brew install ollama<br>ollama pull phi3<br>jupyter notebook slm_agents_ollama.ipynb</pre><p>That’s it. In about 20 minutes, you can run the full algorithm and see the cost-quality charts for yourself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s2ZDBnhcS4LmJ2hUPODHlA.png" /><figcaption>Figure 1: Experimental comparison of task quality and cost for SLM and LLMs. Results reveal that except for summarization rest tasks quality are better for SLM. There are substancial cost saving (~95%) for all but complex tasks</figcaption></figure><p>All relatively simpler tasks improve quality and save cost when implemented with SLM vs LLM (Figure 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YNrbXeQs1FcH4Neogp04jw.png" /><figcaption>Figure 2: Overall statistical view of quality and cost metrics for SLM vs LLM.</figcaption></figure><h4>Where I’m Taking This Next</h4><p>This implementation is a proof of concept — deliberately minimal so the ideas come through clearly. But the production extensions are obvious and exciting:</p><p>The rule-based router in my current implementation is a placeholder. Replace it with a tiny 125M parameter classifier SLM and you’ve got a fully local, sub-millisecond routing layer. Add LoRA fine-tuning via Hugging Face PEFT and you’ll close most of that residual 3–5% quality gap. Implement a fallback escalation threshold — SLM → LLM when confidence score drops below some threshold — and the system becomes self-healing.</p><p>The paper itself points toward a future where your AI agent is less like a single powerful model and more like a well-staffed team: specialists handling routine work, experts called in only when the problem is genuinely hard. That’s not a downgrade. That’s good engineering.</p><h4>The Bigger Picture</h4><p>We’re at an inflection point in how we think about language model deployment. The race to the biggest model made sense when we were exploring what was possible. But production AI is increasingly about what’s <em>practical</em> — reliable, fast, affordable, and maintainable at scale.</p><p>Small Language Models aren’t a consolation prize for teams that can’t afford frontier APIs. They’re a deliberate architectural choice for teams that have thought carefully about what their agents actually need to do.</p><p>NVIDIA’s research makes the case rigorously. My notebook makes it tangibly runnable. The results, I think, speak for themselves.</p><p>The future of agentic AI isn’t one giant brain. It’s a network of specialized minds, each expert in its lane.</p><p><em>The full implementation is available on GitHub: </em><a href="https://github.com/sulbhajain/slm_agentic_future"><em>github.com/sulbhajain/slm_agentic_future</em></a><em>.</em></p><p><em>The research paper referenced is Belcak et al., NVIDIA Research, 2025 (arXiv:2506.02153).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=139af566d3b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Sending Every Query to GPT-4]]></title>
            <link>https://sulbhajain.medium.com/stop-sending-every-query-to-gpt-4-d22b96ac1585?source=rss-82b008267a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d22b96ac1585</guid>
            <category><![CDATA[llm-inference]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-applications]]></category>
            <dc:creator><![CDATA[Sulbha Jain]]></dc:creator>
            <pubDate>Wed, 25 Feb 2026 17:24:37 GMT</pubDate>
            <atom:updated>2026-02-25T17:24:37.306Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Practical Guide to Budget-Aware LLM Routing with PILOT — Paper Review</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*dlwIPZf6uKQyggeInCsnFg.png" /><figcaption><a href="https://arxiv.org/pdf/2508.21141">https://arxiv.org/pdf/2508.21141</a></figcaption></figure><p><em>How a bandit-based router can learn when to use expensive models, when to use cheaper ones, and how to stay within budget constraints in production.</em></p><p>If you’re building LLM products, you already know the tension:</p><ul><li>Better models usually produce better answers.</li><li>Better models also cost more.</li><li>At scale, blindly routing all traffic to your strongest model becomes financially unsustainable.</li></ul><p>The paper <strong>Adaptive LLM Routing Under Budget Constraints</strong> proposes a clean idea: treat model selection as an <strong>online decision problem</strong> where you continuously learn which model to call for each query <em>under a spend limit</em>.</p><p>I implemented a toy, CPU-only version of this idea in a notebook to understand it deeply and test whether the core claims hold up in practice.</p><p>This post explains the intuition, what I built, and what the results actually show.</p><h4>The Problem: One-Size-Fits-All LLM Calls Are Expensive</h4><p>Modern applications often integrate with a single LLM endpoint. It’s simple, predictable, and… deeply inefficient.</p><p>Think about the actual distribution of queries hitting your system:</p><ul><li>Some are trivially easy — formatting, keyword lookup, short Q&amp;A</li><li>Some are moderately complex — summarization, classification, light reasoning</li><li>A small minority are genuinely hard — multi-step reasoning, coding, nuanced generation</li></ul><p>A frontier model like GPT-4 or Claude Opus is overkill for the first two categories. A smaller, cheaper model — something like Haiku or Mistral-7B — handles them just as well at a fraction of the cost.</p><p>The dream: <strong>route easy queries to cheap models, hard queries to powerful ones, and stay within a defined budget.</strong></p><p>The challenge: <strong>you don’t know in advance which queries are “hard.”</strong></p><h4>The Solution: PILOT — Preference-Prior LinUCB for LLM Routing</h4><p>This project implements the ideas from the paper <em>“Adaptive LLM Routing Under Budget Constraints”</em> (Panda et al., 2025), which frames query routing as a <strong>contextual bandit problem</strong> with budget constraints.</p><p>The core insight is elegant: instead of trying to predict query difficulty from scratch, we can learn a <em>preference model</em> — a prior that captures, for a given query embedding, which LLMs have historically performed well. Then we use online learning (LinUCB) to refine these preferences in real time as new queries arrive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*RtyoRF1mgbQr9ZpcJDSO5A.png" /><figcaption><a href="https://arxiv.org/pdf/2508.21141">https://arxiv.org/pdf/2508.21141</a></figcaption></figure><p>Here’s the conceptual architecture:</p><p><strong>1. Preference Pretraining (Offline)</strong> Before going live, the router learns a mapping from query features → expected reward for each model. This is the “preference prior” — a warm start that prevents cold-start failures.</p><p><strong>2. PILOT Router (Online Learning)</strong> Once deployed, the router uses a LinUCB-style policy that balances exploration (trying different models to gather signal) with exploitation (routing to the best known model for this query type). The preference prior is baked into the uncertainty estimates, making it much more data-efficient than standard bandit approaches.</p><p><strong>3. Budget-Aware Policy (ZCL-style)</strong> The real world has budgets. PILOT+Budget enforces a hard cost cap across all queries. As spending approaches the limit, the policy automatically shifts toward cheaper models — accepting some quality degradation in exchange for guaranteed budget compliance.</p><p>This is the key tradeoff the paper makes explicit: <strong>total reward vs. budget compliance.</strong> You can’t always have both, and this system lets you choose your point on that curve.</p><h4>The Core Insight</h4><p>Most teams start with one of two naive strategies:</p><ol><li><strong>Always use the best model</strong> → high quality, painful cost.</li><li><strong>Always use the cheapest model</strong> → lower cost, degraded quality.</li></ol><p>PILOT sits in the middle.</p><p>For each incoming query, it asks:</p><blockquote><em>“Given what I’ve learned so far about this type of query, which LLM should I call </em>right now<em> to maximize reward while respecting budget?”</em></blockquote><p>Formally, this is a <strong>contextual multi-armed bandit</strong> setup:</p><ul><li><strong>Context</strong> = query representation (embedding/features)</li><li><strong>Arms</strong> = candidate LLMs (e.g., GPT-4-class, medium, small)</li><li><strong>Reward</strong> = response quality proxy</li><li><strong>Cost</strong> = per-call model price</li></ul><h4>Why PILOT Is Different from Vanilla LinUCB</h4><p>A standard LinUCB bandit starts cold. It explores a lot early because it knows nothing.</p><p>PILOT introduces a smart warm start:</p><ul><li>First, learn from <strong>offline preference data</strong> (which model tends to win on which query types).</li><li>Use that learned preference signal as a <strong>prior</strong> in the online bandit.</li></ul><p>So instead of beginning with “uniform ignorance,” the router begins with informed beliefs.</p><p>In the notebook, this is implemented as:</p><ul><li>Phase 1: learn query projection into a shared space</li><li>Phase 2: learn model preference embeddings</li><li>Online phase: initialize the bandit with those learned preferences</li></ul><p>This directly tests the paper’s claim that a preference-informed prior should reduce regret versus a cold-start bandit.</p><h4>My Notebook Setup (Toy but Useful)</h4><p>The toy notebook I built runs entirely on CPU with no external API calls — everything uses synthetic data and mock LLMs to illustrate the dynamics. Here’s what you’ll see when you run it:</p><ul><li><strong>PILOT (unconstrained)</strong> achieves the highest total reward but will overspend on a tight global budget</li><li><strong>PILOT+Budget</strong> enforces a true hard cap — zero overspend, with a measurable but reasonable drop in reward</li><li><strong>Baselines</strong> (random routing, always-cheap, always-expensive) provide intuitive reference points</li></ul><p>The multi-panel results figure makes the tradeoff viscerally clear. You’ll see the cumulative reward curves diverge over time, and the budget compliance chart shows exactly where each policy stands at any given query count.</p><p>To keep the implementation reproducible and easy to run:</p><ul><li>Synthetic query embeddings (8D)</li><li>Three mock LLMs with different cost/quality profiles</li><li>Stochastic reward function per query type</li><li>No external APIs</li><li>Runs on CPU in about a minute</li></ul><p>Algorithms compared:</p><ul><li>Random routing</li><li>LinUCB (no prior)</li><li>PILOT (preference prior)</li><li>PILOT+Budget (PILOT + online budget policy)</li></ul><h4>What the Results Show</h4><ol><li><strong>Preference prior helps online learning — </strong>In my run:</li></ol><ul><li><strong>LinUCB cumulative regret:</strong> 8.27</li><li><strong>PILOT cumulative regret:</strong> 7.53</li></ul><p>So the preference-informed router does learn faster / more efficiently than a cold-start baseline, which is exactly what the hypothesis predicts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XSw33IwAnZSV4YVn8oxJIQ.png" /><figcaption><a href="https://github.com/sulbhajain/adaptive-llm-routing">https://github.com/sulbhajain/adaptive-llm-routing</a></figcaption></figure><p>2) <strong>Hard budget compliance requires explicit enforcement — </strong>A key practical finding from implementation:</p><ul><li>A soft budget-aware rule can still drift over a strict global cap.</li><li>I added a <strong>hard affordability filter</strong> in the experiment loop (only selectable arms with cost &lt;= remaining budget).</li></ul><p>After this change, PILOT+Budget enforces a true cap.</p><p>3) <strong>Compliance vs reward is a real tradeoff — </strong>With budget set to <strong>$2.00</strong> over deployment queries:</p><ul><li><strong>PILOT:</strong> cost $4.535, reward 125.546 (over budget)</li><li><strong>PILOT+Budget:</strong> cost $2.000, reward 55.516 (within budget)</li></ul><p>So yes, you can enforce budget exactly — but you should expect a drop in absolute reward under tight caps.</p><p>Interestingly, efficiency stayed competitive:</p><ul><li><strong>PILOT reward/$:</strong> 27.68</li><li><strong>PILOT+Budget reward/$:</strong> 27.76</li></ul><p>This is the operational message: under a strict budget, a router can still spend efficiently even when total output quality is constrained by the cap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IILwZdjtuKtDGZ9YbFD-Gg.png" /><figcaption><a href="https://github.com/sulbhajain/adaptive-llm-routing">https://github.com/sulbhajain/adaptive-llm-routing</a></figcaption></figure><h4>Why This Matters in Production</h4><p>Many real LLM systems fail not because quality is bad, but because <strong>unit economics break at scale</strong>.</p><p>A routing policy like PILOT is valuable because it gives you a controllable knob for:</p><ul><li>Quality</li><li>Cost</li><li>Risk of budget overrun</li></ul><p>Instead of asking “Which single model is best?”, you ask:</p><blockquote><em>“Which model is best for this query </em>at this point in the budget cycle<em>?”</em></blockquote><p>That’s a much more production-native framing.</p><h4>What I’d Improve Next</h4><p>If you want to move this from toy simulation toward real deployment:</p><ol><li>Replace synthetic features with real embedding models.</li><li>Plug in actual local or hosted LLM endpoints.</li><li>Add non-stationarity (query distribution shifts).</li><li>Stress test different budget levels and prior strengths.</li><li>Evaluate with task-specific metrics (not just simulated reward).</li></ol><h4>Why This Matters</h4><p>LLM costs are not going to zero. Even as inference gets cheaper, the gap between small and large models persists — and the volume of queries in production systems keeps growing. Intelligent routing is a first-class engineering concern, not an optimization you do later.</p><p>The PILOT framework offers something rare: a <strong>principled, theoretically grounded approach</strong> to a problem that most teams currently solve with ad-hoc heuristics or not at all. The budget constraint handling, in particular, is something I haven’t seen cleanly implemented in most routing libraries.</p><p>If you’re building anything with multiple LLMs — or if you’re paying a single frontier model to answer questions it’s overqualified for — this is worth your afternoon.</p><h4>Final Takeaway</h4><p>The paper’s core idea is strong: <strong>LLM routing should be adaptive, contextual, and budget-aware.</strong></p><p>The notebook implementation supports two practical conclusions:</p><ul><li>Preference-informed routing beats cold-start routing in online learning efficiency.</li><li>Strict budget compliance is achievable, but only if you enforce hard affordability constraints — and that naturally introduces a quality tradeoff under tight budgets.</li></ul><p>If you’re shipping LLM applications to real users, this is the kind of policy layer that turns “cool demo” into “sustainable product.”</p><p><strong>Check out the full implementation:</strong> <a href="https://github.com/sulbhajain/adaptive-llm-routing">github.com/sulbhajain/adaptive-llm-routing</a></p><p><strong>Read the paper:</strong> Panda et al., “Adaptive LLM Routing Under Budget Constraints” (arXiv: 2508.21141)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d22b96ac1585" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Quiet Revolution in AI Reasoning]]></title>
            <link>https://sulbhajain.medium.com/the-quiet-revolution-in-ai-reasoning-why-the-best-thinking-happens-in-silence-0a66243a25ee?source=rss-82b008267a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a66243a25ee</guid>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-reasoning]]></category>
            <dc:creator><![CDATA[Sulbha Jain]]></dc:creator>
            <pubDate>Mon, 23 Feb 2026 19:01:36 GMT</pubDate>
            <atom:updated>2026-02-23T19:46:40.630Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why the Best Thinking Happens in Silence — Paper Review</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PGLorG155XxEfRVHdx-AoA.png" /><figcaption><a href="https://arxiv.org/pdf/2509.02350">https://arxiv.org/pdf/2509.02350</a></figcaption></figure><p>You’ve probably marveled at how ChatGPT or Claude “thinks out loud” — showing you step-by-step arithmetic, walking you through logical deductions, narrating its thought process before landing on an answer. This is Chain-of-Thought (CoT) reasoning, and for the past few years it’s been the dominant way we’ve gotten AI systems to solve hard problems.</p><p>But a fascinating new survey — <strong>“Implicit Reasoning in Large Language Models: A Comprehensive Survey”</strong> (Li et al., arXiv 2509.02350) — is making the case that this explicit, wordy form of reasoning is just the beginning. The next frontier? Teaching AI to <em>think without words</em> — reasoning entirely inside its own hidden layers, silently, before surfacing an answer.</p><p>This isn’t a philosophical curiosity. It’s a practical leap with enormous implications for cost, speed, and how deeply AI can actually reason.</p><h4>First: What’s the Difference?</h4><p>Let me make this concrete with the paper’s own example.</p><p><strong>Explicit Reasoning (Chain-of-Thought):</strong></p><blockquote>Question: Samantha had 5 packs of markers. Each pack had 12 markers. She gave 9 markers to her friend and lost 3. How many markers does she have?</blockquote><blockquote><em>Step 1: 5 × 12 = 60 markers total. Step 2: 60 − 9 = 51. Step 3: 51 − 3 = 48. </em><strong><em>Answer: 48</em></strong></blockquote><p><strong>Implicit Reasoning:</strong></p><blockquote>Question: [same]<em> → [Layer 1 processes…] → [Layer 2 refines…] → [Layer 3 concludes…] </em><strong><em>Answer: 48</em></strong></blockquote><p>Same answer. But the implicit version never <em>writes down</em> the intermediate steps. The computation happens in the model’s hidden states — its “neural scratchpad” — and only the final answer surfaces as text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3qF_u3-BOEJiPW2M-pPDvA.png" /><figcaption><a href="https://arxiv.org/pdf/2509.02350">https://arxiv.org/pdf/2509.02350</a></figcaption></figure><p>Think of it like this: when a chess grandmaster sees a board, they don’t consciously enumerate every possible move and counter-move. The analysis happens fast, pattern-matched, deep in their cognitive hardware. Explicit CoT is the novice writing out every move. Implicit reasoning is the grandmaster.</p><h4>The Three Paradigms: A New Taxonomy</h4><p>One of the paper’s most valuable contributions is organizing the messy landscape of implicit reasoning research into three clean execution paradigms. Think of these as three different architectural answers to the question: <em>where and how does hidden computation happen?</em></p><h4>1. Latent Optimization — Rewiring the Thinking Tokens</h4><p>This paradigm manipulates the model’s internal representations directly. Instead of “saying” reasoning steps out loud, the model learns to <em>compress</em> them into dense, information-rich latent states.</p><p>Key approaches:</p><ul><li><strong>CoCoMix &amp; Latent Token</strong>: Insert special non-word “concept tokens” into the input stream that carry semantic meaning without being human-readable. Think of them as private shorthand the model writes to itself.</li><li><strong>Coconut (Chain of Continuous Thought)</strong>: Trains the model to reason in a <em>continuous latent space</em> rather than discrete word sequences. ICLR 2025 paper. The model literally thinks in vectors, not words.</li><li><strong>CCoT (Compressed Chain-of-Thought)</strong>: Compresses an entire reasoning chain into just a handful of dense embedding vectors. The full thought, distilled to its essence.</li><li><strong>CoT-Valve &amp; LightThinker</strong>: Dynamically adjust <em>how much</em> the model thinks based on question difficulty. Easy question? Brief latent trace. Hard math problem? Deep, extended computation.</li></ul><h4>2. Signal-Guided Control — Steering With Special Tokens</h4><p>Rather than rewriting the model’s architecture, this paradigm inserts lightweight <em>control signals</em> — special tokens that tell the model “think harder here” or “you can skip the reasoning this time.”</p><p>Key approaches:</p><ul><li><strong>Thinking tokens / Pause tokens</strong>: Dummy tokens inserted into the sequence that give the model extra computation cycles without generating visible text. Like giving a student extra time on an exam — same inputs, better outputs.</li><li><strong>Thinkless</strong>: Uses &lt;short&gt; and &lt;think&gt; tokens to let the model <em>choose</em> whether to reason explicitly or implicitly, depending on task difficulty. The model becomes self-aware of when it needs to deliberate.</li><li><strong>Memory Reasoning tokens</strong>: Multiple signal types — one for “store this fact,” one for “reason about it” — enabling fine-grained cognitive control.</li></ul><h4>3. Layer-Recurrent Execution — Thinking in Loops</h4><p>Most transformer architectures process tokens once through each layer, top to bottom. This paradigm breaks that constraint by introducing <em>recurrence</em> — letting the model loop through the same layers multiple times, iteratively refining its hidden states.</p><p>Key approaches:</p><ul><li><strong>Looped Transformer / CoTFormer</strong>: Share weights across multiple passes. The model essentially “thinks again” using the same parameters, refining its internal representation with each loop.</li><li><strong>Huginn</strong>: A commercial-style architecture where computation depth is dynamic — some tokens get more processing loops than others, based on difficulty.</li><li><strong>RELAY</strong>: Adds explicit recurrent connections between transformer layers, enabling information to flow backwards and be reconsidered.</li></ul><p>The result? Models that can do more cognitive work with fewer parameters, since the same weights serve multiple reasoning “steps.”</p><h4>How Does It Compare? The Benchmarks Tell the Story</h4><p>The survey reviews <strong>70+ benchmarks</strong> used across these methods. Here’s a structured comparison across the most important evaluation dimensions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y9deD2lXuGtTaypxvJGy3Q.png" /></figure><h4>Key Benchmark Takeaways:</h4><ul><li><strong>On GSM8K (grade school math):</strong> Coconut achieves near-CoT accuracy while generating <em>zero</em> intermediate text. This was the first major proof-of-concept that implicit reasoning can match explicit reasoning on structured problems.</li><li><strong>On multi-hop reasoning (HotpotQA):</strong> Layer-recurrent models like the Looped Transformer match or exceed CoT performance because multi-hop retrieval maps naturally to iterative hidden-state refinement.</li><li><strong>On complex math (MATH dataset):</strong> The performance gap persists. Explicit CoT still wins. But this gap is narrowing as methods like CCoT and Soft Thinking compress multi-step reasoning into better latent representations.</li><li><strong>On efficiency metrics (ACU — Accuracy per Computational Unit):</strong> Implicit methods dominate. When you normalize for latency and token cost, implicit reasoning consistently delivers better ROI.</li></ul><h4>The Evidence That Reasoning Really Is Happening Inside</h4><p>A skeptic might ask: isn’t the model just <em>memorizing</em> answers, not actually “reasoning” in its hidden states?</p><p>The survey addresses this head-on with three categories of mechanistic evidence:</p><p><strong>1. Structural Evidence:</strong> When researchers probe intermediate layer activations, they find distinct subtasks being computed at different layers. Early layers handle syntax; middle layers handle factual retrieval; later layers handle logical inference. Reasoning isn’t a monolith — it’s <em>scheduled</em> across depth.</p><p><strong>2. Behavioral Signatures:</strong> Models trained with implicit reasoning show “grokking” — a sudden phase transition where they go from memorization to genuine generalization. They also exhibit step-skipping on easy problems and deliberate depth on hard ones, suggesting authentic cognitive flexibility.</p><p><strong>3. Representation Analysis:</strong> Using probing classifiers and causal interventions, researchers have found that logical inference circuits, reasoning trees, and multi-hop paths are genuinely encoded in hidden states — not just surface-level pattern matching.</p><p>The model isn’t cheating. It’s genuinely doing the work, just quietly.</p><h4>The Honest Challenges (Because No Revolution Is Clean)</h4><p>The paper is admirably honest about where implicit reasoning falls short:</p><p><strong>Interpretability</strong>: When reasoning happens in latent space, you lose the audit trail. This is a serious problem for high-stakes applications — medical diagnosis, legal reasoning, financial decisions — where humans need to verify the logic.</p><p><strong>Reliability</strong>: Without explicit intermediate steps, there’s no natural checkpoint for error correction. The model can confidently give a wrong answer with no visible reasoning chain to inspect.</p><p><strong>Performance on Hard Tasks</strong>: For genuinely complex multi-step problems — olympiad mathematics, intricate logical proofs — explicit CoT still wins. Implicit methods haven’t cracked this ceiling yet.</p><p><strong>Evaluation Inconsistency</strong>: The field lacks standardized benchmarks. Different papers measure different things in different ways, making apples-to-apples comparison hard. The survey calls for unified evaluation suites — and it’s an urgent need.</p><p><strong>Architecture Lock-In</strong>: Many implicit methods are designed for specific model sizes or architectures and don’t generalize. A method that works brilliantly for a 7B parameter model may fail at 70B.</p><h4>Why This Matters for Practitioners</h4><p>If you’re building AI-powered products, this research has direct implications:</p><p><strong>Cost</strong>: Every token generated costs money. Implicit reasoning that compresses a 500-token CoT chain into a 50-token latent process is a 10× cost reduction. At scale, that’s transformative.</p><p><strong>Latency</strong>: Real-time applications — voice assistants, live coding tools, interactive agents — can’t afford to “think out loud” for 30 seconds before responding. Implicit reasoning enables faster, snappier inference.</p><p><strong>Specialized Models</strong>: Instead of one massive model that talks through every problem, implicit reasoning opens the door to lean, efficient models that have deeply internalized domain knowledge and reason compactly within it.</p><p><strong>Hybrid Systems</strong>: The emerging sweet spot seems to be <em>adaptive</em> reasoning — models that default to implicit (fast, cheap) and escalate to explicit (slow, transparent) only when difficulty demands it. Think System 1 / System 2, but inside a single neural network.</p><h4>The Bigger Picture: What Are We Actually Building?</h4><p>There’s something philosophically interesting at the heart of this research. When we insist AI “show its work,” we’re projecting a human pedagogical norm onto a system that doesn’t think the way we do. We taught math students to show their work so teachers could diagnose errors. But an AI’s errors don’t live in the textual trace — they live in the weights.</p><p>Implicit reasoning is, in some sense, AI finally being allowed to think <em>its own way</em> — through activation patterns, vector arithmetic, recursive layer dynamics — rather than performing cognition in a human-legible theater.</p><p>That’s exciting. It’s also a little unsettling. The interpretability gap is real and must be solved before we trust implicit reasoning in consequential domains.</p><p>But the direction is clear. The next generation of language models will reason more, speak less about their reasoning, and become dramatically more efficient in the process.</p><p><strong>The best thinking, it turns out, often happens in silence.</strong></p><p><em>This post is based on “Implicit Reasoning in Large Language Models: A Comprehensive Survey” by Jindong Li et al. (arXiv:2509.02350, September 2025). The survey’s continuously updated repository of implicit reasoning papers is available at </em><a href="https://github.com/digailab/awesome-llm-implicit-reasoning"><em>github.com/digailab/awesome-llm-implicit-reasoning</em></a><em>.</em></p><p>Paper Link — <a href="https://arxiv.org/pdf/2509.02350">https://arxiv.org/pdf/2509.02350</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a66243a25ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Your AI Confidently Lies to You?]]></title>
            <link>https://sulbhajain.medium.com/why-your-ai-confidently-lies-to-you-7ad1d65129f1?source=rss-82b008267a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ad1d65129f1</guid>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[llm-hallucinations]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sulbha Jain]]></dc:creator>
            <pubDate>Mon, 23 Feb 2026 03:08:50 GMT</pubDate>
            <atom:updated>2026-02-23T19:47:37.690Z</atom:updated>
            <content:encoded><![CDATA[<h4>And Why That’s Actually a Math Problem — Paper Review</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gp5km08VB_dNhGGH" /><figcaption>Photo by <a href="https://unsplash.com/@mvrko?utm_source=medium&amp;utm_medium=referral">Marko Korb</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You ask a language model when a historical figure was born. It answers without hesitation. Confidently. Fluently. And completely wrong.</p><p>This is hallucination — and if you’ve spent any time working with LLMs in production, you know it’s not a rare edge case. It’s a feature of the landscape.</p><p>For years, the conversation around hallucinations has been frustratingly hand-wavy. “The model makes things up.” “It’s pattern matching gone wrong.” “We just need more data / better RLHF / bigger models.” None of these explanations fully satisfy, because none of them tell you <em>why</em> the problem is so stubbornly persistent — even in our most capable systems.</p><p>In my recent work, I set out to experiment the work from OpenAI and Georgia Tech researchers. They have developed a comprehensive statistical framework that grounds hallucinations not in mysterious emergent behavior, but in something far more fundamental: the same forces that cause ordinary classification errors.</p><p>The <strong>paper</strong> is available at <a href="https://arxiv.org/abs/2509.04664">arxiv.org/abs/2509.04664</a>,</p><p>The <strong>experimental implementation</strong> is on <a href="https://github.com/sulbhajain/why_llms_hallucinate">GitHub</a>.</p><p>Here’s what I found — and why it should change how the entire community thinks about evaluation.</p><h4>The Core Insight: Generation Is Harder Than Verification</h4><p>Imagine two tasks. In the first, you’re asked to write down the capital of Burkina Faso from memory. In the second, someone shows you five options and asks you to identify which one is correct.</p><p>Intuitively, the second task is easier. And this intuition is provably correct.</p><p>The authors formalize this through what they call the <strong>Is-It-Valid (IIV) binary classification problem</strong>: given an output, can you determine whether it’ correct? they then prove a fundamental inequality:</p><blockquote><strong><em>Generation error rate ≥ 2 × (IIV misclassification rate)</em></strong></blockquote><p>In plain terms: a model will always hallucinate at least twice as often as it would fail to recognize a hallucination if shown one. This isn’t a training failure. It’s a mathematical property of the relationship between generation and classification.</p><p>This single result reframes the problem entirely. Hallucinations aren’t a bug to be patched out. They are an <em>inevitable consequence</em> of asking a model to generate rather than verify — at least under current training paradigms.</p><h4>Three Statistical Roots of Hallucination</h4><p>Once you accept that hallucination is a statistical phenomenon, you can start categorizing its causes clearly. The paper identify three:</p><p><strong>1. Arbitrary facts — no pattern, no learning.</strong> When information has no underlying structure — a celebrity’s phone number, a random birthday, an obscure address — there is nothing for a model to generalize from. It can only memorize. And for facts that appear only once in training data, the model will hallucinate at least as often as those facts actually appear. This is not fixable with scale alone; you cannot generalize from the ungeneralizable.</p><p><strong>2. Poor model fit.</strong> Sometimes the architecture genuinely cannot represent a concept. The classic example: count the letter “r” in “strawberry.” A tokenizer that never sees “r” as an isolated unit creates a structurally broken substrate for this task. Trigram models trying to learn grammar face the same wall. The model isn’t being careless — it literally lacks the representational capacity.</p><p><strong>3. Compounding factors.</strong> Distribution shift (the real world keeps changing after training ends), computational hardness (some correct answers are just hard to compute), and training data errors (“garbage in, garbage out”) all layer on top of the first two causes.</p><p>Each of these has different mitigation strategies. But all of them share a common thread: they are <em>knowable</em> failure modes, not random noise.</p><h4>The Evaluation Ecosystem Is Making Things Worse</h4><p>Here is where the paper’s findings become most actionable — and most uncomfortable for the field.</p><p>They analyzed 10 major LLM evaluation benchmarks. Nine of them use binary grading: the answer is either correct or incorrect. There is no credit for saying “I don’t know.” There is no reward for expressing calibrated uncertainty.</p><p>Think about what this trains models to do.</p><p>If you’re uncertain about an answer and you guess, you have some probability of being right and getting full credit. If you abstain, you get zero. Under binary scoring, <strong>guessing when uncertain is always the rational strategy</strong>. And so, through post-training, models learn exactly that. They learn to project confidence because confidence is rewarded. Honest uncertainty is penalized.</p><p>We haven’t just built models that hallucinate. We’ve built an evaluation ecosystem that <em>selects for</em> hallucination behavior. It is a socio-technical problem as much as a technical one.</p><h4>A Simple Fix With Large Implications</h4><p>The good news: fixing the evaluation problem doesn’t require redesigning benchmarks from the ground up. It requires adding one thing — <strong>explicit confidence targets</strong>.</p><p>Instead of asking “What is the capital of Burkina Faso?”, you ask:</p><p><em>“Answer only if you are more than 75% confident. If you are uncertain, say ‘I don’t know.’ Incorrect answers will be penalized more than abstentions.”</em></p><p>This small change in the evaluation prompt completely restructures the model’s incentives. Suddenly, honest uncertainty becomes a viable and rewarded strategy. The model no longer has to hallucinate to maximize its expected score.</p><p>Paper show that this approach — specifying confidence thresholds and making error penalties explicit — would significantly reduce hallucination incentives in post-training, without requiring any changes to model architecture or training data.</p><h4>Experimental Results</h4><p><a href="https://github.com/sulbhajain/why_llms_hallucinate/blob/main/ollama_experiments.ipynb">Notebook run </a>:</p><pre>Binary Classfication Results:<br>Accuracy: 0.0%<br>  Hallucination rate: 100.0%<br>  IDK rate: 0.0%<br>  Average score: 0.000<br><br>IIV Classification Results:<br>  Accuracy: 5/10 = 50.0%<br>  Error rate: 50.0%</pre><h4>What This Means for Practitioners</h4><p>If you’re building LLM-powered systems in production, here’s how I’d translate these findings into practice:</p><p><strong>Stop expecting hallucinations to disappear.</strong> The statistical forces behind them are real and persistent. Your architecture, your retrieval pipeline, your confidence thresholds — these are the levers you actually control. Design for graceful handling of uncertainty, not its elimination.</p><p><strong>Audit your evaluation setup.</strong> If you’re using binary pass/fail metrics to evaluate your system, you are measuring the wrong thing and likely rewarding the wrong behavior. Add abstention options. Reward calibration. Penalize wrong-and-confident more than uncertain-and-abstaining.</p><p><strong>Distinguish between types of hallucination.</strong> A model hallucinating a birthday (arbitrary fact, no pattern) needs a different solution than a model mis-counting letters (architectural mismatch) or a model giving you outdated information (distribution shift). Lumping them together obscures the cure.</p><p><strong>Prompt for uncertainty.</strong> Even today, with models that weren’t trained with confidence targets, you can prompt for calibrated responses. Explicitly invite the model to say “I’m not sure.” Give it permission to abstain. You’ll often get more reliable outputs as a result.</p><h4>The Bigger Picture</h4><p>What I find most intellectually satisfying about this framework is that it <em>demystifies</em> hallucination. For too long, it’s been treated as something strange and uniquely AI — a kind of digital confabulation with no clear analog in the rest of machine learning.</p><p>But hallucinations arise from the same factors that cause classification errors: insufficient data, model limitations, and misaligned objectives. The moment you see it this way, you stop asking “why does this weird AI phenomenon happen?” and start asking “which of these three statistical factors is dominant in my use case, and what do I do about it?”</p><p>That’s a question practitioners can actually answer.</p><p>The full framework, proofs, and benchmark analysis are in the paper. The code for reproducing the paper’s experiments is on GitHub. And if you’re working on evaluation design, hallucination mitigation, or AI reliability in general — I’d genuinely love to hear how you’re thinking about these problems.</p><p>Because fixing hallucinations isn’t just a modeling challenge. It’s a community challenge. And it starts with measuring the right things.</p><p><strong><em>GitHub</em></strong><em>: </em><a href="https://github.com/sulbhajain/why_llms_hallucinate"><em>github.com/sulbhajain/why_llms_hallucinate</em></a></p><p><strong><em>Paper</em></strong><em>: </em><a href="https://arxiv.org/abs/2509.04664"><em>arxiv.org/abs/2509.04664</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ad1d65129f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-Tuning a Tiny Medical AI That Runs Without the Internet]]></title>
            <link>https://sulbhajain.medium.com/fine-tuning-a-tiny-medical-ai-that-runs-without-the-internet-e7fbf488f920?source=rss-82b008267a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7fbf488f920</guid>
            <category><![CDATA[supervised-fine-tuning]]></category>
            <category><![CDATA[aimed]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[medgemma]]></category>
            <dc:creator><![CDATA[Sulbha Jain]]></dc:creator>
            <pubDate>Sun, 22 Feb 2026 09:27:34 GMT</pubDate>
            <atom:updated>2026-02-22T09:27:34.219Z</atom:updated>
            <content:encoded><![CDATA[<h4>My MedGemma Journey</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kOW6SzmCP6sd2dJ_UQ3GOA.jpeg" /></figure><p>Imagine a rural clinic in a low-connectivity region where a doctor needs a quick, reliable answer to a biomedical question. No cloud. No internet. Just a laptop with a single GPU and a question that could affect a patient’s care.</p><p>That’s the scenario that motivated this project — and it turns out you don’t need a billion-dollar data center to make it work.</p><p>I built <strong>MedGemma BioQA</strong>, a fine-tuned language model for biomedical question answering that runs entirely offline, on a single free-tier Kaggle GPU, in about 15 minutes of training time. Here’s how it works, what I learned, and why it matters.</p><h4>The Problem with “Big AI” in Healthcare</h4><p>Large language models have made remarkable strides in medical reasoning. But the dominant paradigm — sending patient-adjacent queries to a remote cloud API — creates real problems: latency, data privacy concerns, cost, and the fundamental assumption of reliable internet access.</p><p>What if we could push a capable, clinically relevant model to the edge? What if it could run on hardware that already exists in resource-constrained settings?</p><p>That question led me to Google’s <strong>MedGemma</strong> family and a technique called <strong>QLoRA</strong>.</p><h4>What I Built</h4><p><strong>MedGemma BioQA</strong> fine-tunes google/gemma-2-2b-it — a 2-billion parameter instruction-tuned model — on two well-established medical benchmarks:</p><ul><li><strong>PubMedQA</strong>: Yes/No/Maybe questions grounded in PubMed abstracts, testing a model’s ability to reason from research evidence.</li><li><strong>MedQA-USMLE</strong>: Four-option multiple choice questions styled after the US Medical Licensing Examination, testing clinical knowledge and reasoning.</li></ul><p>The result is a compact model that can answer both research and clinical questions accurately, entirely on-device.</p><h4>The Secret Sauce: QLoRA</h4><p>Training a 2B parameter model from scratch is expensive. Fine-tuning the full model is still expensive. QLoRA (Quantized Low-Rank Adaptation) is a technique that makes this practical on consumer hardware.</p><p>Here’s the core idea:</p><p><strong>4-bit quantization</strong> compresses the base model’s weights from 32-bit floats to 4-bit NF4 format, cutting memory usage by roughly 8×. The model becomes small enough to fit comfortably in the memory of a single T4 or P100 GPU — the kind you get for free on Kaggle.</p><p><strong>LoRA adapters</strong> then add a small number of trainable parameters (only the adapter weights, not the frozen base model) injected into the attention and MLP layers. With rank r=16, only about 0.1% of the total parameters are actually trained.</p><p>The result? You get most of the benefit of full fine-tuning at a tiny fraction of the cost.</p><pre>BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_use_double_quant=True,<br>    bnb_4bit_compute_dtype=torch.bfloat16,<br>)</pre><pre>LoraConfig(<br>    r=16, lora_alpha=32, lora_dropout=0.05,<br>    target_modules=[&quot;q_proj&quot;,&quot;k_proj&quot;,&quot;v_proj&quot;,&quot;o_proj&quot;,<br>                    &quot;gate_proj&quot;,&quot;up_proj&quot;,&quot;down_proj&quot;],<br>    task_type=&quot;CAUSAL_LM&quot;,<br>)</pre><h4>Teaching the Model to “Think Like a Doctor”</h4><p>The training data format matters enormously. For PubMedQA, I wrapped each example in Gemma 2’s native chat template, giving the model a clear persona and task:</p><pre>&lt;start_of_turn&gt;user<br>You are a medical research expert. Answer the following biomedical question...</pre><pre>Context: &lt;PubMed abstract sentences&gt;<br>Question: &lt;question&gt;<br>Answer with yes, no, or maybe:&lt;end_of_turn&gt;<br>&lt;start_of_turn&gt;model<br>Answer: yes</pre><pre>This conclusion is based on the research context provided.&lt;end_of_turn&gt;</pre><p>This format teaches the model not just to produce an answer, but to frame it in the context of the evidence — a crucial habit for clinical reasoning.</p><h4>The Gotchas (Learned the Hard Way)</h4><p>Building this on Kaggle’s free GPU tier was an adventure in debugging. Here are the landmines I stepped on so you don’t have to:</p><p><strong>DataParallel crashes with 4-bit models.</strong> Kaggle sometimes exposes two GPUs and tries to use both. 4-bit quantized models do not play nicely with DataParallel. The fix: force single-GPU mode <em>before</em> any PyTorch import with os.environ[&quot;CUDA_VISIBLE_DEVICES&quot;] = &quot;0&quot;.</p><p><strong>The padding token trap.</strong> Gemma’s tokenizer sets pad_token == eos_token by default. This causes subtle loss corruption during training because the model can&#39;t distinguish real end-of-sequence from padding. Fix it explicitly: tokenizer.pad_token = &quot;[PAD]&quot;.</p><p><strong>SFTTrainer API drift.</strong> The TRL library changed its SFTTrainer interface in version 0.9. The old dataset_text_field argument no longer exists. The new pattern: pass processing_class=tokenizer instead.</p><p><strong>Gradient checkpointing conflicts.</strong> Don’t manually call .gradient_checkpointing_enable() when using QLoRA with SFTTrainer — it conflicts with the adapter setup. Set gradient_checkpointing=False in TrainingArguments and let the framework handle it.</p><p>These bugs cost me hours. Documenting them in the README was as important as the model training itself.</p><h4>Results and Evaluation</h4><p>Evaluation uses greedy decoding on a held-out test split of 50 examples per task. Answer extraction is intentionally simple: keyword matching for PubMedQA (yes/no/maybe) and option letter matching for MedQA (A/B/C/D). This keeps evaluation transparent and reproducible without any black-box scoring.</p><p>Training for ~15 minutes on a P100 produces a model that shows clear improvement over the base (instruction-tuned but not medically specialized) model on both tasks. The loss curves tell a clean story — rapid early improvement that plateaus gracefully without overfitting.</p><h4>Why This Matters</h4><p>The technical achievement is modest by frontier AI standards. A 2B parameter model is not going to replace a physician. But that’s not the point.</p><p>The point is <strong>accessibility</strong>. A model that:</p><ul><li>Trains in 15 minutes on hardware anyone can access for free</li><li>Runs entirely offline after training</li><li>Can be reproduced by any developer with a Kaggle account</li><li>Is modular enough to swap in different medical datasets</li></ul><p>…is a model that lowers the barrier to building specialized medical AI tools in settings where the big cloud APIs simply aren’t an option.</p><p>This is what “democratizing AI” actually looks like in practice — not just making models cheaper, but making the entire pipeline reproducible at the bottom of the resource curve.</p><h4>Try It Yourself</h4><p>The full code is open source:</p><p>🔗 <strong>GitHub:</strong> <a href="https://github.com/sulbhajain/med-gemma-bioqa">github.com/sulbhajain/med-gemma-bioqa</a></p><p>To get started:</p><pre>git clone https://github.com/sulbhajain/med-gemma-bioqa.git<br>cd med-gemma-bioqa<br>pip install -r requirements.txt</pre><pre># Train (requires HuggingFace token for Gemma access)<br>python src/train.py</pre><pre># Launch the Gradio demo<br>python src/gradio_app.py --checkpoint ./sft_medical_qa_model</pre><p>The self-contained Kaggle notebook at notebooks/med-gemma-bioqa-kaggle.ipynb runs end-to-end on Kaggle&#39;s free GPU tier with no local setup required.</p><h4>What’s Next</h4><p>A few directions I’m excited to explore:</p><ul><li><strong>Expanding datasets</strong> — integrating MedMCQA (Indian medical licensing exams) and clinical NLP benchmarks like i2b2 to broaden coverage.</li><li><strong>Retrieval-augmented generation</strong> — connecting the model to a local vector store of PubMed abstracts so it can ground answers in up-to-date literature without internet access.</li><li><strong>Quantized deployment</strong> — converting the fine-tuned adapter to GGUF format for llama.cpp inference on CPU-only devices.</li></ul><p>The vision is a full offline clinical QA pipeline that any motivated developer can deploy in a resource-constrained environment. We’re partway there.</p><p><em>If you found this useful, consider starring the repo and sharing with anyone building in the medical AI space. I’d love to hear what you build with it.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7fbf488f920" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>