<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sagar Ganapaneni on Medium]]></title>
        <description><![CDATA[Stories by Sagar Ganapaneni on Medium]]></description>
        <link>https://medium.com/@gsagar?source=rss-7560a6b328fd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MmS8ZSDtzBK98vkF0OmiOg@2x.jpeg</url>
            <title>Stories by Sagar Ganapaneni on Medium</title>
            <link>https://medium.com/@gsagar?source=rss-7560a6b328fd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:11:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gsagar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Quantizing an Analytics Agent Without Losing the Decision]]></title>
            <link>https://medium.com/data-science-collective/quantizing-an-analytics-agent-without-losing-the-decision-by-sagar-ganapaneni-a5579a429fed?source=rss-7560a6b328fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5579a429fed</guid>
            <category><![CDATA[model-compression]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sagar Ganapaneni]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 22:48:15 GMT</pubDate>
            <atom:updated>2026-11-05T06:09:18.599Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A follow-up to “Fitting a Stronger Model Into Less Memory”: what happens when INT4 rounding doesn’t just add noise, but flips which answer an agent picks.</em></p><p>In <a href="https://medium.com/data-science-collective/fitting-a-stronger-model-into-less-memory-a-quantization-walkthrough-by-sagar-d4530c368aa5">Fitting a Stronger Model Into Less Memory</a>, two quantized weights flipped a toy business from retail to repair, and nothing about the arithmetic was wrong. Rounding did exactly what rounding does. That example was narrow on purpose: small enough to check by hand, so the flip couldn’t be waved away as a measurement artifact.</p><p>Here, the same question gets a sharper stand-in. Instead of a business classifier, a two-choice scorer meant to resemble a decision an analytics agent actually makes, such as picking which denominator a retention query should use. The mechanics are the same scale, rounding, and clipping as last time. What changes is what’s riding on the answer.</p><p>My first instinct when learning quantization was to keep reducing the scale. If the values are closer together, the rounding error should get smaller. I was thinking of a scale of 0.01, but with only four codes that leaves us with 0, 0.01, 0.02, and 0.03. We have better spacing and almost no range. A value like 0.5 no longer fits.</p><p>That was the part I needed to work through. The number of bits limits the choices we can store. The scale decides how far apart those choices sit, and values beyond the range have to be clipped. Making the scale smaller does not give us more choices.</p><p>For a model, the next question is what those rounding errors do to the calculation. With an analytics agent, I’m interested in whether they change a choice that reaches the query or the final answer. The model can still produce readable text after choosing a different metric definition.</p><p>A small calculation makes this easier to inspect. The example below is a constructed two-output scorer, evaluated in NumPy, rather than a measurement from an LLM. It has two choices, A and B, with these weights and one shared input. Multiplying each row by the input gives A a score of 0.662 and B a score of 0.639: A wins by 0.023.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*tTFkRKsvIfNbbXD8hj1-6w.png" /><figcaption>setup-two-choices</figcaption></figure><p>Now quantize the weights using four-bit integer codes from −7 to 7, leaving −8 unused. Use zero as the zero-point, round-to-nearest-even, and one scale for the whole matrix. The largest absolute weight is 2.8, so the scale is 2.8 / 7 = 0.4. Divide each weight by 0.4, round it to a code, then multiply by 0.4 to reconstruct it. A now scores 0.720 and B scores 0.760. The choice has switched to B. Nothing was clipped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*vEQE8tVBf6UIe3Ae-_m_Lw.png" /><figcaption>decision-flip-bars</figcaption></figure><p>The 2.8 is worth looking at here. Its input is zero, so it contributes nothing to this particular answer, but it still sets the scale for the other weights. Those smaller weights lose the differences that previously gave A the higher score.</p><p>We can give the first three weights in each row their own scale and put the final weight in a separate group. Doing that restores A on this input and reduces the average squared weight error by about 30 times. These tiny groups illustrate the calculation; production groups also have to account for kernel and storage overhead.</p><p>That fixes the input we started with. Changing the third input from 0.1 to a variable t shows what happens elsewhere. Keep the other inputs fixed at [1, 0.2, t, 0]. The original score difference, A minus B, is 0.028 − 0.05t. It changes sign at t = 0.56.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*tLXt-Da-I_RY5-5XVI_qtA.png" /><figcaption>margin-panels</figcaption></figure><p>Global INT4 chooses B for every positive t. Split INT4 keeps choosing A across the plotted range. INT8 moves the boundary to about 0.3333. So the grouping that fixed our first input still fails elsewhere, despite its smaller weight error. These eight constructed weights show how a decision can change; they cannot estimate an actual agent’s failure rate.</p><p>For a fixed input x, a weight change ΔW changes the output by Δy = ΔW x. The input determines how much the rounding matters. In a full model, earlier layers can also change the inputs to later ones.</p><p>For paid-account retention, accounts that became paid after the opening snapshot should not enter the opening cohort. Choosing the wrong denominator can still produce executable SQL and a plausible percentage. Quantization could change that choice, but this numerical example does not establish that it happened in a real agent.</p><p>I’d test the same checkpoint at BF16, W8A16, and W4A16, keeping the instructions, tools, data, activation precision, and attention-cache precision fixed. W4A16 means four-bit weights and sixteen-bit activations. Replay identical prefixes and tool results to locate differences, then let each version complete the task with the same retry budget.</p><p>Grade against reviewed answers and executed queries; BF16 can be wrong too. Include user corrections and missing evidence. Keep those tests separate from calibration data, which should reflect the schemas and tool responses the agent handles.</p><p>Then measure memory and end-to-end latency. Scales and the attention cache add memory beyond the weights. Kernel performance, tool waits, and extra retries determine whether the smaller model actually saves the user time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*0EHJsKv3y4TDO6atwdi1zQ.png" /><figcaption>release checklist</figcaption></figure><p>For this agent, I’d keep the retention question in the release test. I want to see the right denominator survive the change in precision, alongside a measured improvement in memory or latency.</p><h3>Notes</h3><p><em>A follow-up to: </em><a href="https://medium.com/data-science-collective/fitting-a-stronger-model-into-less-memory-a-quantization-walkthrough-by-sagar-d4530c368aa5"><em>Fitting a Stronger Model Into Less Memory: A Quantization Walkthrough</em></a><em>.</em></p><p><em>The scorer, weights, and input sweep here are constructed for illustration, independently re-derived and verified: the score computations, the per-group rescaling, the ~30× error reduction, and the INT8 crossing point at t ≈ 0.333 all check out exactly against hand and NumPy calculation.</em></p><p><em>W4A16 / W8A16 notation and the BF16/INT8/INT4 release-testing approach follow common LLM-serving practice (four- or eight-bit weights, sixteen-bit activations).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5579a429fed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/quantizing-an-analytics-agent-without-losing-the-decision-by-sagar-ganapaneni-a5579a429fed">Quantizing an Analytics Agent Without Losing the Decision</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turning user interactions into better model behavior]]></title>
            <link>https://medium.com/data-science-collective/a-thumbs-up-is-not-a-training-label-by-sagar-ganapaneni-d40dad0b5cc1?source=rss-7560a6b328fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/d40dad0b5cc1</guid>
            <category><![CDATA[product-development]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sagar Ganapaneni]]></dc:creator>
            <pubDate>Mon, 12 Oct 2026 16:38:39 GMT</pubDate>
            <atom:updated>2026-11-01T03:59:55.208Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 4 of 4. A thumbs up tells you someone liked the answer, which isn’t the same as the answer being right.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3tD7p_Gg-IM8_mBo4DVGmQ.jpeg" /></figure><p>Most model-behavior work doesn’t start with a dataset. It starts with a user who got annoyed enough to correct the same mistake twice in one conversation.</p><p>The obvious place to look is the ratings. But a thumbs up only tells you that someone liked an answer. It does not tell you the answer was right.</p><p>The question that decides whether anything useful comes out of that moment is narrow: what, specifically, should the model learn from it. A correction could expose a reasoning mistake, missing context, an unavailable tool, or a preference that only applies to one user, and each of those calls for a different fix.</p><p>Consider a simplified conversation. The user has already specified paid accounts and a reporting period, then asks for a comparison by region. The agent asks which accounts and period to use. The user repeats the information, and the agent eventually completes the task. It would be reasonable to want the next model to avoid that extra exchange.</p><h3>Start with what the model actually saw</h3><p>Check what the model actually received before you decide anything else. If the application dropped the earlier messages, training the model to remember them will not fix the problem, since it never saw them. If the context was present and clear, the interaction is a useful example of how the model should carry scope across turns.</p><p>Now consider a similar question with no population specified. Here, asking may be the correct next step. I would include both situations in the training and evaluation data. Otherwise, feedback about unnecessary questions could teach the model to guess when it should clarify instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oyagYoZNK5-n1ehER7P8-A.png" /><figcaption><em>Training can’t fix context the model never received.</em></figcaption></figure><h3>Turn corrections into usable examples</h3><p>For each usable interaction, keep the context shown to the model, the relevant tool results, its response, and the user’s correction. Record the model, prompt, and tool versions too. A short label such as “ignored supplied scope” is more useful for diagnosis than a negative rating alone.</p><p>The training pipeline should use only conversations permitted for that purpose, with unnecessary private details removed. A private preference should stay scoped to its user. Current metrics and changing business definitions should keep coming from governed sources rather than being treated as facts the model must memorize.</p><h3>Ratings are not labels</h3><p>A thumbs up can still help identify examples for review, but it does not confirm the answer. The user may have liked the wording without checking the calculation. Silence or abandonment is equally ambiguous. Sample interactions without ratings as well, so the training set does not represent only the people most likely to leave feedback.</p><p>When building a preference pair, both responses must be judged against the same available information. An answer written after the user supplies a missing detail cannot simply be preferred over an earlier clarification: it had information the earlier response did not. Reconstruct a fair comparison, or leave that pair out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4k5gmSpVCSbEHIYk2MH2Tg.png" /><figcaption><em>An answer written after new information isn’t a fair comparison with the clarification before it.</em></figcaption></figure><p>Reviewers can then separate verified demonstrations from preference comparisons. Keep ties and uncertain cases visible instead of forcing a winner. Disagreement about business meaning should go back to the relevant owner before it becomes a training label.</p><h3>Three ways to use the data</h3><p>The three methods below all change model parameters, which sets them apart from saving a preference in application memory. The choice depends on the behavior being improved and the evidence available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6cce8qgZCIvjO9yxpQF7bw.png" /><figcaption><em>Each method can be used on its own.</em></figcaption></figure><p>Supervised fine-tuning, or SFT, trains the model on examples of the response or action we want. In the regional-comparison example, a demonstration can show the agent carrying forward the stated population and choosing the appropriate tool. The dataset should include cases where clarification is necessary too.</p><p>Direct preference optimization, or DPO, uses preferred and rejected responses to the same context. Its objective favors the preferred response relative to the rejected one, usually against a reference model. It does not require a separately trained reward model or an online reinforcement-learning loop, though the quality of the comparison labels still matters just as much.</p><p>Reinforcement learning from human feedback, or RLHF, uses human judgments to guide model training. In a common setup, comparisons first train a reward model to score candidate responses. The agent model then generates responses or tool-use sequences, and reinforcement learning updates it using those scores. Training commonly includes a penalty for moving too far from a reference model. That limits drift, but it does not establish correctness.</p><p>SFT can provide a starting point for preference training, but these are not three compulsory stages. A smaller experiment should establish whether training improves the behavior beyond the best prompt, context, and tool setup already available.</p><h3>Test the reward, not just the answer</h3><p>If you train a reward model, evaluate it on its own. If reviewers consistently favor shorter answers, the reward model may learn to reward brevity even when a qualification is necessary. If task completion gets rewarded without checking evidence, a confident unsupported number may score well. Test those cases explicitly, and keep factual checks separate from judgments about presentation.</p><p>For an agent that takes several steps, the final user rating is not enough to identify which step deserves credit. A correct answer might follow repeated searches or an unnecessary query. Where possible, review the relevant tool choices and the verified outcome alongside the response. A higher reward score alone is not a release result.</p><h3>Set up the experiment properly</h3><p>Before starting a training job, confirm that the chosen model and hosting platform support the required training and serving path. Keep a frozen baseline and version the dataset, review rules, training configuration, and resulting checkpoint or adapter, so the team has a specific, reproducible candidate to evaluate.</p><p>Split the data before tuning, and keep an entire conversation and its near-duplicates in one partition. Otherwise a correction can appear in training while a slightly reworded version appears in the test set.</p><h3>Measure quality and cost together</h3><p>Compare the candidate with the baseline as <a href="https://medium.com/data-science-collective/ai-agent-scored-better-what-actually-improved-by-sagar-ganapaneni-54425c643999">Part 2</a> describes, using unseen conversations, the same source snapshots, tools, and limits, and repeated runs where behavior varies. For this change, check correct completion, unnecessary clarification, appropriate clarification, unsupported answers, and regressions in previously reliable tasks.</p><p>Efficiency needs its own measurement. Fine-tuning does not automatically make the model smaller or faster. Better tool selection may reduce repeated calls, while a different response strategy may use more tokens. Measure response time and total model and tool cost across the workload, including failed attempts, then divide that cost by correctly completed tasks. Report the completion rate alongside it.</p><p>Training a smaller model on a stronger model’s outputs, often called distillation, is a separate experiment. The smaller model still has to meet the required quality checks. Compare any serving savings with the cost of collecting labels, training, evaluation, and maintaining the new version.</p><h3>Ship it like a release</h3><p>Roll out the way <a href="https://medium.com/data-science-collective/ai-agent-scored-better-what-actually-improved-by-sagar-ganapaneni-54425c643999">Part 2</a> describes, with a limited rollout and rollback conditions agreed ahead of time. Training also means keeping the previous model deployable. For adapter-based training, retain the compatible base model and adapter versions; for hosted models, record the deployment identifiers and confirm how long rollback remains available.</p><p>The release record should connect the model to its training data, evaluations, prompt, and tool configuration. New feedback can enter the next reviewed batch. It should not silently update the serving model after every conversation.</p><h3>Back to the regional comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3uo7f6zOwN65hocj6n0-1g.png" /><figcaption><em>Saving an exchange in the first case means nothing if the model starts guessing in the second.</em></figcaption></figure><p>For the regional-comparison example, the candidate should use scope already present, ask when a necessary detail is absent, and complete the task with verified evidence. If it saves an exchange in the first case but starts guessing in the second, the training change needs more work. That is the comparison I would want before expanding the rollout.</p><blockquote><strong><em>Thanks for reading, this is the final article in the four-part series. <br></em></strong><em>Part 1: </em><a href="https://medium.com/data-science-collective/not-every-ai-agent-mistake-costs-the-same-by-sagar-ganapaneni-267911ea0c4a"><em>Not every agent mistake costs the same</em></a><em> <br>Part 2: </em><a href="https://medium.com/data-science-collective/ai-agent-scored-better-what-actually-improved-by-sagar-ganapaneni-54425c643999"><em>The agent scored better. What actually improved?</em></a><em><br>Part 3: </em><a href="https://medium.com/data-science-collective/how-an-agent-should-learn-from-a-wrong-answer-by-sagar-ganapaneni-8f2f555f315c"><em>How an agent should learn from a wrong answer</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d40dad0b5cc1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/a-thumbs-up-is-not-a-training-label-by-sagar-ganapaneni-d40dad0b5cc1">Turning user interactions into better model behavior</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How an agent should learn from a wrong answer]]></title>
            <link>https://medium.com/data-science-collective/how-an-agent-should-learn-from-a-wrong-answer-by-sagar-ganapaneni-8f2f555f315c?source=rss-7560a6b328fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f2f555f315c</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sagar Ganapaneni]]></dc:creator>
            <pubDate>Sun, 11 Oct 2026 16:18:30 GMT</pubDate>
            <atom:updated>2026-10-12T16:44:39.297Z</atom:updated>
            <content:encoded><![CDATA[<p><em>When an agent reports zero for data that hasn’t arrived, the fix isn’t in the prompt.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*coUrjgdbPa9uJr5mdkgBmg.jpeg" /></figure><p>An agent gives a wrong answer, someone corrects it, and we want it to do better next time. Before saving that correction, the first thing to check is where the mistake actually came from. The answer may need a better instruction, but it may also be following an incorrect definition exactly as written.</p><h3>Zero is not the same as missing</h3><p>Take a simplified example. Someone asks how many paid accounts were active at the end of last month. The source contains daily snapshots, but the requested month-end snapshot has not arrived. The query converts the missing value to zero and the agent reports that there were no active accounts.</p><p>The SQL ran successfully. It may even use the right table. But zero means we observed no active accounts, and that is different from not having the observation. A source link and a clear explanation do not fix that difference.</p><p>If the mistake is in a shared query example, adding a reminder to one agent’s prompt leaves the original problem available to other agents. The correction belongs in the example and its checks, with any related definition reviewed as well.</p><h3>What a definition has to spell out</h3><p>A semantic layer makes business definitions reusable. It connects terms such as paid account or active subscription, and month-end balance to the sources and calculation rules behind them. For an agent to use those definitions, the population, unit being counted, time basis, exclusions, and missing-data behavior all need to be clear.</p><p>The business object model describes the relationships underneath those calculations. A business, a billing account, and a subscription are different objects. One business may have several subscriptions, so counting subscriptions will not answer a question about businesses unless that relationship is handled correctly.</p><p>For the month-end question, an earlier snapshot is not a substitute for the requested date. The agent can offer it as a clearly labeled alternative. Even a non-null month-end result needs care, because some records may have arrived while others are still missing. Source-readiness checks have to establish completeness separately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1hV9toe_kwowk-2UyRJg0g.png" /><figcaption><em>What the agent should report depends on what the source actually contains.</em></figcaption></figure><h3>Find where the mistake came from</h3><p>The final answer alone may not tell us which part needs changing. The investigation should narrow that down before we decide how the agent will learn from it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c5njSq5Jk1DoFiwcR1Pw-A.png" /><figcaption><em>The investigation decides where the correction belongs.</em></figcaption></figure><h3>Every correction has a scope</h3><p>Feedback also needs a scope. “I meant paid accounts for this comparison” clarifies the current task. “Remember paid accounts as my usual default” is a personal preference. “This published metric excludes trial accounts” is a claim about a shared definition.</p><p>The first affects the current answer. The second can be saved for that user, with the current request taking precedence. The third needs verification against the metric’s definition and owner before it changes answers for everyone. Saving all three as general knowledge could turn a personal preference into a business rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EBSZ0B7A3znXHjjMKkvk4A.png" /><figcaption><em>Similar-sounding corrections can belong in three different places.</em></figcaption></figure><p>For shared knowledge, I would keep the improvement process separate from answering the current question. The proposed change should carry the original question, available evidence, definition version, and reason for the correction. That gives the reviewer enough context to understand where it applies.</p><h3>Test where a fix should and shouldn’t apply</h3><p>For the missing-snapshot example, useful tests include an absent endpoint, an endpoint containing only null values, an observed zero, and a partially populated snapshot. They should also confirm that an earlier balance does not replace the requested date. These tests check calculation behavior; they do not certify that the live source is complete.</p><p>Verification needs more than the agent agreeing with its own explanation. Executable checks can test the calculation, source checks can establish freshness and completeness, and domain review can confirm the intended business meaning. Each answers a different part of the question.</p><p>It also helps to test where a correction should not apply. Experiment identifiers may be required for an experiment analysis and irrelevant to a general operational question. Making them mandatory everywhere can cause the agent to keep asking for information the user’s task does not need.</p><p>That is part of making the agent more direct. It should proceed when the available context is enough, and ask a focused question when the missing detail would materially change the answer. Clear rules about applicability help it make that decision.</p><p>Retrieval needs the same review. A result can rank highly because it shares the right words or table names while using a different population, denominator, or time basis. The selected calculation needs its supporting constraints too. A high ranking score does not establish that the definition fits the question.</p><p>Corrections tend to add context over time. When trimming it later, repeated background is a reasonable place to start. Removing an exclusion or separating an example from its qualifying explanation changes the evidence the agent receives. A shorter response should still contain the information required to perform the task correctly.</p><h3>When training is worth trying</h3><p>Model training becomes worth testing when the knowledge is correct and available, but the agent still makes the same behavioral mistake. Clearer instructions, better tool interfaces, and fine-tuning on validated examples are possible changes to compare. If training uses a reward, it should credit the agent for recognizing when the evidence does not support a number. Part 4 covers how to train without teaching the wrong lesson. A stale definition or unavailable snapshot still needs attention at its source.</p><h3>What improvement looks like</h3><p>For a shared platform, I would start with one domain and a small set of recurring failures. Domain owners approve changes to business meaning, data scientists evaluate the learning and verification methods, and engineers provide versioning, controlled release, and rollback. The ownership needs to remain clear as more teams adopt the platform.</p><p>Track whether corrected failures recur, whether unrelated workflows regress, and whether users need fewer unnecessary clarifications. Keep each answer traceable to the knowledge version it used. When a definition changes, dependent examples and cached results need review too.</p><p>For the month-end question, the expected improvement is specific. The agent explains that the requested observation is missing and applies that rule to future relevant questions. An observed zero still returns zero, and an unrelated metric keeps its own missing-data policy. Before expanding the change across more domains, confirm that behavior and keep a way to reverse the correction if the evidence changes.</p><blockquote><em>This is the third article in a 4-part series. Next in the series: </em><a href="https://medium.com/@gsagar/a-thumbs-up-is-not-a-training-label-by-sagar-ganapaneni-d40dad0b5cc1"><strong><em>A thumbs up is not a training label</em></strong></a><em>. Earlier: </em><a href="https://medium.com/data-science-collective/not-every-ai-agent-mistake-costs-the-same-by-sagar-ganapaneni-267911ea0c4a"><em>Not every agent mistake costs the same</em></a><em> and </em><a href="https://medium.com/data-science-collective/ai-agent-scored-better-what-actually-improved-by-sagar-ganapaneni-54425c643999"><em>The agent scored better. What actually improved?</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f2f555f315c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/how-an-agent-should-learn-from-a-wrong-answer-by-sagar-ganapaneni-8f2f555f315c">How an agent should learn from a wrong answer</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The agent scored better. What actually improved?]]></title>
            <link>https://medium.com/data-science-collective/ai-agent-scored-better-what-actually-improved-by-sagar-ganapaneni-54425c643999?source=rss-7560a6b328fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/54425c643999</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sagar Ganapaneni]]></dc:creator>
            <pubDate>Sat, 10 Oct 2026 07:36:52 GMT</pubDate>
            <atom:updated>2026-10-12T19:13:57.097Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 2 of 4. A five-point gain isn’t a release decision until you know which tasks moved.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*webV5GL2PJELV4pV2PTasQ.jpeg" /></figure><p>An agent’s evaluation score has moved from 85% to 90%. That looks good, but before replacing the current version, we need to check which tasks improved and which ones stopped working. Some of those failures could be in workflows people already depend on.</p><p>The <a href="https://medium.com/data-science-collective/not-every-ai-agent-mistake-costs-the-same-by-sagar-ganapaneni-267911ea0c4a">previous article</a> in the series covered what an evaluation should check. The next part is deciding whether a change is actually ready to use. That needs a closer look at the results, especially when the overall score improves but individual tasks move in different directions.</p><h3>Match results task by task</h3><p>Consider the campaign audience agent again. It interprets a request, reconciles business records, applies eligibility rules, and saves an audience for review. Suppose we change its instructions and compare both versions on the same 100 tasks, using the same success criteria. All numbers in this article are invented to explain the comparison.</p><p>The old version succeeds on 85 tasks and the new version on 90. When we match the results task by task, 75 pass in both versions, 15 pass only in the new version, and 10 pass only in the old version. None fail in both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QViYjzDkZylLlirgPTLwXA.png" /><figcaption><em>Illustrative numbers. Same 100 tasks, one attempt per version.</em></figcaption></figure><p>So we have 15 tasks that now pass and 10 that used to pass but now fail. The net gain is five. These results come from one attempt per version. We still need to inspect the changed cases and the steps the agent took, then check whether those differences repeat before attributing them to the instructions.</p><p>Maybe the new instructions help with ambiguous requests but cause the agent to miss an exclusion. The totals alone cannot confirm that, so we need to check the actual cases. It also matters how often these workflows come up in production. A test set full of unusual questions can be useful for testing capability, but its average may not reflect what most users experience.</p><p>We should agree on the release criteria before seeing the scores. The team may accept some regressions after reviewing the tradeoff, but a required eligibility rule being violated could block the release. That failure needs to stay visible even if other tasks improved.</p><h3>Run it more than once</h3><p>We also need to run the comparison more than once. An agent can take different paths on the same task, so both versions should get repeated attempts from the same starting conditions. Then we can check whether the improvement holds across those attempts.</p><p>The next chart is a separate illustration. Both candidates average 90 correct tasks, against a baseline averaging 85. Candidate A stays five tasks ahead in every repetition shown. Candidate B falls below the baseline twice, even though its average improvement is identical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BGRqWlfCmKfVft_ZYQu1Ng.png" /><figcaption><em>Six repetitions show variability; they are not a release threshold.</em></figcaption></figure><p>Candidate B needs a closer look because its results vary quite a bit. Candidate A is more consistent in this example, though six repetitions are only for illustration, not a release requirement. We also need the individual task results. A similar total each time does not mean the same tasks kept passing.</p><p>Running 100 tasks six times gives us more information about those 100 tasks. It does not give us 600 independent customer scenarios. We still need representative cases that were not used to tune the agent. Any estimate of uncertainty should compare the versions on the same tasks and account for repeated attempts on each task.</p><h3>Change one thing at a time</h3><p>For this comparison, we changed the instructions, so the data snapshot, business definitions, permissions, tool interfaces, resource limits, and grading rules should stay the same. Otherwise, it becomes harder to tell what caused the difference. Each run also needs a clean starting state. A saved audience or memory from the previous attempt should not help the next one.</p><p>This is a practical concern. In <a href="https://www.anthropic.com/engineering/infrastructure-noise">Anthropic’s study of agentic coding evaluations</a>, changing infrastructure resources alone moved Terminal-Bench 2.0 scores by as much as 6 percentage points while the model and task set stayed fixed. That result concerns coding agents, but it is a useful reason to inspect the environment when comparing other agents too.</p><p>Giving the new version more time, more retries, or a larger model may be a reasonable change. We just need to report the improvement along with those changes and their cost. We cannot attribute all of it to the prompt.</p><h3>The grader needs checking too</h3><p>The evaluator needs checking too. Keep its version and grading rules fixed, and review important disagreements against the actual evidence. If an LLM judge is grading the responses, check whether its grades vary on the same response. Running the agent again does not tell us whether the judge graded it correctly.</p><h3>Keep a held-out set</h3><p>After a few rounds of reading failures and adjusting instructions, we are tuning against those cases. Their improvement is useful, but we need a separate set that has not informed those changes before claiming the agent improved more broadly. Once we use failures from that held-out set to guide the next fix, it is no longer an untouched test set.</p><p>Keep the known failures as regression tests so we can catch them if they return, with separate cases to check broader capability. <a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents">Anthropic’s agent evaluation guide</a> distinguishes these purposes: checking what an agent can do and protecting behavior it already handles.</p><h3>From offline gain to rollout</h3><p>Before rollout, the team should have a clear record of which workflows improved, what still fails, how certain we are, and what the change costs. A limited rollout can then check the result on real traffic. Keep users or accounts assigned to the same version where conversations share state. For consequential actions, we can first compare what the agent proposes without executing it. That shadow evaluation helps check the proposed actions, but it cannot tell us their actual business impact.</p><p>During rollout, track task completion, user corrections, critical errors, response time, and cost, with rollback conditions agreed in advance. If the offline gain does not show up in production, check where the traffic or operating conditions differ from the evaluation.</p><p>For the 85% to 90% result, I would want the ten regressions reviewed before moving ahead. If they include a required exclusion being missed, that needs fixing and another comparison. If the remaining tradeoffs are acceptable and the gains hold across repeated runs and held-out cases, we have a basis for a limited rollout, with production checks to confirm the result.</p><blockquote><em>Next in this series: </em><a href="https://medium.com/@gsagar/how-an-agent-should-learn-from-a-wrong-answer-by-sagar-ganapaneni-8f2f555f315c"><em>How an agent should learn from a wrong answer.</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54425c643999" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/ai-agent-scored-better-what-actually-improved-by-sagar-ganapaneni-54425c643999">The agent scored better. What actually improved?</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Not every agent mistake costs the same]]></title>
            <link>https://medium.com/data-science-collective/not-every-ai-agent-mistake-costs-the-same-by-sagar-ganapaneni-267911ea0c4a?source=rss-7560a6b328fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/267911ea0c4a</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sagar Ganapaneni]]></dc:creator>
            <pubDate>Sat, 10 Oct 2026 03:51:34 GMT</pubDate>
            <atom:updated>2026-10-12T02:24:56.514Z</atom:updated>
            <content:encoded><![CDATA[<p><em>What an agent evaluation should tell you before the agent gets more responsibility.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*82tns5osO-jTV8m7MnUCeg.jpeg" /></figure><p>I want an agent to answer directly when it already has enough context. If it asks an unnecessary follow-up and then answers correctly, that is worth improving. But I would not treat it the same as confidently answering the wrong question.</p><p>The extra question takes the user’s time, especially if they have already provided the information at the same an incorrect answer could affect a business decision. Both need attention, but the consequences are different, and the evaluation should reflect that.</p><h3>Agree on what failure means</h3><p>Before assigning scores, we need to agree on which mistakes are acceptable, which need investigation, and which should stop the agent from taking an action. This gives the team a clear basis for deciding what a good result means.</p><p>Consider an agent preparing a marketing campaign, requires identifying the right audience to target, reconcile records across sources, apply any business rules, and save the audience for review.</p><p>The agent could give a reasonable explanation while using an outdated exclusion rule. It could also prepare the correct audience but fail to save it. In both cases, the request has not been completed correctly, but the fixes are different.</p><p>We need to check the rule it used, the audience it produced, and whether that audience was actually saved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YXN6HlOgj9MLFaUcgwRg9A.png" /><figcaption><em>An evaluation should keep the evidence behind its conclusion. Reviewed failures become new test cases.</em></figcaption></figure><p>The teams building and using the agent need to agree on these expectations. Product managers define the intended experience, domain owners clarify the business rules, and data scientists and engineers turn them into checks with evidence the team can review.</p><p>For the audience agent, that means agreeing on which businesses to include, the required exclusions, how fresh the data needs to be, and what action the agent is allowed to take. Preparing an audience for review and activating a campaign need different release requirements.</p><h3>A higher score can hide worse failures</h3><p>With those expectations in place, we can compare the scores and check the kinds of failures behind them.</p><p>Suppose we compare two versions on the same 100 cases. Version A completes 86 tasks correctly. Version B completes 90. At first glance, B looks better.</p><p>Now separate out tasks where the agent included an ineligible business. `A` has two such failures. `B` has five.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4S8fWnMwSalSo8HNmxFNjg.png" /><figcaption><em>Illustrative numbers. Both versions ran the same 100 cases once.</em></figcaption></figure><p>These are invented results, with one attempt per case. The eligibility violations are counted among the failed tasks.</p><p>`B` completes more tasks correctly, but it also includes ineligible businesses more often. We need to investigate that before deciding to use it. If the release requirement is zero eligibility violations in this test suite, neither version passes.</p><p>Keep these critical failures visible separately from the overall score. We can review tradeoffs in speed, cost, and interaction quality, but improvements there should not offset a required exclusion being missed.</p><h3>Use the trace to know where to look</h3><p>Once we know what failed, we need enough evidence to work out where to investigate.</p><p>An execution trace records the sources, tool calls, arguments, and returned results. That record helps connect an incorrect outcome to the decisions behind it. <a href="https://mlflow.org/docs/latest/genai/eval-monitor/running-evaluation/agents/">MLflow’s agent evaluation workflow</a> supports examining intermediate behavior alongside final outputs.</p><p>For the audience agent, the trace gives us a starting point for checking each failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G9BVslvgoRNXuFVW5bjyVA.png" /><figcaption><em>Where to start looking for each kind of failure.</em></figcaption></figure><p>These are places to start checking. The trace still needs review before we can confirm the cause or assign the fix.</p><h3>Track completion and failure handling separately</h3><p>We should also record task completion and failure handling separately. If the agent stops because it cannot verify eligibility, that may be the correct response, but the user’s request is still unfinished. Track both. Otherwise, we could penalize the agent for stopping correctly or overlook a service problem because it handled the failure well.</p><p>The evaluator needs evidence too. Executable checks can verify saved records and explicit constraints. LLM judges can assess meaning and explanation quality, but their judgments need comparison with expert review. <a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents">Anthropic’s guide to agent evaluations</a> describes combining these grading methods and calibrating LLM judges against human judgment.</p><h3>What the release discussion should cover</h3><p>Before release, compare versions under the same conditions, repeat cases where behavior varies, and review important failure categories separately. Track the versions of the model, instructions, tools, business definitions, and evaluation rules so the team can connect a score change to what was changed.</p><p>Tracing, test execution, and version tracking can be shared across teams. Domain teams still need to own the business expectations and decide which failures are acceptable. The shared platform should make those checks easier to run, with clear ownership when a rule needs updating or a failure needs review.</p><p>For this audience agent, the release discussion should cover which tasks improved, whether eligibility failures increased, what the evidence tells us about the cause, and who will follow up on anything unresolved. If the agent missed a required exclusion, we need that fixed and checked again before giving it more responsibility.</p><blockquote><em>This is the first article in a 4-part series. Next in the series: </em><a href="https://medium.com/data-science-collective/ai-agent-scored-better-what-actually-improved-by-sagar-ganapaneni-54425c643999"><strong><em>The agent scored better. What actually improved?</em></strong></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=267911ea0c4a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/not-every-ai-agent-mistake-costs-the-same-by-sagar-ganapaneni-267911ea0c4a">Not every agent mistake costs the same</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fitting a Stronger Model Into Less Memory: A Quantization Walkthrough]]></title>
            <link>https://medium.com/data-science-collective/fitting-a-stronger-model-into-less-memory-a-quantization-walkthrough-by-sagar-d4530c368aa5?source=rss-7560a6b328fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4530c368aa5</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[quantization]]></category>
            <dc:creator><![CDATA[Sagar Ganapaneni]]></dc:creator>
            <pubDate>Sat, 03 Oct 2026 05:19:42 GMT</pubDate>
            <atom:updated>2026-10-03T05:19:42.133Z</atom:updated>
            <content:encoded><![CDATA[<p>Could I fit a stronger model into less memory and still get the same useful results? That was the question that got me interested in quantization. I understood the storage benefit fairly quickly. What I kept getting stuck on was why a smaller scale would not let me keep all the precision as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*Cur_2fKsERu7Yczc6TfDqQ.png" /></figure><p>A model’s weights are numbers adjusted during training, when the model learns from examples. Inference is when we use that trained model on a new input. Quantization represents those weights (and often a model’s activations, the intermediate values produced while processing an input) using a limited set of values, usually stored with fewer bits. The model still has the same number of weights; some of their values just become approximations.</p><p>I wanted to work through that change in a setting I could relate to. I have worked on matching businesses to industry categories, where similar language can refer to different kinds of businesses. A pump repair service and a pump retailer might both mention pumps. The distinction we need is what the business actually does.</p><p>For this write-up, I’m borrowing that problem and building a tiny scoring model we can calculate by hand. The businesses, input values and weights below are invented for the explanation. They are not customer records or results from my industry-matching model.</p><p>Suppose we already have two input signals between 0 and 1: evidence of repair work and evidence of retail activity. Higher means stronger evidence. We will supply these signals ourselves and leave them unchanged. Our model uses two positive weights, 0.86 and 0.57, and subtracts the retail contribution: score = (0.86 × repair signal) − (0.57 × retail signal).</p><p>For this exercise, a score above 0.30 selects repair; anything else selects retail. This score is not a probability. A real system would need more categories and a way to handle uncertain cases, but two choices let us follow every calculation.</p><p>A bit can hold 0 or 1. Two bits give us four patterns: 00, 01, 10 and 11. We can treat these as integer codes 0, 1, 2 and 3. More generally, b bits give us 2 to the power of b codes: 256 codes for the 8-bit storage used in most quantized models today, and 16 for the 4-bit storage increasingly used for large language models.</p><p>Let’s choose a range from 0 to 1.2 for our two weights. With four codes, we have three gaps, so each gap is 1.2 ÷ 3 = 0.4. That gap is the scale, the step size between one stored code and the next. The table below compares this with the smaller scale I initially thought would solve the precision problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*EEcjFFD3NulQ2Ymn4BaoDg.png" /></figure><p>At scale 0.4, the first weight rounds to code 2 (the full walkthrough is below). The second weight follows the same path: 0.57 ÷ 0.4 = 1.425, which rounds to code 1.</p><p>We store codes 2 and 1. Multiplying them by the scale gives 0.8 and 0.4. This reconstruction is called dequantization. It gives us approximations of the weights, not the original 0.86 and 0.57. The second weight’s approximation (0.17 away) is looser than the first’s, though both stay under the half-scale bound that rounding always respects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*0F0MX4wr3fcjNG3UZpsnxA.png" /></figure><p>If we shrink the scale to 0.01, the largest code represents only 0.03. Both weights now lie outside the range. The gaps are smaller, but we have not gained any choices.</p><p>This was the part I had misunderstood. Changing the scale does not add bits, store more training examples or create room for more predictions. It changes what our existing codes mean.</p><p>In this example, the two weight codes occupy four bits altogether if packed. The scale and other bookkeeping also need storage. With just two weights, that overhead can outweigh the saving; quantization becomes useful when we share that information across many values. In practice this sharing is a spectrum rather than one choice: a single scale per entire tensor is simplest for hardware, one scale per output channel typically works better for convolutional weights, and large language models often use one scale per group of around 128 values. Wider sharing is cheaper to store but makes every value in the group live with the same step size, which is part of why the choice matters.</p><p>Now use the approximated weights in the same scoring rule: score = (0.8 × repair signal) − (0.4 × retail signal). Keep the inputs and the decision threshold unchanged.</p><p>Consider three invented cases: a repair shop with signals (0.9, 0.1), a retailer with (0.2, 0.9), and a business that sells equipment and also repairs it, with (0.6, 0.4). The chart and table below show the original and quantized scores. Values to the right of 0.30 select repair.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*IpsRzZQZSVx9bECzIaaj0A.png" /></figure><p>The repair shop moves from 0.717 to 0.680 and still selects repair. The retailer moves from −0.341 to −0.200, a bigger absolute change, and still selects retail. Both weights shrank by rounding, not clipping, so the bigger move is arithmetic, not an error: the retail weight fell further in relative terms (0.57 → 0.4, a drop of 0.17) than the repair weight did (0.86 → 0.8, a drop of 0.06), and the retailer’s strong retail signal (0.9) multiplies that larger drop. The mixed business is closer to the decision boundary.</p><p>Originally, its score is (0.86 × 0.6) − (0.57 × 0.4) = 0.288. After quantization, it is (0.8 × 0.6) − (0.4 × 0.4) = 0.320. It crosses 0.30, so the selected category changes from retail to repair. This is the one place in our toy example where quantization changes the decision, not just the number.</p><p>Both weights became smaller. The score still increased because the amount we subtract fell by more than the amount we add. Looking at the weights alone would not have told me what happened to the decision.</p><p>We also have not established that the new category is wrong. We would need a correct label for that business. These three cases show disagreement between the models; they do not establish an accuracy loss of one-third. To measure accuracy, we need a labelled evaluation set.</p><p>More bits could give us finer spacing. Three bits provide eight codes, so keeping the range at 0 to 1.2 would give a scale of 1.2 ÷ 7, or about 0.1714. That uses 50% more bits per weight than two-bit storage, before metadata. Changing the scale alone does not have that effect.</p><p>Rounding and clipping are different sources of error, shown side by side below: rounding moves an in-range value to its nearest code, an error that’s always bounded by half the scale; clipping cuts an out-of-range value off at the endpoint instead, an error with no such bound.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*nMWpxKfF-Pj2Le9mPEHt7g.png" /></figure><p>An outlier is a value unusually far from the others. Extending the range to include it spreads the same number of choices farther apart, which is the rounding-error trade-off above applied to every other weight at once. Clipping it instead keeps finer spacing for the other weights but may remove a contribution the model needs. Large language models make this concrete: once models reach a certain size, a small number of activation values can run tens of times larger than the rest, and naive quantization schemes degrade sharply when that happens, which is why those models typically need special handling for a handful of dimensions. I would check the resulting predictions before deciding that an unusual weight was safe to clip either way.</p><p>Our example stores two positive weights and performs the subtraction separately. Real models also contain negative weights. Symmetric quantization centres the range around zero and needs only a scale; asymmetric quantization allows an uneven range using both a scale and a zero-point. A zero-point is the integer code that represents the numerical value zero exactly: not an approximation of zero, but the one code that reconstructs to precisely 0.0. That matters for operations like zero-padding, which depend on an exact zero. In practice, weights are often quantized symmetrically while activations are quantized asymmetrically, partly because pairing two asymmetric ranges introduces extra cross-terms that cost more to compute.</p><p>For a small asymmetric example, use scale 0.4 and zero-point 1. Codes 0, 1, 2 and 3 then represent −0.4, 0, 0.4 and 0.8. To reconstruct a value, calculate (code − zero-point) × scale. To encode one, divide by the scale, round, add the zero-point and clip to the allowed codes. The stored code and the value it represents are different things.</p><p>In post-training quantization, or PTQ, we quantize an already trained model. Weights are usually quantized directly, since their range is already known. Activations depend on the input, so quantizing them typically needs calibration: we run a batch of representative inputs through the model and observe the resulting ranges before fixing a scale.</p><p>Quantization-aware training, or QAT, simulates rounding and clipping during training or fine-tuning so the weights can adjust before deployment. The weights themselves stay in ordinary floating-point form throughout training; only the rounding and clipping effects are simulated, so the model learns around them without ever actually computing in low precision until it is deployed.</p><p>Initially, I assumed QAT would simply be better because the model could adapt. It can help, but the additional training has a cost and improvement isn’t guaranteed, and the size of the gain is specific to the model. PyTorch’s own static-quantization walkthrough illustrates this on a vision model, MobileNetV2 on ImageNet, summarized below. For a large language model, PyTorch has separately reported QAT recovering up to 96% of the accuracy a post-training-quantized Llama 3 8B model lost on a standard benchmark. Both approaches still need evaluation on the task that matters, and “up to” in that second figure is doing real work: it’s a best case, not a guarantee.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*vaKMoH5mXgFAv3jSrAngFA.png" /></figure><p>I would start with PTQ and check whether it already meets the task’s requirements. If the loss is unacceptable, I would investigate where it occurs before adding training. A different range, finer-grained scales, better calibration, or keeping a sensitive part at higher precision may help; QAT is another option, roughly in that order of effort.</p><p>For business matching specifically, I’d weight that held-out comparison toward businesses that are easy to confuse (the real-world equivalent of the mixed business above), and, if the system retrieves a ranked list rather than a single category, check whether the correct industry still appears near the top.</p><p>Memory savings from quantization are close to guaranteed; speed is not. Activations also use memory and can be quantized, cutting it further, but whether execution actually gets faster depends on whether the hardware and software compute efficiently in that lower precision. On older hardware without that support, the overhead of converting values back and forth can leave inference no faster, or even slower, than the original.</p><p>I still want to fit a useful model into less memory. But before using it, I would check which business matches change, whether those changes help or hurt, and how much memory and time we actually save.</p><h3>References</h3><ol><li>MobileNetV2 PTQ/QAT figures: PyTorch, “(beta) Static Quantization with Eager Mode in PyTorch” tutorial.</li><li>Llama 3 8B QAT recovery figure: PyTorch Blog, “Quantization-Aware Training for Large Language Models with PyTorch,” July 2024.</li><li>Activation outlier behavior in large models: Dettmers et al., “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,” NeurIPS 2022.</li><li>General quantization formulas and terminology: Nagel et al., “A White Paper on Neural Network Quantization,” Qualcomm AI Research, 2021; Hugging Face Transformers quantization concept guide.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4530c368aa5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/fitting-a-stronger-model-into-less-memory-a-quantization-walkthrough-by-sagar-d4530c368aa5">Fitting a Stronger Model Into Less Memory: A Quantization Walkthrough</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Context and Domain Expertise Was Priced Under the Old Economics]]></title>
            <link>https://medium.com/data-science-collective/context-and-domain-expertise-was-priced-under-the-old-economics-a394a5e2f82e?source=rss-7560a6b328fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/a394a5e2f82e</guid>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sagar Ganapaneni]]></dc:creator>
            <pubDate>Sun, 06 Sep 2026 16:54:33 GMT</pubDate>
            <atom:updated>2026-09-07T02:02:56.024Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The most popular advice has been to pick a domain and go deep. I’ve spent fifteen years without one. Depth wins by default because it’s the half of the job teams can actually see.</em></p><p>At a high-tech company around 2018, I spent months on what everyone called an inventory model. It was really a cash flow model, and once we said so, the fix stopped being a forecasting problem and became a decision about what we stocked. I didn’t know silicon manufacturing. I did know how to find where money gets stuck.</p><p>Now, the advice goes like this. AI is blurring the lines between roles, so pick a domain and go deep. Become the person who knows the business cold. It sounds like conviction. It looks to me more like insurance on something that’s already losing value.</p><p>In the <a href="https://medium.com/data-science-collective/the-maker-manager-dilemma-in-the-agentic-ai-era-by-sagar-a2321171c35a"><strong>last article</strong></a> I wrote about when to be the player, the coach, or the architect. This one is about what to invest in, which turns out to be harder.</p><p>One idea sits underneath all of this. Companies can only reward what leaves a trace, and domain expertise leaves plenty: an answer with your name on it. The other half of the work leaves almost nothing — a question that redirected a quarter, an error that never happened, a partner who accepted a number they didn’t like. Depth isn’t overrated. It’s just the half that can be measured.</p><p>AI didn’t create that asymmetry. What changed is the price of one side of it. When a plausible answer was slow and expensive to produce, being the person who could produce it was most of the job. That part is getting cheap, and it’s the only claim here that depends on this moment. Most of the rest isn’t new.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F-EwHgrt_AgVzK9zsgq4nw.png" /><figcaption>What is actually being repriced</figcaption></figure><h3>Multiple industries, no domain</h3><p>None of what followed was planned. The only through-line is that every job was a commercial problem nobody had made measurable yet.</p><p>Marketing data science in online travel. Advertising and product analytics at a large software company. A high-tech company — sales ops, pricing, that cash flow model. Digital health: clinical risk, capacity forecasting, performance guarantees that had to survive a clinical review and a merger. Then small-business financial services, launching an audience and measurement product with a team we built from nothing.</p><p>5 industries, many functions. By the specialization logic I should have been mediocre at most of it, and in one or two I was. I wasn’t a generalist, though. I was deep — in measurement, in modeling, in how you optimize and monetize something and shallow in every business I pointed it at. What carried over wasn’t knowledge of the business. It was a way of walking into a problem: find opportunities and levers to pull, partner with the right folks, build the solution, and measure the progress.</p><p>Breadth cost me. In some rooms I was the least knowledgeable person, and I knew it — decisions could have moved faster. I spent months picking up vocabulary a ten-year veteran already had. And someone who never specialized, arguing that specialization is oversold, is not a neutral witness. Discount accordingly. I’ve tried to write the version I’d believe if my career had gone the other way, but I can’t check that from the inside.</p><blockquote><em>The craft is portable. The context isn’t.</em></blockquote><h3>‘Closer to the business’ turned into something else</h3><p>Done properly, closer to the business is three things you can check. Can you read the P&amp;L — where revenue comes from, what has to be true? Do you know how the business talks about its customers? When did you last send a request back with a better question? None of that is easy. All of it is checkable, and it takes a few quarters, not a decade.</p><p>Most people who think they need to specialize were never given that bar. All they got was “know the business,” with nobody saying what that meant. So they turned it into become the business, because that at least fits in a development goal. Meanwhile our time goes to pulling data, and that work has a ceiling. The scarce part is framing the question, and that comes from sitting close to strategy, roadmap, and go-to-market. Not from ten years in one vertical.</p><p>Which leads somewhere unglamorous. Routine questions have to become self-serve, which means building foundations partner teams can use without us — and staying useful to them while they get there. Frame it as handing work back and it fails. It looks like plumbing. It’s what buys back the time.</p><p>There’s a shortcut to that second check, we’re close to the data, and usually nowhere near the customer. That gap is why a clean analysis answers the wrong question. The fix isn’t five more years in the vertical. It’s sitting in on customer calls, reading the customer research instead of skipping to the metrics, watching someone use the product badly.</p><p>Time with customers builds intuition about which question to ask. That intuition is what people are reaching for when they say they need domain expertise. You can build a useful amount of it in a quarter<em>.</em></p><blockquote><em>Closer to the business is a test, not a tenure.</em></blockquote><h3>Trust the expert, then check the certainty</h3><p>My default is the functional expert. They’re closer to the constraint, and to the exception that never made it into a document. Most of the time that’s simply better information. This is about who is more likely to be right, not the highest-paid person in the room. The person closest to the constraint is often the most junior. Whether the organization can see both contributions is a different problem, and I’ll come back to it.</p><p>I flip when I hear the tell<em>: we’ve always done it this wa</em>y.</p><p>The harder version is that I’ve done both — convened the room, then quietly run the decision past the two people I already trusted. It saves time. It also costs you the reason you invited anyone else.</p><p>Experts don’t get blindsided because they know too little. They get blindsided because they’re sure. I’ve been that person too, and the uncomfortable part is that it doesn’t feel like anything. Not knowing enough is a problem you can feel. Being sure isn’t, and no review process catches it.</p><p>They’d answered a version of the question for years, so they answered this one from memory. Then the population shifted, or a definition moved underneath them, or the exception they were known for stopped binding. Nobody checked, because checking looks like doubting the person who knows.</p><p>A benchmark was set when the funnel carried a fraction of today’s volume. Years later the mix has shifted, the number hasn’t, and a team spends a quarter optimizing toward a target nobody has recalculated. The answer arrives faster than the question deserves. Nobody can say when the belief was last validated. The number gets quoted, never rebuilt. That’s what an incumbent does: spend an edge built for conditions that no longer hold.</p><p>That’s why we need someone whose job is to check. Not to challenge the expert on their own ground, but to ask whether the thing everyone is sure about is still true. The person with less depth often catches it first, because they haven’t spent five years being rewarded for the same answer.</p><blockquote><em>Certainty has no error bars.</em></blockquote><h3>The part I keep getting wrong</h3><p>I’ve been on the wrong side of this from the chair that runs the room — hearing the objection, nodding, and letting it pass.</p><p>Challenging a recognized expert is thankless. It goes political fast. One miss and the room gets to say <em>I told you so</em>, and the next person without deep credentials learns to stay quiet. Nobody decides this. It settles.</p><p>The counter-move is unglamorous: listen to someone who isn’t using the right words. Hear the intent. Let them finish before the room moves on. You don’t have to agree. You have to stop treating imperfect vocabulary as grounds for dismissal.</p><p>If you’re the one without the depth, you owe the room something too. Challenge the assumption, not the conclusion. Ask when it was last checked. Bring the method, not the verdict.</p><p>Nobody intends this. One kind of contribution is easy to point at; the other isn’t. So the person who reframes the question needs protection the domain expert doesn’t — not more status, just enough that disagreeing out loud isn’t a bet against their own review.</p><blockquote>What can’t be seen doesn’t get defended.</blockquote><h3>The Innovator’s Dilemma, pointed at a career</h3><p>Christensen’s incumbents don’t lose because they’re lazy. They lose because they’re being rational about a world that’s being repriced underneath them. I’ve started asking whether that’s happening to us.</p><p>The incumbent asset here isn’t a company. It’s whatever earned you your seat. For some people that’s accumulated domain knowledge. For others it’s craft — the model, the pipeline, the analysis nobody else could produce. Both were scarce. Both were priced when a defensible answer was slow and expensive to produce.</p><p>That’s what’s changing, and it’s the visible half that’s moving. When retrieval and drafting and the first plausible answer get cheap, whatever part of your edge was <em>having</em> the answer gets cheaper with them. Knowing which answer is worth having doesn’t.</p><p>There’s a second thing that doesn’t get cheaper, and I think we undersell it. Producing an answer and landing one are different jobs. Getting a model into production, getting a partner to accept a number they don’t like, getting a team to change what it does on Monday — none of that is retrieval. It stays expensive because it runs through people and organizations rather than through a model. The durable advantage belongs to people who were never the most knowledgeable in the room but always got the thing shipped.</p><p>This cuts at specialists and generalists alike, which is the part I find uncomfortable. Both bets can be wrong the same way, priced against a market that has already moved.</p><p>What got you here doesn’t automatically carry you forward. For a company that’s nearly fatal, since its cost structure won’t let it move. For a person it’s just a reason to re-check your own advantage every few years instead of assuming it compounds.</p><blockquote><em>Disruption is destiny for a firm. For a person it’s a choice.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I-kNUxp1BHA1gJtrcOYT3g.png" /><figcaption>Diminishing returns of depth</figcaption></figure><h3>Go deep. Then don’t stay.</h3><p>When someone tells me they want to go deep to get closer to the business, I almost never argue. Depth is how you learn what breaks. Skipping the craft and calling it breadth is fake, and it shows immediately.</p><p>The tilt comes later than people expect. First-line management is often still one problem space. The job that demands range is managing managers. The portfolio widens, and your team covers rooms where you can’t be the expert. That’s when breadth stops being a trait and becomes the work.</p><p>Tourist breadth doesn’t count. The test is simple: did you change an outcome in the new space, or just learn to talk about it? You don’t need to be the deepest person in the vertical. You need to be useful to the decision, next to people who hold what you don’t.</p><blockquote>Depth teaches you what breaks. Breadth teaches you where to look.</blockquote><h3>What AI changes</h3><p>You can see the market pricing this already. Among the roles AI companies are hiring hardest for is the <strong>forward deployed engineer</strong> — someone who lands in a customer’s business with no domain knowledge, works out what the real problem is, builds something, and gets it used. That job description is the invisible half, written down.</p><p>Roles are blending. Everyone is doing work outside their core craft now. The reflex is to grab the identity that sounds hardest to automate<em>: I own the domai</em>n. I understand it. It’s still defending a position rather than repricing it.</p><p>AI compresses production — the pulling, the drafting, the first plausible answer. It doesn’t compress the questions where judgment and data sense have to work together, and it won’t soon. Our craft sits somewhere unusual: close enough to the data to know what’s real, close enough to the business to know what matters. That position gets more valuable as the boundaries soften.</p><p>If every senior bar read<em>s trusted partner because they know the doma</em>in — and most I’ve seen, lean that way — you’ll undervalue the person who entered a new space and changed the outcome. That’s the capability the next few years will ask for, and it’s the harder one to show in a packet.</p><p>This has limits. There are situations where the domain-first answer is simply correct and my framing would have wasted everyone’s time — where the constraint was real, the history mattered, and the outsider’s clean question was the least useful thing in the room. I’ve been that outsider. The argument was never that range beats depth.</p><blockquote>The market is already paying for the half our rubrics can’t see.</blockquote><h3>The question I’d ask now</h3><p>For years the question was: what should I go deep in, so the business takes me seriously? Fair question. It still has an answer. Two more matter now. What part of my edge is getting cheaper? And what am I willing to rebuild instead of defend? The work nobody can see is usually the work worth protecting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a394a5e2f82e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/context-and-domain-expertise-was-priced-under-the-old-economics-a394a5e2f82e">Context and Domain Expertise Was Priced Under the Old Economics</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Maker–Manager Dilemma Is Changing in the Agentic AI Era]]></title>
            <link>https://medium.com/data-science-collective/the-maker-manager-dilemma-in-the-agentic-ai-era-by-sagar-a2321171c35a?source=rss-7560a6b328fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2321171c35a</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[builders]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[manager]]></category>
            <category><![CDATA[design-systems]]></category>
            <dc:creator><![CDATA[Sagar Ganapaneni]]></dc:creator>
            <pubDate>Sun, 16 Aug 2026 20:23:35 GMT</pubDate>
            <atom:updated>2026-08-24T05:58:43.374Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>If AI makes us 5X faster at work that should never have required us, we optimized the wrong thing. The next question is larger: when to be the player, when to be the coach, and when to be the architect.</blockquote><p>Agentic AI is changing an old leadership question. Almost two decades ago, Paul Graham wrote about the difference between the <strong>maker’s schedule and the manager’s schedule</strong>. Makers need long stretches of uninterrupted time. Managers live in meetings, decisions, and shorter blocks of time.</p><p>Andy Grove approached the same problem from another direction in <em>High Output Management</em>: a manager’s output isn’t simply what the manager personally produces. It is the output they enable across their organization. A lot of us were trained on that idea: <strong>leadership is a leverage game.</strong></p><p>I have spent that career close to data science solutions — building them, then leading the people who build them, and more recently designing how agents should use what an organization already knows. After that arc, I have started wondering whether agentic AI changes an assumption underneath both models.</p><p><strong>What happens when the cost of being a maker suddenly collapses? </strong>Not to zero, but enough to change how senior technical leaders spend their time.</p><h3>Seniority has traditionally created distance from the work</h3><p>Early in a technical career, you are close to everything. You query the data, build the model, debug the pipeline, argue about the metric definition, and stare at the weird distribution nobody can explain. That proximity is how intuition gets built. Then your scope grows. Five people become twenty, one problem becomes many, and the work becomes hiring, coaching, prioritization, org design, and decisions with incomplete information. This is exactly what should happen.</p><p>The side effect is less discussed: <strong>as scope increases, distance from the actual work tends to increase with it.</strong> Information starts arriving already summarized. First the analysis, then the summary of the analysis, then the slide that summarizes the summary, then another readout explaining the first one. Eventually it is a status: green, yellow, or red.</p><p>This isn’t always bureaucracy. It is how organizations have made leadership scale. A senior leader could not inspect every analysis, read every codebase, or prototype every interesting idea. The time cost was too high. That is the assumption I think AI is beginning to challenge.</p><h3>AI changes the cost curve of curiosity</h3><p>Suppose someone brings me an unfamiliar technical problem.Five years ago, going deep enough meant finding the right person, waiting on a walkthrough, getting access, and spending hours — sometimes days — just to get context. For a senior leader, that could easily become a bad use of a week.</p><p>Today, an agent can help me review an architecture, explore an unfamiliar codebase, or pressure-test an analysis in under an hour. I still do this, not because I should own the implementation, but because I have learned I cannot design a system I have not felt fail.</p><p>The useful part is not that AI lets me produce more things. It is that it lowers the cost of <strong>getting closer to the problem.</strong> AI doesn’t turn managers back into makers. It lets leaders go deep when they choose to, and that changes what they can afford to understand.</p><h3>The trap is doing more of the wrong work, faster</h3><p>If you spent a large part of your career as a builder, this is hard to resist. Suddenly you can build again, and fast: one more analysis, one more prototype, one more doc, one more agent, one more late-night rabbit hole that used to take a day and now takes 45 minutes. Each artifact may be useful, but producing more is not necessarily leadership leverage. This is where I keep coming back to Grove. <strong>Personal throughput is not the same as leadership leverage.</strong></p><p>That does not mean I should. If AI makes you five times faster, and you spend that capacity on work that should never have required you, you optimized the wrong thing. The question is not whether you can do this faster with AI. It is whether your being here changes the outcome enough to justify it.</p><p>I don’t think the answer is a 50/50 split of maker and manager. The point is not the right percentage of builder time. The point is <strong>where your involvement actually changes the system.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ubzUj8lKjhcSYLbAluSKg.png" /><figcaption>Player-coach-architect</figcaption></figure><h3>The model I find useful: player, coach, architect</h3><p>The leadership model I increasingly find useful is not maker versus manager, but player, coach, and architect. Be the player when firsthand exploration improves your judgment, the coach when another person needs to develop ownership, and the architect when a recurring problem should become a reusable capability.</p><p>A seasoned leader shouldn’t compete with their team to demonstrate that they can still do the work. But they also shouldn’t allow seniority to become an excuse for losing technical curiosity.</p><h3>Be the player when making is how you learn</h3><p>I don’t believe senior leaders should stay in the tools so the room can see that they still can. The point is judgment, not display. There are places, though, where going directly to the work changes the quality of your judgment, especially with AI. You need to use these systems, push them until they fail, and build something with them. Watch an agent produce confident code against incomplete context, get the nuances wrong, or invent a metric that looks executive-ready and is quietly untrue.</p><p>There is a kind of understanding you do not get from a slide about the work. You don’t need to be the person building everything, but you probably do need enough scar tissue from actually using these systems to know what is real, what is hype, and what your organization should be asking. The purpose of making, at this level, is often learning. You go deep enough to develop conviction, then you come back up.</p><h3>Be the coach when ownership is the leverage</h3><p>There is another category where being able to do it yourself barely matters, because someone else needs to own it. This one is harder when you can see the design. I can often tell, fairly quickly, how I would structure the workflow, where the knowledge should live, and why the first agent path will fail. That is exactly when I have to be careful. One of the hardest transitions in leadership is learning that your ability to solve a problem doesn’t mean you should solve it.</p><p>Sometimes the highest-leverage move is giving someone context and then stepping back. Sometimes it is coaching, or connecting two people, or one decision that clears three weeks of ambiguity, or one uncomfortable question. And sometimes it is letting someone else stay with a problem long enough to develop judgment themselves. AI doesn’t change that. If anything, the discipline matters more, because jumping in got cheaper.</p><p>When production is cheap, organizations generate more prototypes, analyses, documents, code, and ideas. A long AI-generated document may add no more value than the prompt that produced it; when that is true, share the prompt.What gets scarce is <strong>judgment about what deserves attention.</strong> What is actually worth solving? What should we stop? What is signal, and what is just AI slop? Those are leadership questions.</p><h3>Be the architect when you see the same problem twice</h3><p>This is the mode I find myself in most often now. A request comes in, someone who knows the domain translates it, remembers the exception that never made it into a doc, does the work, and sends it back. Next week, a slightly different version arrives, and the work is repeated.</p><p>I have asked a different question enough times that it has become a habit: <strong>why does a person need to keep answering this?</strong> Not because the person is slow, but because the knowledge required to answer it is still sitting in a few heads — the exception, the constraint, the thing that makes a clean answer wrong. Once you see that, the work changes. You stop trying to staff the repetition and start asking whether the repetition should exist.</p><p>That is a different kind of leverage. You aren’t solving the question, and you aren’t just assigning someone to solve it. <strong>You are changing the system so a whole class of questions gets easier.</strong> If I see the same problem twice, should I delegate it again — or should I redesign the system? The highest form of leverage isn’t always doing the work faster. <strong>Sometimes it is making the work disappear.</strong></p><h3>Delegation still matters. Orchestration is now part of the job.</h3><p>For years, one of the most important things I had to learn as a manager was delegation: don’t solve everything yourself, find the right person, give them context and ownership, and let them solve it. I still believe that. There is now another fork. When a problem shows up, I ask whether I should do it, whether someone on my team should do it, whether AI should help one of us do it, or whether we should build something so nobody has to do this by hand again.</p><p>That last question is the one I now spend the most time on. If agents are another form of leverage, you are no longer only coordinating people. The useful shift is less about who does the task, and more about how the work should move through the system — and what must be true before you trust a machine to move it.</p><h3>Experience may become more valuable, not less</h3><p>People say AI spreads expertise around, and it does, to some extent. There is another side we talk about less. Give two people a system that can produce ten plausible answers, and the experienced person’s edge may no longer be producing the eleventh. It may be seeing that nine of the ten are wrong.</p><p>Experience tells you where assumptions usually break: when a metric doesn’t smell right, when a clean architecture will not survive the org that has to run it, which question the customer is actually asking, and when the problem you were asked to solve isn’t the one worth solving. It also tells you the unglamorous parts — access, evaluation, ownership, and what happens when the agent is confidently wrong in front of a business partner. That is how you decide <strong>where to spend attention.</strong></p><p>So I don’t think expertise gets less important as AI gets more capable. I think its value moves earlier: from producing answers to framing the problem, from creating every artifact to deciding what is trustworthy, from doing the work to designing systems that keep doing good work. AI lets experienced leaders apply that judgment closer to the work without owning every output.</p><h3>The question is changing</h3><p>I no longer think the useful question is how much of my week should be “maker time” versus “manager time.” That split feels incomplete, and I don’t want speed to pull leaders back into being high-leverage individual contributors. Stay close enough to the technology to develop conviction, far enough above the task to see the system, and humble enough to know when someone else should own the answer.</p><p>For a long time, the leadership question was <strong>what should I delegate?</strong> That still matters. Two more sit next to it now: <strong>what should I explore myself because it will improve my judgment?</strong> And the one I find most useful: <strong>what should we turn into a system so that neither I nor the team needs to keep doing it?</strong></p><p><em>It is about moving between player, coach, and architect — and knowing, in a given week, which one the situation actually requires.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2321171c35a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/the-maker-manager-dilemma-in-the-agentic-ai-era-by-sagar-a2321171c35a">The Maker–Manager Dilemma Is Changing in the Agentic AI Era</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Being a Stranger to Your AI]]></title>
            <link>https://pub.towardsai.net/stop-being-a-stranger-to-your-ai-62c27ee2a410?source=rss-7560a6b328fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/62c27ee2a410</guid>
            <category><![CDATA[model-context-protocol]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[context-engineering]]></category>
            <dc:creator><![CDATA[Sagar Ganapaneni]]></dc:creator>
            <pubDate>Sun, 15 Mar 2026 01:59:12 GMT</pubDate>
            <atom:updated>2026-03-30T02:25:49.063Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>You use multiple AI tools. Each one knows a piece of you. None of them knows the whole picture. That led me to build a memory layer to help address the problem.</em></h4><p>You’ve spent months with your AI tools. Not training them deliberately, just using them. Every conversation, every correction, every preference you expressed. ChatGPT knows your tech stack. Claude knows how you like your answers. Gemini knows what you’re reading.</p><p>But switch tools, and that context doesn’t follow. You re-introduce yourself: your role, your projects, your preferences — to every new AI, every new session.</p><p><em>“I’m a data scientist working on…”</em> , for the fifth time this week, to the fifth different AI.</p><p>This isn’t about any single tool having bad memory. The problem is that your personal context is <strong>fragmented across platforms, </strong>each tool holds a partial, sometimes contradictory slice of who you are. No single tool has the complete picture, and none are designed to share.</p><p>I got tired of the fragmentation. So I built <strong>MemOS</strong>: a local-first, open-source memory layer that consolidates AI memories from ChatGPT, Claude, and Gemini into a single store and makes them accessible to any MCP-compatible client.</p><pre>ChatGPT memories ─┐<br>Claude memories  ─┼──▶ ~/.memos/memories.db ──▶ Any MCP client<br>Gemini memories  ─┘       (local SQLite)        Claude Desktop · Cursor · Windsurf · ...</pre><p>But MemOS isn’t just an aggregator. Once your memories are in one place, a more interesting question emerges: <strong>which memories should your AI actually see?</strong></p><h3>Why Not Just Use Claude’s Memory?</h3><p>If you use a single AI tool exclusively, its native memory is probably fine. MemOS is for people who use <strong>multiple AI tools</strong> daily and want consistent context across all of them.</p><p>Beyond fragmentation, three concerns drove this project:</p><p><strong>Privacy.</strong> Cloud-hosted memories mean your preferences, projects, and work patterns live on someone else’s infrastructure. For users in regulated industries, that’s a non-starter.</p><p><strong>Auditability.</strong> I wanted to know <em>exactly</em> what my AI tools had stored about me. Not “we’ve learned your preferences”, the actual data, searchable and auditable.</p><p><strong>Portability.</strong> When a new tool launches tomorrow, I want to bring my context with me. Memory should be infrastructure I own, not a feature I rent.</p><h3>How It Works</h3><p>MemOS has three layers: <strong>adapters</strong> that parse platform exports, a <strong>store</strong> that persists and indexes memories locally, and an <strong>MCP server</strong> that exposes them to AI clients.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MNTA5KNWOchp0KJlr7d39g.jpeg" /></figure><h3>Getting Your Memories Out (The Hard Part)</h3><p>T<strong>he most friction-heavy part is getting your data out of each platform.</strong> No major AI platform offers a programmatic API for exporting memories. The process is manual — clicking through settings menus, waiting for download emails, extracting ZIP files. This is the single biggest UX limitation, and MemOS can’t fix it unilaterally. Until platforms offer export APIs, and at this point, they’re not incentivized to, because portable memory reduces lock-in and this friction remains.</p><ul><li><strong>ChatGPT:</strong> Settings &gt; Data Controls &gt; Export data → extract memories.json from the ZIP</li><li><strong>Claude:</strong> Settings &gt; Capabilities &gt; Memory &gt; export</li><li><strong>Gemini:</strong> Google Takeout &gt; select My Activity &gt; Gemini Apps (the top-level “Gemini” option is for Gems, not chat history)</li></ul><p>Data export steps is manual, it does handle everything downstream: parsing idiosyncratic export formats, normalizing them into a unified schema, classifying and tagging each memory, deduplicating across sources, and indexing for fast retrieval. Re-importing is safe, MemOS deduplicates by content and only adds new entries. The import is batch-based rather than real-time, partly by design (batch imports are simpler and auditable) and partly by constraint (no platform offers sync APIs or webhooks).</p><pre>./memos.sh import --source chatgpt --input ~/Downloads/memories.json<br>./memos.sh import --source claude  --input ~/Downloads/memory.json<br>./memos.sh import --source gemini  --input ~/Downloads/gemini_memories.json</pre><h3>The Store</h3><p>Memories live in a SQLite database at ~/.memos/memories.db. I chose SQLite over Postgres or a vector database deliberately: for a single-user, local-first tool, it&#39;s the right default. It ships with Python, requires zero configuration, and <strong>FTS5 virtual tables</strong> provide full-text search with porter stemming out of the box. A vector DB would enable semantic search (FTS5 won&#39;t find &quot;machine learning&quot; when you search &quot;AI&quot;), but it would mean managing an embedding pipeline and a separate process. That trade-off wasn&#39;t worth it for v1 — semantic search via local embeddings is on the roadmap.</p><p>Each memory carries: content, source (chatgpt/claude/gemini/manual), category (auto-classified into professional/ personal/preference /project/ relationship), tags (auto-extracted keywords), a sensitive flag, a confidence score, and the original timestamp.</p><p>Classification uses <strong>keyword-based scoring</strong> rather than an LLM call. An LLM could classify more accurately — a memory like “coffee with my manager to discuss the Q3 roadmap” could reasonably be professional, relationship, or preference, and keyword matching doesn&#39;t always pick the best one. But LLM classification would require either an API key (cloud dependency, per-call cost) or a local model (hardware requirements). Keyword scoring runs in under a millisecond, works entirely offline, produces deterministic results, and handles the common cases well. The storage layer is designed so embedding-based classification can be layered on later without migration.</p><p>The store also provides near-duplicate detection (Jaccard similarity), staleness detection, per-source/category aggregations, and full JSON export.</p><h3>The MCP Server</h3><p>This is where the value lives. MemOS includes an MCP server that exposes eight tools over stdio:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l6NvqnhqCP1_5bFUWDWY2w.png" /></figure><p>All read operations automatically apply exclusion rules — sensitive memories, excluded tags, excluded categories, and keyword filters are stripped before reaching the AI client.</p><p>MCP is the protocol Anthropic introduced for tool integration, now adopted by Claude Desktop, Cursor, Windsurf, and others. By implementing MemOS as an MCP server, it works with all of them — no per-client integration code. Configuration is a single JSON block:</p><pre>{<br>  &quot;mcpServers&quot;: {<br>    &quot;memos&quot;: {<br>      &quot;command&quot;: &quot;/path/to/MemOS/memos.sh&quot;,<br>      &quot;args&quot;: [&quot;serve&quot;]<br>    }<br>  }<br>}</pre><p>Restart your client, ask <em>“What do you know about me?”</em>, and it pulls from memories spanning every platform you’ve imported. One caveat: if ChatGPT recorded that you live in San Francisco and Claude recorded that you moved to New York, MemOS currently stores both without flagging the contradiction. Cross-source conflict detection is a planned feature.</p><h3>Context Engineering: Not Just What Your AI Knows, but What It Should See</h3><p>There’s a concept gaining traction in AI tooling: <a href="https://www.promptingguide.ai/guides/context-engineering-guide"><strong>context engineering</strong> </a>— deliberately shaping what information reaches a model at inference time. Most discussion focuses on system prompts and RAG pipelines. But there’s a dimension that gets less attention: <strong>personal context curation</strong>.</p><p>Once your memories are unified, you don’t want <em>all</em> of them flowing into every interaction:</p><ul><li>You discussed a health condition with an AI during a vulnerable moment. Do you want it surfacing when you ask for help with a cover letter?</li><li>You had an emotionally honest conversation about burnout. Useful then. Not something you want injected into a coding session.</li><li>A six-month-old memory says you work at Company A. You’ve since moved to Company B. The stale memory is now actively misleading.</li></ul><p>The problem isn’t that AI tools remember too much or too little — it’s that there’s <strong>no curation layer</strong> between raw memory and the context reaching the model.</p><h3>How MemOS handles this</h3><p>MemOS provides four layers of context curation, from coarse to fine-grained:</p><p><strong>1. Context packs</strong> — named bundles scoped to specific categories. A “coding” pack pulls professional and project memories. A &quot;writing&quot; pack pulls personal and preference. You control which slice of your identity reaches each interaction without deleting anything.</p><pre>./memos.sh pack create &quot;coding&quot; --categories professional,project --exclude-tags health<br>./memos.sh pack inject &quot;coding&quot; | pbcopy</pre><p><strong>2. Sensitive flags</strong> — mark individual memories that should never reach an AI client. Therapy notes, health information, emotional conversations. The memory stays in your store; no AI tool receives it.</p><p><strong>3. Exclusion rules</strong> — broad, declarative filtering stored in ~/.memos/rules.json. Exclude by tag, category, keyword, or sensitivity flag. Rules compose with OR semantics and apply automatically to all MCP responses, pack injections, and prompt generation.</p><pre>./memos.sh rules set exclude_tags &quot;health,therapy&quot;<br>./memos.sh rules set exclude_categories &quot;personal&quot;</pre><p><strong>4. Selective cleanup</strong> — edit and delete commands for surgical updates. Found stale info? Edit in place. Something that shouldn&#39;t exist? Delete it. Far more granular than the &quot;delete all&quot; nuclear option most platforms offer.</p><p>As AI tools get better at remembering, the value shifts from <em>how much they know</em> to <em>how thoughtfully that knowledge is applied</em>. You discussed burnout at 2am. That was the right context then. It’s not the right context for a code review the next morning. MemOS lets you make that distinction without losing the memory.</p><h3>The CLI</h3><p>Beyond the MCP server, MemOS ships with a 15-command CLI:</p><pre>./memos.sh stats                                    # Counts by source and category<br>./memos.sh health                                   # Staleness, duplicates, gaps<br>./memos.sh search &quot;python&quot;                          # Full-text search with FTS5<br>./memos.sh audit                                    # Scan for emails, phones, API keys<br>./memos.sh prompt &quot;write a cover letter&quot; | pbcopy   # Memory-aware prompt generation<br>./memos.sh export --output backup.json              # Full JSON export</pre><p>The health command scores your store and surfaces actionable issues — stale memories, near-duplicates, category gaps. The audit command scans for PII patterns (emails, phone numbers, API keys, SSNs) using regex — fast and deterministic, though it won&#39;t catch sensitive information expressed in natural language. A more robust approach would use NER, but that reintroduces the cloud/local model dependency trade-off. The prompt command wraps a task with relevant memories, producing an enriched prompt that already knows your background.</p><h3>How to Get Started</h3><pre>git clone https://github.com/123saga/memos<br>cd memos<br>bash setup.sh</pre><pre>./memos.sh import --source chatgpt --input ~/Downloads/memories.json<br>./memos.sh import --source claude  --input ~/Downloads/memory.json</pre><pre>./memos.sh stats<br>./memos.sh health</pre><p>Add the MCP config to Claude Desktop, Cursor, or Windsurf and restart. No Docker, no cloud accounts, no API keys. The install takes under a minute.</p><h3>Closing Thoughts</h3><p>MemOS started as a personal frustration — I was tired of being a stranger to every new AI tool. But building it surfaced a deeper question: <strong>who should own the context that AI tools accumulate about you, and who should decide how it’s used?</strong></p><p>Right now, the answer to both is the platform. Your memories live on their infrastructure, in their format, accessible only through their interface. You can’t search them programmatically, audit them for sensitive data, or decide that your coding assistant should know your tech stack but not your therapy notes.</p><p>MemOS is an attempt to separate memory from the tools that create it. Aggregate your context, own it locally, curate what each tool sees, and carry it forward as you adopt new tools.</p><p>This is a vanilla implementation of this idea. The export process is manual and the classification is rule-based. But the curation layer is real, the sensitivity flags, exclusion rules, context packs and selective cleanup are all very helpful in owning the memory these AI tools capture about you.</p><p>The principle is simple: personal context should be infrastructure you own, not a feature you rent. If this resonates, I’d welcome contributions, feedback, and conversation.</p><p><strong>GitHub:</strong> <a href="https://github.com/123saga/memos">github.com/123saga/memos</a> use it, fork it, build on it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62c27ee2a410" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/stop-being-a-stranger-to-your-ai-62c27ee2a410">Stop Being a Stranger to Your AI</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Same Model, Same Prompts, Different Answers]]></title>
            <link>https://medium.com/@gsagar/same-model-same-prompts-different-answers-dd1810ad88f4?source=rss-7560a6b328fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd1810ad88f4</guid>
            <dc:creator><![CDATA[Sagar Ganapaneni]]></dc:creator>
            <pubDate>Sun, 22 Feb 2026 04:57:25 GMT</pubDate>
            <atom:updated>2026-02-22T16:23:27.808Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>How batch size introduces noise into LLM evaluations</strong></h3><p>This is the last one in the series. Parts <a href="https://medium.com/towards-artificial-intelligence/i-changed-one-string-and-my-models-score-dropped-70-points-19b69abde8c9">1</a>, <a href="https://medium.com/@gsagar/my-model-knew-the-answer-but-wasnt-allowed-to-finish-717af0354501">2</a> and <a href="https://medium.com/@gsagar/over-half-my-models-correct-answers-were-scored-as-wrong-2c3ada1b632f">3</a> were about large, systematic effects. Stop sequences and token budgets can swing scores by 70 percentage points. Parsing strictness creates 25 to 52 point gaps. Those are dramatic numbers that change conclusions about models.</p><p>This experiment is about something quieter. Does the batch size you use during inference, meaning how many prompts you process in parallel, actually change the measured accuracy?</p><p>The short answer is yes. The effect is around 7 to 10 percentage points. Whether that matters depends on how tight your leaderboard gaps are.</p><h3><strong>Why Batch Size Could Matter At All</strong></h3><p>When you process one prompt at a time, the model sees exactly that prompt’s tokens. When you batch 20 prompts together, shorter sequences get left-padded to match the longest one in the batch. The attention mask handles this so the model “ignores” the padding, but the underlying floating-point arithmetic is not identical. Different tensor shapes, different accumulation order, slightly different rounding behavior.</p><p>Under greedy decoding, this should not matter. The highest-probability token is the highest-probability token regardless of how the tensor was padded. But under sampling, you are drawing from the probability distribution. Tiny shifts in that distribution can flip which token gets selected, especially in regions where the model is choosing between several plausible next tokens.</p><p>With greedy decoding, I already knew from Part 1 that batch size would not move accuracy at all. There would be nothing to write about. So for this experiment, I switched to sampling with temperature=0.7 to give the effect a chance to appear.</p><h3>The Setup</h3><p>Same 40 questions, both models, chain-of-thought prompt, 512 max tokens. But this time with do_sample=True and temperature=0.7 instead of greedy. I ran batch sizes of 1, 5, 10, and 20.</p><p>I also did something that I wish more evaluation papers would do. I ran batch_size=1 twice. Two completely separate runs with identical settings.</p><p>Here is why that matters. Under sampling, even two runs at batch_size=1 will produce different outputs. That is just the inherent randomness of temperature-based decoding. If I compare batch_size=20 against batch_size=1 and see different outputs, I cannot tell whether that is a batch-size effect or just sampling noise.</p><p>But if I also know how much the two batch_size=1 runs differ from each other, I have a baseline for sampling variance alone. Any divergence at higher batch sizes beyond that baseline is attributable to padding and floating-point effects.</p><h3>What Happened</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*63h9y90ZkrB5VAzc432wcQ.png" /><figcaption>Experiment - Batch-Size / Concurrency Sensitivity</figcaption></figure><p>Qwen’s strict accuracy moved around a bit: 0.725 at batch_size=1, 0.70 at batch_size=5, 0.75 at batch_size=10, 0.675 at batch_size=20. The spread across batch sizes was 7.5 percentage points.</p><p>Mistral showed slightly more variation: 0.40 at batch_size=1, dropping to 0.30 at batch_size=5, recovering to 0.35 at batch_size=10, then back down to 0.30 at batch_size=20. A 10 percentage point spread.</p><p>But look at the lenient accuracy panel. Both models hold rock steady at roughly 0.90 to 0.925 regardless of batch size. This is the key insight. The batch size effect is mostly changing surface-level formatting, not the actual answer content. The model still produces the right answer. It just wraps it slightly differently depending on which tokens sampling selected, and the strict parser catches a few more or fewer mismatches.</p><p>The throughput panels show what you would expect from batching: bigger batches, faster inference. Qwen’s total wall time dropped from about 225 seconds at batch_size=1 to about 35 seconds at batch_size=20. A 6x speedup. The question is just whether you are paying an accuracy tax for it.</p><h3><strong>Putting All Four Experiments Together</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qKNT2yYYXsc2hJzqay2jcA.png" /><figcaption><em>How Much Does Each Knob Move Accuracy?</em></figcaption></figure><p>Here is where all four experiments land:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Qp3QbXZVtxJhCoE-EZdOw.png" /></figure><p>Every single one of these is larger than the typical gap between models on popular leaderboards.</p><p>The batch size effect (7 to 10 pp) is real but modest compared to the other three. This makes intuitive sense. Stop sequences and token budgets act as hard cutoffs that prevent entire sections of output from existing. Parsing strictness reclassifies large numbers of correct answers as wrong. Batch size just nudges individual token probabilities by tiny amounts. Most of those nudges are inconsequential. Occasionally one flips a token near the answer boundary and changes a strict match. The effect exists, but it is inherently smaller.</p><h3>Take Away From This Series</h3><p>The pattern across all four experiments is consistent. The things people treat as fixed infrastructure in their evaluation harnesses are actually experimental variables that move scores.</p><p>I am not saying benchmarks are useless. I use them all the time. What I am saying is that before you compare Model A against Model B, you need to make sure the plumbing is identical between the two runs. Because if it is not, the plumbing difference can easily be larger than the model difference, and you will draw the wrong conclusion.</p><p><strong>Report your eval config. </strong>Stop sequence, token budget, parsing strategy, and batch size should all be documented alongside the model name and prompt template.</p><p><strong>Set the token budget high enough that truncation rate is near zero. </strong>Then verify by checking whether increasing it further changes the score.</p><p><strong>Prefer lenient matching as the primary metric for factual QA. </strong>Strict matching penalizes formatting, not knowledge. Report both if possible.</p><p><strong>Use greedy decoding when you can. </strong>It eliminates batch size as a confound entirely.</p><p><strong>Check your stop sequence against your prompt format. </strong>A newline stop and a chain-of-thought prompt are fundamentally incompatible, and nothing in the pipeline will warn you about it.</p><p><em>All experiments run in Google Colab on a single A100.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd1810ad88f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>