<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chamath Palihawadana on Medium]]></title>
        <description><![CDATA[Stories by Chamath Palihawadana on Medium]]></description>
        <link>https://medium.com/@chamathpali?source=rss-3e30c36b8db5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XUVjZtPwzTS-GtvxPQHk6w.jpeg</url>
            <title>Stories by Chamath Palihawadana on Medium</title>
            <link>https://medium.com/@chamathpali?source=rss-3e30c36b8db5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 06:54:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chamathpali/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Judge the trajectory, not just the output]]></title>
            <link>https://medium.com/@chamathpali/judge-the-trajectory-not-just-the-output-6172ea10090f?source=rss-3e30c36b8db5------2</link>
            <guid isPermaLink="false">https://medium.com/p/6172ea10090f</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Chamath Palihawadana]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 12:19:02 GMT</pubDate>
            <atom:updated>2026-11-03T12:19:02.904Z</atom:updated>
            <content:encoded><![CDATA[<p>Teams evaluate LLM systems mostly by grading the system’s output. How the output is graded depends on the amount of time and resources invested, from small deterministic checks when there is a clear right or wrong answer to expensive and time-consuming human reviews.</p><p>LLM-as-a-judge sits in between: a model reads the final answer and returns a verdict, ideally a binary pass or fail calibrated against a domain expert. All three grade the answer, not the process by which the agent got to its output.</p><p>For a single completion, that is fine, because the output is all there is. But as we move towards multi-agent and long-horizon workflows, the ways in which our agents can fail and the opportunity for slight deviations in trajectory grow significantly, which can reduce the likelihood of the output being accurate and consistent.</p><p>This article expands on the need for evals that measure agent trajectory, some of the agentic mistakes we have observed, and the system we have created to monitor the behavior of some of our agents.</p><figure><img alt="The same run, graded twice. The output view has one thing to grade. The trajectory view has every step that produced it." src="https://cdn-images-1.medium.com/max/1024/1*KQ6eJ7A9_0tqFsKKs_Wi1Q.png" /><figcaption><em>The same run, graded twice. The output view has one thing to grade. The trajectory view has every step that produced it.</em></figcaption></figure><h3>Why this matters</h3><p>We are building Glassray, a platform that monitors agents in production to catch silent failures and generate evals from production data. Most of the failures it catches look like this: the output is fine, and the trajectory behind it is wrong.</p><p>We evaluated an agent that answers security questionnaires from a company’s knowledge base. A correct-sounding answer that does not come from the customer’s documents is worth nothing on a security questionnaire.</p><p>During one of our evaluations, the agent answered an ISO 27001 certification question confidently and correctly. Every output check passed it.</p><p>Then we opened the trace.</p><p>The agent had never searched the knowledge base. Earlier in the conversation, the customer had asked about SOC 2, and the agent had answered correctly. When the ISO 27001 question arrived, it assumed a company with SOC 2 probably has ISO 27001 too. The knowledge base had nothing to confirm that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D2dpL7cU3Z84nTmI.png" /><figcaption><em>This pass was assembled from chat history alone, with no tool call and no source verification.</em></figcaption></figure><p>Judged on the final response, there was nothing to catch. Open the trace and you get a question about one certification, a conversation history about a different one, no retrieval call, and no source check before the agent answered.</p><p>Its rules told it to ground every answer in the knowledge base; this answer was implied from the conversation instead. That shortcut holds only while nothing changes. When the customer replaces a policy or lets a certification lapse, the same shortcut produces an outdated answer that sounds no less confident, and a final-answer check still sees a plausible response.</p><h3>The gap between accuracy and reliability</h3><p>The problem is not unique to one agent. In the last two years, model capability and accuracy have improved rapidly. What has not kept pace is reliability: whether an agent that can solve a task actually solves it consistently, every time, without silent shortcuts or missed steps.</p><p>Princeton’s reliability study (<a href="https://arxiv.org/abs/2602.16666">Rabanser et al., ICML 2026</a>) makes the gap concrete. They ran 15 agents over two benchmarks, GAIA’s 165 tasks and the verified 26-task subset of τ-bench airline, five times each. Accuracy across the agents ranged from 17% to 80%. Their overall reliability score, a composite of consistency, predictability, and robustness, sat in a narrow band from 0.69 to 0.87, and the most accurate agent was not the most reliable one. Two years of capability gains produced only small improvements in reliability. One detail matters for what comes later: across runs, agents were consistent about which actions they took and inconsistent about the order they took them in. The study’s own summary of the shift is to stop asking “How often does the agent succeed?” and start asking “How predictably, consistently, robustly, and safely does it behave?” Its verdict on why that matters: “an agent that succeeds on 90% of tasks but fails unpredictably on the rest may be a useful assistant yet an unacceptable autonomous system.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NOlf9k4dzRa3Kk3oU8gDHA.png" /><figcaption>Accuracy spreads across 63 points between the weakest and strongest agent. Overall reliability moves 0.18. Data: HAL reliability leaderboard, August 2026, 15 agents averaged over two benchmarks.</figcaption></figure><p>If agents fail unpredictably, the next question is what those failures look like. MAST, a taxonomy built from over 1,600 annotated multi-agent traces across seven frameworks, catalogs 14 recurring failure modes (<a href="https://arxiv.org/abs/2503.13657">Why Do Multi-Agent LLM Systems Fail?</a>). The systems it studied failed on 41% to 87% of their tasks. The most common failures are trajectory-shaped: repeating a step that already ran (16% of failures), stating one plan and taking a different action (13%), failing to notice the task is done (12%). None of these show up in the final answer. An output check sees a wrong answer or a right one; it cannot see a right answer that was reached by repeating work, contradicting its own reasoning, or overshooting the task.</p><p>A wrong output tells you the run failed. A trajectory tells you where it failed, and whether the same failure is hiding in runs that happened to produce a correct output. The longer the trajectory and the more agents involved, the more places a silent failure can sit while the final answer still looks right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PCnkr8YZWO2MdKTN_Qf_LA.png" /><figcaption>A schematic run whose final answer is correct. Graded on the output it passes. The trajectory shows a repeated step and a plan the agent never carried out.</figcaption></figure><h4>The failures we found</h4><p>On the questionnaire agent, we went through the traces with Glassray and flagged the runs that deviated from the intended trajectory. Three failure shapes kept coming back:</p><p><strong>Wrong-source retrieval. </strong>Retrieval happened, so a “did it retrieve?” check passes. But the agent searched the wrong place: an internal wiki for a question about the current API, or an old document where a newer version exists. The answer looks reasonable and rests on the wrong evidence. You can only spot it by comparing which source the answer needed against which source the trace actually consulted.</p><p><strong>Phantom grounding. </strong>The answer presents a claim as something it looked up, and the trace contains no retrieval behind it. The ISO 27001 answer is this one. The agent never retrieved a source and framed the claim as if it had. To catch it, take each claim framed as looked-up and search the trace for the lookup. The failure is an absence, so the output never shows it.</p><p><strong>Silent fill-in.</strong> The agent states a fact confidently without attempting a lookup, and unlike phantom grounding, nothing in the answer claims a lookup happened. A policy, document, or value just appears in the answer as if it had always been known. Because the assumption is often right, nothing surfaces the behavior until the assumed value goes stale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4oHZTd5liNYEHJ0NfNfBKw.png" /><figcaption>The three failure types, graded twice: in every case the output view passes and the trace view catches it. Glassray records each confirmed case as a deviation.</figcaption></figure><p>All three produce plausible answers, and some produce correct ones. Output evals would mark all as passing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4490341873462a0d2d8ca23e6a0a415b/href">https://medium.com/media/4490341873462a0d2d8ca23e6a0a415b/href</a></iframe><p>The first row is why output evals stay in the stack. The other five rows are the gap.</p><h3>How to evaluate the trajectory</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a8UdUX5Onr7n5THKlE2joA.png" /><figcaption><em>A behavior spec compiles into deterministic predicates and scoped LLM checks. The predicates run first; the model is called only for the clauses they cannot reach.</em></figcaption></figure><p>Every agent follows instructions. A security-questionnaire agent’s system prompt says “always cite sources” and “never guess configuration values.” A customer-support agent’s says “verify the account before making changes.” Every one of these converts directly to a check you can run against a trace.</p><p>Does the trace contain the behavior in the agent’s instructions? If the answer cites a source, that retrieval must exist in the trace. If the answer states a certification or policy fact, a source lookup must precede it. If a required value was not provided, the trace must show where it came from.</p><p>A useful formalization of this is a behavioral spec: a document that lists what the agent must and must not do, written in natural language, decoupled from any specific tool-call sequence. “Ground every answer in the knowledge base” is a behavioral spec clause. “Call search_documents at step 3” is a golden path. One describes conduct; the other prescribes a route.</p><p>A behavioral spec turns into two kinds of checks.</p><ul><li>Some clauses reduce to deterministic predicates: Did a retrieval tool call occur before the final answer? Is the cited document ID present in the trace? Does the count of search calls fall within bounds? These are string matches and sequence assertions, free to run, and they cover most of the spec.</li><li>The remaining clauses need judgment: does the answer’s claim actually match the content of the retrieved source? Is the agent’s summary faithful to the document it cited? These become scoped LLM calls, one clause at a time, each with a binary verdict and quoted evidence.</li></ul><p>That split controls cost and reliability. Deterministic checks are free and perfectly consistent. LLM checks are expensive and stochastic, so you want as few of them as possible, each scoped as narrowly as possible.</p><p><a href="https://github.com/MaanavKhaitan/behavior-judge">behavior-judge</a> is an open-source project that demonstrates this architecture end to end. It compiles a behavioral spec into an intermediate representation of deterministic predicates and scoped semantic checks, runs the predicates first, and calls the LLM only where the predicate layer cannot reach.</p><h3>How we evaluate agentic trajectories</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8P4X3-6yACR3cqGf8jSZWg.png" /><figcaption><em>How a trajectory eval runs at Glassray: from the agent’s flows and prompts to a confirmed deviation.</em></figcaption></figure><p>At Glassray, we generate a behavior spec for each agent and flow (a group of agents). These files contain the goal, instructions, and guardrails set for that process. For each one of those rules, we generate a small rubric with a combination of deterministic checks, such as a regex or an identifier to look for, and non-deterministic checks for an AI judge to evaluate.</p><p>We then pass the entire trace to a model that goes through those evaluations. The deterministic checks run on the full trace, whereas the AI judge has access to a skeleton of the full trace (the steps and agents involved) and a summary of each step, with the ability to investigate the full context of the relevant sections.</p><p>Through this process, we discover deviations, which you can think of as errors. The three failures above are examples of deviations.</p><p>Once we suspect a deviation, we create a detection recipe for a small model to find it across a larger cohort of traces. The recipe defines what counts as a match, what does not, common false positives, the evidence the checker must quote, and what to return when the evidence is unclear.</p><p>Run the recipe against one trace at a time, with a constrained verdict (present or absent) plus the quoted evidence.</p><p>A condensed version of our phantom-grounding recipe:</p><pre>match:      the answer states a fact as retrieved or looked up, AND the trace<br>            contains no retrieval or tool call that returned that fact<br>not match:  the fact is framed as an assumption, or as a question back to the<br>            user; retrieval happened but returned a superseded document (that<br>            is wrong-source retrieval, a separate recipe)<br>evidence:   quote the answer&#39;s claim AND name the retrieval that is absent<br>if unclear: return absent (false alarms erode reviewer trust faster than a<br>            rare miss, and misses are backstopped by sampling)</pre><p>We ran this end to end on the questionnaire agent’s own traffic. Glassray had recorded the confirmed cases as deviations; a strong model turned one into a recipe like the block above; we labeled 24 of the agent’s real traces as an answer key, with the evidence quoted per trace; and each candidate model then ran the recipe over the identical extracted traces, three times each.</p><h3>What’s hard about it</h3><p><strong>Trusting the judge.</strong> LLM judges are inconsistent. They need calibration before you can trust their verdicts, and the calibration has to be redone whenever the model changes. For open-ended quality scoring (is this answer good?), that problem is hard to solve.</p><p>Trace checks are narrower. They check whether a specific relation holds inside one document: does the cited source appear in the trace, does a retrieval precede the claim? The checks are close to binary, which makes them easier to validate.</p><p>You can validate without labeled data. Take a trace that passed, break it deterministically (delete the retrieval span or swap the cited document for a different one), and check that the judge flags the broken copy while still passing the original. Because the break is deterministic, you know the right answer without labeling anything. When we do calibrate against labels, the bar is agreement on a small adjudicated set before the judge’s verdicts count, and we weight misses most heavily.</p><p><strong>Cost. </strong>Running an LLM over every trace is expensive if you send the full trajectory in a single call. Most of the work does not need an LLM. Checking “does this retrieval exist in the trace” is a string match. Checking “did the agent call the search tool before answering?” is a sequence assertion. Reserve the LLM for what only an LLM can check (does the answer’s claim match the content of the retrieved source), scope it to one claim at a time, and use a cheaper model.</p><p>We benchmarked four models and cut the cost of the per-trace check by 1.5x while catching more deviations. The full comparison is in <a href="https://www.glassray.ai/blog/cheap-model-trace-judge">We found a 1.5x-cheaper trace judge that catches more real failures</a>.</p><p>Trajectory evals are not there to replace output evaluation; they complement it. You still want to know whether answers are good, and a final-answer eval is the right tool for that question. It cannot tell you whether a good-looking answer came from the customer’s documents or from a guess that happened to land. The trace is where that gets answered.</p><p>Start smaller than any of that. Pull your last 20 traces and read them against the three failure types.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6172ea10090f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We put Claude on the night shift]]></title>
            <link>https://medium.com/@chamathpali/we-put-claude-on-the-night-shift-08b560be386b?source=rss-3e30c36b8db5------2</link>
            <guid isPermaLink="false">https://medium.com/p/08b560be386b</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[developer-productivity]]></category>
            <dc:creator><![CDATA[Chamath Palihawadana]]></dc:creator>
            <pubDate>Thu, 06 Aug 2026 17:50:33 GMT</pubDate>
            <atom:updated>2026-08-06T17:50:33.143Z</atom:updated>
            <content:encoded><![CDATA[<p>Every codebase has a pile of work that is nobody’s idea of a good time. The small UI tweak that matters, but never quite makes the sprint. The broken edge case in a date filter. The lint error everyone has learned to ignore. The PR that was ready to merge until main moved underneath it.</p><p>We are a small team building <a href="https://www.linkedin.com/company/glassray/">Glassray</a>, and we want to move fast. But for us, moving fast does not mean asking engineers to spend more hours on repetitive work. It means protecting their<strong> time for the hard problems, the product decisions, and the engineering judgment</strong> that actually move the company forward. So we moved the boring work to the night shift.</p><h3>The routine that creates PRs while we sleep</h3><p>Each engineer on the team has their own Claude routine running roughly once an hour overnight. The routine only picks up Linear tickets assigned to that engineer and explicitly tagged for Cloud Agent. Once it takes a ticket, it claims it so the next run does not duplicate the same work. That setup matters. Claude is not pulling random work from a shared backlog. It is working inside the <strong>same ownership model </strong>we already use as a team: assigned tickets, clear scope, and a human who is responsible for reviewing the final PR.</p><p>From there, the routine reads the ticket, checks the comments, finds the relevant part of the codebase, and makes the smallest correct change that stays within scope. That last part matters too. We are not asking it to rethink the product, redesign a flow, or make broad architectural decisions. We are asking it to clear the kind of work that is important, but repetitive enough that<strong> it should not consume a human engineer’s best hours.</strong></p><p>For user-facing changes, it captures before-and-after screenshots. For backend changes, it includes the passing command output. Then it opens a PR with the root cause, the fix, and the evidence, before moving the ticket to “Waiting for review”. One ticket per run, then it stops. An empty queue is a good night’s work, not an invitation for the agent to go invent more things to do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P3d3kxXW6h1STiHRe8oIEw.png" /><figcaption>Claude routine flow</figcaption></figure><h3>The best part is the feedback loop</h3><p>Our PRs also get an automated review from <a href="https://www.linkedin.com/company/greptile/">Greptile</a> , which leaves real, specific feedback on the diff. The routine reads that feedback and acts on it. Claude opens the PR. Greptile reviews it. Claude pushes fixes that address the comments. That loop continues until the review comes back clean.</p><p>By the time a PR reaches a person, it has already been<strong> written by AI, reviewed by AI, and revised against that review.</strong> The most common round trip in code review, leaving a comment and waiting for a fix, mostly happens before a human even opens the tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*HHwt-CMbr8FF2Kfy2QTg0Q.png" /><figcaption>PR Checks for peace of mind</figcaption></figure><h3>The setup is not magic. That is why it works</h3><p>Claude is not just giving us suggestions in a chat window. It runs in a cloud environment where it can access the repo, inspect the codebase, make changes on a branch, run commands, and prepare a pull request. That makes it powerful, but it also makes the boundaries more important.</p><p>We do not give it a vague instruction like “go improve the app.” It works from assigned and tagged Linear tickets, one at a time. Each ticket needs a clear scope, a clear expected behaviour, and a clear definition of done.</p><h3>Then we hit the next bottleneck</h3><p>Once the first routine started working, a new problem became obvious. The PRs were better prepared, but they could still go stale. A branch that was clean at 1am might have conflicts by 9am because main kept moving. That meant engineers were still spending time doing the kind of maintenance work we were trying to avoid.</p><p>So we added a companion routine. This one does not create new work. It looks after the work that already exists. It walks through open PRs, checks each one against main, and if it finds a merge conflict, it resolves it carefully, commits the fix, pushes the branch, and leaves a GitHub comment explaining exactly what changed. This routine exists for one reason: <strong>to protect the time saved by the first one.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xwt0UktPBNkmj5sqMREiHQ.png" /><figcaption>Second Routine for merge conflicts</figcaption></figure><h3>What moving fast actually looks like</h3><p>For a small team, speed does not come from working more hours. It comes from protecting the hours you already have. At <a href="https://www.linkedin.com/company/glassray/">Glassray</a>, the hard work is not fixing another lint error or resolving the same merge conflict twice. The hard work is building trust into production AI agents.</p><p><strong>We want engineers spending their best hours on the problems that make the product better:</strong> helping teams understand when their AI agents fail, why they failed, and whether the proposed fix is actually safe to ship.</p><p>That is what moving fast looks like for us. Not agents merging code while everyone sleeps. Not a fully autonomous engineering team. Just a small team being deliberate about where human attention should go.</p><p>We wake up to a short stack of PRs that are tested, conflict-free, and already cleaned up against review feedback. Each one is waiting on a single human decision.</p><p>Running an agent off the leash is easy. Putting it on a leash short enough that you trust the output is the real work.</p><p>So where do you draw the line between agent autonomy and human judgment?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08b560be386b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>