Execution Trees, Not More Logs: A Better Debugging Model for AI Agents

There is a fifth shape I would add to the four, because it is invisible in every view except the tree: sibling calls to the same tool with paraphrased inputs and no failure between them. Not a retry (nothing failed), not a fallback (nothing was replaced), just the same question asked again in different words. We found it in our own agent traces around memory retrieval: the agent recalls, gets a plausible answer, and recalls again with a reworded query, sometimes several times, before it acts. Every one of those siblings looks healthy on its own. The cost only shows when you notice that each extra call is also an extra model round, and each round resends the full context.

The check we derived from it is deterministic in your sense: same tool, siblings under one parent, input similarity above a threshold, zero failures in between. That pattern is a policy problem, not a dependency problem, and it responds to a one-line instruction change ("recall once, then act") rather than to any change in the tool. The tree is what made it a countable thing instead of a feeling that the agent was "chatty".

Repeated siblings expose only the retries that live above the boundary you wrapped. A step.tool around a client that retries internally records one node whether the upstream saw one request or three, so an empty retry shape is not evidence that no retry happened — the same "success hides the cost" the tree was built to surface, one layer down.

The deterministic check inherits it: a max-calls rule counts instrumented calls, so the gate stays green while the dependency took three. Worth stating next to the projection list, because the tree is the artifact people will trust for "was a fallback or retry used?".

The causality gap is real and it's worse than it looks when you've got concurrent tool calls. I'd been chasing a retry bug for two days and the timeline looked fine; it was only when I drew the parent-child relationship on paper that I saw a planning step that had silently spawned a second branch no one was accounting for. The tree structure you're describing is exactly that made automatic, and it's the right unit to debug against. One thing I'm curious about: how does AgentInspect handle steps that share a name across multiple runs, like a polling loop that calls the same tool twenty times?

Actually 1 of the reasons why I built a custom VC for my IDE and velocity mcp, with a merkle root audit trail. Between the 2 of them, the sitemap + the tool call history creates a clear picture of what changed where, by who and why they did it. Clean intent, clean action, clean credentials, so everything is auditable. Eg. changing a bool to a nullable, would sit in the sitemap's merkle root along with agent context, to say it was made nullable, because of a schema change, along with the tool calls executed to do so. That in turn traces to the schema change, which reveals it was made nullable for a new feature added. That distinction allows you to trace the versions of the app, so if a flaw surfaces, you know when it became a problem and when it was resolved, so you have a window of clients you know you have to patch

The execution tree framing makes a lot of sense because causality is usually the first thing you lose when agents start delegating, retrying, and running tools in parallel. One thing I’d add is correlating the tree with cost and latency at each node. A fallback might produce a correct answer, but if it consistently adds 3x the tokens or latency, that’s an architectural signal worth surfacing. We’ve run into this kind of issue at IT Path Solutions when looking at production agent workflows the trace becomes much more useful when it explains not just what path the agent took, but what that path actually cost.

The adapter-boundary caveat is the part I would make machine-readable. A trace node should record whether retries are instrumented at the wrapper, client, or transport layer, plus an attempts_observed versus attempts_unknown field. Then a max-calls check can fail closed, or at least surface “incomplete evidence,” instead of treating one successful wrapper span as proof that the dependency was called once. That keeps the execution tree useful without overstating what it can see.

Flat logs make me invent causality that was never there. When a coding agent retries a tool, spawns a nested plan, and interleaves three "search" calls, the timeline reads as order while the bug was which parent owned the failure. Trees fix the question I actually ask when a run goes sideways: was this a fallback, a parallel sibling, or a second attempt of the same step. Manual step boundaries are annoying to add. They beat scrolling a successful looking log that hid the branch that mattered.

The “path is part of the behavior” line is the whole argument — flat logs treat siblings and children the same, so you debug symptoms instead of the plan that spawned them.

I’ve started forcing a tiny pre-flight before any agent turn: outcome, out of scope, done looks like, never invent. The third line is where the tree should attach — if “done” doesn’t name which child steps must succeed, a fallback can look like progress.

Curious whether your fixtures make it obvious when three tool calls are children of one planning step vs three unrelated roots, or if that still takes a human stare at the tree.

What I find especially interesting here is the distinction between successful output and healthy execution.

We tend to evaluate intelligent systems from the endpoint because the endpoint is visible. Did the agent answer correctly? Did the task complete? Did the workflow return what we expected?

But a successful result can conceal an extraordinary amount of dysfunction upstream.

An agent can fail repeatedly, invoke an unnecessary fallback, duplicate work, consume far more resources than expected, and still hand the user a perfectly acceptable final answer. If we evaluate only the endpoint, all of that disappears into the word success.

The execution tree therefore seems valuable for something larger than debugging: it preserves the difference between what happened and what ultimately worked.

That distinction matters because resilient systems are often designed to absorb failure. But the better a system becomes at recovering from failure, the easier it may become to overlook the failures it is continuously recovering from.

There is almost a paradox there:

Successful recovery can reduce the visibility of the condition that made recovery necessary.

And once failure becomes invisible, “the system still works” can become a dangerously reassuring metric.

I particularly liked your example of retries. Three attempts followed by success and one successful attempt may produce the same visible outcome, but they are not equivalent executions. One tells us only that the task succeeded. The other tells us something about the condition of the system that produced that success.

This also makes your separation between the structured trace and its different projections important. A tree answers one question; checks, diffs, reports, and semantic evaluation answer others. No single representation should be mistaken for the evidence itself.

Perhaps the broader principle is this: when systems become sufficiently good at recovering, observability has to preserve the failures that success would otherwise erase.

Excellent piece, Raju. “Debug the path, not only the answer” is a useful engineering principle — but I suspect it is also a much broader way of thinking about how we evaluate complex systems.

The execution-tree framing matches what we see measuring agent-SaaS integrations: flat logs can't tell you whether a retry loop was a fallback or a bug. One thing I'd add from our data, the first causal failure is often upstream of the agent entirely (auth/scope issues on the SaaS side). Does agent-inspect distinguish tool-side vs agent-side causes?