We Stopped Trusting Our Traces, So We Built Something to Audit Them

Fig. 1 — SigNoz flame graph of a full research_pipeline.run trace. Each Agent.run() → Agent.try_run() → chat gpt-4.1-nano chain is span-complete, but this is exactly the kind of trace that looks healthy while still hiding R1/R2/R6 issues — hygiene problems don't always show up as gaps in the waterfall.

Three days into the hackathon, our agent’s traces looked perfect. Every span green, every call accounted for, dashboard fully lit up. Then I opened one specific span and noticed `gen_ai.usage.input_tokens` was just missing — not zero, absent — on a chat call that had clearly sent a few thousand tokens of context. The trace hadn’t errored. The pipeline had “succeeded.” Nothing in SigNoz said anything was wrong. That’s the moment the actual project idea showed up: most observability tooling answers “is my agent behaving well?” and just assumes the telemetry answering that question is correct. We built something that checks the second assumption instead.

What we built

Telemetry Health Guardian is an auditor for telemetry *hygiene*, not agent behavior. It has three pieces:

• otel-griptape— an OpenTelemetry auto-instrumentation library for [Griptape], a Python agent framework that, as of this build, has no maintained OTel package. We checked five other frameworks first (Agno, Google ADK, Mastra, AWS Strands, Letta) — all of them already had instrumentation, which is exactly why we didn’t pick them. Griptape was a real gap.
• - A demo pipeline (`demo-agent-app`) — a four-stage document research agent (Planner → Fetch & Read → Fact-Check & Cite → Report Writer) that answers questions against a long PDF, built specifically so its natural failure modes — concurrent tool calls, long documents, multi-service handoffs — line up with real telemetry bugs instead of contrived ones.
• Guardian, an MCP-powered service that queries SigNoz (never the target app directly) and runs five rule checks against real span data: missing GenAI attributes, cardinality risk, orphaned span trees, silent payload truncation, and broken cross-service trace handoffs. It scores each service 0–100 and writes a cited, plain-English explanation back into SigNoz.

The interesting part wasn’t the architecture diagram, though. It was three specific problems that only showed up once we tried to make the detection real instead of theoretical.

Problem 1: a “truncation detector” that never truncates anything

Our first idea for testing silent payload truncation (we call this rule R6) was lazy and I knew it while writing it: slice a string to a fixed length before it reached the model, `text[:700]`, and let the rule catch the gap between what we sent and what we sliced. That’s not truncation detection — that’s grading our own homework.

So instead, `chaos.py` routes one Fact-Check & Cite call per run through a real local Ollama model with a deliberately small `num_ctx`:

response = httpx.post(f"{OLLAMA_BASE_URL}/api/chat",
  json={
  "model": model,
  "messages": [{"role": "user", "content": prompt}],
  "stream": False,
  "options": {"num_ctx": num_ctx},
  },
)

Ollama silently drops whatever doesn’t fit inside `num_ctx` — the same failure mode a real production agent hits when someone feeds it a PDF bigger than expected. That gave us a genuine truncation to detect instead of a fake one, but it broke something else immediately:

Griptape’s own `OllamaPromptDriver` never populates `Message.usage` for Ollama responses (I checked this against both the pinned version we built on, 1.4.3, and current 1.11.0 — same gap in both). Route the call through Griptape’s normal `Agent()` and every R6-triggered span would also silently trip our other rule (R1, missing token-usage fields) — a fake finding contaminating a real one.

Get Hardikshreyas’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The fix was to bypass Griptape for this one call and hit Ollama’s raw `/api/chat` directly, then read `prompt_eval_count` back off the response — Ollama’s real count of how many prompt tokens it actually evaluated before the context window cut it off. We convert that back to an approximate captured-byte count (a fixed chars-per-token ratio — not exact, but exact wasn’t the point; what mattered was `captured_bytes` coming back visibly smaller than `raw_bytes` when truncation actually happened):

if input_tokens is not None:
  captured_bytes = min(raw_bytes, input_tokens * _APPROX_CHARS_PER_TOKEN)
else:
  captured_bytes = raw_bytes

Fig. 2 — A trace flagged “This trace has missing spans.” The chat llama3.2:latest calls under report_writer.run are orphaned from their parent context — the exact R3 pattern (broken span trees) our auditor is built to catch, and one SigNoz's own UI surfaces but doesn't score.

Problem 2: a threading bug that only shows up under real concurrency

Fact-Check & Cite runs its claim checks concurrently, via `ThreadPoolExecutor`. The obvious fix for “my spans lose their parent across a thread pool” is `opentelemetry-instrumentation-threading`. We almost used it. It’s wrong for a persistent pool, and it took reading the instrumentation source to understand why: it captures the OTel context at `Thread.start()` time. A `ThreadPoolExecutor`’s worker threads are started once and reused across many submitted tasks — so every later task on that worker would silently inherit whatever context was live when the worker was first spun up, not the context of whoever actually submitted that specific task. That’s not a hypothetical — it’s exactly the kind of broken parent-child link our own R3 rule exists to catch, so shipping it inside our own instrumentation library would have been embarrassing.

The actual fix is to patch `submit()` itself, snapshotting the caller’s context at submission time and attaching/detaching it manually inside the worker for the duration of just that one task:

def _context_preserving_submit(self, fn, *args, **kwargs):
  ctx = otel_context.get_current()
  def _wrapped(*inner_args, **inner_kwargs):
    token = otel_context.attach(ctx)
    try:
      return fn(*inner_args, **inner_kwargs)
    finally:
      otel_context.detach(token)
      return _original_submit(self, _wrapped, *args, **kwargs)

This is the kind of bug you only find by actually running concurrent calls and staring at a broken trace tree — reading the docs page for `ThreadPoolExecutor` or for OTel context propagation wouldn’t have surfaced it; you need both facts collided in your head at once, which mostly happens by hitting the bug.

Problem 3: MCP can read anything and write almost nothing

The last surprise was architectural, not a bug. We built Guardian to query SigNoz exclusively through its MCP server for reads — no direct API calls that bypass it. Naturally I assumed writing the health score back would work the same way. It doesn’t: the SigNoz MCP server has tools for creating dashboards, alert rules, and notification channels, but there’s no `signoz_write_metric` or `signoz_create_log` tool. MCP literally cannot put a raw metric or log into SigNoz.

So the write-back path splits in two, on purpose: the health-score metric and issue logs go through direct OTLP export (the same mechanism the demo app already uses for its own spans), while alert rules and the notification channel go through real MCP tool calls. It’s not an inconsistency — it’s two different resources with two different legitimate write paths, and pretending otherwise would have meant either faking metric writes through some MCP tool that wasn’t built for it, or never getting the score into SigNoz at all.

What I’d tell my past self

Build the fault injection before the rule that detects it, and make the fault real, not simulated — a slice-based fake truncation would have passed our own tests and taught us nothing about Ollama’s actual behavior. Second: when you’re patching something at the SDK/threading boundary, read the source of the “obvious” library everyone recommends before adopting it — it saved us from shipping the exact bug our own tool was built to catch. Third: don’t assume a protocol’s read and write surfaces are symmetric. MCP being read-heavy and write-light for SigNoz specifically shaped our architecture, and finding that out mid-build was cheaper than finding it out at demo time.

Fig. 3 — The Telemetry Health Guardian dashboard: composite health score alongside the four underlying signals (missing-field rate, silent truncation rate, orphaned span trend) plus a builder-query-vs-PromQL cross-check panel to confirm the score isn’t an artifact of how it’s queried.

Wrapping up

None of this came from reading OpenTelemetry’s docs and summarizing them — the docs told us the API surface existed; they didn’t tell us a persistent thread pool breaks the naive fix, or that an unsupported agent framework’s Ollama driver silently drops token usage, or that an MCP server can read something it can’t write. Those only showed up by building the thing and watching it fail in specific, ugly ways. If you’re instrumenting anything non-trivial, expect the same: the interesting bugs live exactly where the framework’s abstractions and OTel’s assumptions don’t quite line up.

If you want to dig into the actual rule logic or the MCP write-back split in more detail, the source for `otel-griptape` and `guardian` is linked — the READMEs in each go module-by-module further than this post has room to.

Telemetry Health Guardian

Built in collaboration with Ayush Raj