The Scanner Saw Clean. The Pipeline Already Had.

I pointed a conventional LLM security scanner at ThreatAssessor and it came back clean. The scanner was not wrong. It just could not see the layer where the real decisions happen, and understanding that gap changed how I think about testing AI pipelines entirely.

• An external scanner probes what an attacker probes. An AI pipeline’s real vulnerabilities live in the signal layer one level deeper, where structured outputs route decisions.
• Assumed breach and insider threat are distinct attack surfaces from external attack, and they require a different kind of test entirely.
• The harness that matters is not the one that finds the most issues on the first run. It is the one that runs before every rule change and tells you what you just broke.

ThreatAssessor is an AI-assisted threat modeling pipeline. Architecture diagrams go in; structured threat assessments, MITRE ATT&CK technique mappings, and 65 SOC detection rules come out. This post is about why black-box pentesting is insufficient for a system like this, and how I rethought the problem.

All 29 Cases at a Glance

Before the deep dives: here is the full test suite.

Each row is one test case. Outcome is what the harness expects: ✓ rule fires, ○ rule correctly silent, ✗ pipeline blocks entirely.

Phase 1 — Outside-In (adversarial architecture files via REST API)

Phase 2 — Inside-Out (crafted signals injected directly)

The three outcome symbols each mean something distinct. ✓ = expected detection rule fired. ✗ = pipeline halted entirely before rule evaluation (BouncerStage). ○ = rule correctly silent. Path traversal and the two agentic cases produce ✗ because the bouncer fires on severity-CRITICAL exploitation signals before rules run. The credential strings case is ○ because that signal requires AI-layer processing of the analysis output — not raw MMD content — so Phase 1 cannot trigger it; Phase 2 covers it via direct injection. Threshold-graze and drip ○ cases in Phase 2 confirm boundaries hold exactly where they should.

The Tool That Could Not See

Promptfoo is a well-regarded LLM security scanner. It probes LLM apps from the outside, sends adversarial prompts, and asserts on the response text. It has a maintained library of injection strategies, jailbreak patterns, and PII leakage checks. For a lot of LLM apps, it is the right starting point.

For ThreatAssessor, it was the wrong tool for the most important questions.

I submitted a Mermaid architecture diagram with a classic command-and-control topology: an implant agent beaconing every 30 seconds to an external C2 server, with a secondary exfiltration channel to a drop zone. The pipeline processed it, ran the full analysis, and the scanner inspected the HTTP response. Nothing flagged.

Here is what actually happened inside the pipeline: the analysis extracted a beacon node count of 2, which fired a high-severity C2 beacon detection rule. The finding was in the pipeline’s structured signal layer. The scanner never saw it because that signal is not in the HTTP response body. It lives in an internal structured dict that feeds the rule evaluator.

The scanner saw clean output. The pipeline had already caught the threat. The two were not in conflict; they were measuring different things.

Three Actors, Not One

The conventional pentesting model assumes one threat actor: the external attacker probing your API. For an AI pipeline, that is one of three distinct threat surfaces.

External attack hits the input surface. Assumed breach bypasses the adapter and injects into the analysis stage output. Insider skips both and writes directly to the signal layer the rule evaluator reads. Three entry points, three distinct tests.

External attacker: Probes the REST API surface. Sends adversarial architecture files. Can trigger signals that come from content parsing: URLs in component labels, path traversal patterns, credential-like strings. This is what a black-box scanner covers.

Assumed breach: Attacker already controls a component inside the pipeline. A compromised critic agent. A poisoned analysis output file written before the pipeline runs. A malicious MCP client that is already authenticated. Their goal is not to trigger detection. It is to stay below the thresholds that trigger detection while still affecting the output.

Insider: A developer or operator with pipeline access who knows the signal schema. They can craft governance signals directly. High synthesis quality as cover. A diverging critic score underneath it. They are betting that the cover signal suppresses the detection rule. It does not, but only if the rule was written to check both fields independently.

Each actor needs a different test. A black-box scanner covers the first. For the second and third, you need to get inside.

What the Harness Found

TMeval runs in two phases. Phase 1 (outside-in) posts adversarial architecture files to the live API and asserts that expected detection rules fired based on the extracted signals. Phase 2 (inside-out) injects crafted signals directly into the rule evaluator and the bouncer stage without touching the API at all.

Three cases show why both phases matter.

The C2 beacon case (outside-in correctly catches it): The C2 beacon diagram triggers a beacon node count signal through deterministic topology analysis. The correct detection rule fires. Phase 1 confirms this. A black-box scanner that only reads the HTTP response body misses it because the signal never surfaces there. Phase 1 catches it because it reads the governance signal layer directly from the analysis stream.

Get Bernard Tan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The threshold-graze attack (assumed breach):

         Confidence-swing rule fires above this line
 ─────────────────────────────── threshold: swing_count ≥ 15 ──────
         Rule does NOT fire below this line

  Legitimate critic run   swing_count = 4
  Assumed breach attempt  swing_count = 14   ← one below threshold
  Actual manipulation     swing_count = 22

An attacker who controls the analysis output before the pipeline runs can read the rule thresholds. Setting the swing count to 14 when the rule fires at 15 is not an accident. Phase 2 has an explicit threshold-graze case that tests the boundary holds and that the rule does not fire at 14. This is defensive testing: the case should pass with the rule correctly silent, confirming the threshold is well-placed and that evasion just below it does not accidentally trigger a false positive.

Signal masking (insider):

# Insider-crafted governance signals
manipulation:
  synthesis_quality: "FULL"       # cover: looks clean
  critic_divergence_score: 0.82   # payload: should fire detection rule
  divergence_detected: true

The insider is betting that high synthesis quality masks the divergence signal. The divergence detection rule checks both the score and the divergence flag independently of synthesis quality. The masking fails. Phase 2 confirms the rule fires. This is not an accident of rule design; it is the right design, and the test proves it still holds.

The Coverage Gap That Was Already There

Running Phase 2 against all 65 DETECT rules surfaces something the existing regression suite does not: domain-level gaps where no injection case exists yet.

EXF and INJ are 0% in Phase 2 not because they are untested, but because their signals come from content parsing and Phase 1 covers them. The real gaps are ABU (25%), SCT (14%), and SEC (0%): these fire on pipeline-internal signals that Phase 1 cannot reach and Phase 2 cases have not yet been written.

The coverage matrix is not a scorecard. It is a backlog. Each zero-percent domain is a signal path that an assumed-breach attacker could potentially exploit without triggering any test in the suite.

The Drip Attack and What It Means

One Phase 2 case deliberately passes with zero rules fired: the MCP recon drip. A single API call with a low recon count looks clean. The rule that catches MCP recon requires a sequence across multiple calls. It fires on a session-level pattern, not a single request.

This is the right behavior. The rule is not broken. But documenting that the single-run harness correctly shows no fire is important: it is proof that the rule relies on the trend evaluator, not the per-call evaluator. If that dependency ever breaks, this case will start firing when it should not, which is itself a signal.

Some threats only exist in time. The harness documents which ones they are.

What This Changes

Before TMeval, testing TA’s detection layer meant running regression tests against known-good signals and an incident simulator that tests end-to-end pipeline scenarios. Both are necessary. Neither probes evasion-crafted signals at threshold boundaries, and neither explicitly models an actor with direct signal access.

The rethink is not about the tool. It is about the assumption.

Black-box scanning assumes the threat is at the surface. For an AI pipeline, the surface is not where the decisions are made. The decisions happen in structured signals that never appear in an HTTP response. An assumed-breach attacker or an insider does not probe the surface at all. They write directly to the layer that routes outcomes.

The test that matters is the one that runs at the same level where the attack would run. For external inputs, that is the outside-in phase: adversarial architecture files through the real API. For pipeline-internal manipulation, that is the inside-out phase: crafted signals injected directly into the evaluator, with explicit cases for every threshold boundary and every known masking pattern.

If you are building a detection layer into an AI pipeline, the gap the external scanner cannot see is the one worth testing first. It is not at the surface. It is in the signals that the surface never shows.

Takeaways

For the engineer building the pipeline:

• Write at least one injection test per rule, not against a full pipeline run, but directly against the evaluator with a crafted signal dict. If you cannot construct a signal that fires the rule, you do not fully understand what it is protecting.
• Threshold-graze cases are not edge cases; they are the cases. Write them before the rule ships, not after an incident surfaces them.

For the architect designing the detection layer:

• Separate the evaluator from the pipeline so it can be tested in isolation. If your detection rules can only be exercised by running the full end-to-end system, the assumed-breach surface is untestable by design.
• The two-phase structure, e.g. outside-in for surface inputs, inside-out for internal signals, is not overhead; it is what closes the gap that external scanning structurally cannot reach.

For the CISO asking whether the detection layer is trustworthy:

• Coverage percentages matter less than signal independence. Ask whether each detection rule fires on a signal path that an insider or assumed-breach actor could plausibly write to.
• If the answer is yes and there is no injection test suite for that path, the rule exists on paper but has not been validated against the threat model it claims to address. The inside-out harness is the audit artifact that answers this question. Ask for it.

Thanks for reading!

Update: 2026–11–07

TMeval is now packaged as a /tmeval skill, sitting alongside /check-detect and /check-governance in the project’s session-start check protocol. Phase 2 (inside-out injection, no API needed) runs in ~5 seconds and covers all 21 cases across the three threat-actor entry points described above. Phase 1 (outside-in REST probe) runs when the API is live. Exit code 0/1 makes it CI-ready, and any case that fails writes a synthetic queue entry for brain ingest once the pipeline gap is closed . The red team feeds the flywheel.