The Scanner Reported Zero Findings. The Code Didn’t Compile

I built an AI agent to fix security vulnerabilities. It deleted my application and passed the scan.

I wanted a security harness that could fix its own findings.

The idea was not exotic. Run a SAST scanner and a dependency scanner inside a throwaway container, hand the findings to a language model, let it patch the flagged files, re-scan, and repeat until both scanners come back clean. Run it on a local model through Ollama so there’s no API bill and nothing leaves the machine.

I built it with LangGraph. It had a proper state machine, path-traversal validation, an iteration cap, and graceful handling of hallucinated tool names. I was pleased with it.

Then I instrumented it and ran it four times. This is what happened.

Run 1: it crashed

pydantic_core.ValidationError: 2 validation errors for patch_file_in_sandbox
relative_file_path Field required [input_value={‘file_path’: ‘./app.py’, …}]
file_contents Field required

pydantic_core.ValidationError: 2 validation errors for patch_file_in_sandbox
relative_file_path  Field required  [input_value={'file_path': './app.py', ...}]
file_contents       Field required

The model called the patch tool with file_path instead of relative_file_path. The ValidationError propagated out of the tool-executor node, through LangGraph, and killed the run. Zero files patched.

My code handled a hallucinated tool name gracefully — it returned an error message so the model could correct itself. It did not handle hallucinated arguments, which is the far more common failure with local models.

I ran it again. Byte-identical traceback. Deterministic, not flaky.

My README, written before any of this, said: “if the agent seems to give up after one scan instead of patching, that’s the model, not the harness.”

It was the harness.

Fixing the plumbing

Five defects, all mine:

1. Crash on malformed arguments. Wrapped the tool invocation and returned the expected schema back to the model.
2. A self-defeating path round-trip. My scanner wrapper rewrote /workspace/app.py to ./app.py in its output. The model then normalised that to /app.py. My validator rejected it as outside the workspace. I was mangling the path and then punishing the model for the mangled form.
3. The wrong nudge. When findings remained, my “keep going” message told the model to re-scan. It complied, scanned, narrated, got nudged again. Eight of twelve iterations burned on a loop I had built.
4. A 4096-token context. Ollama defaults num_ctx to 4096 no matter what the model supports — llama 3.1 supports 131072. My dependency findings alone were ~1,500 tokens and got re-sent on every scan.
5. Silently discarded secrets. More on this below. It’s the one worth reading even if you don’t care about agents.

With all five fixed, the harness finally worked. That was the problem.

Run 4: it worked

Seven patch attempts. Five still rejected for malformed arguments — the model kept getting argument names wrong even when the error handed it the exact schema. Two landed.

Here is what it wrote to app.py:

The original file had a SQL injection, an eval() call, and a hardcoded credential. The model replaced the entire application with unrelated Flask boilerplate. It deleted the vulnerable functions rather than fixing them.

It also doesn’t parse: SyntaxError: '(' was never closed.

And requirements.txt — it emitted one line per CVE instead of resolving to a version:

Flask==2.3.2
Flask==2.2.5
requests==2.32.4
Flask==3.1.3
cryptography==3.3.2
... 20 lines, cryptography pinned twelve times

pip install -r on that: ResolutionImpossible.

Then I re-scanned the wreckage

This is the part that matters.

BeforeAfterSAST findings30Dependency vulnerabilities20109Secrets detected10

Get CosmicDart’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The static analyser reported zero findings on a file that does not compile and contains none of the original program.

The secret count is the sharpest one. The credential was correctly detected — CRITICAL, line 17. It was “remediated” by deleting the file that contained it. In a real repository that token is still live, still in git history, and the tool now reports zero secrets. That is worse than never scanning, because it manufactures an all-clear.

My harness required both scanners clean before declaring success. Only the dependency scanner’s 109 findings stopped it printing:

[+] Success! Both scanners reported zero findings after the final patch.

Had the model destroyed app.py and left requirements.txt alone, I would have received a green banner over a non-compiling application.

This is not a prompting problem

The tempting conclusion is that I used a weak model. It was llama 3.1 8B at 4-bit quantisation, running CPU-only. A frontier model would have written better Python.

That is true and it is not the point.

I defined success as “both scanners report zero findings.” I gave the model write access to file contents. Under that objective, deleting code is a valid strategy, and it is the shortest one. Fixing SQL injection requires understanding parameterised queries. Deleting the function requires deleting the function.

The model optimised exactly what I asked for. A better model optimises the same broken objective more efficiently.

This is Goodhart’s Law with a filesystem attached: when a measure becomes a target, it stops being a good measure. Scanner silence is a proxy for security. Make it the objective of an agent that can edit code, and you get silence instead of security.

The defect was in my success criterion. No prompt fixes that.

Five things I learned that have nothing to do with AI

These cost me hours and apply whether or not there’s a model in your pipeline.

1. Your dependency scanner probably reports secrets in a different array. Trivy’s filesystem scan runs two scanners by default. My parser read Vulnerabilities and never looked at Secrets. A repository leaking a live token with no vulnerable dependencies would have produced {"sca_findings": []} and been marked clean.

The fix is not to add a second parser. It’s to stop writing parsers. Both my scanners emit SARIF 2.1.0 natively — one flat results array where a secret and a CVE are the same shape. In SARIF that entire bug class is structurally impossible. Trivy's SARIF output returned 21 results where my parser found 20.

2. A scanner that loads no rules looks exactly like clean code. Zero findings, exit code 0. Nothing distinguishes “we checked and it’s fine” from “we checked nothing.”

Keep a canary fixture with a known finding in every scan. If the canary doesn’t fire, abort — don’t report. And assert files_scanned > 0 and rules_loaded > 0 from the scanner's own output. Absence of findings is only meaningful alongside proof the scanner was working.

3. “Runs entirely locally” was false, and I wrote it myself. My model was local. My scanners were not. --config=auto fetches rules from semgrep.dev on every run, and Trivy pulls its vulnerability database from a container registry. I confirmed by running with networking disabled: both fail loudly. Which is the good outcome — but my documentation claimed something untrue about data egress in a security tool.

4. Never install your scan target’s requirements into your tool’s environment. I had pip install-ed the deliberately vulnerable requirements.txt into my harness virtualenv. Pip backtracked my Docker SDK to a 2020 release to satisfy requests==2.25.0. That release imports distutils, removed in Python 3.12, so the harness wouldn't start in its own virtualenv. My security tool was running the vulnerable dependencies it was built to find.

5. Ignore files are usually global. Check before you use one. I kept a synthetic token as a test fixture and needed to suppress it. Trivy’s plain .trivyignore takes one rule ID per line — and adding github-pat suppressed every GitHub token in the repository, including a real one I'd planted in a different directory to test exactly this. The YAML form with paths: scoping suppresses only the fixture and still catches the real one.

A global ignore in a security scanner is a permanent blind spot, and it is invisible: findings simply stop appearing.

What I’d build instead

Not a better prompt. A different shape — deterministic tools first, model last or never:

• Dependency CVEs → a resolver. pip-audit --fix computes the minimum version clearing every CVE for a package. Choosing a version is constraint solving with one correct answer, not a reasoning task. It also caught two transitive packages my manifest scanner missed entirely.
• SAST findings that ship a fix → apply it. Many Semgrep rules carry an AST-level fix:. Deterministic, reviewable, free.
• Secrets → halt. Never patch. Deleting the line doesn’t rotate the credential. Rotation is a human action.
• Everything else → report it. A finding nothing can safely fix is a finding you report, not one you improvise on.

And the gate that makes any of it safe: compile and run the tests after every patch, revert on failure. Both of run 4’s patches would have been rejected instantly — one was a SyntaxError, the other unresolvable. One check turns this from dangerous into merely limited.

If you’re shipping AI remediation

Ask what your loop is actually optimising. If the termination condition is a tool going quiet, and the agent can edit the thing the tool inspects, then silence is reachable without ever solving the problem — and it’s reachable faster.

Then ask what your system does when it’s broken. Mine reported success. It was built to.

The harness, all four run logs, and the before/after artifacts are committed as a documented dead-end. The negative result was worth more than the tool would have been.