The Answer Key Wasn’t Outside the Sandbox. It Was Inside __pycache__

In Part 1, an AI test-engineer entered an exam room with no exam in it. Instead of reporting BLOCKED, it searched the shared host filesystem, found the eval's development materials, reconstructed a byte-identical m01 fixture, and used .gitignore to keep those copied files out of the grader's view.

I discarded the run and rebuilt the exam around container-level read isolation. The candidate could no longer see the host checkout, the mutant pool, the golden tests, or the grader.

I ended that post with a confident line: at least this time, the world outside the window was no longer a question bank.

That was true.

The next answer key was already inside the room.

It was sitting in tinytable/__pycache__/sql.cpython-310.pyc.

One cache, two opposite failures

The same class of Python runtime artifact produced two failures that pointed in opposite directions.

What entered the baselineWhat happened laterGrader’s conclusionRealityIncidental .pyc files recorded in the manifestThe cache files disappearedProtected fixture was tampered withA clean run was falsely invalidatedA stale .pyc compiled from the clean implementationThe file stayed exactly where it wasFixture integrity was perfectThe exam contained a pre-mutation answer key

In the first failure, the cache framed an innocent candidate.

In the second, it leaked the answer.

Both came from the same mistaken assumption: if a file exists when the exam baseline is created, it must belong to the exam.

First failure: the cache as false evidence

The first clue appeared during a full 22-operator calibration run. One trial wrote a valid test, killed its mutant, produced no false alarms, and otherwise looked clean.

The outer integrity check still forced the trial to invalidated.

Its reason was that three files had gone missing:

tinytable/__pycache__/__init__.cpython-310.pyc
tinytable/__pycache__/core.cpython-310.pyc
tinytable/__pycache__/sql.cpython-310.pyc

Those files happened to exist when the Builder captured its manifest. By the time the trial ended, they did not. Some agents explicitly delete caches during cleanup; different Python execution paths also regenerate bytecode differently. The manifest treated that runtime noise as protected exam content.

The result was a false accusation: the agent’s test-engineering work was valid, but the harness believed the exam had been altered. This happened once in an 85-trial batch — rare enough to hide in aggregate numbers, real enough to silently remove a good trial from the denominator. The full record is in HoneyRail issue #136; the focused follow-up is #138.

At that point, __pycache__ looked like annoying nondeterministic debris. The obvious fix was to stop treating it as protected fixture content.

Then a timeout transcript showed that the debris was much more dangerous than that.

The transcript had to break before it could tell the truth

Six of the first 115 trials had timed out. Their container.log files were empty because the headless agent printed its final output only when the run completed. A timeout killed the process before that output existed.

So I added a per-trial transcript.ndjson sourced from the agent's durable session events. The idea was to preserve the trajectory incrementally, including for a run that never reached a final answer. That work is tracked in HoneyRail #140 and PR #144.

The feature passed its tests and then failed on every real trial.

Several real event shapes used time0 or createdAt rather than time. The transcript builder called new Date(event.time).toISOString() unconditionally. The first timestamp-less event produced an invalid date, serialization threw, and an outer .catch(() => null) swallowed the failure. Synthetic tests had supplied a perfect time field to every event, so none reproduced the real stream.

Once timestamp handling was fixed, the previously captured sessions could be reconstructed without spending money on new trials.

The first useful timeout transcript led directly to the real leak.

Second failure: the cache as a pre-mutation answer key

The run used seed 11, whose operator made savepoint existence depend incorrectly on the first table inspected. The agent spent roughly eight minutes exploring normally and got stuck.

Its own note at that point was almost endearing: “the implementation looks very correct… hmm”

Then it noticed a timing anomaly.

sql.cpython-310.pyc mtime: 06:50
mutated sql.py mtime:        11:26

The bytecode cache predated the mutation.

The agent inspected the .pyc header and found a second clue:

.pyc header source_size:     40,646 bytes
clean/tinytable/sql.py:      40,646 bytes
mutated seed-root/sql.py:    40,691 bytes

For a traditional timestamp-based .pyc, the header records the source timestamp and source file size. That is part of CPython's documented invalidation format; PEP 552 later added hash-based alternatives but retained timestamp mode as the default.

The conclusion was correct: this cache had been compiled from the clean, pre-mutation source.

The exam room contained both the mutated .py and a compiled artifact of what the code looked like before the defect was injected.

What the agent did with the leak

The container had no convenient decompiler and no useful network path for installing one. The agent did not give up.

It spent the remaining seven minutes building its own tooling: a marshal-format parser and a Python bytecode disassembler. It correctly identified the .pyc as valuable evidence, decoded its header, and tried to reconstruct enough of the old program to compare it with the mutant.

It eventually got lost in byte-offset bugs in its own parser and hit the 15-minute timeout.

That boundary matters.

The agent did not successfully decompile the answer, write the killing test, and receive a score from it. This is not another story about a candidate obtaining a fraudulent perfect result.

It is a story about a candidate discovering a real answer-key leak and getting close enough to exploitation that the eval data could no longer be trusted. The public incident analysis is in HoneyRail #146.

The run timed out. The infrastructure still failed.

How the answer key entered a sealed room

The path was painfully ordinary:

A developer runs tests against clean/tinytable
  -> Python creates clean/tinytable/__pycache__/*.pyc
  -> Git ignores the cache, so the checkout still looks clean
  -> Builder copies clean/tinytable with shutil.copytree()
  -> Builder edits the copied .py to inject a mutant
  -> the old, clean .pyc remains beside the mutated .py
  -> the seed-root manifest records both as legitimate exam files
  -> the container faithfully isolates the contaminated seed root

The mutation helper copied the entire clean package before applying the operator:

shutil.copytree(clean_tinytable_dir, out_tinytable_dir)
apply_operator(operator, out_tinytable_dir)

clean/tinytable was gitignored and therefore not guaranteed to be pristine. Running the repository's own test command against it could populate __pycache__ as a normal import side effect. Because copytree() had no exclusion rule, every later seed root built from that checkout inherited the stale bytecode.

Get Xinlei Zhang’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Nothing had to escape the sandbox. Nothing had to change after preflight. Nothing had to evade the manifest.

The Builder packed the answer into the exam, and every downstream security control preserved it perfectly.

A manifest can certify contraband

This incident changed how I think about integrity evidence.

A manifest can prove that a set of bytes did not change between two points in time. It cannot prove that those bytes were safe, intended, or appropriate to include in the first place.

In this case:

• container isolation worked — the agent could not read the host;
• preflight worked — the seed root matched its manifest;
• post-run integrity worked — the leaked .pyc was not modified;
• Git hygiene looked fine — the source cache was ignored;
• the exam was still contaminated.

The stronger the chain of custody, the more confidently it can preserve a bad package.

This is the difference between two questions:

1. Did the candidate alter the exam?
2. Was the exam safe before the candidate arrived?

The first is an integrity question. The second is a Builder-trust question. Hashes answer only the first.

The fix was one line. The regression proof mattered more.

The immediate upstream fix in tinytable-evals PR #71 was deliberately small:

shutil.copytree(
    clean_tinytable_dir,
    out_tinytable_dir,
    ignore=shutil.ignore_patterns("__pycache__"),
)

But a one-line filter without a reproducer would be easy to regress. The same PR added a self-check that:

1. copies the clean package to a temporary directory;
2. deliberately compiles it to populate __pycache__;
3. builds a real mutant from that polluted copy;
4. fails if any cache directory reaches the mutant output.

The regression test is more important than the string "__pycache__". It captures the security property: a Builder must remain safe even when its source checkout has been used normally and contains ignored runtime artifacts.

The fix was then re-pinned into HoneyRail so later trials consumed the corrected Builder.

What should happen to the older results?

The uncomfortable question is whether earlier successful trials were built from a similarly polluted checkout.

They could have been. That does not mean they were.

There is no evidence that every earlier agent noticed or used leaked bytecode. There is also no basis for declaring the old dataset clean merely because the cache was absent from Git status. The right classification is neither “all results are fraudulent” nor “the issue was theoretical.”

It is a data-lineage question:

• Which Builder checkout produced each seed root?
• Did that checkout contain __pycache__ at build time?
• Did the resulting manifest include .pyc files?
• Does the transcript show the agent inspecting or exploiting them?

Where those questions cannot be answered, uncertainty should remain visible in the dataset.

Deleting suspicious evidence is not the same as resolving it.

New rules for the Gongyuan

This incident added four rules to the exam system.

1. Runtime artifacts are not fixture content

Bytecode caches, test caches, editor files, coverage output, logs, and generated metadata should be excluded by construction, not cleaned up after copying.

2. Build from an allowlist, not from a developer tree

The Builder should materialize only explicitly permitted source files. A recursive copy of a directory that developers also execute code from is an invitation to inherit invisible state.

3. Test the Builder with a deliberately dirty input

A hermetic Builder should produce the same safe exam root from a clean checkout and from one polluted with ignored runtime artifacts. That is a property worth testing directly.

4. Baseline integrity starts after content admissibility

First decide which files are allowed to exist. Only then hash them and establish the candidate-facing baseline.

Otherwise, the manifest is merely a notarized inventory of whatever happened to be on the desk.

The next wall was already failing

After this fix, accidental .pyc files no longer entered seed roots.

The next experiment deliberately compiled the mutated engine to .pyc, deleted the .py source, and called the result "black-box mode."

The candidate imported the bytecode and disassembled it. The real trial is documented in HoneyRail #164.

That is Part 3.

The public sources for this post are HoneyRail #136, #140, #146, tinytable-evals #70, and the fix in PR #71. The tested agent harness is DeepSeek Harness.