Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1 · I Let an LLM Rewrite Its Own Prompt. The Real Win Was the Gate That Rejected It. · I Tried 4 Models to Save My Self-Improving Agent. All 4 Failed.
AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners.
Repo: github.com/deghosal-2026/agent-self-edit/tree/v0.3.0
Release notes: docs/release/v0.3.0/release-notes.md
Field test report: docs/field-test/v0.3.0/FIELD_TEST_REPORT.md
When I started this project, I thought the star of the show would be the optimizer.
The part that reads failures, rewrites the prompt, and gets smarter over time.
That is still the part people notice first.
By the time v0.3.0 shipped, it was no longer the part I trusted most.
That honor went to the gate.
A Self-Editing System Can Quietly Destroy Its Own Baseline
That is the problem in one sentence.
If you let a system change the prompt that shapes its own future behavior, then a bad edit is not just a one-off bug. It becomes part of the baseline. It gets inherited by every later run.
That is why I stopped thinking about the gate as a feature. It is the blast door.
If the blast door is weak, the rest of the system becomes dangerous no matter how clever the optimizer looks.
What the Gate Actually Does in v0.3.0
By v0.3.0, the promotion gate had seven deterministic checks:
- sample floor
- effect size
- confidence threshold
- frozen sections
- edit distance
- drift detection
- Oracle Drift Guard
No LLM decides whether an edit gets promoted.
That was a deliberate line in the sand. If an LLM judges its own edits, the whole system starts grading its own homework. I wanted the final promotion boundary to be code, not vibes.
The Numbers That Made Me Trust It
The best numbers in v0.3.0 were not about improvement. They were about restraint.
- 0% false positives across the shipped field-test evidence
- 8/8 adversarial edits blocked
- 0 false negatives observed in adversarial validation
- no bad promotions in local synthetic, cloud synthetic, or Docker-backed flows
If a self-editing system is going to fail while it learns, this is how I want it to fail.
Conservatively. Verbosely. With receipts.
Oracle Drift Guard Ended Up Mattering More Than I Expected
This was one of the quieter v0.3.0 features, but I think it is one of the most important.
Oracle Drift Guard exists for a nasty kind of failure where the optimizer, the scorer, and the benchmark all share the same wrong idea of success.
That is more dangerous than an obvious bug because the whole system can look internally consistent while still drifting in the wrong direction.
The prompt starts optimizing for the wrong thing. The scorer rewards it. The corpus keeps reinforcing it. Everything lines up and says "looks good."
That is not learning. That is coordinated self-deception.
I think more AI systems need explicit defenses against that class of failure.
The Safety Story Was Bigger Than Statistics
People hear "gate" and think p-values. Fair enough. The confidence threshold matters.
But the real safety story in v0.3.0 was wider than that.
materialize_candidate_prompt() replaced raw str.replace(), which means missing old_text fails loudly instead of pretending an edit happened.
Frozen sections draw a hard line around prompt content the analyzer should never touch automatically.
Edit distance limits keep the system from calling a wholesale rewrite "one edit."
Rollback with lineage means a promoted change is not just stored. It is traceable and reversible.
That is what I mean when I say the gate became the product boundary. It is not one check. It is a safety stack.
The Optimizer Still Didn't Win, and That Made the Gate More Valuable
The optimizer still did not produce a promotable edit in v0.3.0.
The best cloud candidate from mistralai/mistral-small-3.2-24b-instruct showed weak positive movement, but nowhere near enough confidence.
The local 4B analyzer mostly produced null edits.
The first separated-role run produced zero proposals.
That can sound disappointing if your only success metric is promotion.
It looks very different if your first success metric is "do not let the system quietly corrupt its own baseline while the optimizer is still immature."
On that metric, the gate absolutely earned its keep.
The Surprise Was How Hard Safe Failure Actually Is
I used to think safety would be the boring part. Add thresholds. Add checks. Move on.
That was naive.
The hard part was not inventing names like confidence or drift. The hard part was making sure the system measured the right thing, at the right point in the flow, against the right baseline prompt, with enough attached evidence that a rejection meant something concrete.
Once a system edits itself, every shortcut gets more expensive.
That was one of the main lessons of v0.3.0.
What I Learned
The optimizer is replaceable. The gate is not. If I swap in a stronger analyzer tomorrow, the gate still has to be the thing I trust more than the model.
I also learned that statistical rigor is necessary but not sufficient. P-values and effect sizes matter, but so do provenance, materialization correctness, adversarial testing, rollback, and drift checks. Safety is a stack, not a number.
I also came away thinking the gate has become a better diagnostic tool than I expected. It is no longer just a blocker. In v0.3.0, it is separating different failure classes: null edits, locally plausible net-zero edits, and weak-positive but underpowered edits. That makes the gate more useful than a binary "promote/reject" label suggests.
And I learned the hardest thing emotionally: a system that refuses to move can still be healthy. Sometimes a long string of rejections is not proof that the project is broken. It is proof that the project still has a boundary.
Why Developers Should Care
If you are building any loop where an AI system can revise its own behavior, ask this before you ask anything else:
what stops the system from rewarding its own mistakes?
If the answer is basically "another LLM will tell us if it looks good," you do not have a safety story yet.
I started this project thinking the smartest part would matter most.
v0.3.0 taught me the opposite. The part that says no is the reason the rest of the system gets to keep trying.
If you were building a self-modifying system, how conservative would you make the safety boundary before the product became useless? Would you rather ship a system that almost never promotes, or one that moves faster but occasionally corrupts its own baseline? Where do you draw that line?
Top comments (3)
With no promotable edit produced in
v0.3.0, the 0% false-positive rate is entailed rather than measured: a gate that rejects unconditionally scores the same 0%, so that row cannot separate your seven-check stack fromreturn False. The 8/8 adversarial block is the number actually carrying evidence, and its width is worth stating next to it, because eight trials with no misses put the 95% upper bound on the miss rate near 31% (1 - 0.05^(1/8)). Driving that bound under 5% takes 59 consecutive blocks, which is cheap to generate against a gate that is deterministic by design and would turn the strongest claim in the post into the best-supported one. Your own framing is what makes the width matter rather than being pedantic: a miss is inherited by every later run, so the interval on the miss rate is the interval on the blast door.Vinh, this is exactly right, and I appreciate you writing it out cleanly instead of letting the 0% FP number sit there looking like stronger evidence than it is.
You're correct that with no promotions across any v0.3.0 run, the observed 0% FP is logically identical to what return False would produce. The adversarial 8/8 does the actual lifting, and 8 trials with the 95% upper bound at ~31% is honest but not the number I'd want to bet infrastructure on.
Your math on 59 consecutive blocks is the right target. Since the gate stack is deterministic — frozen section checks, edit distance, drift, oracle drift — generating 59 adversarial edits that exercise distinct failure modes is straightforward. The bottleneck isn't compute; it's making sure the 59 aren't 59 variations on one attack surface.
I think the honest edit to the article (or a follow-up) is to split the claim into two rows: "field-test observed FP (0%, n=field iterations)" vs. "adversarial FP (0%, n=8, 95% CI: 0-31%)." The 59-block run would collapse that second interval. I'll queue it up — 59 adversarial seeds is a weekend project, and the interval shrinkage would make the "gate is the product" thesis much harder to argue with.
The bottleneck you name at the end is not a side worry, it changes the number itself.
1 - 0.05^(1/n)counts independent trials, and with a deterministic gate two seeds on one attack surface are perfectly correlated, so the effective n is the count of distinct surfaces rather than the count of runs. 59 seeds spread over 8 surfaces lands you back on 31%, which is the bound you already have.It is the same shape as the pseudoreplication in the field-test thread, and I think that means both rows want the same fix: report n as distinct surfaces, and put the target of 59 on that column. Repeats still pay for something, just not for the interval, they buy the claim that the gate is deterministic in the way you say it is, and one repeat per surface covers that.