This is article 3 in a series about building PlannerCritic, an open-source engine where one LLM writes a plan and a second LLM reviews it. Article 1 covers the 157-goal field test. Article 2 is about the critic severity bug. This one is about the most uncomfortable thing the field test revealed: the planner has a structural problem that no model upgrade fixes.
132 concrete blockers across 63 strict goals. Three defect families. I tried gpt-4o. Same pattern. The fix is deterministic validation, not more parameters.
The Pattern
By the 10th strict goal, I noticed it. By the 50th, I could predict the failure before the critic printed it. By the 100th, I stopped being surprised and started being annoyed.
The planner kept making the same three mistakes. Not occasionally. Not randomly. Every single strict goal that failed did so because of one of three defect families.
Unverified dependencies (57 blockers): The plan declares a precondition that no earlier task establishes. The planner knows what should be true. It doesn't arrange the steps to make it true.
Real example — from ai-03-model-serving-migration:
[BLOCKER] unverified_dependencies — task=cutover_traffic_100
"Cutover traffic from SageMaker to vLLM (100%) verification of
latency SLO is dependent on prior traffic cutover stages being
established but lacks clear confirmation of stability before
proceeding."
The plan says "cutover 100% of traffic" but no earlier task verifies that the 10% and 50% stages were stable.
Unsafe sequencing (46 blockers): Tasks are ordered before their hard prerequisites. The cutover runs before the verification. The backup completes after the migration.
Real example — from ai-02-embedding-index-migration:
[BLOCKER] unsafe_sequencing — task=backfill_vectors
"Backfill operation cannot proceed until the index is verified
for quality; it is ordered incorrectly in the sequence."
The plan puts the backfill before the quality check that should gate it.
Weak rollback (18 blockers): High-blast-radius steps lack rollback. The planner includes rollback on routine steps but omits it on cutover, teardown, and failback.
Real example — from db-10-multi-tenant-split:
[BLOCKER] weak_rollback — task=dual_write_setup
"The dual-write setup task's rollback only switches to
single-write mode without addressing potential inconsistencies
during the transition."
The rollback exists but it is not credible. Switching back to single-write does not undo the data inconsistencies that dual-write may have introduced.
I Tried a Bigger Model
The natural instinct: use gpt-4o. I kept waiting for it to bail me out.
I tested it both ways — gpt-4o planner with mini critic, and gpt-4o for both roles.
Same defect pattern. Better prose. Same structural mistakes.
Unverified dependencies. Unsafe sequencing. Weak rollback.
That was the moment I stopped blaming the model size. The planner wasn't dumb. That was the annoying part. It was plausible. It knew the right steps. It just couldn't close the dependency graph or enforce the ordering.
I did not have a smaller-model problem. I had a planning-structure problem.
Why the Loop Can't Fix It
The revision loop is designed to converge. The critic reports blockers. The planner revises.
But the planner tends to fix one blocker and introduce another. It reshuffles task order without closing the dependency gap. It adds rollback to the wrong task.
After 2 revisions — the median across 33 strict goals — the planner stops making meaningful changes. The convergence detector fires. The engine escalates.
The loop works as designed. The planner is the bottleneck. I kept expecting the revision loop to converge. It didn't, because the planner can't fix a structural problem by rewriting the prose.
Worth being clear about what failed here. The critic reliably found the same blockers across revisions — the failure was the planner's inability to structurally repair, not the critic's judgment. That's a different defect from the one in Article 2, where the critic's severity calibration was the problem.
The Fix Is Not More Parameters
The highest-leverage fix is a precondition closer: a deterministic linter that runs after the planner produces a draft and verifies that every precondition is actually established by an earlier task. If a task says "requires replica_verified," there must be a prior task that produces it.
This single pass would eliminate 64 of 132 blockers (48%) without asking the LLM to get smarter.
The remaining blockers — unsafe sequencing and weak rollback — need either better prompt engineering, additional deterministic validation, or genuinely better reasoning about ordering and risk.
What the Research Says
This is not just my observation. The academic literature converges on the same finding.
The "Why Reasoning Fails to Plan" paper (arXiv 2601.22311) shows that LLM agents select actions based on local evaluation without considering future consequences. In knowledge graph traversals, single-step greedy policies select myopic traps more than 55% of the time. The authors prove step-wise reasoning is provably insufficient for long-horizon planning.
The PlanGenLLMs survey (arXiv 2502.11221) evaluates LLM planning across four criteria — completeness, executability, optimality, representation. LLMs consistently fail at ensuring plans are executable: preconditions not met, steps out of order.
The field is converging on hybrid approaches — LLM plus deterministic validation plus classical planning techniques. Not LLM alone.
What You Should Take Away
If you are building agents that plan over multiple steps:
- Test your planner on a real corpus. The 157-goal run revealed a pattern that 3 demo goals would never show.
- Measure defect types, not just pass/fail. If all your failures are in one family, you have a specific gap.
- Do not assume a bigger model will fix structural problems. The planning gap is a reasoning limitation, not a language ability limitation.
- Add deterministic validation before trusting the LLM to self-correct. The revision loop is useful, but it is not a substitute for structural checks.
Article 3 of 5 in the PlannerCritic series.
Series: Article 1: "I Ran 157 Agent Plans Against a Real LLM" · Article 2: "I Told My LLM Critic to Be Adversarial" · Article 4: "The Field Test Found 10 Issues" · Article 5: "I Tried to Prompt-Inject My Own Engine"
Links:
- Repo: github.com/deghosal-2026/planner-critic-engine
- README: README.md
-
PyPI:
pip install planner-critic - Field Test Results: 157 goals across 35 domains
- Field Test Plan: 156-goal corpus
- Architecture: architecture-v0.1.0.md
- User Guide: quickstart.md
- CHANGELOG: CHANGELOG.md
Top comments (28)
The three families line up exactly with the gate-coverage seams from the part 2 discussion - and they confirm the worry was justified.
unverified_dependencies is the precondition family (does any earlier step establish what this task requires), unsafe_sequencing is the ordering family. weak_rollback is the sneakiest, and your dual-write example is why: the rollback exists - a naive "rollback present" gate passes it. What is missing is credibility. Switching back to single-write does not undo the consistency window dual-write opened.
That is the distinction that decides whether a gate shifts a defect family or just renames it: presence vs evidence. A gate that counts rollback steps is gameable - the planner will add a rollback step to satisfy it (you watched it add rollback to the wrong task). A gate that requires the rollback to be a reachable sibling with its own verification - undo step, then verify-state-after-undo - is much harder to game, because the planner has to construct an actual recovery path instead of naming one.
The precondition closer's 48% supports the deeper point: cross-task properties are checkable deterministically because they are evaluated on the whole plan graph, and that is precisely what greedy token-by-token generation cannot hold in view - the same reason gpt-4o reproduced the pattern, and the same myopia one level up when the revision loop "fixes one blocker and introduces another" (local repair, no global view).
One suggestion for the loop side: since repair is myopic, consider making the convergence criterion family-based rather than count-based - stop after the first revision whose blocker-family histogram is unchanged, instead of waiting for zero blockers. It is cheaper, and it distinguishes "reshuffling" from "repairing".
Thank you — this is the most productive single comment across all three articles. You identified two gaps in one message, and both are now tracked.
Rollback credibility is the single biggest determinism gap in the engine. The rollback_present gate checks for existence, not for credibility. You are right that a gate requiring undo-then-verify would be much harder to game. We structured a field test to measure this gap across 8 domains and 3 credibility patterns (unreachable, self-dependent, inconsistent-state) — github.com/deghosal-2026/planner-c.... The results will define the acceptance criteria for a new deterministic rollback_credibility gate.
The family-based convergence criterion is the kind of cheap high-leverage signal the loop is missing today. F-06 catches identical text, oscillation detection catches structural cycling, but neither catches a frozen family distribution. We opened a benchmark to measure how many LLM revisions this signal would save across the 85+ strict-goal traces — github.com/deghosal-2026/planner-c.... If savings cross 20%, it becomes a fifth termination signal in the loop controller. Your observation that it distinguishes reshuffling from repairing is the core insight.
Two small notes now that both are concrete, one for each.
Credibility patterns: unreachable, self-dependent and inconsistent-state all describe the rollback path's own integrity. The dual-write case adds a fourth: post-consumed recovery - the undo restores the pre-write state correctly, but a sibling task already consumed that state inside the window between write and rollback. The path is internally consistent and still wrong at t+1. A cheap addition to the acceptance criteria: "does the restored state satisfy the constraints of every later task" - it turns a single-step check into a graph query, the same shape as the precondition closer, so it should stay deterministic.
Convergence signal: watch the period-2 case. F-06 catches identical text and oscillation catches structural cycling, but if the family histogram itself oscillates A->B->A->B, a last-revision comparison keeps seeing change and the signal never fires. Compare against a bounded window (same histogram as two revisions ago) and the reshuffling-vs-repairing distinction survives without adding a new false-negative class.
Both are one-line additions to the harness, which is the right size for the "presence is cheap, evidence is graph-shaped" budget.
Two excellent notes, both filed. Post-consumed recovery went into Rollback credibility gate as the fourth credibility defect alongside unreachable, self-dependent, and inconsistent-state — including your restored-state-must-satisfy-later-tasks graph check, implemented deterministically like the precondition closer. And you're right about the period-2 blind spot: F-06 compares consecutive text and structural oscillation compares signatures, so a family histogram cycling A-B-A-B slips past everything. Tracked as Family-histogram oscillation detection, with a bounded lag-window comparison and a retrospective benchmark over stored traces before the signal ships enabled. Both sized as one-mechanism changes, per your budget argument. Thanks again.
Both sized as one-mechanism changes — that's the right instinct, it keeps the review surface small and each gate independently testable. The bounded lag-window is the right call for #217, with one thing to watch: if the planner legitimately produces a bimodal family distribution (two alternating strategies across a task, e.g. interleaved analysis/code families), a fixed window will fire on steady-state alternation, not just oscillation. Running the retrospective benchmark over stored traces before enabling should surface exactly that case. Curious whether the oscillation class shows up cleanly once legitimate alternation is filtered.
Thank you — the bimodal worry was exactly right, and it shipped as a guard before I replied because it was too good a catch to leave in a thread. The oscillation detector now carries a progress rule: if the newest revision reduced total blocker mass against its predecessor, a lag-p repeat is treated as legitimate alternation on an improving trajectory and never fires; flat-mass repeats still fire, fail-safe, as escalation-only (#229, companion to #217). It landed with a synthetic bimodal fixture and a self-test over the stored traces.
To your actual question — whether the oscillation class shows up cleanly once legitimate alternation is filtered — the stored traces show no false fires after the guard, though no genuine period-2 cycling has surfaced in recorded data yet either, so the detector today is benchmark-proven rather than field-proven. If a live run ever trips it, the surrounding trace should be well worth reading.
The deterministic precondition check sounds like the highest-leverage layer. I’d keep it separate from the critic’s prose score and make the linter emit a dependency graph plus a failing task or edge. Then the planner can revise against a concrete missing edge, while rollback checks can remain a distinct validation pass.
This is how the linter already works, happily. The deterministic layer is fully separate from the critic's prose score: each gate emits a typed finding carrying task_id, reason_code, message, and a concrete suggested_fix (gates/init.py), and plancritic can render the plan as a Mermaid dependency graph (viz/graph.py), so a failure reads as a specific task or edge rather than commentary. The planner revises against exactly those findings. All of it shipped in v0.2.0. Thanks for articulating the separation so crisply — it is the load-bearing design choice.
That separation also gives you a clean replay boundary. I’d make each typed finding stable and machine-actionable—task/edge ID, reason code, observed state, suggested fix, and evidence references—then let the planner revise against the finding while the critic remains advisory. One useful guard is to version the finding schema and retain the precondition snapshot, so a later rerun can tell whether the plan changed because the task changed or because the gate logic changed.
This lands well, and thank you for pushing it two levels down from architecture to record format. Partial credit today: findings already carry an id, target task id, plan version, machine-readable reason code, and suggested fix. What does not exist yet is precisely your list — edge-level targeting so ordering defects name the producer-consumer pair instead of one task, an observed-state field, evidence references back to the precondition facts or edges that fired, an independent finding-schema version, and the retained precondition snapshot that lets a rerun attribute a verdict change to the task changing versus the gate logic changing.
All of it is now filed as the machine-actionable finding contract (#243), including your replay-boundary test as an acceptance criterion: same plan plus bumped gate versions must be attributable to gate-logic change, changed plan plus same gates to plan change. The critic-stays-advisory split you describe is already how the loop consumes these findings, so this is schema work, not a redesign.
The post makes a useful distinction between a feature working once and a system being dependable. I’d add an explicit failure-mode checklist so the next contributor can see which assumptions are intentional and which ones still need evidence.
Thank you — working once versus being dependable is exactly the distinction the field tests were supposed to buy us. Fair point on the checklist: the material exists but is scattered across the Known Gaps section of the README, the design decision log (design-decisions.md), and the field-test reports — no single place tells a contributor which assumptions are intentional versus which still need evidence. Filed as a docs issue: Failure-mode checklist — a living register where every known failure mode gets an explicit class, a rationale or evidence link, and a tracking milestone. Thanks for the suggestion; it should have existed from day one.
The biggest takeaway here is that better models don’t automatically mean better planning. If the underlying structure isn’t validated, the same dependency, sequencing, and rollback mistakes will keep coming back. Deterministic validation feels like a much more reliable solution than simply throwing a bigger model at the problem.
Thank you for reading closely. You put your finger on the part I would most want a skimming reader not to miss — the gpt-4o-for-both-roles test is what turns "add deterministic validation" from advice into a finding. Same model, same defect families, just more fluent wrong plans. Glad the post held up as a takeaway.
The "fixes one blocker and introduces another" pattern is exactly what I see in retry loops on scheduled agent jobs - each revision patches the symptom the critic named, and by revision 3 the plan has drifted from the original goal. Curious whether you tried letting the critic propose the repair itself, or if that would just move the structural problem one level up.
We run the same loop with the model proposing its own repairs, and yes — the structural problem does move up one level, just not in a helpful direction at first: the revision loop gets good at satisfying the critic. Each repair targets the vocabulary of the finding ("add a rollback step", "reference the precondition"), the plan text drifts, and by round two or three the plan is internally consistent and critic-pleasing while the goal it was built for has quietly receded. What you are describing — patches the named symptom, drifts from the original goal — is the loop optimizing the wrong objective, and it gets worse the more the critic talks.
The fix that held for us is to make the level above the critic deterministic: keep the original goal as an external artifact the loop re-checks against (not the plan's own prose), and key the convergence signal on the plan graph rather than the critic's verdict — stop when the blocker-family histogram stops changing, not when the critic is quiet. That distinguishes reshuffling from repairing, and it turns "drifted from the original goal" into a measurable state instead of a judgment call. So letting the critic propose repairs is not the dead end; it just moves the problem to the level above — which is exactly where the deterministic checks have to live.
Sorry for the late reply — and thanks to pm25coder for covering the middle ground meanwhile. Direct answer: no, the critic never proposes repairs here, by design. When the loop finds blockers, the planner role revises against the findings list, and two repairs skip models entirely — deterministic topological reordering and a precondition closer that inserts the missing establishment tasks. The critic audits; it does not author. Your instinct about the problem moving up a level matches what the revision-loop failure mode actually looks like, which is why the convergence signal now watches the blocker-family histogram for reshuffling-instead-of-repairing — cycling detection with a legitimate-alternation guard (#217, #229) — rather than trusting a quiet critic. The retry-drift pattern you describe on scheduled jobs is the same failure wearing a cron hat; the histogram signal should port directly if you want to try it.
The useful test here is whether the plan has a balance sheet. If a task consumes replica_verified or stable_50pct, I want to see where that asset was created earlier. Otherwise the LLM is just writing a plausible operations memo with missing entries.
"48% of blockers eliminated by a deterministic linter, without asking the LLM to get smarter" is the line that reframes the whole problem. Half the failure mode wasn't a reasoning gap at all, it was a graph closure check any compiler-adjacent tool could do, and nobody needed gpt-4o for that.
The gpt-4o test is what makes this convincing though, not the linter itself. Running the same architecture with a stronger model and getting identical defect families is the actual proof that this was structural, not capability. Without that test, "add deterministic validation" reads
as an assumption; with it, it reads as a finding.
The rollback example is the one that'll stick with me, a rollback that exists but doesn't undo the actual damage is worse than no rollback in a way, because it looks handled in the plan review and isn't.
Thank you — you stated the epistemics better than the article did: without the gpt-4o-for-both-roles control, add-deterministic-validation is just an assumption; with it, it's a finding. And your rollback observation is exactly where the next work goes — a rollback that exists but does not undo looks handled in review and isn't. That gap is now tracked in Rollback credibility gate, while the precondition closer itself shipped in v0.2.0. Appreciate the close read.
the gpt-4o-for-both-roles test is the whole thing and people are going to skim right past it. same model both roles, same defects. that's your proof.
we ran the same idea at way bigger scale. terminal bench 2.1, 89 tasks, 5 runs each, 445 trials, nothing dropped. only variable changed was the harness. 78.9 to 85.4 on the same model.
structure moved it further than a model generation would have.
your 48 percent linter number lines up with what we see too. most of what looks like reasoning failure is just context that got too big. hand a child agent a small slice with the preconditions already true and the model stops screwing it up. probably why all three of your defect families come out dependency shaped.
harness is MIT and every trial log is public if you want to run your taxonomy against our failures: github.com/harbor-framework/termin...
i co-founded backboard, so, grain of salt.
Thank you — and yes, let's run it. Your setup isolates the same variable we did, structure with the model held fixed, at five times the scale, and the 78.9 to 85.4 move alongside our 48 percent linter number is a nice convergent result from opposite directions. I would genuinely like to point our defect taxonomy at your public trial logs and see how the three families partition those 445 trials — if dependency-shaped failures dominate there too, that is strong cross-harness evidence. I'll dig through the logs and follow up with results. Engine side lives at planner-critic-engine, v0.2.0 tagged. Grain of salt acknowledged, receipts noted.
The “precondition closer” feels like the key architectural shift here.
Once a planner is allowed to generate the plan freely, I’m not sure the LLM should also be responsible for proving that the plan is structurally valid.
It makes me think of planning as two separate contracts:
LLM: propose what should happen
Validator: prove that it can happen in this order
And the same idea seems applicable to rollback: instead of asking the planner to “add a rollback,” the validator could require every high-blast-radius action to declare what state it restores and what evidence proves that restoration is possible.
That separation seems much more robust than another revision loop.
Your two-contract framing is the right name for where the engine has been heading — six deterministic gates already prove structural validity while the LLM critic stays advisory on everything probabilistic, which is why the precondition closer could eliminate blockers without making any model smarter. But you identified the piece where the split still leaks: rollback. Today the planner writes rollback prose, and the credibility gate infers intent from surrounding structure. Your version is stronger — require every high-blast-radius action to declare what state it restores and what evidence proves restoration possible.
Filed as typed rollback restoration contracts (#245): declarative restored-state and restoration-evidence fields on RollbackStep, consumed by the gate when present and derived as today when absent, with a migration path that promotes declaration from advisory to required once corpus coverage earns it. Thank you — the "propose versus prove" sentence went into the issue nearly verbatim.
Glad the framing was useful. The “propose vs prove” split feels especially powerful because it gives you a clean boundary for where probabilistic reasoning ends and deterministic guarantees begin.
I’m also curious to see how the typed rollback contract performs against the inconsistent-state cases you mentioned. That seems like the harder test of whether the validator is actually proving recoverability rather than just checking that a rollback exists.
The clustering into three repeatable families is the useful part. I hit the same wall: a stronger model just made the wrong plans more fluent, not more correct, and only a deterministic check between plan and execution moved the failure rate. Did the three families stay stable across task types?
They stayed remarkably stable — I re-ran the domain cross-tabulation over the stored traces tonight to answer this properly rather than from memory. Current corpus: 269 blocker findings across 37 domains and 7 reason codes. The three families are near-universal: unsafe_sequencing accounts for 99 blockers across 30 of 37 domains, unverified_dependencies 95 blockers across 30 of 37, weak_rollback 57 blockers across 26 of 37. Together that is about 93 percent of all blockers; the remainder are rare, concentrated gate-level codes like dependency_cycle (one instance) and unsafe_ordering (two).
Two honest caveats: domain here is the goal-id prefix, so it is a proxy for task type rather than a controlled variable, and per-domain counts vary widely. But nothing suggests family membership shifts with task type — which matches your experience that a stronger model made wrong plans more fluent rather than different-shaped. Happy to share the raw cross-tab if useful.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.