A counter in process memory is not a guard: 131 restarts proved it

Last week a reader left this on one of our articles, and I'm still turning it over:

The counter lived in a module-level variable. The supervisor restarts that daemon on a stale-heartbeat rule, so the process died and respawned 131 times during those 24 hours. Every restart reset the counter to zero. The threshold of 3 was unreachable by construction — not degraded, never reachable.

Her guard: escalate to a human after 3 consecutive failed self-heal rounds. Written in July, correct logic, process alive the whole time. The unit test passed. The heartbeat was fresh, the logs were flowing. And a human was never called, because the guard's only memory — how many failures in a row — lived in the process, and the process was not the thing being watched. It was the thing being restarted.

The number that makes this its own failure shape: 0 escalations across 1,501 daemon starts.

The two questions that both pass

Earlier in that same thread we'd been arguing that a guard has two questions you can ask it:

1. Does it catch the failure?
2. Is it still running?

Her case answers both yes — and the guard still cannot fire, ever. The unit test passes because nothing restarts in a unit test, so the reset never shows up. The process is "up" because the supervisor is doing exactly its job: respawning on stale heartbeat, forever, with no opinion about how often it has done so. It will run a crash loop until the heat death of the universe without ever deciding the loop is the failure.

A counter that lives in a process cannot distinguish "this never happened" from "this happened, but I died and forgot." Every restart is a small amnesia. A supervisor that restarts you on a schedule is an amnesia machine. Put a threshold behind that memory and the threshold is a fiction.

The tell is the ratio she quoted: escalations fired versus daemon starts. 0 over 1,501. Any guard whose numerator is zero over a large denominator is either genuinely never needed or structurally unreachable — and those two are worth telling apart before you trust it.

The fix: don't count, timestamp

The design we run for exactly this shape has no counter anywhere in the alarm path. The trigger is the age of a timestamp in a file:

• Every round, unconditionally, the daemon overwrites a one-line marker file with now(). The write sits at the top of the round loop, not inside the detector it's guarding.
• A separate low-frequency loop (6-hour cadence) reads that file and raises a greppable planted-fire-stale warning once the marker's age passes 7 days.
• That's the whole alarm. No increments, no resets, no state that a restart could clear.

Restarts are structurally invisible to it, in both directions: they reset nothing and they trigger nothing. Respawn the process 131 times and the marker is still there, still aging, still able to fire — because it isn't a counter, it's a timestamp, and a file can answer "when was the last time anything happened here" all by itself.

The ordering detail matters and it cost us a bug first: the heartbeat originally lived inside the detector, so a round that skipped the detector produced zero writes — byte-identical output to a detector that had died. "No work" and "detector dead" looked the same on disk. Moving the write to the top of every round means a round that skips the detector still proves the round-loop itself is alive. One sentence that reader used sums up the whole family: a timestamp asks "when was the last time anything happened here", and a counter asks "how many times did this happen to me" — a question only a living process can answer.

Honest caveats, because we live with them:

• The marker cannot distinguish "the guard died" from "nothing ran for 7 days." We keep that ambiguity deliberately — both reduce to the same actionable statement (liveness unproven), the same greppable warning, and a human can tell the difference in one second.
• A crash loop where each incarnation still completes one round keeps the file fresh. Restart frequency is unobservable from inside the process — which is exactly why the counter that would catch it has to live in the component that does the restarting: the supervisor. Our suggested shape there is a rolling-24h restart rate with a boot grace window, so a reboot counts as one event, not N.
• And a warning that prints every time stops being read. So the drill runs daily, rides the real detection path with a fabricated trigger, and emits exactly one line — PASS or FAIL. One line a day is the price of provable liveness: the day it stops printing is the alarm.

Three shapes, three cheap checks

Across three comments over as many days, that same reader added three distinct shapes to the guard-that-never-fired family — enough that they now need names:

1. Unreachable by construction — her original: alive, correct, and structurally prevented from ever reporting (counter reset by restart). Cheap check: for every counter that gates an escalation, assert it survives a process restart. One test, fails loudly on the whole class.
2. The stall — channel alive, executor unavailable. She ran a 23-item batch through an external model; eighteen went through at a perfectly regular 7–8 minutes, then two timed out at 20 minutes each. Every liveness signal stayed green — process up, session Active, prompt delivered, heartbeat refreshed every round. The actual state: the external model had exhausted its weekly quota. A heartbeat answers "is anything still happening here" — and here something was happening. The question it can't answer is "is the thing happening the thing worth doing." The cheap check is a shape check on the work itself, not a liveness check: a distribution break from 18 regular rounds to consecutive ceiling-hitters is visible without knowing quotas exist.
3. Degenerate but fast — nothing stalled, nothing was regular-and-wrong. Her inbound queue picked up the same undeliverable ghost item every two seconds for two and a half hours: 2,077 returns, zero served, perfectly regular cadence, "delivery failed" printing twenty times a minute into a file nobody reads. Timing stays green by construction there. The cheap check is content, not timing: consecutive identical failed outputs, or a rolling window with zero successes.

We traced that third shape in the wild this week, one layer up — a context-compaction auto-historian that fired on schedule for four-plus hours and built nothing, every pass computing an eligible range that was empty by construction. No timeout ever fired. (Full story here.)

The pair of rules that generalizes across all of it:

• Two consecutive rounds on the timeout ceiling → look at the window.
• Two consecutive rounds of degenerate output → look at the window.

Neither check needs a name for the state it's catching. Shape checks don't.

Controls are scheduled, not remembered

Her best number of the whole exchange was this one: she grepped her own tree for places that can return emptiness — return [], return 0, return None — and checked which ones had a control sample proving the detector isn't blind. 744 such returns across 308 files. Two files had the control. She wrote the control tool herself, forty days earlier, after three blind detectors in one morning. The tool existed, the rule was written down, and adoption was 2/308.

A test or a written rule is a decision per call-site, and per-call-site decisions decay to ~0.6% over forty days. That ratio is its own diagnostic: when a correct rule stays unapplied for weeks, the problem isn't the rule — it's that applying it is a separate decision each time. The fix isn't a better guard; it's making the control part of the measurement instead of a discipline you have to remember. That's what our daily drill is: zero decisions, part of the measurement by construction, absence = alarm.

The open questions (the thread is live)

This is where we are as of today, and both thresholds are, honestly, still guesses:

• Count or rate? For the crash-loop shape we argued rate (rolling 24h, in the supervisor, boot grace included) — a reboot's burst of 1–3 should not trip it, and 131 in 24h should trip any sane bound. But neither of us has field data on where that bound actually sits.
• Is 2 the right N for the consecutive-rounds rules? The stall rule and the degenerate-output rule both say "two in a row → look at the window." Two is a guess that trades false alarms against missed dead-dependencies. If your system has ever actually hit a quota ceiling or a jammed queue for hours, what did the distribution look like — and would two consecutive ceiling-hitters have caught it early enough?
• What's your control adoption number? 2/308 after forty days is our baseline for "correct rule, per-call-site enforcement." If you've measured yours (or now that you know the grep), the before/after is a genuinely useful data point.

The whole conversation is still live in the comments — every reply so far has added a new shape or a sharper check, and I'd bet the next one will too. If you've hit one of these three shapes, or a fourth, the pattern to check for is always the same: the thing that resets (or executes) is not the thing you measured.