The Health Check Failed. The Deploy Went Live Anyway.
A postmortem from our own test run, and why two correct parts added up to a bug
The test was supposed to be the easy one. Push a commit whose health route returns a 500, and watch the platform refuse it while the old version keeps serving. This time the platform under test was our own.
On 2026–09–27, during a local run of Levelrail against a real app, the readiness check failed after 60 seconds with a ReadinessFailed condition. So far, correct. Then the ingress sent traffic to the new, failing container anyway. The previous container kept running, with nobody routed to it. And the platform recorded the deploy attempt as succeeded.
The system knew the container was unhealthy, and it sent users there.
Where the bug lived
Nobody had written a wrong line. Two pieces each did something reasonable.
The application controller tracked readiness correctly and wrote a Ready condition. The ingress, in a different part of the system, chose its target with a simpler rule: the container named after the desired image, as soon as it ran. Running, but not ready.
The readiness verdict existed, and the router never read it. The deploy attempt record had the same problem in miniature: it got written when the container started, so it never changed to failed when readiness failed later.
Two components that each hold half of a safety guarantee do not add up to a guarantee. Someone has to own the whole path from “probe failed” to “no traffic”.
What we changed
One pull request, two changes:
- The routing step now gates on the application controller’s stored Ready condition. If the new release is not ready, traffic falls back to the previous release that is still running, instead of cutting over early.
- A failed readiness probe now flips the deploy attempt to
failed, so the history shows what happened.
We added tests for both. One fails readiness and checks the routing choice. One checks that a fresh deploy with failing readiness records the failure. One, at the storage layer, checks that the newest succeeded attempt flips to failed.
The neighbors of the bug
The same run turned up nearby problems, which is the usual pattern once you start trying to break something on purpose.
Get GDSKS’s stories in your inbox
Join Medium for free to get updates from this writer.
A rollback to a mistyped image tag returned exit code 0. The container could not start, the ingress dropped the route, and the app went offline while the old container was still running. The CLI now refuses a tag that is not present locally, names the tags it does know, and points at the right command.
A www redirect also disappeared after an env import, because every app save deleted and re-inserted the domain rows that redirect and certificate settings hang off. We fixed that one too.
None of these show up in a feature list. They showed up because we ran the platform against a real app and tried to make it fail.
What it checks now
Beyond routing on readiness, the controller verifies that the running container’s image matches the pinned digest the deploy intended. If it does not, the app reports Ready: False with ImageDigestMismatch and the previous release keeps serving. It never reports Deployed while the running content differs from what you shipped.
You can check an outcome from a script. The exit code of this command tells you the result:
levelrail-cli apps deploys wait webZero means healthy. Seven means failed, canceled, superseded or blocked. Six means the wait timed out.
What is still open
One item from that run was still open when we wrote the original post: confirming end to end that apps deploys wait reports a readiness failure correctly now that the attempt flips to failed. We expect it works as a side effect of the fix, and we have not verified it on its own.
Levelrail is young. The project labels single-server deployments production ready, and newer features are still beta. If you try it, run it beside your current setup for a full business cycle with real traffic. That is what surfaces the boring failures.
The code and the issue tracker are at https://github.com/glincker/levelrail
Which part of your own deploy path decides what counts as “ready”, and does the router actually read it?
Originally published at https://levelrail.com/blog/health-check-passes-dead-container/
