Docker restart Costs You a Gap. Here Is the Count | Levelrail

152 requests around a rolling restart, 111 failures around a stop and start

Cover image: restarting a Docker app without downtime

Sometimes an app just needs a fresh process. A leaked connection, a stuck worker, an environment variable you changed. The habit is docker restart, and it works like this: kill the container first, start it second. Every request in between fails.

I wanted the actual number, so I ran a request loop against a demo app, restarted it two different ways, and counted. The app ran on Levelrail, a self-hosted deployment platform I am building. All output below is from a real run.

The loop

One request every 200 ms, printing the status code, running for the whole experiment:

while true; do curl -s -o /dev/null -w '%{http_code} ' shop.localhost:8081/; sleep 0.2; done

The app has a readiness check, which is the gate that makes a restart safe: traffic only moves to the new container once /healthz answers with a success status. I only measured restarts on an app that has one, so I do not claim the same without it.

Restart, the rolling way

levelrail-cli apps restart shop

The app overview with the more actions menu open and Restart as the first item

The command returns quickly, but the restart itself is asynchronous. I sampled docker ps once a second while it ran:

14:47:24 shop-ff866c55=Up 27 seconds
14:47:25 shop-057b1fe5=Up 1 second shop-ff866c55=Up 28 seconds
14:47:36 shop-057b1fe5=Up 12 seconds shop-ff866c55=Up 40 seconds

A new container (057b1fe5) appeared one second after the command, while the old one (ff866c55) stayed up. The pattern is: new container, readiness gate, switch, old one held. The switch happened about 10 seconds after the new container started.

Between 14:47:20 and 14:47:50 the loop sent 152 requests, and all 152 returned 200. Across every restart I did up to 14:56:00, including the failing ones below, the loop sent 2,581 requests with 0 non-200 responses.

A restart is not a deploy, so it does not appear in the deploys list. Look in the timeline:

levelrail-cli apps timeline shop

The same job with stop and start

levelrail-cli apps stop shop
levelrail-cli apps start shop

stop ran at 14:56:55 and start at 14:57:36. Stop took roughly 20 seconds to take effect. Between 14:57:15 and 14:57:47, 111 requests failed. That is about 32 seconds, mainly 404 No app is serving this address plus a handful of timeouts.

Get GDSKS’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Stop then start is not a zero downtime restart. On a running app, use apps restart.

What if the new container is bad?

Zero downtime is only worth something if a bad restart does not take you down. I set an environment variable that makes the demo app answer 500 on /healthz, then restarted:

levelrail-cli env set shop BAD=1
levelrail-cli apps restart shop
levelrail-cli apps status shop

TYPE               STATUS   REASON           MESSAGE
Ready              False    ReadinessFailed  readiness probe: never became ready (context deadline exceeded), last attempt: GET http://127.0.0.1:52838/healthz returned 500, expected 200-299

The new container never passed readiness, so traffic never moved. Between 14:48:00 and 14:50:00 the loop sent 593 requests, all 200, all from the old container. The restart fixed nothing and broke nothing.

What if it passes readiness and then dies?

The harder case. I made the demo app exit with code 1 ten seconds after start (CRASH_AFTER=10) and restarted. About 2 minutes later:

TYPE               STATUS   REASON                    MESSAGE
Ready              False    ExitedDuringStableWindow  Container exited with code 1 after 10s, during its stable window after becoming ready

pending changes: env CRASH_AFTER (run: levelrail-cli apps apply shop)

The platform started the candidate again and again, and docker ps showed it flipping between Up and Exited (1). It never took over traffic. The previous container stayed up, and the loop sent 816 requests between 14:51:30 and 14:54:15 with 0 non-200.

apps status gives the reason and the exit code, and apps logs --tail 12 shows the last lines the process wrote. One caution: do not rely on apps deploys wait here. It reported "never passed readiness", which is the wrong cause for a crash.

Rough edges I would want to know about

• apps restart returns exit code 0 as soon as it accepts the restart, even if the restart then fails readiness or crashes. Check apps status after it.
• apps deploys wait is not meaningful for a restart. Right after a restart it answered healthy instantly about the old deploy.
• After the failed restarts above, apps deploys list showed my first deploy (v1) as failed, and it stayed failed after the app recovered. Treat deploy history as unreliable after a failed restart.
• Restarting a stopped app does nothing: the command exits 0, the timeline says Restarted, and the app stays Suspended. Use apps start.
• A restart silently applies pending env changes, and apps status kept printing pending changes even after that.
• The dashboard Restart button has no confirm step and no completion toast.

All of these are tracked for the next release. A restart also leaves 3 to 4 containers around for a one replica app for up to the 5 minute hold window.

The short version

Give the app a readiness check, then use apps restart. In my run, 152 requests around a healthy restart all returned 200, and 0 requests failed across 2,581 in total. Stop then start cost 111. Poll apps status and your request loop to see how it went, not the deploys list.

The project is open source: https://github.com/glincker/levelrail

What do you do today to check that a restart did not drop a request?

Originally published at https://levelrail.com/blog/restart-docker-app-without-downtime/