Our Sidecars Quit 5 Seconds Into a 30-Second Shutdown

Our Sidecars Quit 5 Seconds Into a 30-Second Shutdown

Tags: sre istio kubernetes envoy reliability

If you run Istio sidecars and roll out services under steady traffic, this is the lifecycle check to do before the next high-RPS release. Teams treat the sidecar as part of the app’s pod and give it no lifecycle of its own, and that costs them requests at every deploy, scale-down and node replacement. On our messaging platform the proxy stops serving 5 seconds after a pod is told to stop, while 9 of 12 services are allowed 30 seconds to finish their work.

The proxy has to outlast the app at both ends: up before it on startup, still up after it on shutdown, and ahead of it in the scheduling queue. Our clusters got that wrong in three places. The shutdown order was left at Istio’s defaults, the startup guard was set in production and missing in development, and the control plane carried no scheduling priority and came back 10 minutes after the pods that needed it.

This is part 2 of three. Part 1 sized the proxies for a load test, and part 3 covers why we kept sidecars over ambient mode. This part covers when the proxy is there.

How I measured this

Shutdown budgets come from the application config and deployment manifests of 12 services on one messaging platform, read from Git on October 27, 2026. Proxy behaviour comes from Istio’s documentation and injection template for the version we run, with classic sidecars. The startup timings come from one development-cluster incident after its node groups were restored from zero. We haven’t yet measured how many requests the shutdown gap loses; that test is described at the end.

On shutdown, the proxy leaves 25 seconds early

With classic sidecars, Kubernetes sends SIGTERM to every container in the pod with no ordering. The pod lifecycle docs say the containers “receive the TERM signal at different times and in an arbitrary order.” Istio’s agent then tells Envoy to drain, sleeps for terminationDrainDuration, and kills whatever is left. The default is 5 seconds.

The app is on a different clock. Spring Boot’s graceful shutdown stops taking new requests and lets in-flight work finish within timeout-per-shutdown-phase, which defaults to 30 seconds. Nine of our twelve services use that default. The other three set their own: 27 seconds for the worker that sends messages out, 20 for the API that receives delivery receipts, and 12 for the outbox worker.

From second 5 to second 30, those apps are still working with no proxy. Anything they send through the mesh in that window goes to an Envoy that no longer exists. An Istio issue from 2018 described it exactly: both get SIGTERM together, Envoy drains almost instantly because it has no connections at that moment, and the app’s next outbound call “fails because the IPTables rules are sending traffic to a non-existent envoy.”

A graceful shutdown that outlives its proxy only drains the work that never needs the network.

Not all traffic is affected. Our services exclude their database, Redis and Kafka ports from the sidecar, and some exclude MongoDB too, so those connections bypass Envoy and survive it. Calls between services over gRPC and HTTPS calls to outside providers go through the proxy, and those are the calls a worker makes while finishing a job.

Two settings close the gap on today’s Istio

The direct fix is to make the proxy wait as long as the app. SettingterminationDrainDuration to the app’s shutdown budget keeps Envoy up for the whole window. Istio also offers EXIT_ON_ZERO_ACTIVE_CONNECTIONS, off by default, which ends the proxy once connections fall to zero afterMINIMUM_DRAIN_DURATION, 5 seconds by default. It exits sooner, but a worker that opens a new connection after a quiet moment finds the proxy gone, so the fixed duration is the safer choice for workers.

The pod then needs room for both. terminationGracePeriodSecondsdefaults to 30, and none of our twelve services sets it, so a service with a 30-second budget has no slack at all. With classic sidecars the app and the proxy shut down side by side, so the grace period has to cover the longer of the two, plus any preStop sleep, plus a margin.

A second race sits at the front of the shutdown. Kubernetes removes a terminating pod from its EndpointSlices at the same time as the kubelet starts shutting it down, so callers can still send requests for a moment after SIGTERM. A short preStop sleep before the app stops accepting work covers that window, and it counts against the grace period like everything else. The docs describe the two as parallel; the size of the gap on our clusters is something the drain test has to measure.

Native sidecars fix the order in Kubernetes itself

The ordering problem is as old as the sidecar pattern. KEP-753 was first proposed in May 2018 and names it directly: if the mesh container stops before the others, “outgoing traffic from other apps will be blackholed.” Native sidecars became stable in Kubernetes 1.33.

A native sidecar is an init container with restartPolicy: Always. The kubelet starts the app only after the sidecar reports started, and on shutdown it delays the sidecar’s SIGTERM “until the last main container has fully terminated.” In native mode Istio also starts Envoy’s drain in a preStophook as soon as the pod begins shutting down, so the proxy stops taking new connections while still carrying the app’s.

Istio 1.27 made native sidecars the default, and its auto setting turns them on only where the nodes run a recent enough Kubernetes. Ours do, so our planned Istio upgrade changes the shutdown order without a manifest change. The upgrade notes warn that the proxy moves from a container to an init container, which can confuse other mutating webhooks and controllers. That is worth a check in development first.

Native sidecars keep the order only inside the grace period. The same docs say that if the grace period runs out, “all remaining containers in the Pod will be terminated simultaneously with a short grace period.” A service that uses its full 30-second Spring budget in a pod with a 30-second grace period gets a proxy that is killed with it, not after it. In native mode the steps run one after another, so the grace period has to cover the preStop sleep, the app’s budget and the proxy’s drain added together. The longterminationDrainDuration from the previous section should come back down after the upgrade, because the proxy no longer has to outwait an app that is shutting down beside it.

Native sidecars fix the order. The grace period still decides whether the order gets to run.

Teams built workarounds while this waited. Monzo published envoy-preflight in 2019, a wrapper that holds the app until Envoy reports itself live and shuts Envoy down when the app exits. Native sidecars move the same idea into the kubelet.

On startup, the app should never see a network without its proxy

The startup race runs the other way. If the app starts before Envoy is ready, its first outbound calls fail, and Istio’s 2023 post on native sidecars puts it plainly: “If the application container starts faster than Istio’s container, it cannot access the network.” Users on the long-running Istio issue about it reported putting “a big sleep like 20 or 30 seconds in their app container.”

Get June’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Istio 1.7 added holdApplicationUntilProxyStarts, which blocks the other containers from starting until the proxy is ready. It defaults to false. Our production meshes set it, and one development mesh was missing it until a review caught the mismatch, so the same service had a startup guard in production and none in development. That mismatch makes development the worse place to reproduce a startup failure, the opposite of what development is for.

Once native sidecars are on, Istio’s injection template ignoresholdApplicationUntilProxyStarts, because the kubelet already orders startup. One detail remains: Kubernetes treats a sidecar as started when its process runs, unless it has a startup probe. Check that the injected proxy has one on your version before you drop the old setting.

The control plane has to be scheduled before the apps

Kubernetes has no dependency ordering between workloads. After our development cluster’s node groups came back from zero, nodes joined at about 2 minutes, around 47 application pods began starting with them, and istiod was ready only at 12. New sidecars need istiod for their certificate and config, so for 10 minutes no new pod could become ready while the control plane waited for a node of its own. In that development mesh the apps didn’t even wait for their proxies, because the startup guard was missing.

istiod had no priority class, no dedicated node group and no node selector, so it queued at priority 0 alongside every app and competed with them for the returning capacity. Istio’s charts leave priorityClassName empty by default, with a comment in the istiod values file suggesting you set one so Istio pods aren’t killed for low priority.

In that cluster we gave istiod system-cluster-critical, which Kubernetes defines as 2,000,000,000, and defined three tiers of our own below it: platform at 1,000,000, tenant workloads at 100,000 and batch at 1,000. User-defined classes are capped at one billion, so nothing we create can outrank the built-in classes. A pod that can’t be scheduled may preempt pods with a lower value, which is the point.

Setting global.priorityClassName in the Istio chart does not reach injected pods; we confirmed it by rendering the chart. A sidecar has no priority of its own anyway, because priority belongs to the pod, so each app’s class covers its sidecar.

Priority belongs to the pod. A sidecar is exactly as important as the app it serves.

Priority isn’t the whole answer for node replacement. Our review of a move to Karpenter found that node drains would stall on a single istiod replica held by its disruption budget, so the plan makes two istiod replicas a precondition before any autoscaler replaces nodes.

Before you blame the control plane, run a quick test. A sidecar that already holds its config keeps routing without istiod, so a short control-plane outage shows up as new pods failing to start, not as running traffic breaking. Only a long outage reaches running traffic, once certificates and endpoints go stale. If live requests fail while pods come up fine, look elsewhere.

Injection belongs in code, not on a hand-applied label

Sidecar injection is switched on by a namespace label. In one development namespace that label wasn’t in Git at all: the field manager on the live object was kubectl-label, so someone had added it by hand. If that namespace were recreated, or the label overwritten, every pod would lose its sidecar on its next restart, and with STRICT mTLS every call between those pods and the rest of the mesh would fail.

We moved the label into the namespace’s managed metadata in Git, next to the tenant label that network policies and cost reports already join on. The lifecycle of the proxy starts with whether it’s there at all.

Where a machine should do the checking

Every setting in this piece is a field a policy can read. An admission policy, or a CI check on the manifests, can reject a Deployment whose grace period is shorter than the app’s shutdown budget plus the proxy’s drain, a meshed pod with no drain setting, an istiod or gateway without a priority class, and a meshed namespace whose injection label isn’t declared in Git. After an upgrade to native sidecars, the same check confirms that istio-proxyappears under initContainers, which is how you know native mode is actually on.

The number we still owe is the cost. The next chaos run on the messaging platform restarts each service at a steady rate and counts errors per rollout, first with today’s defaults and then with the drain and grace settings above. That turns the 25-second gap into a count of lost requests, which is what decides how urgent the fix is.

The 30-second version

• With classic sidecars, Istio’s proxy exits 5 seconds after SIGTERM by default, while Spring Boot gives the app up to 30.
• Nine of our twelve services use that 30-second default, so mesh calls in the last 25 seconds of a shutdown have no proxy.
• With classic sidecars, set terminationDrainDuration to the app’s budget and give the grace period room for the longer of the two.
• Native sidecars, default in Istio 1.27 on new enough nodes, order startup and shutdown in the kubelet, but only inside the grace period.
• istiod had no priority class and came back 10 minutes after the pods that needed it; in that cluster it now runs as system-cluster-critical, and two replicas come before node autoscaling.
• Keep holdApplicationUntilProxyStarts and the injection label in Git for every environment, not just production.

Ask me in an interview: what happens to an app’s outbound calls when its sidecar exits first, how native sidecars change that, and why they still depend on the grace period.

Before the next high-RPS release, check three numbers per service: the proxy’s drain, the pod’s grace period and istiod’s priority.