Envoy’s CPU Limit Is A Thread Count. 500m And 1000m Both Mean One.
Envoy’s CPU Limit Is A Thread Count. 500m And 1000m Both Mean One.
Tags: sre istio envoy kubernetes capacity-planning
After spending two days wrong about a 503, I stopped theorising about the service mesh and measured it. The first thing the numbers said was that a change I had been ready to recommend would have done nothing at all.
The gateway that fell over during an earlier load test had a 500m CPU limit. The fleet standard elsewhere is 1000m. Doubling it looks like the obvious repair, and it buys exactly zero additional throughput, because both values round to the same integer.
Meanwhile the control plane I had blamed for a ten-minute outage was sitting at 2 millicores against a 500m limit, serving 16 sidecars, with roughly 250 times more headroom than it was using.
Two numbers, both wrong in my head, in opposite directions. This is what I now believe about sizing a mesh.
The limit is a thread count, not a budget
For Envoy, and this covers sidecars and gateways alike, the container’s CPU limit does something no other workload’s limit does. Istio injects it through the Kubernetes downward API, which rounds a CPU limit up to a whole number of cores, and hands that integer to Envoy as --concurrency.
That integer is the worker thread count. Throughput then moves in steps, not smoothly.
Every value on that first step behaves identically. A sidecar at 200m and a gateway at 1000m both run a single Envoy worker. The difference between them is how much CPU that one worker is allowed to burn, which is a throttling question, not a parallelism one.
Raising a gateway from 500m to the fleet standard of 1000m looks like a capacity fix and changes nothing measurable. It is the same one thread with a higher ceiling.
This is why the load-test failure I wrote about earlier was a CPU problem wearing a memory problem’s clothes. One worker pinned under half a core could not drain the queue at 1,500 requests per second, queued bodies pushed memory upward, and the container hit its memory wall. Adding memory would have moved the death, not prevented it.
Note what the fleet standard would and would not have done there. At1000m that gateway gets twice the quota for the same single worker, so it survives longer and then fails in the same shape. Changing the shape takes a second thread, and a second thread takes a limit above one core.
Verify the thread count from inside the pod with pilot-agent request GET server_info and read concurrency back. Never infer it from the values file, because the rounding happens after the value you wrote.
The control plane does not scale with traffic
The premise worth fixing before any sizing conversation: istiod never touches a request. It watches the Kubernetes API, computes configuration, and pushes it to proxies over xDS.
Double your traffic and istiod’s load is unchanged. Double your services and it climbs. The three mesh components sit in different places entirely.
istiod tracks service and endpoint counts, configuration churn, and the number of connected proxies. Watch pilot_proxy_convergence_timerather than CPU, because rising convergence is the signal that arrives first.
A gateway Envoy tracks ingress request rate and TLS handshakes, and does not care how many backend services exist. A sidecar tracks the request rate through its own pod, plus its config scope, which is also what drives its memory.
That last one matters more than people expect. An unscoped sidecar holds every endpoint in the mesh, so its memory grows with cluster size whether or not its pod gets busier. A Sidecar resource with explicit egress hosts cuts that sharply.
A correction I owe my own incident review
In the write-up of that 503 I said istiod’s 500m limit made it thin when every sidecar in the cluster reconnected at once. The measurement says otherwise.
Get June’s stories in your inbox
Join Medium for free to get updates from this writer.
istiod was using 2 millicores and 48 MiB, against limits of 500m and 1 GiB, while serving 16 sidecars across 42 pods. For a cluster that size the limit is generous. CPU throttling was almost certainly not why the control plane took ten minutes to come back.
The scheduling gap explains those ten minutes. The resource limit does not.
The same number,
500m, was fatal on an ingress gateway and irrelevant on the control plane. Nothing about the number tells you which case you are in.
Those two pods sit at opposite ends of this article. The gateway’s 500m set a thread count under real traffic and killed it. The control plane’s 500m was a throttle ceiling for a process that never came close to it.
Requests reserve capacity that nothing uses
Limits shape throughput. Requests shape scheduling, and that is where the waste hides.
On four two-core nodes the mesh reserves about 17 percent of the cluster in order to run at about 1 percent. That reserved capacity cannot be scheduled to anything else.
A node can read 90 percent requested while sitting at 10 percent utilised, and every pod going Pending against it is queued behind capacity nobody is using.
The per-sidecar arithmetic is 50 millicores reserved against 3 used, sixteen times over. Dev traffic was near zero when I measured, so the ratio flatters the point, and requests should cover a realistic peak rather than a quiet Tuesday. It is still worth revisiting.
The two fields do unrelated jobs, and one of them does two. Requests are what the scheduler sets aside. Limits are the throttle ceiling and, for Envoy alone, the worker count. That overloading is what makes this so easy to tune in the wrong direction.
Sizing from a target request rate
Six steps, in the order that keeps you honest.
- Measure your own cost per request. Published figures of roughly 0.3 to 0.35 vCPU per 1,000 requests per second per proxy are a starting estimate, not your number. Yours depends on protocol, payload size, mTLS and telemetry config. Divide sidecar CPU rate by request rate at a known load.
- Multiply by the proxies in the path. A mesh-internal call costs Envoy CPU twice, once outbound and once inbound. Ingress to a service is a gateway plus one sidecar. Sizing that counts the server side alone is short by half.
- Pick the thread count first, then the limit. Decide how many workers you need at peak, then set the limit to that integer. Doing it the other way round is how you end up with one thread and a comfortable-looking number.
- Set requests from steady state, not peak. Requests drive scheduling and over-requesting wastes cluster capacity. Limits drive burst and concurrency.
- Size sidecar memory from config scope. Scope the sidecar with explicit egress hosts and memory stops growing with the cluster.
- Size istiod from proxy count and churn. Roughly 1 vCPU and 1.5 GiB per thousand proxies, and watch convergence time rather than CPU.
Step 3 is the one that would have saved us the load-test outage, and step 1 is the one everybody skips.
The 30-second version
For Envoy, limits.cpu is rounded up to whole cores and handed to the proxy as its worker thread count, so 200m, 500m and 1000m all produce exactly one thread and the fleet-standard bump changes nothing. istiod is the opposite case: it never touches a request, so it does not scale with traffic at all, and mine measured 2 millicores against a 500m limit while serving 16 sidecars. Requests are the other half of the mistake, reserving about 17 percent of a small cluster to run at about 1 percent. Size the thread count first and the limit second, count every proxy in the request path, and verify concurrency from inside the pod rather than from the values file.
Ask me in an interview: why raising an Istio gateway’s CPU limit from 500m to 1000m adds no throughput, and how to tell whether a mesh component’s ceiling is a thread count or a throttle before you change it.
The same field means two different things depending on which pod you put it in, and nothing in the manifest warns you which.
