“Not Enough Resources” Was True Three Times. Never The Same Resource.
“Not Enough Resources” Was True Three Times. Never The Same Resource.
Tags: sre istio envoy kubernetes capacity-planning
A backend engineer left a comment on our load test that I have been thinking about ever since. He asked whether the Envoy upstream idle timeout was set shorter than the load balancer’s 350 seconds.
He was pointing at the right layer. Every mechanism in the question was wrong. And he was right about two other things I had to concede in writing.
The round he was asking about had come back at 29.44 percent success. The round above it, 4.67 percent. The cause turned out to be 500m of CPU and 512Mi of memory on a pod that neither of us had been looking at.
What actually died
The ingress gateway. One pod, thirteen restarts, exit code 137, OOMKilled, back-off restarting the proxy container 57 times over 84 minutes.
Its limits were 500m CPU and 512Mi memory, replicaCount: 1, autoscaling disabled.
The first constraint is the CPU, not the memory. A 500m limit makes the sidecar derive ISTIO_CPU_LIMIT=1, which starts Envoy with --concurrency 1. One worker thread, pinned to half a core. At 1,500 requests per second it cannot drain the queue, so requests bank up, the queued bodies push memory upward, and the process walks into the 512Mi wall.
The resource that failed and the resource that mattered were not the same resource.
Raising the memory wouldn’t have fixed it. It would have died later, or started returning 503s first, and then hit the same wall, because nothing about a bigger heap gives Envoy a second worker thread. This is the shape of the whole problem: “it OOMKilled, give it more memory” is a true observation that leads to the wrong repair.
How we knew it was not the timeout
The 1,000 rps round did the refuting on its own. It pushed 300,000 requests through 4,020 new connections, which is 74.6 requests per connection, which means a connection sat idle about 4.0 seconds between uses. That is one eighty-seventh of 350 seconds. Both failing rounds were tighter still.
Then the reset counters settled it.
AWS defines those two counters precisely. One counts resets generated by the target, the other resets generated by the load balancer. The load balancer generated none, in any window. Our own gateway sent every one of them.
Here is where I had to give ground, though. Keep-alive genuinely is an L7 concept, and the balancer is L4, so its 350 seconds is a TCP flow idle timer rather than a keep-alive setting. The interaction he was aiming at, an L4 timer tearing down an L7 connection, is real. It just didn’t fire here. He was also right that the 350 seconds can’t be changed: on a TLS listener AWS documents it as fixed.
And the timer did do exactly one thing that afternoon. When the gateway died it took roughly 6,000 live keep-alive connections with it, the generator kept opening new ones at about 1,197 per second, and flow state piled up to ten times the pool size. The 350-second timer’s entire contribution was cleaning up that debris.
He aimed at a real mechanism that happened not to be firing. That is a much better guess than most of us make about layers we cannot see.
Nobody owned that pod
This was one gateway out of seven. The other six all ran with autoscaling on, minimum two replicas, maximum five.
This one had autoscaling off and a single replica. Not a decision anybody made twice, and not a decision anybody revisited, because a gateway sits in the gap between the team that owns the application and the team that owns the platform. It appears on neither dashboard as something with a capacity budget.
Round two, different resource
Six weeks later, a second campaign on the same platform, and the ceiling moved to the application.
Get June’s stories in your inbox
Join Medium for free to get updates from this writer.
Somebody had set spring.threads.virtual.enabled: true. Before that flag the service ran on the Tomcat platform thread pool, which defaults to 200. That number had never been written down as a capacity decision, and it was the only admission control in the request path. Virtual threads switch it off, and the next limit up is max-connections at 8,192, which nobody had overridden.
In-flight requests per pod went 3, then 292, then 1,967 on average, against an unchanged 2 GiB memory limit. Four pods were OOMKilled 20 seconds apart, and each death pushed its share onto the survivors, so the last one died carrying more than that.
What killed them is worth naming exactly, because it wasn’t a JavaOutOfMemoryError. The live set was 1.09 GiB against a 1.5 GiB heap ceiling, so as far as the JVM was concerned it had room. The kernel killed the container when its total footprint crossed the 2 GiB cgroup limit. Meanwhile max-connections, the only ceiling still in force, sat at 8,192, roughly four times above what the memory could take, so it never got a turn.
Second campaign, second answer to “not enough resources,” and it was not the same resource as the first.
The seam
Vegeta recorded 41,232 failures with a 503. Every single one came from a sidecar. The application returned zero.
That sentence is the whole article. The only component that spoke during the failure was the proxy in between, and the proxy is the component neither team instruments.
It runs in both directions. The application engineers can’t see CloudWatch or the mesh, so a gateway starving on half a core is invisible to them and “the resource is not sufficient” is the most precise thing they can honestly say. I can’t see their object graph, so I couldn’t tell them which allocation was filling the heap. In the write-up after the second campaign I had to ask them two questions to interpret my own test: whether it was measuring the end-to-end pipeline, and whether a fairness limit that was declared in configuration and read by no code had been intended as backpressure.
I was running a load test and I couldn’t tell, without asking, what it was measuring.
The resource nobody predicted
There was a third answer, and it belonged to neither the gateway nor the heap.
Scheduling. An API pod costs 800m once the 200m sidecar is counted, and no node had room for a fifth replica. The obvious read is that the API tier is eating the cluster. It isn’t. One worker class alone commits 6,300m across seven pods, and the four API pods together account for 3,200m.
Commit rate across the cluster sat at 91.6 percent while measured idle CPU was 3 to 7 percent. Both numbers are real and they describe different things. One is the scheduler’s ledger. The other is the hardware. Argue about capacity without saying which one you mean and you’ll talk past each other for an afternoon.
What I write down now, before a load test
Whose ceiling are we testing. If the answer is “the application,” then the gateway, the sidecar, the connection pool and the node all need their limits enumerated first, because any one of them will fail before the application does and the result will be reported as the application’s number.
Every limit in the path, with its current value. Not the ones we think are interesting. All of them, including the proxy’s, because on this platform the proxy is where two of the three failures actually landed.
None of this is shipped yet. The gateway sizing and the application-side cap are both still open, and I am publishing the diagnosis anyway, because the flag is already on in two sibling services with the same missing override and the arithmetic doesn’t care whether I’ve fixed mine.
And the proxy gets a named owner. That’s the fix that generalizes. The gateway did not fail because someone chose 500m badly. It failed because it was the only pod in the request path that no capacity conversation had ever included.
The 30-second version
Three rounds of load testing produced three different binding constraints: a gateway pinned to one Envoy worker by a CPU limit, an application heap with no admission control after a virtual-threads flag removed it, and a scheduler with no room for another replica because the worker tier had the commit. Every failure was correctly described as “not enough resources” and every one of them named the wrong resource, including once by me.
The developer who guessed Envoy was closer than the phrasing suggests. He was reaching for the layer he could not measure, which is the same layer I couldn’t attribute allocations inside.
Ask me in an interview: why raising the memory limit on that gateway would have moved the failure without fixing it, and what I would have instrumented instead.
The pod that fails first is usually the one nobody argued about.
