Scaling for Memory: Right-Sizing Containers on Azure for Memory Bound workloads

TL;DR. Standard cloud autoscalers scale worker count based on requests, queue depth, or CPU, assuming each job has similar resource needs. Our workload broke that assumption: memory usage depended on the size of each file being processed. So we built a memory-aware scaler with Azure Functions and Service Bus that starts one right-sized Azure Container Instance per message. This post explains why standard scaling fell short, how the design works, and what we learned the hard way.

The workload

As part of the Transportation Data Exchange Initiative (TDEI) infrastructure we built, we needed to handle workloads that scale with the size of the input. Users submit geospatial datasets, and our services process each one. Validation checks a dataset for structural and topological correctness, formatting converts it between formats, and quality-metrics calculation measures how good the data is.

These services have different logic, but they share one trait that drives everything else in this post: the memory a job needs depends on the size of its input file.

The following chart shows how peak memory usage grows with the size of the input file.

Formatter profiling: peak memory usage vs. input file size

Three properties shape the design:

1. Jobs are long-running. They take much longer than a typical web request. For example, the formatter can take up to 6 minutes for a 250 MB file. The work is not CPU-bound, so a machine with more CPU doesn’t help.
2. Arrival is bursty. Long quiet periods are followed by several submissions at once. Keeping App Service instances running for those bursts would be an unnecessary cost.
3. Size is known before the job starts. The input file size is available up front, so we can decide how much memory to give a job before it runs. That turns out to be the key to the whole design.

That combination set our design goal: provision a machine that is just big enough for the job, run the job to completion, then decommission the machine. Nothing sits idle and nothing is oversized, so we only pay for the memory a job actually needs, for as long as it needs it.

What makes a job memory-bound

“Memory-bound” gets used loosely. We mean memory capacity: the working set has to fit in RAM, or the job fails. This is different from being limited by memory bandwidth or CPU. A CPU-bound job slows down when starved of CPU. A capacity-bound job gets killed by the out-of-memory (OOM) handler. Adding more replicas doesn’t fix that, because each replica still needs enough RAM for the whole input.

Algorithms tend to fall into this category when they need most of the input resident at once, so peak memory is a multiple of input size:

• Whole-document parsing. Loading JSON or GeoJSON into an in-memory object tree often costs several times the file size.
• Graph algorithms. Connectivity and network validation need the full node-and-edge structure, plus adjacency data, before they can produce an answer.
• Spatial operations. Building spatial indexes and intersecting geometries across features.
• Joins, sorts and group-bys that don’t fit a streaming model.
• Deduplication and uniqueness checks. Hash sets over every ID or geometry seen.
• Cross-reference validation. Checking every reference against every definition needs both sides in memory.

Contrast that with per-record transforms, hashing or encoding. Those stream through the data, memory stays flat as the input grows, and ordinary horizontal scaling works well.

The useful part is that the memory demand of a job is predictable from the input, before the job starts. File size is a decent proxy for it, so if we size the worker to fit the job, we don’t have to guess.

Why standard autoscaling doesn’t fit

Most Azure scaling tools, including KEDA scale rules, Container Apps replica scaling and App Service autoscale, are built to answer one question: how many copies of this worker should be running? They scale on signals like queue depth, HTTP concurrency or CPU. That works well when every unit of work costs about the same. It breaks down for us in three ways:

• Uniform replicas. Every replica gets the same CPU and memory. To avoid OOM on the largest file, you size every replica for the worst case, and you pay for that memory on every small job too. Size for the median instead, and large files crash.
• Wrong signal. Queue depth says how many messages are waiting, not how much memory the next one needs. CPU is nearly irrelevant to a capacity-bound job.
• Memory metrics arrive too late. Scaling on observed memory pressure is reactive. By the time the metric moves, the process is often already dead.

Here is how the options stacked up for us:

• KEDA / Container Apps scale rules: no per-message sizing; scales to zero; risky on scale-in because a job may be cut off before it finishes. Every replica is the same size.
• AKS + KEDA: no per-message sizing; partly scales to zero; risky on scale-in. It also adds cluster operations overhead.
• Azure Functions (Service Bus trigger): no per-message sizing; scales to zero; cannot run jobs to completion because of memory and execution-time ceilings.
• Function App + ACI (what we built): per-message sizing, scale to zero and runs to completion. The trade-off is custom code that we own.

The approach: one container per message, sized to fit

The idea is to stop treating the request queue as a signal for “how many workers” and instead treat each request as an order for a specific machine: sized to its input, used once, then retired.

Simplified Architecture Diagram of the solution

1. A function app manages the machines. An Azure Function looks at the pending requests, provisions a container for each one that needs it, and decommissions containers that have finished. There is no fleet to size or keep warm: the function creates capacity when work arrives and removes it when the work is done.

Get Suresh Devalapalli’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Each new machine is sized from the request’s input file size:

1. Calculate the peak memory requirement of the algorithm.
2. Find the smallest container instance size whose memory meets or exceeds the value calculated in step 1.
3. Provision that container instance.

Each machine is an Azure Container Instance (ACI), created with exactly the required memory and tagged with the request it was provisioned for.

2. A request leaves the queue only when its job is fully done. Our jobs run for minutes, and every request must be handled to completion. So the function only looks at the queue (a read-only peek) and never removes anything. The container it provisions picks up the request and removes it from the queue only after it has finished processing. If the machine fails for any reason, the request is still in the queue, and the next run of the function provisions a fresh machine to retry it.

The queue is the record of what is left to do, so a crashed job needs no special recovery logic. It simply hasn’t been completed yet.

3. Finished machines are shut down and decommissioned. Once a container has completed its work, the function deletes it. Nothing lingers, so cost tracks actual work.

Two guardrails keep this safe. A cap on concurrent containers bounds cost, and tags on each container let the function tell which requests already have a machine, so the same request is never provisioned twice.

The hard parts

The happy path is about a hundred lines of code. Nearly all of the real engineering went into what happens when the world doesn’t cooperate.

Peeking doesn’t lock messages. We peek instead of receive, so the scaler never holds a message it can’t finish. The cost is that a message can vanish between the peek and the moment a container is ready: another consumer takes it, or it expires. To handle this, we re-check that the message still exists while the container is being provisioned, and again once provisioning succeeds. If the message is gone, we delete the container we just made.

Duplicates. The scaler runs every minute and provisioning takes a while, so the same message shows up in consecutive peeks. Checking tags on existing containers, plus deduplicating within a single peek batch, stops us from launching two containers for one job.

Orphans. A running container whose subscription has since drained is wasted money. We detect it from its tags and delete it.

Knowing when a container is really done. We delete a container only when both its container state and its provisioning state are terminal, so we never delete one that’s still winding down.

Fairness. One busy subscription shouldn’t starve the others. The first pass provisions at most one message per subscription, and the second pass fills any remaining capacity.

The function timeout. The timer function runs under a 10-minute limit on the Consumption plan. Batch sizing and bounded parallelism keep a run from overshooting it.

Results

Before. Each service was provisioned for the worst-case input and deployed on Azure App Service plans that ran around the clock. Because bursts are short and most files are far smaller than the worst case, the machines were mostly idle. Typical utilization sat around 10%: only a tenth of the machine’s run time was doing real work. Put differently, we were paying roughly 10x what the work actually required.

After. With the scaler in place, machines exist only while a job runs and are sized to the job’s input. Costs dropped by about 8x, cutting the client’s bill substantially.

• Deployment: before, always-on App Service sized for the worst case; after, one right-sized container per job, removed when done.
• Typical utilization: before, ~10%; after, close to 100% while a job runs, with the machine shut down afterward.
• Relative cost: before, 1x (baseline); after, about one-eighth.
• Maximum input file size: before, 1 GB; after, 2 GB (16 GB is technically possible).

Bigger inputs, too. Cost wasn’t the only limit. Before, the largest input we could handle was capped by the biggest machine the App Service plans could offer. To keep monthly costs down, the client had settled on 32 GB machines, which meant a maximum input of 1 GB. With per-job containers, memory is no longer a fixed ceiling chosen up front: each job gets what it needs. We raised the supported input size to 2 GB. Technically, we can go as high as 16 GB, but the client preferred a gradual rollout, raising the limit as the need arises.

Trade-offs

This design is not free, and it isn’t right for every workload.

• Latency. Container instances sometimes take up to a minute or two to provision. That’s fine for jobs that run for minutes, but wrong for latency-sensitive work.
• Custom code to own. The scaler becomes another piece of code to maintain and monitor.
• Platform limits. ACI has per-region quotas and per-group CPU and memory ceilings. These limits need to be monitored constantly, and the scaler’s parameters adjusted to match.
• Poor fit for many tiny, fast jobs (container startup dominates), or workloads that scale fine horizontally at a uniform size.

The pattern fits when all of these hold: the work is memory-bound, its size is known before it starts, each job runs to completion, and a startup delay of up to about a minute is acceptable.

Open questions

We’re still working on a few things, and we suspect others have opinions:

• Settling instead of peeking. Peeking keeps the scaler simple but forces the presence checks above. Would locking and settling messages (or using sessions) be cleaner, and how would it change failure handling?
• Event-driven triggers. If platforms add per-execution sizing to event-driven jobs, this scaler could become unnecessary. Where does that stand today, and is anyone already doing it?
• Retry on OOM. Should a job that hits an OOM error be retried automatically at a larger size?
• Squeezing out the last drop of memory. Because of the minimum size limits on Container Apps, we could sometimes have assigned two small-file jobs to one container instance. Is that optimization worth it for the cost savings?