Why May a Model Get Slower on a Bigger Machine?

When two libraries both assume they own every CPU core, adding cores can make things slower. Here’s why, and how to spot it.

Not every model runs on a GPU. A small image model, the OCR step in a document pipeline, an embedding job that runs every night: for work like this, a GPU is often wasted money, so it runs on plain CPUs, in a Kubernetes pod, a cheap cloud machine or a serverless function.

Let’s suppose that’s your setup. You trained a small PyTorch model that finds products in photos. To make it fast on CPU, you exported it to ONNX and now run it with ONNX Runtime. But only the model moved. The code around it, which prepares each photo before it goes in and turns the model’s raw output into results afterwards, was written with PyTorch, so PyTorch is still there doing those small jobs. That’s very common: you export the model and keep the rest.

On a cloud machine with 4 CPUs, it gets through 1,000 photos in about 40 seconds. Then the catalogue grows, so you move to the same kind of machine with 8 CPUs. You don’t expect it to be twice as fast, but you do expect it to be faster.

Instead, the same 1,000 photos now take about 59 seconds.

You look at where the time goes, and it gets stranger. Preparing a photo, which took under 2 ms on the smaller machine, now takes longer than running the model.

Nothing in your code changed. The model didn’t change. You only added CPUs.

What PyTorch is actually doing

When PyTorch runs on a CPU, it splits each operation across several threads, so that work on a whole photo or a whole batch finishes sooner. By default, it sizes that group of threads to the machine, roughly one thread per CPU. In the setup above, that meant 3 threads on the 4-CPU machine and 7 on the 8-CPU one.

After each operation, those threads don’t go to sleep straight away. For a short while, they keep checking, in a tight loop, whether new work has arrived. This is called spinning. A spinning thread does nothing useful, but to the operating system it looks busy, so it keeps its core.

That’s deliberate. Waking a sleeping thread takes time, and PyTorch usually runs many small operations back-to-back. If its threads slept after every one, every operation would start with a wake-up delay. Spinning keeps them ready. It’s a good design, as long as PyTorch has the CPU to itself.

In our setup, it doesn’t. ONNX Runtime runs the model with its own group of threads on the same machine, so you end up with more threads than cores, and the trouble starts on PyTorch’s side. Each PyTorch step splits across 7 threads, and the step isn’t done until the slowest one finishes. With more threads than cores, some of those 7 have to wait for a turn. The ones that finished early don’t step aside: they spin, holding the very cores the late ones need.

So PyTorch mostly slows itself down. Its small steps around the model crawl, and the model loses a little time too, because spinning threads also take cores ONNX Runtime wants.

A step is only as fast as its slowest thread, and the early finishers spin instead of making room.

And the bigger machine meant more PyTorch threads: 7 instead of 3. Most likely, that’s why the 8-CPU machine was hit, and the 4-CPU one wasn’t. The extra cores didn’t give the model more room. They gave PyTorch more threads to fill.

Let’s suppose you tell the threads to sleep

There’s a setting for exactly this. The library PyTorch uses to manage its threads, OpenMP, reads an environment variable called OMP_WAIT_POLICY. Set it to PASSIVE, and idle threads go to sleep right away instead of spinning.

On the 8-CPU machine, same 1,000 photos:

The small jobs around the model got about 15 times faster, and the model itself sped up a little too, because it no longer fought for cores. The whole run went from about a minute to 22 seconds.

Problem solved? Not quite. Here’s the same setting with a plain PyTorch model, where PyTorch runs the model itself:

When PyTorch does the heavy work, spinning is exactly what keeps it fast. Turn it off, and the same model runs 3.7 times slower. The setting that rescued one setup wrecked the other.

Get Varun Rajput’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

So the verdict on this setting: it’s a good fix when PyTorch only does side jobs next to another library, and a bad one when PyTorch runs the model. It also applies to the whole program and has to be set before PyTorch starts, so you can’t switch it on for one model and off for another.

The better fix keeps the spinning and gives PyTorch fewer threads, so it never crowds the cores in the first place. That’s what the real project ended up doing, as you’ll see below.

Let’s suppose you run four copies

This one bites even when PyTorch is the only library in the process. To serve more requests, a common move is to run several copies of the model on one machine, as separate processes, each taking a share of the work.

Every copy sizes its threads for the whole machine, as if it were alone. On the 8-CPU machine that’s 7 threads per copy, so four copies bring 28 threads to 8 cores, all of them spinning whenever they’re idle.

Each copy thinks it has the whole machine.

The same project measured this too, with a plain PyTorch model on the 8-CPU machine:

Four copies should finish sooner than one. With default settings, they took about 23 minutes, 50 times longer. Give each copy one thread, and four copies beat one, the way you’d expect.

It’s the same mistake as before, just bigger: every copy planned for a machine it didn’t have. So if you run several copies on one machine, split the cores between them. With 8 cores and 4 copies, each copy gets about 2 threads, not 7. In this test, even one thread each was enough.

What I’d take away

A lot of libraries assume they own every core on the machine: PyTorch, ONNX Runtime, OpenVINO, NumPy’s math libraries, OpenCV. Each one is fast on its own, because that’s how it was tested. Put two of them in one process, or run several copies side by side, and they compete for the same cores. The habits that make each one fast alone start getting in each other’s way.

So when a bigger machine makes something slower, or more copies make it much slower, suspect a fight over cores before you suspect your code. Count the threads, not just the CPUs.

For the engineers

• Spotting it. On Linux, ls /proc/<pid>/task | wc -l gives a process’s thread count. Far more threads than cores, high CPU usage and low throughput together point to oversubscription. Per-stage timings help too: here the small steps around the model got 15× slower while the model barely changed.
• torch.set_num_threads(n) caps PyTorch’s threads at runtime, for the whole process. OMP_NUM_THREADS only works if it’s set before torch is imported, and code that calls torch.set_num_threads later overrides it. YOLO did, which is why the environment variable didn’t help there.
• OMP_WAIT_POLICY=PASSIVE also has to be set before torch loads. Use it when PyTorch only does side jobs, not when it runs the model (3.7× slower here).
• Several worker processes (gunicorn or uvicorn workers, a multiprocessing pool): give each one about cores // workers threads. DataLoader workers already get this right: PyTorch sets each worker to 1 thread.
• ONNX Runtime has its own controls: intra_op_num_threads in SessionOptions, and the session config entry session.intra_op.allow_spinning set to "0". I didn’t test these here.
• Containers. os.cpu_count() reports the host’s CPUs, not your pod’s CPU limit, so a library can size its thread pool for a 64-core node inside a pod limited to 2 CPUs. I didn’t measure this, but it’s the same fight, bigger.
• Benchmark like a user. My first runs imported torch before YOLO, which real users don’t do, and that alone changed the process’s thread count. A benchmark has to load libraries in the same order your users do.
• What I didn’t profile. I didn’t trace which thread waited on which. The explanation above fits every measurement: only PyTorch’s settings fixed it, PyTorch’s own steps slowed most, PyTorch alone was fine, and four PyTorch copies collapsed. But it’s an inference from those results, not a trace.
• Limits of these numbers. Intel Xeon with hyperthreading: “4 CPUs” and “8 CPUs” are 2 and 4 physical cores. Not tested: AMD, ARM, bigger machines. I can’t fully explain why the 4-CPU machine escaped. It also showed a separate two-speed pattern (runs took either ~32 s or ~49 s at random) that I haven’t explained, so its 40 s is a median across both.

If you are setting this up and want to talk through your specific setup, feel free to reach out: topmate.io/varun_rajput_1914 or https://www.youtube.com/@TagAlongWithVarun