Member-only story
Parallelizing LLM Tasks
From one GPU’s tile grid to vLLM clusters and DeepSeek’s DeepInfer
A language model answering a question does two very different jobs, and it does them on the same hardware. First it reads your prompt, all of it, in one gulp. Then it writes its answer one token at a time, each token a full trip through billions of weights. The first job is a sprint. The second is a long walk in which the GPU spends most of its time waiting for memory and very little of it doing math.
Everything clever about LLM serving is a response to that mismatch. Batching exists because of it. vLLM’s scheduler exists because of it. And when DeepSeek open-sourced DeepInfer, the engine behind its production service, the architecture it describes is that same mismatch taken to its logical end: two separate fleets of GPUs, one for each job, each shaped around the work it receives.
This post follows the mismatch from the bottom up. It starts with a toy GPU drawn as a grid of blocks, so you can see the shape of each job. It then shows how batching folds many conversations into one pass, how vLLM spreads a model across a cluster, and finally how DeepInfer departs from vLLM, the project it was originally built on.
