The KV Cache

Why a 16 GB model can’t fit comfortably on a 24 GB GPU

If you’ve ever tried to run a language model locally, you’ve probably hadthis experience: you check the model size, you check your GPU, the numbers look fine — and then it crashes.

The missing piece is almost always the KV cache. It’s the memory a model uses to remember your conversation, and almost nobody explains it to beginners.

Let’s build it up slowly, starting from a single token.

The problem: models are forgetful

An LLM writes one token (roughly one word-piece) at a time. To pick the next token, it looks back at every token that came before.

So if you’re at token 1,000, the model needs to process tokens 1 through 999 to decide what comes next. Then at token 1,001, it processes tokens 1 through 1,000. Then 1 through 1,001.

You can see the problem. You’d be redoing almost the same work over and over. Generating a 1,000-token answer would mean roughly half a million redundant calculations.

Part 1: Why the cache exists

KV caching is the data structure. Prompt / prefix caching is one way of reusing it.

An LLM writes one token at a time. (A token is roughly a word-piece — “cat” is one token, “unbelievable” might be three.)

To choose the next token, the model looks back at everything that came before. So at token 1,000 it processes tokens 1–999. At token 1,001 it processes tokens 1–1,000. Then 1–1,001.

You can see the waste. Almost all of that work is identical to what it just did.

The fix is simple. For each token, the model computes two things it needs later: a Key and a Value. And those never change. The Key for token 5 is the same whether you’re on token 6 or token 6,000.

So compute them once, keep them in GPU memory, and reuse them.

That stored pile of Keys and Values is the KV cache.

It’s like taking notes in a meeting. Instead of asking everyone to repeat themselves every time you want to make a point, you glance at your notes. Much faster — but the notes take up space on your desk.

That space is what this whole post is about.

Part 2: The unit — one token, one user

Everything reduces to one atom: the cost of remembering a single token for a single person.

Here’s how you find it: (simplified version).

bytes per token = 2 × layers × (kv_heads × head_dim) × bytes_per_number

2.1 One layer KV cache..

Multiple all these and you get 16 bytes per token.

2.2 KV Cache growth

2.3 Llama 3 with 8B parameter.

Older models used MHA: every query head had its own matching K and V head. A 32-head model stored 32 copies of K and V per token, so the cache grew with head count and ate GPU memory on long context.

GQA keeps the 32 query heads but shares K and V across groups — Llama 3 8B uses 8 KV heads, so 4 queries share each KV. Quality stays close to MHA; the cache is about 4× smaller, which is why long-context serving is practical now.

From its config:

Layer(s): 32 layers (32 query heads).
KV heads: 8
head dimension 128
Running in BF16, so 2 bytes.

2 × 32 × (8 × 128) × 2
= 2 × 32 × 1,024 × 2
= 131,072 bytes
= 128 KiB

One token, one user, on Llama 3 8B, costs 128 KiB. Let’s call this as baseline size of each token.

Write that number down. Everything else in this post is that number multiplied by something.

2.4 When is KV cache updated.

The cache is written after the K,V projection and before QK⊤. Attention never sees a key that has not been appended.

Two details that keep this from getting sloppy:

• Q is not cached. Only K_t​ and V_t​ go in. Next step will compute a new Qt+1.
• If the cache were updated after attention, the current token could not attend to itself (and implementations would be wrong).
• Append first, attend second.

Part 3: A short conversation

Now make it 100 tokens — about 75 words, one paragraph:

128 KiB × 100 = 12.5 MiB

12.5 MiB. That’s it.

For comparison, the model weights take up about 15 GiB. So one paragraph of conversation is 0.08% of what the model itself occupies. Completely invisible.

This is worth sitting with, because beginners get it wrong in both directions. Some ignore the cache entirely — fine at 100 tokens, fatal at 100,000. Others panic about it — unnecessary at 100 tokens.

The truth is that it depends entirely on length. So let’s grow it.

Part 4: A longer conversation

Same model, same GPU, still one person. The only thing changing is how long they talk:

Nothing changed but the length of the conversation. The cache went from a rounding error to nearly the size of the model itself.

Get Anandha L Ranganathan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Two things to notice:

First, it’s perfectly linear. No cliffs, no surprises. 1,000× the tokens is exactly 1,000× the memory. That’s what makes this easy to reason about — once you know your per-token number, you can do the whole thing in your head.

Second, context_length means prompt plus everything generated so far. It grows while the model is talking. If you send a 500-token question and it writes 200 tokens back, you’re at 700 — and climbing with every word.

Part 5: Adding more people

Now the second multiplier. Each conversation needs its own cache — your Keys and Values are useless to me, because we typed different words.

So they simply add up:

KV cache = bytes_per_token × users × tokens_each

At 4,000 tokens each on Llama 3 8B (500 MiB per person):

If you’re running a model on your own machine, users is 1. Always. This multiplier only matters when you’re serving other people.

Here’s the part that makes everything click.

You usually don’t want to know “how much memory will this use?” You want to know “what can my GPU actually handle?”

So rearrange it:

Step 1:  free VRAM   = total VRAM − model weights − workspace
Step 2:  token budget = free VRAM ÷ bytes per token
Step 3:  max users    = token budget ÷ context length

Let’s do a 24 GB card running Llama 3 8B:

Total VRAM       24 GiB
Model weights  − 15 GiB   (8.03B params × 2 bytes)
Workspace      −  2 GiB   (activations, CUDA overhead)
                ────────
Free            ~7 GiB

7 GiB ÷ 128 KiB = 57,344 tokens

Your GPU is not a “how many users” machine. It is a 57,344-token bucket.

And you get to choose how to pour it out:

Every row uses the exact same memory. There is no setting that gives you both long contexts and many users. The bucket is the bucket.

That trade — length versus concurrency — is the entire job of an inference engineer. And it’s why “how many users can this GPU serve?” is an unanswerable question until you say how long the conversations are.

Part 6: Flip it around — your GPU is a token bucket

Here’s the part that makes everything click.

You usually don’t want to know “how much memory will this use?” You want to know “what can my GPU actually chandle?”

So rearrange it:

8,192 × 128 KiB = 1 GiB

One gigabyte. For one user. And that’s on top of the ~16 GB the model weights already occupy.

Why this ruins your day

Say you have a 24 GB GPU. The weights eat 16 GB, leaving you about 7 GiB.

7 GiB ÷ 128 KiB per token ≈ 57,000 tokens of budget

You can serve one user with a 32k-token context. Or seven users at 8k each. Not seventy. Just Seven users.

This is why serving LLMs is expensive, and why “how much VRAM do I need?” never has a simple answer — it depends entirely on how long your conversations are and how many you run at once.

Part 7: What it costs in dollars

A 24 GB cloud GPU rents for roughly $0.50–$1.00 per hour, depending heavily on provider and region. Call it $0.75. .

$0.75/hour ÷ 57,344 tokens ≈ $0.000013 per token, per hour

Which means:

One user with a maxed-out context is renting your whole machine.

That’s the real reason long-context pricing works the way it does, and why cached prompts on hosted APIs expire quickly. Anthropic’s prompt cache, for example, defaults to a 5-minute lifetime. Idle memory isn’t free, and someone has to pay rent on it.

Here the billing is on GPU and not HBM, why ? .

Physically: the KV cache lives in HBM. That is the scarce resource. When HBM is full of cached tokens, you cannot fit more sequences on that card.

Commercially: nobody sells “HBM-hours.” Cloud bills a GPU-hour. HBM comes bundled with the GPU. So the rent you pay is for the whole card, even if compute is idle and only memory is occupied.

KV cache consumes HBM. You pay for that HBM by renting the GPU that owns it. Idle cache still occupies the card, so you still pay the GPU-hour.

For more information about caching cost of each model you can get it online.

Part 8: How models fight back

Once you understand that the per-token number drives everything, the last decade of research makes sense. It’s all been about shrinking that one number.

GQA : Share the Keys and Values. Older models gave every attention head its own K and V. Grouped-Query Attention makes several query heads share one set. Llama 3 8B has 32 query heads but only 8 KV heads — a 4× smaller cache for almost no quality loss. Nearly every modern model does this.

Sliding Window: Forget old tokens Does every layer really need to see something from 100,000 positions back? Often not. Gemma 3 uses 5 “local” layers that only look back 1,024 tokens for every 1 “global” layer that sees everything. Google measured the cache overhead dropping from around 60% to under 15%. (Caveat: this breaks the simple formula — you have to count local and global layers separately.)

MLA: Compress them. DeepSeek squeezes the Keys and Values into a single small vector instead of storing both. DeepSeek-V3 has 671 billion parameters but uses about 69 KiB per token — barely half of what Llama 3 8B needs.

Quantization/Dequantization . Storing the cache in FP8 instead of BF16 roughly halves it. Most serving frameworks support this with one flag. While storing it in KV cache, apply the quantization and store it in smaller version that reduces the memory. While projecting the K, Q and V matrix, apply the de-quantization and bring it back to original size and continue the calculation.

Quick reference

Memory per token, BF16, single sequence: