<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rinit Jain on Medium]]></title>
        <description><![CDATA[Stories by Rinit Jain on Medium]]></description>
        <link>https://medium.com/@rinitjain9?source=rss-82d3a738e07a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OHmcjqZDFBPlCrM1ReM_AA.jpeg</url>
            <title>Stories by Rinit Jain on Medium</title>
            <link>https://medium.com/@rinitjain9?source=rss-82d3a738e07a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 05:16:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rinitjain9/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Inside a Modern AI Inference Platform]]></title>
            <link>https://pub.towardsai.net/inside-a-modern-ai-inference-platform-8846434bb14b?source=rss-82d3a738e07a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8846434bb14b</guid>
            <category><![CDATA[dllm]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Rinit Jain]]></dc:creator>
            <pubDate>Fri, 11 Sep 2026 17:01:03 GMT</pubDate>
            <atom:updated>2026-09-15T13:48:57.906Z</atom:updated>
            <content:encoded><![CDATA[<h4>How GPUs, model servers, LLM-D and Kubernetes work together</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_obzZdPdDJ4FRTsOV3LPrQ.png" /></figure><blockquote><strong>TL; DR</strong></blockquote><blockquote>The whole stack exists to answer one question: <strong>where does a request’s state live, and what does it cost?</strong></blockquote><blockquote>Capacity is set by two independent ceilings — how many requests fit in memory, and how many the latency target allows. The lower one binds.</blockquote><blockquote>GPU utilisation is a misleading metric. Judge a deployment on <strong>goodput</strong>: requests completed within your latency targets.</blockquote><blockquote>Most production incidents are diagnosable from three symptoms — bad TTFT, bad TPOT, or stalling generations — each pointing at a different layer.</blockquote><blockquote>Cost per million tokens is driven overwhelmingly by <strong>concurrency and cache hit rate</strong>, not by hardware choice.</blockquote><blockquote>Build this in stages. Every layer should be forced by a measured limit, never adopted preemptively — and for many teams, a hosted API remains the right answer.</blockquote><p>Someone types a question and presses send. A second later, words start appearing.</p><p>Between those two moments sits a system with more layers than almost anything else in modern infrastructure: an API gateway, a router that understands token economics, an orchestrator managing GPU nodes, an inference engine juggling hundreds of concurrent sequences, a memory manager allocating attention state in pages, and a model whose parameters may be spread across a dozen accelerators.</p><p>This article does two things. First it lays out that stack quickly enough to be useful on its own — no prior reading assumed. Then it deals with the part that only matters once the architecture is settled: <strong>how you actually run it.</strong> Sizing it, measuring it, diagnosing it at three in the morning, paying for it, and deciding which pieces you need at all.</p><h3>Part 1 — The stack, in one pass</h3><p>Five layers, each solving a problem created by the one below.</p><h3>The model and the hardware</h3><p>A large language model is an architecture plus a very large set of learned parameters called weights. Memory needed is roughly parameters × bytes per parameter, so a 70-billion-parameter model in 16-bit precision needs about 140 GB — before anything else.</p><p>That memory has to be <strong>VRAM</strong>, the high-bandwidth memory attached to the GPU itself. System RAM is not a substitute; having 512 GB of it does not help an 80 GB accelerator. Four resources govern everything that follows: <strong>compute capacity</strong>, <strong>memory capacity</strong>, <strong>memory bandwidth</strong> (how fast data moves between VRAM and the compute units), and <strong>network bandwidth</strong> (how fast GPUs and machines exchange data).</p><h3>Making the model fit</h3><p>When weights exceed one GPU, there are four responses. <strong>Quantization</strong> stores parameters in fewer bits — 8-bit roughly halves the footprint against 16-bit, 4-bit halves it again, with some accuracy cost. <strong>Offloading</strong> keeps part of the model in CPU memory, which solves capacity at a steep bandwidth price. <strong>Sharding</strong> splits one model across several GPUs. <strong>Replication</strong> makes complete copies.</p><p>The last two are constantly confused and solve opposite problems: sharding addresses <em>model</em> capacity, replication addresses <em>workload</em> capacity. Sharding comes in flavours — tensor parallelism splits the operations inside each layer and demands a fast interconnect, so it suits GPUs within one machine; pipeline parallelism gives each GPU a contiguous group of layers and communicates far less, so it tolerates spanning machines.</p><h3>Serving many users from one model</h3><p>A request runs in two phases with opposite hardware appetites.</p><p><strong>Prefill</strong> processes the entire prompt at once. All prompt tokens are known, so the work is highly parallel — prefill is <strong>compute-bound</strong>.</p><p><strong>Decode</strong> generates the answer one token at a time, each token depending on all those before it. To produce a single token, the GPU must read the model’s entire weight set from memory. Decode is <strong>memory-bandwidth-bound</strong>, and the compute units sit largely idle waiting.</p><p>That last fact is the basis of the entire serving economy. If reading 140 GB of weights yields one token, it is wasteful — so the engine <strong>batches</strong>, processing the next token for many sequences in one pass. The same weight read now produces dozens of tokens. Modern engines use <strong>continuous batching</strong>, rebuilding the batch before every forward pass so finished requests leave and waiting ones join immediately.</p><p>Crucially, the weights are read-only and shared by everyone. What is per-request is the <strong>KV cache</strong>.</p><h3>The KV cache</h3><p>Attention requires, for each new token, the keys and values of every preceding token. Those never change once computed, so the engine stores them rather than recomputing. That store is the KV cache, and it is the dominant dynamic consumer of GPU memory.</p><p>Its size is fixed per token by the architecture and grows along two axes: context length and concurrency. For a large model it can run a few hundred kilobytes per token — meaning a hundred users with 8,000-token conversations can consume more memory than the model weights themselves.</p><p>Because sequence lengths are unpredictable, engines allocate it in small fixed-size <strong>blocks</strong> on demand rather than reserving a maximum-length region per request. That technique — <strong>PagedAttention</strong> — eliminates most of the waste and is where a large share of modern serving throughput comes from. When the pool fills, running requests get preempted.</p><p>The consequence people underestimate: <strong>concurrency is capped by cache memory, not compute.</strong> “How many users can this GPU serve?” is unanswerable without a context length attached.</p><h3>Reusing state across requests</h3><p>Production traffic is repetitive. Every request carries the same system prompt; every chat turn resends the whole conversation; many users query the same document.</p><p>Attention is causal, so a token’s state depends only on what precedes it — meaning two requests beginning with identical tokens produce an identical cache for that shared portion. <strong>Prefix caching</strong> retains it so the second request skips that prefill entirely. Engines identify reusable blocks by hashing each one together with the hash of everything before it, so a match implies the entire preceding sequence matched.</p><p>This is why variable content at the <em>front</em> of a prompt is expensive: a timestamp at the top changes the first block’s hash, which changes every hash after it, destroying every match.</p><h3>Routing, orchestration, and inference-aware platforms</h3><p>Once there are many replicas, classic load balancing fails. Its assumptions — that requests cost roughly the same, are short, and that servers are interchangeable — are all false here. One request can cost a hundred times another, occupy memory for a minute, and be nearly free on the one replica holding its cached prefix. An <strong>LLM-aware router</strong> scores candidates on cache locality, cache utilisation and queue depth measured in tokens.</p><p><strong>Kubernetes</strong> runs the fleet: placing servers on nodes with the right accelerators, replacing failures, gating traffic on readiness probes, scaling replicas. It never touches a token. Its blind spot is that it models an inference server as a generic container with GPUs attached.</p><p><strong>Inference-aware platforms</strong> such as <strong>llm-d</strong> close that gap, making a pool of model servers a first-class object with pluggable, state-aware scheduling — and going further by running prefill and decode on separate GPU pools, transferring cached state between them, and letting that state spill beyond one GPU into host memory, disk or shared cluster storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e6d2H2oeHaQRIvFMEY_cVQ.png" /><figcaption>LLM Inference Stack Observability Diagram</figcaption></figure><h3>Part 2 — Operating it</h3><p>Everything above is architecture. What follows is the part that decides whether it works.</p><h3>1. Sizing: how much hardware do you actually need?</h3><p>The first question anyone asks, and it has a real answer — with a catch.</p><p>Work down from total memory:</p><pre>Total VRAM                          640 GB   (8 × 80 GB)<br>− model weights (70B, FP16)        −140 GB<br>− runtime, buffers, overhead        − 40 GB<br>                                    ───────<br>= KV cache pool                      460 GB</pre><p>Then divide by what one request consumes. At roughly 320 KB per token for this model class, a request averaging 4,000 tokens of context needs about 1.3 GB:</p><pre>460 GB ÷ 1.3 GB  ≈  350 concurrent requests</pre><p>That is your <strong>memory ceiling</strong>. Now the catch: it is not the only ceiling.</p><p>Concurrency also has a <strong>latency ceiling</strong>. Larger batches raise throughput but slow every individual response, so if your TPOT target is 50 ms per token, there is a batch size beyond which you cannot meet it — regardless of available memory. That number comes from benchmarking, not arithmetic, because it depends on the model, the hardware and the request mix.</p><p><strong>Your real capacity is the lower of the two.</strong> Which one binds tells you what to do about it: memory-capped means quantize, shard, or shorten context limits; latency-capped means add replicas, because more memory on the same GPU will not help.</p><p>Then subtract headroom. A deployment sized exactly to peak has no room for a traffic spike, and inference replicas take <strong>minutes</strong> to start — image pull, weight loading, engine initialisation. You cannot scale into a spike reactively, so you either hold spare capacity or accept degradation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SqFiKTai6Xb6A0YY0Se_XA.png" /><figcaption>GPU Capacity and Concurrency Infographic</figcaption></figure><h3>2. Setting targets</h3><p>Three numbers describe user experience, and a single average conceals all of them.</p><p><strong>TTFT — time to first token.</strong> How long before anything appears. Driven by queueing and prefill. This determines whether the interface feels alive.</p><p><strong>TPOT — time per output token.</strong> How fast the answer reads out once started. Driven by decode. Comfortable reading is roughly 30–50 ms per token; below that, further gains go unnoticed.</p><p><strong>ITL — inter-token latency.</strong> The gap between individual streamed tokens. Related to TPOT but exposes <em>variance</em> — the stutters caused by other requests’ prefill work, which an average hides completely.</p><p>The number that ties them together is <strong>goodput</strong>: requests completed <em>while meeting your targets</em>. A server reporting excellent token throughput while every user waits eight seconds for a first token has superb throughput and terrible goodput.</p><blockquote>Throughput measures what the hardware did. Goodput measures what the users got. Optimise the second.</blockquote><p>Set targets by workload, not by aspiration. Interactive chat needs fast TTFT and can tolerate moderate TPOT. Batch document processing barely cares about either and should be tuned purely for throughput. Voice needs both to be tight. Running these on shared infrastructure with one configuration means serving all of them badly.</p><h3>3. What belongs on a dashboard</h3><p>Start with what does <em>not</em>: <strong>GPU utilisation</strong>.</p><p>That figure reports whether any kernel was executing, not whether the compute units did anything useful. A GPU in a decode loop, stalled on memory reads with most of its arithmetic idle, reports near-100% happily. High utilisation with low token throughput is not a healthy server — it is a bandwidth-starved one, and the fix is a bigger batch, not a bigger GPU.</p><p>What to graph instead:</p><p><strong>KV cache utilisation, per replica.</strong> The best single indicator of how close a server is to trouble, because it is the resource that actually runs out. It is also your load-balance check — a wide spread across replicas means the router is not seeing real load.</p><p><strong>Queue depth in tokens, not requests.</strong> Five queued requests of 200 tokens is nothing; five of 30,000 is a wall everyone behind will wait through.</p><p><strong>Preemption rate.</strong> Non-zero means the cache pool is exhausted and running requests are being evicted. This is the clearest signal that you are past capacity.</p><p><strong>Prefix cache hit rate.</strong> If it is far below what a single server achieves, routing is destroying locality. If it is near zero on repetitive traffic, look for variable content at the front of your prompts.</p><p><strong>TTFT and TPOT at p95 and p99, never the mean.</strong> Failures live in the tail. A router that is right 90% of the time and catastrophic otherwise looks fine on average.</p><p><strong>Tokens per second, input and output separately.</strong> They represent different work and different costs.</p><h3>4. Benchmarking before traffic exists</h3><p>You cannot size a deployment from a spec sheet, because capacity depends on your request shapes.</p><p>Benchmark with <strong>your</strong> distribution of prompt and output lengths. A model benchmarked at 512-in/128-out will behave nothing like the same model serving 20,000-in/500-out, and the published number will mislead you badly.</p><p>Sweep request rate upward and watch where latency breaks. The useful output is not a single throughput figure but a <strong>curve</strong>: goodput against load, with the knee marked. That knee is your operating point, and the gap between it and your provisioned capacity is your headroom.</p><p>Two mistakes to avoid. Benchmarking with an unrealistically high cache hit rate — sending the same prompt repeatedly gives you numbers you will never see in production. And benchmarking at fixed concurrency rather than fixed arrival rate, which hides queueing entirely, since queueing is exactly what happens when arrivals exceed service.</p><h3>5. Diagnosis: it is slow, what do you check?</h3><p>Most incidents present as one of a few symptoms, and each points at a specific layer.</p><p><strong>TTFT is high, TPOT is fine.</strong> The problem is before generation: queueing, admission or prefill. Check queue depth in tokens, then prefix cache hit rate — a dropped hit rate turns cheap requests into expensive ones instantly. Then check load balance; one saturated replica while others idle is a routing problem, not a capacity problem.</p><p><strong>TPOT is high, or output stutters.</strong> Generation itself is slow. Either the batch is too large — throughput was traded for latency — or large prefills are interrupting decode. Chunked prefill smooths the second; reducing concurrency addresses the first.</p><p><strong>Requests stall mid-generation or fail under load.</strong> Cache exhaustion, almost certainly. Check preemption rate and cache utilisation. Fixes are to reduce concurrency, cap maximum context, quantize the KV cache, or add capacity.</p><p><strong>Throughput is low but GPU utilisation is high.</strong> Bandwidth-bound with too small a batch. Raise concurrency limits. This is the classic misread that leads teams to buy more GPUs they do not need.</p><p><strong>One replica is hot, others idle.</strong> Routing. Either it is counting connections instead of measuring load, or session affinity has concentrated traffic on one server.</p><p><strong>Everything degraded right after a deploy or scale-up.</strong> Cold caches. New replicas start with nothing, and warm up over minutes. Expected, temporary, and worth confirming rather than chasing.</p><p><strong>Pods stuck Pending.</strong> GPU fragmentation. A Pod requesting eight GPUs needs eight free on one node; sixteen scattered two-per-node across eight machines will not do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IsKPSZYFn51UAOQZky-hOw.png" /><figcaption>Inference Performance Troubleshooting Matrix</figcaption></figure><h3>6. What it costs, and what moves the number</h3><p>GPUs are the dominant cost, and the metric that matters is <strong>cost per million tokens</strong>:</p><pre>                          GPU cost per hour<br>cost per M tokens  =  ───────────────────────  × 1,000,000<br>                       tokens produced per hour</pre><p>The numerator is largely fixed once you have chosen hardware. Almost all the leverage is in the denominator — which means cost optimisation is throughput optimisation.</p><p>In rough order of impact:</p><p><strong>Concurrency.</strong> The single largest lever. Running at batch sizes near your latency limit rather than well below it can change the number several-fold, because the expensive weight read is amortised across everything in the batch. A GPU serving 20 concurrent requests when it could serve 200 costs roughly ten times more per token.</p><p><strong>Prefix cache hit rate.</strong> In workloads with large shared prompts, hits eliminate most of the prefill work. Restructuring prompts so stable content sits at the front is close to free and can be worth a great deal.</p><p><strong>Model size and precision.</strong> A smaller or quantized model produces more tokens per GPU-hour. The real question is whether quality holds for your task — often it does, and the saving is large.</p><p><strong>Idle provisioned capacity.</strong> Headroom held for spikes is paid for continuously. This is a real and unavoidable cost, but it should be a deliberate number, not an accident.</p><p><strong>Context length.</strong> Every token of context consumes cache memory for the life of the request, reducing how many requests fit. Trimming conversation history is a direct capacity gain.</p><p>Note what is <em>not</em> on this list: choosing a marginally faster accelerator. Hardware matters, but a well-tuned deployment on modest hardware routinely beats a poorly-tuned one on better hardware, because the difference between batch-of-20 and batch-of-200 is larger than the difference between GPU generations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rN1OPiRBEkWylLqscWLbzA.png" /><figcaption>Cost Per Million Tokens Infographic</figcaption></figure><h3>7. Should you be running this at all?</h3><p>Worth asking before the adoption path, because for many teams the honest answer is no.</p><p>Hosted APIs remove every problem in this article. No GPUs to provision, no cache to tune, no Pods stuck Pending at midnight. You pay per token and someone else runs the platform. For most applications, at most volumes, that is the correct choice — and the engineering time saved is usually worth more than the unit-price difference.</p><p>Self-hosting starts to make sense on a few specific grounds. <strong>Volume</strong>, once you are large enough that per-token pricing exceeds the fully-loaded cost of running your own hardware, including the people who operate it. <strong>Data control</strong>, where requests cannot leave your infrastructure for regulatory or contractual reasons. <strong>Model control</strong>, where you need a fine-tuned or open-weight model that no provider hosts. <strong>Predictable, shaped traffic</strong>, where you can keep GPUs busy — idle accelerators are the fastest way to make self-hosting more expensive than an API. And <strong>latency</strong>, where you need the model physically close to something.</p><p>The trap is underestimating the operational cost. GPUs are the visible expense; the engineering time to tune, monitor, upgrade and debug this stack is the one that gets left out of the comparison and then dominates it.</p><blockquote>Self-host because you have a reason that survives arithmetic, not because it feels like the serious choice.</blockquote><h3>8. Where to start</h3><p>The stack described in Part 1 is what a large deployment looks like after years of pressure. Building it upfront is a mistake. Each layer is complexity, and complexity is only worth paying for when a measured limit forces it.</p><p>A reasonable order:</p><p><strong>Start with one model server on one GPU.</strong> Enable prefix caching. Measure your actual prompt and output distributions — most teams are surprised by them.</p><p><strong>Tune before you scale.</strong> Adjust memory allocation and concurrency limits until you reach either the memory or latency ceiling. Many deployments stop here and should.</p><p><strong>If the model does not fit, make it fit.</strong> Quantize first — it is cheaper than more hardware and often costs nothing in practice. Shard only if it still does not fit.</p><p><strong>If one server is not enough, add replicas</strong> behind a simple load balancer. Confirm the fleet-wide cache hit rate has not collapsed; if it has, that is your signal to add cache-aware routing.</p><p><strong>Add Kubernetes when operating the fleet by hand becomes the bottleneck</strong> — when you have enough replicas that failures, rollouts and placement need automating.</p><p><strong>Add inference-aware orchestration when the workarounds hurt more than the platform would.</strong> Cache-aware routing when locality is being wasted. Disaggregation when prefill interference is visibly damaging inter-token latency and you have the interconnect to support moving cache between machines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GGwsTIsBd3M5HmIwfLtCyw.png" /><figcaption>Measurement-Driven Adoption Ladder</figcaption></figure><blockquote>Every layer in this stack should be forced by a number you measured, not adopted because it appeared in an architecture diagram.</blockquote><h3>9. Mistakes that show up repeatedly</h3><p><strong>Sizing on model weights alone.</strong> The most common error in the field. A 140 GB model does not fit comfortably on 160 GB of VRAM, because concurrent requests need cache and there will be almost none left.</p><p><strong>Optimising the average.</strong> Users experience the tail. A median TTFT of 200 ms with a p99 of nine seconds is a bad experience for a meaningful number of people every hour.</p><p><strong>Treating GPU utilisation as the health metric.</strong> Covered above, and worth repeating because it drives real purchasing decisions.</p><p><strong>Injecting variable content at the top of prompts.</strong> A session ID above the system prompt can cost you the entire prefix cache.</p><p><strong>Aggressive scale-down.</strong> Removing a replica discards a warm cache that costs minutes to rebuild. Scale down slowly and conservatively.</p><p><strong>One configuration for mixed workloads.</strong> Interactive and batch traffic want opposite tuning. Separate pools serve both well; one shared pool serves both badly.</p><p><strong>Liveness probes that fire during startup.</strong> An inference Pod takes minutes to load weights. A short liveness timeout kills it mid-load, forever, and looks like a broken image rather than a misconfigured probe.</p><h3>The one idea</h3><p>Every layer of this stack is an answer to the same question: <strong>where does a request’s state live, and what does it cost?</strong></p><p><strong>Model weights</strong> are state that must fit in memory — which gives us <strong>quantization </strong>and <strong>sharding</strong>. <strong>Attention state</strong> is per-request state that must be stored rather than recomputed — which gives us the <strong>KV cache</strong>, and <strong>paged allocation</strong> to store it efficiently. That state turns out to be shareable between requests — which gives us <strong>prefix caching</strong>. Being shareable makes its <em>location</em> valuable — which gives us<strong> cache-aware routing</strong>. Being located on a specific Pod makes it fragile — which is why a restart is expensive in a way orchestrators do not model. And making it survive, move and tier across a cluster is precisely what inference-aware platforms are for.</p><p>Compute is rarely the limit. Memory, and the movement of state through it, almost always is.</p><p>That is also the practical takeaway. When a deployment misbehaves, the productive question is not “do we need more GPUs?” It is: <strong>which resource is exhausted, and where is state going that it should not?</strong> Is the cache pool full? Is a cache being recomputed that already existed somewhere? Is a phase of the request interfering with another? Nearly every answer in this series is a specific instance of that question.</p><p>The tools will change. Engines will get faster, orchestration will absorb more of what is bolted on today, and some of the specific projects named here will be superseded. The four resources will not change, and neither will the fact that a request’s cost is determined by its state.</p><p>You now have the map. The rest is measurement.</p><h3>The series</h3><p><strong>1. </strong><a href="https://pub.towardsai.net/what-really-happens-when-you-click-send-on-chatgpt-2307e3ede14e"><strong>What really happens when you click ‘Send’ on ChatGPT</strong></a> — A journey through modern AI Infrastructure</p><p><strong>2. </strong><a href="https://pub.towardsai.net/what-do-you-do-with-a-model-thats-too-big-for-your-gpu-443bbbe70d4d"><strong>What Do You Do With a Model That’s Too Big for Your GPU?</strong></a> — Quantization, Sharding and Parallelism Explained</p><p><strong>3. </strong><a href="https://pub.towardsai.net/how-does-one-gpu-serve-hundreds-of-users-at-the-same-time-b999e60a440e"><strong>How Does One GPU Serve Hundreds of Users at the Same Time?</strong> </a>— Inside an LLM inference server</p><p><strong>4. </strong><a href="https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7"><strong>The KV Cache Explained: Why Long Conversations Get Expensive</strong></a> — How LLMs remember context without recomputing everything</p><p><strong>5. </strong><a href="https://pub.towardsai.net/why-is-your-llm-recomputing-the-same-prompt-1-000-times-a-day-675e0cdab4b3"><strong>Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?</strong></a> — Prefix caching, radix trees and block hashing explained</p><p><strong>6. </strong><a href="https://pub.towardsai.net/why-traditional-load-balancing-breaks-for-llms-40311f7b29e2"><strong>Why Traditional Load Balancing Breaks for LLMs</strong></a> — Building an LLM-aware router</p><p><strong>7. </strong><a href="https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c"><strong>Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster</strong></a></p><p><strong>8. </strong><a href="https://pub.towardsai.net/llm-d-explained-69f7d0b2b035"><strong>LLM-D Explained</strong></a> — How modern AI infrastructure routes, schedules and scales LLM inference</p><p><strong>9. </strong><a href="https://pub.towardsai.net/inside-a-modern-ai-inference-platform-8846434bb14b"><strong>Inside a Modern AI Inference Platform</strong></a> — The full stack end-to-end</p><p><strong>Sources</strong></p><ul><li><a href="https://docs.vllm.ai/">vLLM documentation</a></li><li><a href="https://arxiv.org/abs/2309.06180">Efficient Memory Management for LLM Serving with PagedAttention</a></li><li><a href="https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/">NVIDIA — Mastering LLM Techniques: Inference Optimization</a></li><li><a href="https://llm-d.ai/docs/architecture/architecture">llm-d — Architecture overview</a></li><li><a href="https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/">Kubernetes — Schedule GPUs</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8846434bb14b" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/inside-a-modern-ai-inference-platform-8846434bb14b">Inside a Modern AI Inference Platform</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM-D Explained]]></title>
            <link>https://pub.towardsai.net/llm-d-explained-69f7d0b2b035?source=rss-82d3a738e07a------2</link>
            <guid isPermaLink="false">https://medium.com/p/69f7d0b2b035</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[dllm]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Rinit Jain]]></dc:creator>
            <pubDate>Fri, 11 Sep 2026 15:01:03 GMT</pubDate>
            <atom:updated>2026-09-15T13:48:22.068Z</atom:updated>
            <content:encoded><![CDATA[<h4>How modern AI infrastructure routes, schedules and scales LLM inference</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eazOz8MHZPImxjIwwm7H8Q.png" /></figure><blockquote><strong>TL; DR</strong></blockquote><blockquote>Kubernetes knows which Pods exist but nothing about what is inside them. A custom router knows what is inside but cannot create or scale them. llm-d closes that gap.</blockquote><blockquote>It is a <strong>layer</strong>, not a replacement: vLLM stays the engine, Kubernetes stays the orchestrator, and llm-d makes the orchestration inference-aware.</blockquote><blockquote>The <strong>InferencePool</strong> makes a group of model servers a first-class Kubernetes object, and an <strong>endpoint picker</strong> replaces round-robin with scheduling that scores prefix-cache locality, KV cache load and SLA.</blockquote><blockquote><strong>Prefill/decode disaggregation</strong> runs the two phases on separate GPU pools so each can be sized for the resource it actually consumes — at the cost of transferring KV cache between them.</blockquote><blockquote>Cache stops being trapped in one Pod’s VRAM. It can be offloaded to host memory and disk, or shared across instances with a global index.</blockquote><blockquote>This is scale machinery. Below a certain size it is complexity you do not need.</blockquote><p>A production LLM deployment ends up with two components that each hold half the truth.</p><p>Kubernetes knows which inference servers exist, which are ready, and where they run. It knows nothing about what is happening inside them — not tokens, not batches, not cache occupancy. To the control plane, a server at 95% cache utilisation and one at 10% are two healthy Pods.</p><p>A custom router knows exactly that missing half. It tracks queue depth, cache pressure and which replicas hold useful cached prefixes. It cannot create a server, replace a dead one, or add capacity.</p><p>So the platform’s two most important decisions are made by components that cannot see each other’s information. Kubernetes terminates a Pod during a rollout without knowing it held the hottest cache in the fleet. Its built-in load balancing distributes by connection, which is precisely the behaviour that fails for inference — so you bypass it with a router bolted on beside the platform rather than built into it.</p><p><strong>llm-d</strong> is an attempt to remove those workarounds by making the orchestration layer itself understand inference.</p><h3>1. What llm-d is, and what it is not</h3><p>llm-d is a Kubernetes-native distributed inference serving stack, built as a layer on top of three things that already exist: <strong>vLLM</strong> as the inference engine, <strong>Kubernetes</strong> as the orchestrator, and the <strong>Gateway API Inference Extension</strong> as the routing foundation. It is open source under Apache 2.0, built by people from the Kubernetes and vLLM projects.</p><p>The clarifications matter, because the name suggests more replacement than there is:</p><p><strong>It does not replace vLLM.</strong> Batching, the KV cache, PagedAttention, prefix caching — all of that continues to happen inside vLLM exactly as before. llm-d decides which vLLM instance handles a request and how those instances are deployed.</p><p><strong>It does not replace Kubernetes.</strong> Pods, Deployments, scheduling, probes and Services all still apply. llm-d adds inference-specific objects and behaviour on top.</p><p><strong>It does not make a single GPU faster.</strong> Everything it offers is about coordinating many servers.</p><blockquote>vLLM decides how a request uses a GPU. Kubernetes decides where servers run. llm-d decides which server, in what role, holding what state.</blockquote><h3>2. The missing Kubernetes object</h3><p>Start with the most concrete gap.</p><p>A Kubernetes <strong>Service</strong> groups Pods behind a stable address and distributes traffic among them, effectively round robin. It works because it assumes the Pods are interchangeable — which, once each one holds a different cache, they are not.</p><p>The <strong>Gateway API Inference Extension</strong> adds an object built for this: the <strong>InferencePool</strong>. It represents a group of model-serving Pods as a first-class thing the platform understands, with an associated model and an associated way of choosing between its members.</p><p>The important difference is not the grouping. It is that an InferencePool comes with a pluggable decision-making component instead of fixed round-robin behaviour:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K22PIfCuQUUB3LVfNn0Deg.png" /><figcaption>Kubernetes Routing: Round Robin vs Scored Inference</figcaption></figure><p>That component is called the <strong>endpoint picker</strong>, and it is where the routing logic from a hand-rolled router now lives — inside the platform, using standard Kubernetes objects, rather than beside it.</p><h3>3. Scheduling that knows what a request costs</h3><p>llm-d’s <strong>inference scheduler</strong> is its implementation of the endpoint picker, tuned for vLLM. It works in two stages that are worth separating because they answer different questions.</p><p><strong>Filters</strong> eliminate candidates that cannot serve the request at all — wrong model, not ready, wrong role. This is correctness.</p><p><strong>Scorers</strong> rank the survivors. This is optimisation, and it is where the signals become concrete. The scheduler uses operational telemetry from vLLM to make decisions that are aware of prefix-cache state, KV cache load, request role, and service-level objectives.</p><p>In other words, exactly the signals that make LLM routing work — where the cached prefix already lives, how close a server is to running out of cache memory, how backed up it is — but sourced from the engine through a defined interface rather than scraped by something you built yourself. Teams can add their own scorers where the defaults do not fit.</p><p>A fair question at this point: how does the scheduler know what any given Pod holds in its cache? It does not inspect them per request — that would be far too slow for something on the critical path. Instead the workers continuously report their own state, and the scheduler maintains a running picture from that stream, supplemented by knowing which prefixes it has recently sent where. This means its view is always slightly out of date, which is why locality is treated as a score to weigh rather than a fact to obey.</p><p>The practical difference is not that better decisions become possible. It is that they stop being bespoke. The routing intelligence becomes a component of the platform with a supported way to extend it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fVXjV3PfeVmcWndgTMP0jA.png" /><figcaption>Scheduling Pipeline: Filter, Score, Select</figcaption></figure><h3>4. Splitting the request in two</h3><p>Now the part that changes deployment architecture rather than just routing.</p><p>A request has two phases with opposite hardware appetites. <strong>Prefill</strong> processes the whole prompt at once — a large burst of parallel work that is limited by compute. <strong>Decode</strong> generates one token at a time, reading the model’s weights from memory on every step, limited by memory bandwidth.</p><p>Run both on the same server, as a normal deployment does, and two problems follow.</p><p><strong>Neither phase gets hardware suited to it.</strong> The server must be adequate at both, so it is optimal for neither.</p><p><strong>They interfere.</strong> A large prefill occupies the GPU long enough that every request currently generating on that machine stutters. One user submitting a long document degrades inter-token latency for everyone else on that server.</p><p><strong>Disaggregation</strong> separates them: one pool of GPUs runs prefill workers, another runs decode workers, and a request visits both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vSWZxBMRT21f1HRRh8xUqw.png" /><figcaption>Aggregated vs Disaggregated GPU Architectures</figcaption></figure><p>Each pool can now be sized, tuned and scaled for the resource it actually consumes, and prefill work no longer interrupts anyone’s token stream. The reported gains are meaningful — throughput improvements on the order of tens of percent on identical hardware, and better inter-token latency because the interference is gone.</p><p>One thing that surprises people: <strong>both pools still hold the full model.</strong> A prefill worker and a decode worker each run the complete set of weights, because each performs a full forward pass — one over many token positions, the other over a single position. Splitting the phases does not split the model. So disaggregation does not save weight memory, and a naive split can even use more GPUs than the aggregated deployment it replaced. The gain comes from specialisation and from removing interference, not from a smaller footprint.</p><h3>5. The catch: the cache has to move</h3><p>If prefill happens on one machine and decode on another, something must travel between them.</p><p>Prefill’s entire output is the KV cache — the attention state for every prompt token. Decode cannot start without it. So disaggregation requires transferring that cache from the prefill worker’s GPU memory to the decode worker’s, and for a long prompt that is gigabytes.</p><p>This is the whole engineering problem of disaggregation, and it is why the idea is recent despite being obvious. It only works with a transfer path fast enough that moving the cache costs less than the interference it avoids. llm-d uses high-performance transport for this — NVIDIA’s NIXL over fast interconnects such as InfiniBand and RDMA — rather than ordinary networking.</p><p>The consequence is a real precondition, not a footnote:</p><blockquote>Disaggregation pays off when prompts are long enough that avoided interference outweighs the transfer, and when the interconnect is fast enough to make the transfer cheap. Neither is automatic.</blockquote><p>For short prompts on commodity networking, an aggregated deployment can easily be the better choice. This is scale machinery, and it assumes infrastructure to match.</p><h3>6. A request, end to end</h3><p>Putting it together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CY9ueAuQLgXf2yM1vgfvag.png" /><figcaption>LLM Inference Routing and Feedback Flow</figcaption></figure><p>Step 3 is LLM-aware routing, now inside the platform. Steps 4 and 5 are the disaggregation. Step 7 is what keeps the scheduler’s picture current.</p><h3>7. Cache that outlives the Pod</h3><p>This is the most consequential idea in llm-d, and it resolves a limitation that has been accumulating through every layer below.</p><p>Until now, cached attention state has been trapped. It lives in one Pod’s GPU memory. It cannot be used by another replica. It is evicted when that GPU’s pool fills. And it is destroyed entirely when the Pod restarts — a replacement Pod is functionally identical and economically much worse until it warms up.</p><p>llm-d uses vLLM’s <strong>KVConnector</strong> interface to make the cache a pluggable, layered resource rather than a fixed property of one GPU. Two schemes are described:</p><p><strong>Independent offloading.</strong> A Pod’s cache spills beyond its GPU into host memory and local disk. Entries evicted from VRAM are demoted rather than deleted, and can be pulled back faster than recomputing them. This is local, cheap to operate, and adds capacity without adding GPUs.</p><p><strong>Shared caching.</strong> Cache is transferred between instances and backed by shared storage with a <strong>global index</strong>, so a prefix computed by one worker can be found and reused by another.</p><p>That second one is the significant shift. It means the cluster can know where cached state lives, and the scheduler can route to it — or fetch it — rather than recomputing it because it happened to land on the wrong machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YhW6pLBshxBNyIryqQZ7MQ.png" /><figcaption>Multi-Tier Cache Hierarchy Diagram</figcaption></figure><p>The trade-off is stated plainly in the design: shared caching offers higher performance at the cost of a considerably more complex system to operate. Independent offloading is the low-effort option; the shared, globally indexed cache is the high-ceiling one.</p><p>Worth tracing the arc, because it is the through-line of this entire stack. Attention state began as a within-request optimisation, so a token would not be recomputed on every step. Prefix caching made it reusable across requests on one server. Cache-aware routing made its location a factor in where requests go. And here it becomes a cluster-level resource — tiered, transferable, and able to outlive the process that produced it.</p><h3>8. Scaling roles, not just replicas</h3><p>Splitting prefill and decode creates a scaling problem that ordinary autoscaling cannot express.</p><p>The two pools consume different resources and are stressed by different traffic. A workload of long documents and short answers is prefill-heavy. Long conversational generation is decode-heavy. And the correct <strong>ratio</strong> between the pools depends on that mix, which changes through the day.</p><p>Scaling on a single aggregate metric cannot handle this — adding replicas to the wrong pool spends money without relieving the bottleneck. What is needed is an autoscaler that measures each instance type’s capacity, understands that different request shapes impose different loads, observes the current traffic mix, and computes the right combination of prefill and decode capacity to meet the SLO.</p><p>llm-d describes exactly this as <strong>variant autoscaling</strong>, and it is an area still under active development — worth knowing as the direction rather than treating as settled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HJUS1ibW6ljQ_A3t0P_M0A.png" /><figcaption>Autoscaling Dashboard: Traffic Mix and Pool Ratios</figcaption></figure><h3>9. What breaks</h3><p>Disaggregation adds moving parts, and it is worth knowing where the new failure modes are.</p><p><strong>A decode worker dies mid-generation.</strong> Its KV cache goes with it, and in-flight responses on that worker fail. Prefill work already done for those requests is lost too — the same class of failure as before, now with an extra hop involved.</p><p><strong>A prefill worker dies before transfer.</strong> The request has produced nothing usable and must be retried from the beginning.</p><p><strong>The transfer path degrades.</strong> Because cache movement sits on the critical path between phases, interconnect problems appear as latency rather than as errors — which makes them harder to attribute. A disaggregated deployment on a congested network can perform worse than the aggregated one it replaced.</p><p><strong>The cache index goes stale.</strong> Shared caching depends on knowing where entries live. If that view drifts, the scheduler routes for hits that are not there, and the cost is silent: correct answers, quietly recomputed.</p><p>None of these is disqualifying, but together they make the point that this architecture buys performance with operational complexity.</p><h3>10. Do you actually need this?</h3><p>An honest answer, because the ecosystem is loud.</p><p><strong>Probably not, if</strong> you run one model on a handful of GPUs, your traffic is modest, prompts are short, or you have no fast interconnect. A single vLLM deployment behind a simple router will serve you well, and it is far easier to operate.</p><p><strong>Probably yes, if</strong> you run many replicas across many GPUs, your workload has large shared prefixes, prompt and output lengths vary widely, you have fast interconnect between nodes, and GPU cost is large enough that a double-digit efficiency gain is worth real operational effort.</p><p>llm-d is not the only implementation of these ideas — NVIDIA’s Dynamo pursues a similar architecture from a different angle. What they share is the underlying claim, and that claim is the durable part:</p><blockquote>At scale, inference cannot be treated as a stateless workload behind a load balancer. Where state lives has to become a first-class input to routing, scheduling and scaling.</blockquote><p>The specific projects will change. That conclusion will not.</p><h3>The architecture is complete</h3><p>Step back and look at what has been built.</p><p>A model too large for one GPU is quantized or sharded, its parameters spread across devices that coordinate through fast interconnect. An inference engine loads it once and serves many users at a time, batching their work continuously so that one expensive read of the weights produces many tokens. It keeps attention state in paged blocks so memory is not wasted, and retains prefixes so repeated prompts are not recomputed. A scheduler routes each request to the server most likely to serve it well, weighing cached state against memory pressure. Kubernetes runs the fleet — placing servers on the right hardware, replacing failures, gating traffic on readiness. And an inference-aware layer ties routing, scheduling and cache into one system that understands the workload, splitting requests by phase and letting cached state move between machines.</p><p>Every architectural question this series raised now has an answer.</p><p>Which leaves the question that only appears once the architecture is done. Building this system and <strong>operating</strong> it are different problems. The remaining questions are not about what to build:</p><p>How do you size it before traffic exists? What do you put on a dashboard, given that GPU utilisation is misleading and averages hide the failures that matter? What does it cost per million tokens, and which decisions move that number most? When it degrades at three in the morning, how do you find out which of these layers is responsible?</p><p><strong>How do you run this thing?</strong></p><p>That is the last piece.</p><h3>The series</h3><p><strong>1. </strong><a href="https://pub.towardsai.net/what-really-happens-when-you-click-send-on-chatgpt-2307e3ede14e"><strong>What really happens when you click ‘Send’ on ChatGPT</strong></a> — A journey through modern AI Infrastructure</p><p><strong>2. </strong><a href="https://pub.towardsai.net/what-do-you-do-with-a-model-thats-too-big-for-your-gpu-443bbbe70d4d"><strong>What Do You Do With a Model That’s Too Big for Your GPU?</strong></a> — Quantization, Sharding and Parallelism Explained</p><p><strong>3. </strong><a href="https://pub.towardsai.net/how-does-one-gpu-serve-hundreds-of-users-at-the-same-time-b999e60a440e"><strong>How Does One GPU Serve Hundreds of Users at the Same Time?</strong> </a>— Inside an LLM inference server</p><p><strong>4. </strong><a href="https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7"><strong>The KV Cache Explained: Why Long Conversations Get Expensive</strong></a> — How LLMs remember context without recomputing everything</p><p><strong>5. </strong><a href="https://pub.towardsai.net/why-is-your-llm-recomputing-the-same-prompt-1-000-times-a-day-675e0cdab4b3"><strong>Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?</strong></a> — Prefix caching, radix trees and block hashing explained</p><p><strong>6. </strong><a href="https://pub.towardsai.net/why-traditional-load-balancing-breaks-for-llms-40311f7b29e2"><strong>Why Traditional Load Balancing Breaks for LLMs</strong></a> — Building an LLM-aware router</p><p><strong>7. </strong><a href="https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c"><strong>Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster</strong></a></p><p><strong>8. </strong><a href="https://pub.towardsai.net/llm-d-explained-69f7d0b2b035"><strong>LLM-D Explained</strong></a> — How modern AI infrastructure routes, schedules and scales LLM inference</p><p><strong>9. </strong><a href="https://pub.towardsai.net/inside-a-modern-ai-inference-platform-8846434bb14b"><strong>Inside a Modern AI Inference Platform</strong></a> — The full stack end-to-end<br>(Next Article)</p><p><strong>Sources</strong></p><ul><li><a href="https://llm-d.ai/docs/architecture/architecture">llm-d — Architecture overview</a></li><li><a href="https://llm-d.ai/docs/architecture/Components/inf-scheduler">llm-d — Inference scheduler</a></li><li><a href="https://llm-d.ai/docs/architecture/Components/disagg_prefill-decode">llm-d — Disaggregated prefill/decode</a></li><li><a href="https://gateway-api-inference-extension.sigs.k8s.io/">Kubernetes Gateway API Inference Extension</a></li><li><a href="https://developers.redhat.com/articles/2025/05/20/llm-d-kubernetes-native-distributed-inferencing">Red Hat Developer — llm-d: Kubernetes-native distributed inferencing</a></li><li><a href="https://developer.nvidia.com/blog/deploying-disaggregated-llm-inference-workloads-on-kubernetes/">NVIDIA — Deploying disaggregated LLM inference workloads on Kubernetes</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69f7d0b2b035" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/llm-d-explained-69f7d0b2b035">LLM-D Explained</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster]]></title>
            <link>https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c?source=rss-82d3a738e07a------2</link>
            <guid isPermaLink="false">https://medium.com/p/424f0dcfab8c</guid>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[llm-inference]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Rinit Jain]]></dc:creator>
            <pubDate>Wed, 09 Sep 2026 21:01:00 GMT</pubDate>
            <atom:updated>2026-09-15T13:53:28.821Z</atom:updated>
            <content:encoded><![CDATA[<h4>How AI workloads actually run across a GPU cluster</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4YJQKEH9stIACHC0i9380A.png" /></figure><blockquote><strong>TL; DR</strong></blockquote><blockquote>Kubernetes does not make inference faster. It runs, places, monitors, replaces and scales the servers that do — and publishes a trustworthy list of which ones are ready.</blockquote><blockquote>GPUs are allocated to Pods as <strong>whole, indivisible units</strong> through vendor device plugins. This makes GPU scheduling behave very differently from CPU scheduling.</blockquote><blockquote>A Pod cannot span machines, so a single-node model replica is one Pod — but a replica too large for one machine becomes a group of Pods that must be created and replaced together.</blockquote><blockquote>An inference Pod takes <strong>minutes</strong> to become ready, because it must pull a large image and load model weights into VRAM. Nearly every operational difference follows from this.</blockquote><blockquote><strong>Readiness probes</strong> are what gate traffic, and they are the mechanism that keeps requests away from a server still loading.</blockquote><blockquote>When a Pod dies, its KV cache and prefix cache die with it. The replacement is correct but cold.</blockquote><blockquote>Kubernetes autoscaling on CPU is meaningless here; the useful signals come from the inference server itself.</blockquote><p>A router that makes good decisions needs an accurate list of inference servers: which exist, which are healthy, which are ready to receive traffic. In production that list is never stable. Replicas are added under load and removed when it drops. Servers crash. Deployments roll out new model versions. Hardware fails and nodes are drained.</p><p>Something has to own that churn — create the servers, place them on machines with the right GPUs, watch their health, replace the dead ones, and publish an accurate picture of what is running.</p><p>That is orchestration, and in practice it means <strong>Kubernetes</strong>. This article covers what it actually does for an inference deployment, and — more usefully — the specific places where GPU and LLM workloads behave unlike the web services Kubernetes was designed around.</p><h3>1. What Kubernetes is, and what it is not</h3><p>Kubernetes is a system for running containerised workloads across a cluster of machines. You describe the state you want — five replicas of this server, each with four GPUs, reachable at this address — and it works continuously to make reality match that description.</p><p>The important clarification first, because it is the most common misconception:</p><blockquote>Kubernetes does not participate in inference. It never touches a token, a batch or a KV cache. It starts the process that does, gives it hardware, and gets out of the way.</blockquote><p>Your throughput comes from the inference engine. Your memory efficiency comes from the engine’s cache management. Kubernetes contributes none of that. What it contributes is everything around it: placement, health, replacement, scaling, networking and rollout.</p><p>If you run one model on one server, you do not need it. The value appears when you have many GPUs, several models, replicas that fail, and traffic that changes.</p><h3>2. The vocabulary, kept minimal</h3><p>Five terms are enough to follow the rest.</p><p><strong>Cluster</strong> — a set of machines managed together, plus a <strong>control plane</strong> that holds the desired state and drives the cluster toward it. When we say “Kubernetes decides”, we mean the control plane.</p><p><strong>Node</strong> — one machine. Some have GPUs, most do not.</p><p><strong>Container</strong> — your inference server and its dependencies packaged into an image.</p><p><strong>Pod</strong> — the smallest thing Kubernetes schedules. One or more containers sharing a network identity and lifecycle, placed on a single node. For inference, a Pod is typically one container running vLLM or a similar engine. A Pod is not a server you maintain; it is a disposable instance of one.</p><p><strong>Deployment</strong> — a declaration of how many identical Pods should exist. Ask for five, and the control plane maintains five, replacing any that die.</p><p>Two more appear later: <strong>Service</strong>, which gives a stable address over a changing set of Pods, and <strong>probes</strong>, which report whether a Pod is healthy and ready.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rKTfz4LMgpL9YMjj0yv_gQ.png" /><figcaption>Kubernetes GPU and CPU Cluster Diagram</figcaption></figure><h3>3. How a Pod gets a GPU</h3><p>This is the first genuine difference from ordinary workloads.</p><p>Kubernetes understands CPU and memory natively, and it treats them as <strong>divisible</strong>. You can request half a CPU core. Two Pods can share a core, and the kernel will time-slice between them.</p><p>Kubernetes knows nothing about GPUs out of the box. They are exposed by a <strong>device plugin</strong> — software from the hardware vendor that runs on each GPU node, discovers the devices and advertises them to the cluster as a schedulable resource. A Pod then requests them the way it requests memory:</p><pre>resources:<br>  limits:<br>    nvidia.com/gpu: 4</pre><p>But GPUs are advertised as <strong>whole, indivisible units</strong>. You cannot request half a GPU by default. If a Pod asks for one, it gets exclusive use of one; no second Pod is placed on it.</p><p>This has a direct consequence people run into immediately: <strong>a small model wastes a large GPU.</strong> A 7B model on an 80 GB accelerator has the whole device to itself and uses a fraction of it. There are mechanisms to subdivide GPUs — time-slicing, MPS, and hardware partitioning such as MIG — but they are opt-in, come with real trade-offs, and none of them is the default.</p><p>For LLM serving this is usually acceptable, because the standard approach is the opposite: rather than several models sharing a GPU, one model instance takes whole GPUs and serves many users through batching.</p><h3>4. Not all GPU nodes are equal</h3><p>Asking for “four GPUs” is rarely specific enough. A cluster may contain several accelerator generations, machines with different GPU counts, and — most importantly — different interconnects between the GPUs inside them.</p><p>That last point matters more than it appears. A model sharded across four GPUs has those GPUs in constant communication during inference. Four GPUs linked by a fast in-server interconnect and four GPUs that merely happen to be on the same node are very different propositions for performance.</p><p>Kubernetes gives you a few controls for this:</p><p><strong>Labels and node selectors</strong> — nodes are labelled with what they are (accelerator type, region, pool), and Pods select the labels they need.</p><p><strong>Affinity rules</strong> — a richer form of the same idea, including <em>anti</em>-affinity, which is how you keep the replicas of one Deployment from all landing on the same node and dying together with it.</p><p><strong>Taints and tolerations</strong> — an inversion of the above. A taint marks a node as repellent by default; only Pods that explicitly tolerate it are placed there. GPU nodes are almost always tainted, because otherwise ordinary CPU workloads would drift onto your expensive hardware and occupy it.</p><blockquote>Taints keep the wrong workloads off GPU nodes. Selectors and affinity put the right ones on the right GPU nodes.</blockquote><h3>5. When one replica needs more than one Pod</h3><p>A question that comes up as soon as models get large: if a model is sharded across eight GPUs, is that eight Pods or one?</p><p>Usually one. A Pod requests all the GPUs its model instance needs, and the inference engine coordinates them internally as a single logical server. Eight GPUs in one Pod, one address, one endpoint.</p><p>That works because a <strong>Pod cannot span machines</strong>. It is scheduled onto exactly one node, so this arrangement holds only while the whole model fits in a single server’s GPUs.</p><p>When a model needs more GPUs than any one machine has, that breaks. The replica must now span nodes, which means multiple Pods — a leader and its workers — that together form <strong>one</strong> model instance. And that is a genuinely awkward thing to express, because a Deployment is built for identical, independent, interchangeable Pods, and these are none of those. They have distinct roles, they must all be running before any of them is useful, and if one dies the whole group is broken rather than four-fifths working.</p><p>Kubernetes has grown purpose-built resources for exactly this shape of workload, which schedule a leader-plus-workers group as a single unit and restart it as a unit. The takeaway for now is the constraint itself:</p><blockquote>A single-node model replica is one Pod. A multi-node replica is a group of Pods that must be created, scheduled and replaced together.</blockquote><p>This is also why the topology concerns from the previous section matter so much. Keeping a replica inside one node keeps it inside one Pod, and keeps the whole arrangement far simpler.</p><h3>6. Why your Pod is stuck in Pending</h3><p>A near-universal first experience, and worth understanding because the cause is structural.</p><p>Pending means the scheduler cannot find a node satisfying the Pod&#39;s requirements. For CPU workloads this is usually a mild shortage that resolves as things shift around. For GPU workloads it is often absolute, and indivisibility is why.</p><p>A Pod requesting eight GPUs needs <strong>eight free GPUs on a single node</strong>. A cluster with sixteen idle GPUs spread two-per-node across eight machines cannot schedule it. There is plenty of capacity in aggregate and none in the shape required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ujyg2DOOXUdaXPqfA9PoNg.png" /><figcaption>Kubernetes GPU Fragmentation vs Consolidation</figcaption></figure><p>The same fragmentation problem that appears inside GPU memory reappears at cluster scale. Free capacity is not useful unless it is the right shape.</p><h3>7. Health, readiness, and the minutes-long startup</h3><p>Kubernetes checks Pods with <strong>probes</strong>, and the distinction between two of them is the most operationally important thing in this article.</p><p><strong>Liveness</strong> asks: is this process still functioning? If it fails, the Pod is killed and replaced.</p><p><strong>Readiness</strong> asks: should this Pod receive traffic right now? If it fails, the Pod stays alive but is removed from the pool of destinations. This is the mechanism the router from the previous layer ultimately depends on.</p><p>Now the LLM-specific part. A web application container starts in seconds. An inference Pod does not:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kuSbosvmk-_dJp9q_MGJOA.png" /><figcaption>Container Startup Timelines Compared</figcaption></figure><p>This takes <strong>minutes</strong>, not seconds.</p><p>Where the weights come from is worth a note, because it is the stage that dominates. Baking them into the container image makes the image enormous — pulling hundreds of gigabytes onto a fresh node is slow, and every new replica pays for it. The common alternative is to keep the image lean and fetch weights at startup from object storage or a shared volume, with a node-local cache so that later Pods on the same machine start much faster than the first. Either way, this step is usually the reason a replica takes minutes rather than seconds to appear, and it is the first place to look when scale-up feels too slow.</p><p>The startup time creates two requirements.</p><p>First, readiness must gate traffic strictly. A Pod that is running but still loading weights will fail every request sent to it. Readiness is what prevents that, and it is why <strong>“the Pod is up”</strong> and <strong>“the Pod can serve”</strong> are entirely different statements for inference workloads.</p><p>Second, liveness must not fire during startup. A liveness probe with a short timeout will decide the still-loading Pod is broken, kill it, and start a fresh one — which will also be killed, forever. This is why <strong>startup probes</strong> exist: they suspend liveness checking until the process has finished coming up. Getting this wrong produces an infinite crash loop that looks like a broken image and is actually a misconfigured timeout.</p><h3>8. How traffic reaches a Pod — and where the router fits</h3><p>Pods are disposable and their addresses change, so clients never target them directly. A <strong>Service</strong> provides a stable name and address in front of a changing set of Pods, and the control plane keeps its list of endpoints updated as Pods become ready or stop being ready.</p><p>This is exactly the list the router needed. Readiness feeds endpoints; endpoints feed whatever routes traffic.</p><p>Which raises the obvious question: <strong>a Service already load-balances across Pods, so why is a separate router needed at all?</strong></p><p>Because a Service balances by connection, in effect round robin. That is the precise behaviour that fails for inference — it cannot tell a 200-token request from a 30,000-token one, and knows nothing about which replica holds useful cached state.</p><p>So a production deployment usually has both, doing different jobs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PR_bydpuZcA27MvM4Jlq2w.png" /><figcaption>Kubernetes Routing: Round Robin vs Smart Routing</figcaption></figure><p>The router is a normal workload in the cluster. It uses Kubernetes to <em>discover</em> which server Pods are ready, then makes its own decision about which one to use — deliberately bypassing the Service’s built-in balancing. Kubernetes supplies membership; the router supplies judgement.</p><h3>9. Rolling out a new version</h3><p>Deploying a change means replacing Pods, and Kubernetes does this gradually — bring up new Pods, wait for readiness, retire old ones. For a web service it is routine. For inference, three things bite.</p><p><strong>Surge costs GPUs.</strong> A rolling update starts new Pods before removing old ones, so during a rollout you need capacity for both. When a Pod holds four GPUs, surging one replica means four additional GPUs must be free. On a full cluster the rollout simply stalls, with new Pods Pending.</p><p><strong>Rollouts are slow.</strong> Each new Pod takes minutes to become ready, and they proceed in batches. A rollout across many replicas is measured in tens of minutes.</p><p><strong>In-flight requests need draining.</strong> When a Pod is told to stop, Kubernetes gives it a grace period and then kills it. The default is thirty seconds, which is fine for HTTP requests that finish in milliseconds — and not fine for a generation that has been streaming for a minute and is not done. Too short a grace period cuts responses off mid-sentence. Long-running generation needs a longer window and a server that stops accepting new work while finishing what it has.</p><h3>10. When a Pod dies</h3><p>A Pod crashes — out-of-memory, a hardware fault, a node drained for maintenance. Kubernetes notices the desired state is unmet and creates a replacement.</p><p>That replacement is correct in every respect Kubernetes cares about. It is the same image, the same configuration, the same GPU count. It is also completely cold.</p><p>Everything the dead Pod had accumulated in GPU memory is gone:</p><ul><li>The KV cache for every in-flight request, which is why those requests fail rather than migrate.</li><li>The <strong>prefix cache</strong> — every system prompt, every document, every conversation history it had computed and retained.</li></ul><p>The replacement Pod starts with an empty cache and takes minutes to reach ready. Once it does, it begins recomputing prefixes that were, until recently, already sitting in memory a few racks away. Every request routed to it during that window pays full prefill cost.</p><blockquote>A replaced Pod is functionally identical and economically much worse, for as long as it takes to warm up<em>.</em></blockquote><p>Two practical implications. <strong>Restarts are expensive in a way Kubernetes does not model</strong> — it sees a healthy replacement, while your latency percentiles see a cold one. And because Pods are treated as interchangeable, nothing in the orchestration layer knows that the Pod it just terminated was the one holding your hottest prefix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xETlrMcDVTK4r0ZF5CPVRw.png" /><figcaption>Pod Restart Cache Recovery Timeline</figcaption></figure><h3>11. Autoscaling, and why it is harder here</h3><p>Kubernetes can add replicas as load rises. The standard mechanism scales on CPU utilisation, which for inference is close to meaningless — the CPU is doing tokenisation and request handling while the GPU does the work. A saturated inference server can show modest CPU usage.</p><p>So scaling must be driven by signals from the inference server itself. The useful ones are the same ones a router uses: <strong>queue depth in tokens</strong>, <strong>KV cache utilisation</strong>, <strong>running sequences</strong>, and <strong>TTFT against your target</strong>. Cache utilisation is often the best single trigger, because it is the resource that actually runs out.</p><p>Then there is the timing problem, which is more fundamental.</p><p>Scaling out a web service takes seconds, so you can react to load as it arrives. Adding an inference replica takes minutes — and if no GPU node is free, the cluster autoscaler must provision a new machine first, which takes longer still. By the time capacity arrives, the traffic spike may be over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uFO4676ID-Tt3mCJrVjQkA.png" /><figcaption>Pod Restart Performance Timeline</figcaption></figure><p>The consequences are unavoidable rather than clever. You <strong>over-provision</strong>, keeping headroom that costs money in order to absorb spikes. You <strong>scale on leading indicators</strong> — queue growth rather than saturation — to buy back some of the delay. You <strong>scale down conservatively</strong>, because removing a replica discards a warm cache you will pay minutes to rebuild. And where traffic is predictable, you scale on a schedule rather than reacting at all.</p><h3>12. What you get, and what you still do not</h3><p>Kubernetes gives an inference platform a great deal. Pods land on machines with the right accelerators and the right topology. Dead replicas are replaced without anyone waking up. Readiness keeps traffic away from servers that cannot serve. Rollouts happen gradually and can be reversed. Capacity follows load, if slowly. Most valuable of all, there is one authoritative, continuously updated answer to <em>which inference servers exist and which are ready</em> — which is exactly what the routing layer required.</p><p>What it does not give you is any understanding of the workload.</p><p>Kubernetes schedules on the resources you declared when you deployed: GPUs, CPU, memory. Those numbers are fixed at deploy time and describe the container, not what is happening inside it. It has no concept of a token, a batch, a KV cache block or a cached prefix. A Pod at 95% cache utilisation and a Pod at 10% are, to the control plane, two healthy Pods.</p><p>That is not a criticism. It is a design decision that makes Kubernetes general enough to run anything. But it means the orchestration layer and the routing layer each hold half of the truth, and neither holds both.</p><h3>The next problem</h3><p>Look at what we have actually built.</p><p>Kubernetes knows which Pods exist, which are ready, and where they run. It knows nothing about what is inside them.</p><p>The router knows what is inside them — cache utilisation, queue depth, which prefixes are warm. It knows nothing about creating, replacing or scaling them.</p><p>So the two most important decisions in the system are made by components that cannot see each other’s information. Kubernetes terminates a Pod during a rollout without knowing it held the hottest prefix cache in the fleet. It scales up on a metric it does not understand, delivered by a custom pipeline someone had to build. Its Service load-balances in exactly the way we established is wrong, so we bypass it with a router that lives beside the platform rather than within it.</p><p>Every one of these is a workaround. Each one works, and each one exists because the orchestration layer models an inference server as a generic container with a GPU attached.</p><p>The alternative is to make orchestration itself inference-aware — for the platform to understand that these Pods serve models, hold caches, and have a load that is measured in tokens rather than connections. That means treating a pool of inference servers as a first-class object, routing with knowledge of what each holds, scheduling with knowledge of what each is doing, and being able to place the two phases of a request on different hardware when that helps.</p><p><strong>What does an orchestration layer look like when it is built for LLM inference specifically?</strong></p><p>That is the last architectural piece of the stack.</p><h3>The series</h3><p><strong>1. </strong><a href="https://pub.towardsai.net/what-really-happens-when-you-click-send-on-chatgpt-2307e3ede14e"><strong>What really happens when you click ‘Send’ on ChatGPT</strong></a> — A journey through modern AI Infrastructure</p><p><strong>2. </strong><a href="https://pub.towardsai.net/what-do-you-do-with-a-model-thats-too-big-for-your-gpu-443bbbe70d4d"><strong>What Do You Do With a Model That’s Too Big for Your GPU?</strong></a> — Quantization, Sharding and Parallelism Explained</p><p><strong>3. </strong><a href="https://pub.towardsai.net/how-does-one-gpu-serve-hundreds-of-users-at-the-same-time-b999e60a440e"><strong>How Does One GPU Serve Hundreds of Users at the Same Time?</strong> </a>— Inside an LLM inference server</p><p><strong>4. </strong><a href="https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7"><strong>The KV Cache Explained: Why Long Conversations Get Expensive</strong></a> — How LLMs remember context without recomputing everything</p><p><strong>5. </strong><a href="https://pub.towardsai.net/why-is-your-llm-recomputing-the-same-prompt-1-000-times-a-day-675e0cdab4b3"><strong>Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?</strong></a> — Prefix caching, radix trees and block hashing explained</p><p><strong>6. </strong><a href="https://pub.towardsai.net/why-traditional-load-balancing-breaks-for-llms-40311f7b29e2"><strong>Why Traditional Load Balancing Breaks for LLMs</strong></a> — Building an LLM-aware router</p><p><strong>7. </strong><a href="https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c"><strong>Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster</strong></a></p><p><strong>8. </strong><a href="https://pub.towardsai.net/llm-d-explained-69f7d0b2b035"><strong>LLM-D Explained</strong></a> — How modern AI infrastructure routes, schedules and scales LLM inference<br>(Next Article)</p><p><strong>9. </strong><a href="https://pub.towardsai.net/inside-a-modern-ai-inference-platform-8846434bb14b"><strong>Inside a Modern AI Inference Platform</strong></a> — The full stack end-to-end</p><p><strong>Sources</strong></p><ul><li><a href="https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/">Kubernetes — Schedule GPUs</a></li><li><a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/">Kubernetes — Configure liveness, readiness and startup probes</a></li><li><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/">Kubernetes — Taints and tolerations</a></li><li><a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html">NVIDIA — GPU Operator documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=424f0dcfab8c" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c">Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Traditional Load Balancing Breaks for LLMs]]></title>
            <link>https://pub.towardsai.net/why-traditional-load-balancing-breaks-for-llms-40311f7b29e2?source=rss-82d3a738e07a------2</link>
            <guid isPermaLink="false">https://medium.com/p/40311f7b29e2</guid>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[load-balancing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Rinit Jain]]></dc:creator>
            <pubDate>Wed, 09 Sep 2026 16:01:01 GMT</pubDate>
            <atom:updated>2026-09-15T13:56:59.309Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building an LLM-aware router</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OXR0nB0zD_s-B7uU57Fabg.png" /></figure><blockquote><strong>TL; DR</strong></blockquote><blockquote>Classic load balancing assumes requests are roughly equal, short-lived, and that servers are interchangeable. For LLM inference, all three are false.</blockquote><blockquote>Two requests can differ by more than a hundred times in compute and memory cost, and one may occupy a server for a minute while the other finishes in two seconds.</blockquote><blockquote>Counting connections tells you nothing useful. Ten short requests can be a lighter load than two long ones.</blockquote><blockquote>The signals that matter are <strong>KV cache utilisation</strong>, <strong>queue depth in tokens</strong>, and <strong>cache locality</strong> — where the state for this request already lives.</blockquote><blockquote>Locality and load pull in opposite directions, so a router has to score them against each other rather than optimise either alone.</blockquote><blockquote>Output length is unknown at routing time, which makes every routing decision a decision under uncertainty.</blockquote><blockquote>A request’s two phases want opposite things from a server, so one placement decision has to serve two incompatible workloads.</blockquote><p>You have a working inference server. It batches efficiently, manages its memory carefully, and caches prompt prefixes so repeated work is skipped. Traffic grows past what one machine can handle, so you run twenty replicas and put a load balancer in front.</p><p>And performance gets strange. Some requests are fast, others inexplicably slow. One replica is pinned while another idles. The prefix cache hit rate, which was excellent on a single server, has collapsed.</p><p>Nothing is broken. The load balancer is doing exactly what it was designed to do. The problem is what it was designed for.</p><h3>1. What a load balancer assumes</h3><p>Load balancing is a solved problem for ordinary web services, and it is solved because ordinary web traffic has convenient properties.</p><p><strong>Requests cost roughly the same.</strong> One API call is much like another. Variation exists, but it is bounded — no request is a hundred times more expensive than its neighbour.</p><p><strong>Requests are short.</strong> Milliseconds to a second or two. A bad placement decision is corrected almost immediately by the next request.</p><p><strong>Servers are interchangeable.</strong> Any server can handle any request equally well. State lives in a database, not in the server.</p><p><strong>Load is observable from outside.</strong> Because requests are similar and brief, the number of open connections is a decent proxy for how busy a server is.</p><p>Every strategy in the standard toolkit — round robin, least connections, weighted variants — is built on these four. And LLM inference violates all four.</p><h3>2. Assumption one: requests cost roughly the same</h3><p>Consider two requests arriving at the same moment.</p><pre>Request A     200 input tokens  →     50 output tokens<br>Request B  30,000 input tokens  →  2,000 output tokens</pre><p>To the load balancer these are identical: one HTTP request each. In reality they are not close to comparable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xYXTg3MYKFJgJfDMInG8Tw.png" /><figcaption>Round-Robin Balances Counts, Not Load</figcaption></figure><p><strong>Prefill.</strong> Request B has 150 times more input to process. That work is real GPU compute and it happens before either request produces a single token.</p><p><strong>Memory.</strong> The KV cache — the attention state the server must hold for the whole life of a request — scales with token count. At a few hundred kilobytes per token for a large model, A occupies tens of megabytes. B occupies several gigabytes. On a server with limited spare VRAM, B might consume a meaningful fraction of the entire pool while A is a rounding error.</p><p><strong>Duration.</strong> A finishes 50 decode steps and releases everything, likely within a couple of seconds. B runs 2,000 decode steps, holding its multi-gigabyte cache the entire time — possibly a minute or more.</p><p>So one request is briefly negligible, and the other occupies a large share of a server’s memory for a long time. A round-robin balancer alternates between them as if they were the same thing, and load diverges within seconds of starting.</p><h3>3. Assumption two: requests are short</h3><p>A normal request is over before a bad decision matters. An LLM request is not.</p><p>If you route a 4,000-token generation to a server that is already saturated, that request will be slow for its entire lifetime — potentially a minute — and it will make everything already running on that server slower for the same period, because they now share memory and batch slots with it.</p><p>Routing mistakes in LLM serving are not transient. They persist, and they compound.</p><h3>4. Assumption three: servers are interchangeable</h3><p>This is the assumption that breaks most interestingly, and it follows directly from prefix caching.</p><p>Inference servers keep the computed attention state for prompt prefixes they have already seen, so that a later request beginning with the same tokens can skip that work. This means each replica accumulates <strong>its own</strong> cache, containing whatever it happens to have processed. Those caches are not shared between replicas.</p><p>So the replicas are no longer interchangeable. A request whose prefix is already cached on Server A can skip most of its prefill there and answer almost immediately. Sent to Server B, the same request pays the full cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WzHJ1qOBGroOO9dJbtgsnw.png" /><figcaption>Cache-Aware Routing: Server A vs Server B</figcaption></figure><p>The consequence is counter-intuitive enough to state plainly:</p><blockquote>The best destination is not necessarily the server with the most spare capacity. It may be the server that already holds the right state.</blockquote><p>There is also a compounding effect that explains the collapsed hit rate from the opening. Round robin spreads requests evenly, which means the same system prompt gets prefilled and cached on every replica independently. Twenty replicas hold twenty copies of the same prefix, each consuming memory, and every replica’s cache is churning through unrelated traffic. Even distribution actively destroys locality.</p><h3>5. Assumption four: load is observable by counting</h3><p>Least-connections routing sends each request to whichever server has the fewest open connections. It fails here for the same reason round robin does — a connection is not a unit of work.</p><pre>Server A   10 requests   short contexts       20% of cache pool used<br>Server B    2 requests   very long contexts   85% of cache pool used</pre><p>Least connections sends the next request to Server B, which has one fifth the connections and four times the memory pressure. It is the worst available choice, and the algorithm cannot see it.</p><p>The general problem: every classic strategy measures <strong>how many things a server is doing</strong>. What matters for an inference server is <strong>how much memory those things are consuming, and for how much longer</strong>. Those are different quantities, and the first is not a proxy for the second.</p><h3>6. What the router actually needs to know</h3><p>If counting connections does not work, what does? An LLM-aware router bases decisions on the state of the inference servers themselves. Three signals do most of the work.</p><h3>KV cache utilisation</h3><p>The single most useful number a router can have.</p><p>An inference server’s real capacity limit is memory. Model weights occupy a fixed portion of VRAM; what remains is the pool from which every active request draws attention state. When that pool fills, the server stops admitting new requests — and starts evicting running ones, which appears to users as responses stalling mid-generation.</p><p>So cache utilisation is a direct measure of how close a server is to trouble, in a way that connection counts and even GPU utilisation are not. A server at 90% cache occupancy should receive very little new work regardless of how few connections it holds.</p><h3>Queue depth, measured in tokens</h3><p>Requests waiting to be admitted matter, but <em>how many</em> are waiting is the wrong unit. Five queued requests carrying 200 tokens each is a trivial backlog. Five carrying 30,000 tokens each is a substantial amount of prefill that everyone behind it will wait through.</p><p>Measuring the queue in <strong>tokens rather than requests</strong> gives an estimate of how long a new arrival would actually wait — which is what TTFT depends on.</p><h3>Cache locality</h3><p>The value signal from Section 4: does this server already hold cached state for this request’s prefix?</p><p>Determining this cheaply is the practical challenge. A router cannot inspect every replica’s cache on every request. The common approaches are to have servers report which prefixes they hold, or to hash the request’s prefix and route consistently, so identical prefixes deterministically land on the same replica and locality emerges without the router tracking anything.</p><p>Two further signals matter in larger deployments: <strong>model identity</strong>, since a replica can only serve requests for a model it has loaded, and <strong>health and readiness</strong>, since a server still loading weights into VRAM must receive nothing at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PhVAsGIPxbzsOYMlUhv4IA.png" /><figcaption>LLM-Aware Routing vs Classic Load Balancing</figcaption></figure><h3>7. The conflict at the centre of it</h3><p>Here is where LLM routing stops being a matter of collecting better metrics.</p><p>The two most valuable signals <strong>disagree</strong>. Cache locality says: send this request to the server that already has its prefix. Load says: send it to the server with free memory. When one server holds the hot prefix, those are frequently different servers.</p><p>Following locality alone is a trap. If a single system prompt dominates your traffic, every request has the same locality preference, and pure locality routing herds all of them onto one replica while nineteen sit idle. The cache hits would be superb and the server would fall over.</p><p>Following load alone is the round-robin failure from Section 4 — perfectly balanced, and destroying reuse.</p><p>So a router scores rather than optimises. Something like:</p><pre>score(server) = w₁ × prefix_match<br>              − w₂ × cache_utilisation<br>              − w₃ × queued_tokens</pre><p>Send the request to the highest score. The weights encode the trade-off: a cache hit is worth a great deal, but not worth sending work to a server that is nearly out of memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wLp49NhYf-uV0KYXEhdUEw.png" /><figcaption>Routing Trade-Offs: Prefix Versus Load</figcaption></figure><p>The useful mental model is that locality is treated as <strong>a strong preference that yields under pressure</strong>. Prefer the warm server until it is loaded enough that the hit is no longer worth having — at which point the request goes elsewhere and, incidentally, warms a second replica.</p><h3>8. Prefill and decode want opposite things</h3><p>One more tension worth naming, because it shapes where this field is going.</p><p>The two phases of a request have different needs. Prefill is a large burst of compute that wants a server with spare arithmetic capacity. Decode is a long, memory-bandwidth-bound trickle that wants a server with free cache memory and room in its batch. A single request needs both, in sequence — and routing it to one server means choosing a machine that must be good at both, at whichever moment it happens to arrive.</p><p>Worse, the two interfere. A large prefill landing on a server occupies the GPU long enough that every request already generating there sees its token stream stutter. So routing a long prompt to a replica is not just a load decision about that request; it degrades the experience of everyone currently mid-response on that machine.</p><p>Some systems respond by pushing this to its logical conclusion and running prefill and decode on <strong>separate pools of GPUs</strong>, routing each phase independently and transferring the cached state between them. That is a substantial architectural change and a topic in its own right — but the tension that motivates it is visible right here, in a router trying to make one placement decision for two incompatible workloads.</p><h3>9. The part the router cannot know</h3><p>Everything above depends on estimating what a request will cost. The router can measure half of that.</p><p><strong>Input length is known.</strong> The prompt is right there. Prefill cost and initial memory footprint can be estimated well.</p><p><strong>Output length is not.</strong> Whether the model produces 20 tokens or 4,000 is not knowable until it happens. And output length determines how long the request holds memory and how many decode steps it consumes — which is most of its total cost.</p><p>So routing is a decision under uncertainty, and systems handle it with estimates rather than answers: a declared max_tokens gives an upper bound, historical averages per endpoint or per customer give a prior, and continuous feedback lets the router correct as servers report their real state.</p><p>The honest framing is that an LLM router does not compute optimal placement. It makes a good decision from partial information and relies on fresh feedback to correct.</p><h3>10. Sticky routing for conversations</h3><p>Multi-turn chat deserves a specific mention, because it is where locality pays off most and is easiest to capture.</p><p>Each turn of a conversation re-sends everything before it. Turn five’s prompt is turns one through four plus a new message — meaning its prefix is <em>exactly</em> what the server computed on turn four. Route turn five back to that same replica and almost the entire prompt is a cache hit.</p><p>Route it elsewhere and the whole conversation is prefilled again from scratch, and it gets worse every turn, because the history keeps growing.</p><p>So routing by session identity is a large and cheap win. Two caveats worth knowing. It creates hot spots — a few very active conversations can concentrate load — so stickiness should still yield to a genuinely overloaded server. And it is only ever an optimisation: if the replica has restarted or evicted the entry, the request must still be correct, just slower.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BtvTXUdOk-zq-YKEl2lphw.png" /><figcaption>Sticky Routing vs. Round Robin Caching</figcaption></figure><h3>11. Where the router sits, and what it costs</h3><p>The router is on the critical path of every request, so it has a budget: single-digit milliseconds. Decisions must come from state it already holds, not from queries made at request time.</p><p>That means it maintains a continuously updated view of the fleet, refreshed by servers reporting their own state — cache utilisation, queue depth, running sequences, readiness. Inference engines expose these as metrics, which is what makes this style of routing practical rather than theoretical.</p><p>Two properties follow, and both matter later. The router’s view is <strong>always slightly stale</strong>, so routing has to tolerate acting on information that is a moment out of date. And the router must <strong>know which servers exist right now</strong>, which is a harder requirement than it sounds.</p><h3>12. What to measure</h3><p><strong>TTFT at p95, not the mean.</strong> Routing failures show up in the tail. A router that is right 90% of the time and catastrophically wrong the rest looks fine on average.</p><p><strong>Prefix cache hit rate across the fleet.</strong> If it is far below what a single server achieved, routing is destroying locality.</p><p><strong>Load imbalance.</strong> Compare cache utilisation across replicas. A wide spread means the router is not seeing real load — which usually means it is counting something instead of measuring it.</p><p><strong>Queue wait time</strong>, separated from prefill time. They have different causes and different fixes.</p><h3>The next problem</h3><p>We now have a router that makes decisions from live server state: which replicas have memory, which are backed up, which hold useful cached prefixes.</p><p>All of which assumes something we have never examined. The router holds a list of inference servers, believes they are alive, and has current state for each one.</p><p>Where does that list come from?</p><p>In production it is not fixed, and it is not stable. Replicas are added when traffic rises and removed when it falls. Servers crash and are replaced. Deployments roll out new model versions, draining old replicas while new ones load weights — a process that takes minutes, during which those servers exist but must receive nothing. Hardware fails. Nodes are drained for maintenance.</p><p>The fleet the router is routing to is being continuously created, destroyed and reshaped underneath it. Something has to run those inference servers, place them on machines with the right GPUs, watch their health, replace the dead ones, add capacity under load, and publish an accurate list of which are ready to receive traffic — updated constantly and reliably enough that the router can trust it.</p><p><strong>What actually operates a fleet of GPU servers, and how does it keep an accurate picture of what is running?</strong></p><p>That is the orchestration layer, and it is the next piece of the stack.</p><h3>The series</h3><p><strong>1. </strong><a href="https://pub.towardsai.net/what-really-happens-when-you-click-send-on-chatgpt-2307e3ede14e"><strong>What really happens when you click ‘Send’ on ChatGPT</strong></a> — A journey through modern AI Infrastructure</p><p><strong>2. </strong><a href="https://pub.towardsai.net/what-do-you-do-with-a-model-thats-too-big-for-your-gpu-443bbbe70d4d"><strong>What Do You Do With a Model That’s Too Big for Your GPU?</strong></a> — Quantization, Sharding and Parallelism Explained</p><p><strong>3. </strong><a href="https://pub.towardsai.net/how-does-one-gpu-serve-hundreds-of-users-at-the-same-time-b999e60a440e"><strong>How Does One GPU Serve Hundreds of Users at the Same Time?</strong> </a>— Inside an LLM inference server</p><p><strong>4. </strong><a href="https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7"><strong>The KV Cache Explained: Why Long Conversations Get Expensive</strong></a> — How LLMs remember context without recomputing everything</p><p><strong>5. </strong><a href="https://pub.towardsai.net/why-is-your-llm-recomputing-the-same-prompt-1-000-times-a-day-675e0cdab4b3"><strong>Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?</strong></a> — Prefix caching, radix trees and block hashing explained</p><p><strong>6. </strong><a href="https://pub.towardsai.net/why-traditional-load-balancing-breaks-for-llms-40311f7b29e2"><strong>Why Traditional Load Balancing Breaks for LLMs</strong></a> — Building an LLM-aware router</p><p><strong>7. </strong><a href="https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c"><strong>Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster</strong></a><strong><br></strong>(Next Article)</p><p><strong>8. </strong><a href="https://pub.towardsai.net/llm-d-explained-69f7d0b2b035"><strong>LLM-D Explained</strong></a> — How modern AI infrastructure routes, schedules and scales LLM inference</p><p><strong>9. </strong><a href="https://pub.towardsai.net/inside-a-modern-ai-inference-platform-8846434bb14b"><strong>Inside a Modern AI Inference Platform</strong></a> — The full stack end-to-end</p><p><strong>Sources</strong></p><ul><li><a href="https://docs.vllm.ai/en/latest/serving/metrics.html">vLLM — Production metrics</a></li><li><a href="https://gateway-api-inference-extension.sigs.k8s.io/">Kubernetes Gateway API Inference Extension</a></li><li><a href="https://arxiv.org/abs/2309.06180">Efficient Memory Management for LLM Serving with PagedAttention</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40311f7b29e2" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/why-traditional-load-balancing-breaks-for-llms-40311f7b29e2">Why Traditional Load Balancing Breaks for LLMs</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?]]></title>
            <link>https://pub.towardsai.net/why-is-your-llm-recomputing-the-same-prompt-1-000-times-a-day-675e0cdab4b3?source=rss-82d3a738e07a------2</link>
            <guid isPermaLink="false">https://medium.com/p/675e0cdab4b3</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[sglang]]></category>
            <category><![CDATA[prefix-caching]]></category>
            <dc:creator><![CDATA[Rinit Jain]]></dc:creator>
            <pubDate>Wed, 09 Sep 2026 12:01:02 GMT</pubDate>
            <atom:updated>2026-09-15T13:55:46.625Z</atom:updated>
            <content:encoded><![CDATA[<h4>Prefix caching, radix trees and block hashing explained</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w3dJtl7g0I00TNflqrI2PQ.png" /></figure><blockquote><strong>TL; DR</strong></blockquote><blockquote>Real LLM traffic is enormously repetitive. System prompts, chat history and shared documents mean requests overlap heavily at the start.</blockquote><blockquote>Attention is causal, so a token’s cached state depends only on the tokens <strong>before</strong> it. Two requests with an identical prefix produce an identical cache for that prefix.</blockquote><blockquote><strong>Prefix caching</strong> keeps that state after a request finishes so the next request can reuse it, skipping prefill entirely for the shared portion.</blockquote><blockquote>vLLM identifies reusable state by <strong>hashing each cache block together with the hash of everything before it</strong>. SGLang organises the same information as a <strong>radix tree</strong>.</blockquote><blockquote>The gain shows up as lower time-to-first-token and reclaimed prefill capacity — often dramatic for long shared prompts.</blockquote><blockquote>The cost is memory: blocks kept for future reuse are blocks unavailable to active requests.</blockquote><p>Here is a pattern that shows up in almost every production LLM deployment.</p><p>An application has a system prompt — instructions, tone, policy, tool definitions, maybe a few examples. It might be 4,000 tokens. Every single request begins with it. The user’s actual question is 20 tokens on the end.</p><p>So the server processes those 4,000 tokens. Then the next request arrives and it processes the same 4,000 tokens again. Then again. A thousand times an hour, computing an identical result and discarding it each time.</p><p>This article is about why that happens, why it is fixable, and how modern inference engines fix it.</p><h3>1. First, what gets recomputed</h3><p>A quick definition, because the rest depends on it.</p><p>When a model processes text, each token produces a <strong>key</strong> and a <strong>value</strong> vector at every layer. Generating subsequent tokens requires the keys and values of everything before them, so the engine stores that state rather than recomputing it. This store is the <strong>KV cache</strong>, and building it for the input prompt is the <strong>prefill</strong> phase — the work done before the first output token appears.</p><p>Prefill is expensive and scales with prompt length. For a large model, processing several thousand prompt tokens can take a noticeable fraction of a second, and that time lands directly on the user’s time-to-first-token.</p><p>Normally, when a request finishes, its cache is released and the state is gone. That is the behaviour we are about to question.</p><h3>2. Real traffic overlaps far more than you would expect</h3><p>Repeated prefixes are not a corner case. They are the dominant shape of production traffic.</p><p><strong>System prompts.</strong> Every request from an application carries the same instruction block. Often thousands of tokens, identical every time.</p><p><strong>Multi-turn conversations.</strong> Models are stateless. Turn five of a chat re-sends turns one through four in full. The prefix of each turn is the entire conversation so far — and it was computed on the previous turn.</p><p><strong>Shared documents.</strong> In retrieval and document-analysis workloads, many users ask different questions against the same document. The document is the prefix.</p><p><strong>Few-shot examples.</strong> Prompts with fixed demonstrations before the variable part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p4QU4k1O7ZuHWnPH8fcvTw.png" /></figure><p>In workloads like these the shared portion is frequently the large majority of the prompt.</p><h3>3. Why reuse is possible at all</h3><p>The overlap alone does not justify reuse. What justifies it is a property of how attention works.</p><p>Attention in a generative model is <strong>causal</strong>: a token can attend to the tokens before it, never to the tokens after. Which means a token’s key and value depend only on itself and on everything preceding it — and on nothing that follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Zs_TNtqQm6kapLmNuUfEg.png" /><figcaption>Casual Attention and Prefix cache Reuse</figcaption></figure><p>The consequence is exact and worth stating carefully:</p><blockquote>If two requests begin with the identical sequence of tokens, the KV cache for that shared portion is identical — regardless of what either request contains afterwards.</blockquote><p>Nothing appended later can retroactively change it. That is what makes the state safely reusable rather than merely similar.</p><p>Note the strictness. The prefix must match <strong>from the very first token</strong>, and it must match exactly. Shared text in the middle of two prompts is not reusable, because the state at that position depends on the different text preceding it. Prefix caching earns its name precisely.</p><h3>4. What a cache hit actually buys</h3><p>Suppose an application has a 4,000-token system prompt and users send short questions.</p><p>Without prefix caching, every request prefills 4,020 tokens. With it, the first request prefills 4,020 and each subsequent one prefills roughly 20 — the shared 4,000 are read from cache.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ztQwdr4QvV6Os0kCLGGXdw.png" /></figure><p>Two things improve, and they are worth separating.</p><p><strong>Time to first token drops sharply.</strong> Prefill was the dominant term for the request; almost all of it disappears. To put rough numbers on it: if a deployment prefills at a few thousand tokens per second, those 4,000 shared tokens were costing on the order of a second of TTFT on every request, before the model even looked at the user’s question. A hit removes nearly all of it. For long shared prompts this is the difference between a sluggish interface and an immediate one.</p><p><strong>Prefill capacity is reclaimed.</strong> The GPU cycles that were being spent recomputing identical state become available for real work. Across a busy server this raises throughput even for requests that never hit the cache.</p><p>The saving scales with how much of the prompt is shared. A 100-token system prompt is not worth much. A 20,000-token document that fifty users are querying is worth an enormous amount.</p><h3>5. The unit of reuse is a block</h3><p>Modern engines do not store a sequence’s cache as one contiguous region. They split it into fixed-size <strong>blocks</strong>, each holding a set number of tokens — 16 is a common default — allocated on demand.</p><p>That choice, made originally to avoid wasting memory, turns out to be exactly what prefix caching needs. Reuse operates at block granularity: a block is either identical to one already in memory, or it is not.</p><p>The practical consequence is that reuse is <strong>block-aligned</strong>. If two prompts share 4,005 tokens and the block size is 16, the engine can reuse the first 250 complete blocks — 4,000 tokens — and recomputes the partial remainder. A trivial rounding loss, but it explains why matches are reported in blocks rather than exact token counts.</p><h3>6. How the engine recognises a reusable block</h3><p>This is the mechanism at the centre of the article.</p><p>The engine needs to answer one question quickly: <em>have I already computed the state for this exact block, in this exact position, with this exact history?</em> All three conditions matter.</p><p>A hash of just the block’s own tokens is not enough. The same 16 tokens appearing at the start of one prompt and in the middle of another have completely different state, because the state depends on everything preceding them. Hashing only the contents would produce a false match and silently corrupt the output.</p><p>So vLLM <strong>chains the hashes</strong>. Each block’s hash is computed from its own tokens <em>together with the hash of the preceding block</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KQ8AX-Qu0-WmK7QHkKKL3A.png" /><figcaption>Chained Block Hashes for KV Cache Reuse</figcaption></figure><p>Each hash therefore identifies not “these 16 tokens” but “these 16 tokens, arrived at through exactly this history.” Two blocks match only if their entire preceding sequence matched, which is precisely the condition that makes reuse correct.</p><p>Those hashes go into a lookup table pointing at physical blocks. When a request arrives, the engine hashes its prompt block by block and looks each one up.</p><h3>7. Matching stops at the first difference</h3><p>Lookup walks forward from the first block and stops as soon as one misses.</p><pre>Cached:  [ A ][ B ][ C ][ D ][ E ]<br>New:     [ A ][ B ][ C ][ X ][ Y ]<br>           ✓    ✓    ✓    ✗<br>         └── reused ──┘ └ computed ┘</pre><p>This is exactly the behaviour multi-turn chat needs. Each new turn shares every block of the conversation so far and diverges only at the newly appended message. The reused portion grows with the conversation, so prefill cost per turn stays roughly flat instead of climbing with history length.</p><p>It also answers the question in this article’s title. When a deployment has prefix caching enabled and is <em>still</em> recomputing everything, the cause is almost always <strong>variable content at the front of the prompt.</strong> A timestamp, session ID, user name or A/B flag injected at the top of an otherwise-fixed system prompt changes block 0 — which changes hash₀, which changes every hash downstream, which destroys every match for the entire prompt. One dynamic line at the top can cost you the other four thousand tokens.</p><p>The fix is mechanical: <strong>put stable content first, variable content last.</strong> Fixed instructions, tool definitions and examples at the top; anything that changes per request at the bottom.</p><h3>8. A different structure: radix trees</h3><p>vLLM’s hash table is one way to organise this. SGLang takes another, called <strong>RadixAttention</strong>.</p><p>A radix tree stores strings by shared prefix: common beginnings become shared branches, and divergence creates a fork. Applied to token sequences, the cached prefixes form a tree where each path from the root is a cached sequence, and any shared prefix is stored once on a shared path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vcecefJH2Y6GL1kfetDYbA.png" /><figcaption>RadixAttention Shared Prefix Tree</figcaption></figure><p>Both approaches answer the same question and both evict with LRU. The difference is structural: a hash table gives direct lookup of a known block, while a tree makes the <em>relationships</em> between cached prefixes explicit — you can see which sequences share what, which branches are hot, and where a new request would attach. SGLang uses that visibility for cache-aware scheduling, ordering requests to maximise reuse.</p><p>For a practitioner the takeaway is not which is better. It is that both engines cache prefixes, both evict LRU, and both benefit from the same prompt structure.</p><h3>9. Eviction, and why this is cheaper than it sounds</h3><p>Cached prefixes cannot be kept forever. Memory is finite and the pool is shared with active requests.</p><p>The elegant part of the design is that a finished request’s blocks are not immediately wiped. They are marked free and left in place. If a new request hashes to them before they are needed elsewhere, it gets a hit at no cost. If memory pressure arrives first, they are reclaimed, least-recently-used first.</p><p>So prefix caching largely runs on memory that would otherwise sit idle. It does not reserve a separate pool; it defers cleanup. That is why engines can enable it by default — in the common case it costs nothing and sometimes saves a great deal.</p><p>The cost appears under pressure. When the server is saturated, blocks retained for possible reuse are blocks unavailable to active sequences, so the engine faces a real trade-off between <strong>memory for reuse</strong> and <strong>memory for concurrency</strong>. Hot, highly-shared workloads want more retention; workloads with unique prompts want none.</p><h3>10. When it does not help</h3><p>Prefix caching is close to free, but it is not universal.</p><p><strong>Unique prompts.</strong> Workloads where every request is genuinely different get nothing. All lookups miss, and the retained blocks are pure overhead.</p><p><strong>Variable content at the front.</strong> The single most common reason a cache appears to do nothing — see Section 7.</p><p><strong>Cache too small for the working set.</strong> If the hot prefixes exceed available memory, entries are evicted before they are reused and the hit rate collapses.</p><p><strong>Short prompts.</strong> Saving a few hundred tokens of prefill is not material.</p><p><strong>Decode is untouched.</strong> Prefix caching removes redundant prefill. Generation still runs one token at a time, at the same speed. A request that spends most of its time generating will barely notice a cache hit.</p><p>One operational note: because entries are keyed on token IDs, only genuinely identical content matches, and no content crosses between requests. What is observable is <em>timing</em> — hits are faster than misses. In shared multi-tenant deployments that is worth being aware of.</p><h3>11. What to measure</h3><p>Two numbers tell you whether this is working.</p><p><strong>Prefix cache hit rate</strong> — the share of prompt blocks served from cache. Engines expose this directly. If it is near zero on a workload you expect to be repetitive, look for variable content at the front of your prompts before looking anywhere else.</p><p><strong>TTFT, at p50 and p95</strong> — hits and misses have very different profiles, so the average conceals the behaviour. A deployment with a high hit rate typically shows a fast median and a long tail of misses, and the gap between them is roughly what the cache is worth.</p><h3>The next problem</h3><p>Prefix caching changes the economics of a request in a way that has a consequence beyond the server it runs on.</p><p>Two identical requests no longer cost the same. One that lands where its prefix is cached skips almost all of its prefill and answers immediately. The same request landing elsewhere pays the full cost. The difference can be large.</p><p>Which is fine on a single server. It is not fine on twenty.</p><p>In a real deployment, traffic arrives at a load balancer sitting in front of many replicas, and each replica has its own cache holding its own set of prefixes. Those caches are not shared. Server A may hold the state for a request that would be nearly free to serve there — while a round-robin balancer, knowing nothing about any of this, sends it to Server B to be computed from scratch, evicting something useful in the process.</p><p>And cached state is only part of it. LLM requests were never interchangeable to begin with. A 200-token prompt with a 20-token answer costs almost nothing. A 30,000-token prompt with a 4,000-token answer occupies a large share of a server’s memory for a long time. A load balancer counting connections cannot tell them apart, and counts them the same.</p><p>So the routing layer is making decisions about wildly unequal work, with no idea where the valuable state lives.</p><p><strong>What would a load balancer have to know about an inference server to route to it well?</strong></p><p>That is the next layer of the stack.</p><h3>The series</h3><p><strong>1. </strong><a href="https://pub.towardsai.net/what-really-happens-when-you-click-send-on-chatgpt-2307e3ede14e"><strong>What really happens when you click ‘Send’ on ChatGPT</strong></a> — A journey through modern AI Infrastructure</p><p><strong>2. </strong><a href="https://pub.towardsai.net/what-do-you-do-with-a-model-thats-too-big-for-your-gpu-443bbbe70d4d"><strong>What Do You Do With a Model That’s Too Big for Your GPU?</strong></a> — Quantization, Sharding and Parallelism Explained</p><p><strong>3. </strong><a href="https://pub.towardsai.net/how-does-one-gpu-serve-hundreds-of-users-at-the-same-time-b999e60a440e"><strong>How Does One GPU Serve Hundreds of Users at the Same Time?</strong> </a>— Inside an LLM inference server</p><p><strong>4. </strong><a href="https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7"><strong>The KV Cache Explained: Why Long Conversations Get Expensive</strong></a> — How LLMs remember context without recomputing everything</p><p><strong>5. </strong><a href="https://pub.towardsai.net/why-is-your-llm-recomputing-the-same-prompt-1-000-times-a-day-675e0cdab4b3"><strong>Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?</strong></a> — Prefix caching, radix trees and block hashing explained</p><p><strong>6. </strong><a href="https://pub.towardsai.net/why-traditional-load-balancing-breaks-for-llms-40311f7b29e2"><strong>Why Traditional Load Balancing Breaks for LLMs</strong></a> — Building an LLM-aware router<br>(Next Article)</p><p><strong>7. </strong><a href="https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c"><strong>Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster</strong></a></p><p><strong>8. </strong><a href="https://pub.towardsai.net/llm-d-explained-69f7d0b2b035"><strong>LLM-D Explained</strong></a> — How modern AI infrastructure routes, schedules and scales LLM inference</p><p><strong>9. </strong><a href="https://pub.towardsai.net/inside-a-modern-ai-inference-platform-8846434bb14b"><strong>Inside a Modern AI Inference Platform</strong></a> — The full stack end-to-end</p><p><strong>Sources</strong></p><ul><li><a href="https://docs.vllm.ai/en/latest/design/prefix_caching.html">vLLM — Automatic prefix caching</a></li><li><a href="https://arxiv.org/abs/2312.07104">SGLang — RadixAttention / Efficiently Programming Structured Language Model Programs</a></li><li><a href="https://arxiv.org/abs/2309.06180">Efficient Memory Management for LLM Serving with PagedAttention</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=675e0cdab4b3" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/why-is-your-llm-recomputing-the-same-prompt-1-000-times-a-day-675e0cdab4b3">Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Does One GPU Serve Hundreds of Users at the Same Time?]]></title>
            <link>https://pub.towardsai.net/how-does-one-gpu-serve-hundreds-of-users-at-the-same-time-b999e60a440e?source=rss-82d3a738e07a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b999e60a440e</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[llm-inference]]></category>
            <dc:creator><![CDATA[Rinit Jain]]></dc:creator>
            <pubDate>Mon, 07 Sep 2026 19:01:01 GMT</pubDate>
            <atom:updated>2026-09-15T13:51:22.121Z</atom:updated>
            <content:encoded><![CDATA[<h4>Inside an LLM inference server</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ogPYNUlCmxzb75F_AU78FQ.png" /></figure><blockquote><strong><em>TL;DR</em></strong></blockquote><blockquote>The model weights are loaded into GPU memory <strong>once</strong> and shared by every request. Only the per-request state is duplicated.</blockquote><blockquote>A request runs in two phases with opposite hardware behaviour: <strong>prefill</strong> is compute-bound, <strong>decode</strong> is memory-bandwidth-bound.</blockquote><blockquote>During decode the GPU reads the entire model from memory to produce <strong>one token</strong>. Batching amortises that read across many requests, which is why it works so well.</blockquote><blockquote><strong>Continuous batching</strong> schedules at the token level instead of the request level, so finished requests leave and new ones join mid-flight.</blockquote><blockquote>Concurrency is usually capped by <strong>KV cache memory</strong>, not by compute.</blockquote><blockquote>Latency needs two numbers, not one: <strong>TTFT</strong> and <strong>TPOT</strong>. Raising batch size improves throughput and worsens both.</blockquote><p>A single GPU can serve hundreds of concurrent users against a model with tens of billions of parameters. That sounds implausible. The model is enormous, the GPU has one set of compute units, and every user expects a response streaming back within a second or two.</p><p>The obvious approach — give each user their own GPU — fails immediately on arithmetic. A hundred concurrent users would mean a hundred copies of a model that might occupy 140 GB each. Nobody does this.</p><p>What actually happens is that one copy of the model serves everyone, and a piece of software called the <strong>inference server</strong> works very hard to keep the GPU busy. This article is about how that software works: queues, schedulers, batching, and the specific reasons LLM inference is harder to schedule than almost any other workload.</p><h3>1. Three things that are easy to confuse</h3><p>Worth being precise about the vocabulary before going further, because these terms get used interchangeably and they are not the same thing.</p><p><strong>The model</strong> is the artifact: an architecture plus billions of learned parameters (weights). It does not run anything by itself.</p><p><strong>The inference engine</strong> is the component that executes the model on the GPU and manages memory, batching and scheduling. <strong>vLLM</strong>, TensorRT-LLM and SGLang are inference engines.</p><p><strong>The model server</strong> is the process that exposes the engine over the network — accepting HTTP requests, tokenizing input, streaming output back. In practice engines like vLLM ship with a server, so one process does both jobs.</p><blockquote><em>The model is what gets executed. The engine decides </em>how<em> and </em>when<em> it gets executed. The server turns that into a service.</em></blockquote><h3>2. The key insight: weights are shared, state is not</h3><p>Here is the single fact that makes multi-user serving possible.</p><p>The model weights are <strong>read-only</strong> during inference. Nothing about processing your request modifies them. So they are loaded into GPU memory once, and every concurrent request reads the same copy.</p><p>What <em>is</em> per-request is much smaller: the tokens of the conversation, some position bookkeeping, and the <strong>KV cache</strong> — the attention state the model builds up as it processes your tokens, which lets it avoid recomputing everything on each generation step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V6wkclKcZw8SVQEcCyGnfA.png" /></figure><p>So, the cost of an additional user is not “another model”. It is “another slice of KV cache”. That slice is small compared to the weights — but as we will see in Section 11, it is what ultimately caps concurrency.</p><h3>3. The path a request takes inside the server</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2-EkMsH_smMMcZo5XyMfkw.png" /></figure><p>Two things are worth noticing about this diagram.</p><p>First, <strong>the loop is not the whole request.</strong> Everything from the scheduler down runs again for every single token generated. A 500-token answer goes around that loop 500 times.</p><p>Second, <strong>the scheduler runs before each pass, not once per request.</strong> That is the design decision the rest of this article hangs on</p><h3>4. Two phases, two completely different problems</h3><p>Inference is not prompt → answer. It runs in two phases, and they stress different parts of the hardware.</p><p><strong>Prefill</strong> processes the input prompt. If you send 5,000 tokens, all 5,000 are processed to build the initial attention state. Crucially, they can be processed <strong>together</strong> — token 4,000 does not need to wait for token 3,999, because the whole prompt is already known. That is a large, dense matrix operation with a lot of parallel work.</p><p><strong>Decode</strong> generates the answer one token at a time. Each token depends on every token before it, including the ones just generated. There is no way to compute token 10 before token 9 exists. Each step processes exactly <strong>one</strong> token position per sequence.</p><pre>PREFILL                      DECODE<br>5,000 tokens at once         1 token at a time<br>    ████████████             █<br>    ████████████             █<br>    ████████████             █<br>  lots of parallel work    almost none</pre><p>This asymmetry is the source of nearly every difficulty in LLM serving.</p><h3>5. Why decode is slow, and why it is slow for a surprising reason</h3><p>Intuitively you would guess decode is slow because generating text is hard work. That is not it. Decode is slow because of <strong>memory bandwidth</strong>.</p><p>To produce one output token, the GPU must read the model’s weights out of VRAM and pass the token through them. All of them. Generating a single token from a 70B model in 16-bit precision means moving roughly 140 GB from memory into the compute units.</p><p>Take a rough number for on-GPU memory bandwidth of around 3,000 GB/s:</p><pre>140 GB ÷ 3,000 GB/s ≈ 47 ms per token<br>                     ≈ 21 tokens/second</pre><p>That is the ceiling for a single request, and it has nothing to do with how much arithmetic the GPU can do. The compute units are mostly idle, waiting for weights to arrive. Decode is <strong>memory-bandwidth-bound</strong>.</p><p>Prefill is the opposite. It processes thousands of token positions against that same single weight read, so there is far more arithmetic per byte loaded. Prefill is <strong>compute-bound</strong> — it can actually saturate the GPU’s arithmetic units.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A7d7-LlpY0ZdeEoyetyLpA.png" /><figcaption>Prefill vs Decode: Utilization Explained</figcaption></figure><blockquote>Prefill is limited by how fast the GPU can calculate. Decode is limited by how fast the GPU can read.</blockquote><h3>6. Batching: the fix falls out of the previous section</h3><p>Look again at that 47 ms weight read. The GPU spent it fetching 140 GB of weights to serve <strong>one</strong> user’s token. The compute units were idle for most of it.</p><p>So bring more users. If eight requests are all mid-generation, the server can process their next token <em>in the same forward pass</em>. The weights get read once, and that read now serves eight tokens instead of one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mezJ1B0uV6dEqyBwdreVwA.png" /></figure><p>Throughput rises close to linearly for a while, and per-user latency barely moves, because the expensive part — the weight read — was already being paid for. This is why <strong>batching is not a minor optimization in LLM serving. It is the entire economic basis of it.</strong></p><p>The scaling is not free forever. Each additional sequence adds its own KV cache reads and attention work, so at some point the pass becomes genuinely compute- or memory-limited and per-token latency starts climbing. But the first stretch of that curve is close to free throughput, and every serving system is built to exploit it.</p><h3>7. Static batching, and why it fails</h3><p>The simplest implementation collects requests until the batch is full, runs them together, and returns when all are done. This is <strong>static batching</strong>.</p><p>It is a poor fit for LLM traffic for two reasons.</p><p><strong>Requests wait to start.</strong> With a batch size of eight, the seventh request sits idle until an eighth arrives. Latency depends on other people’s traffic.</p><p><strong>Requests wait to finish.</strong> Output lengths vary enormously — one user asks for a yes/no answer, another for a 2,000-word essay. In a static batch, everyone waits for the longest generation. Slots belonging to finished requests sit empty, doing nothing.</p><pre>Static batching (time →)<br><br>A ████░░░░░░░░░░░░   done early, slot wasted<br>B ████████░░░░░░░░   done, slot wasted<br>C ████████████████   the straggler<br>D ██████░░░░░░░░░░   done, slot wasted<br>                  ↑<br>       batch releases only here</pre><p><strong>Dynamic batching</strong> patches the first problem: launch when the batch is full <em>or</em> when a timeout expires, whichever comes first. That bounds the waiting-to-start cost. It does nothing about the second problem, which for LLMs is the bigger one.</p><h3>8. Continuous batching: schedule tokens, not requests</h3><p>The real fix is to stop treating a batch as a fixed group. Introduced as <strong>iteration-level scheduling</strong> in the Orca paper and now standard, <strong>continuous batching</strong> re-forms the batch before every single forward pass.</p><p>A request that finishes leaves immediately, and a waiting request takes its slot on the very next step.</p><pre>Continuous batching (time →)<br><br>A ██████<br>B ████<br>C ████████████<br>D     ██████████<br>E         ██████<br>F              ████</pre><p>Nothing waits for a batch boundary, because there are no batch boundaries. This is what “continuous”, “in-flight” and “iteration-level” batching all refer to — the same idea under different names.</p><p>The gain is large precisely because output lengths vary so much. The more variance in generation length, the more slots static batching wastes and the more continuous batching recovers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g2ZHuXaBDKsPsD9HWCWHNw.png" /><figcaption>Static vs Continuous Batching</figcaption></figure><h3>9. What the scheduler actually decides</h3><p>Given that the batch is rebuilt every step, the scheduler has a real decision to make each iteration. Roughly:</p><ul><li>Which waiting requests can be <strong>admitted</strong> — is there enough free KV cache memory to accept them?</li><li>Should this step run <strong>prefill</strong> for a newly admitted request, or <strong>decode</strong> for requests already generating?</li><li>If memory is exhausted, should a running request be <strong>preempted</strong> to free space?</li></ul><p>That last one is worth noting. When the KV cache fills up, the engine can evict a running sequence — either dropping its cached state (to be recomputed later) or swapping it out — and resume it once memory frees up. From the outside this looks like a request that suddenly stalls mid-generation.</p><p>The first two decisions create a direct conflict, which is the subject of the next section.</p><h3>10. Prefill and decode fight each other</h3><p>Prefill is a big, compute-hungry operation. Decode is a stream of small, latency-sensitive steps.</p><p>Put them in the same engine and they interfere. When a request arrives with a 20,000-token prompt, running its prefill occupies the GPU for a noticeable stretch. Every user currently mid-generation sees their token stream <strong>pause</strong> for the duration.</p><p>The result is a genuine trade-off:</p><ul><li>Prioritize prefill → new users get fast TTFT, existing users see stuttering output.</li><li>Prioritize decode → existing users stream smoothly, new users wait longer for their first token.</li></ul><p><strong>Chunked prefill</strong> is the standard mitigation. Instead of processing a 20,000-token prompt in one enormous step, the scheduler splits it into chunks and interleaves them with decode steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JWrcAUORVy2jXtPoSDV8ZQ.png" /><figcaption>Chunked Scheduling</figcaption></figure><p>This smooths inter-token latency for existing users at a small cost to the new request’s TTFT, and it lets a single batch contain both prefill and decode work — which keeps the compute units busier than pure decode ever could.</p><p>The same tension, taken to its conclusion, leads some systems to run prefill and decode on <strong>entirely separate GPU pools</strong>. That is a topic for a later article.</p><h3>11. What actually limits concurrency</h3><p>Here is where most intuitions are wrong. Ask an engineer what caps the number of concurrent users on a GPU and they will usually say compute. It is almost always <strong>KV cache memory</strong>.</p><p>Recall the memory budget:</p><pre>VRAM = model weights + KV cache + runtime buffers</pre><p>The weights are fixed. Whatever remains is the KV cache pool, and every active sequence draws from it in proportion to its context length.</p><p>The consequence is that <strong>your maximum concurrency depends on how long your users’ conversations are.</strong> A deployment comfortably serving 200 concurrent requests at 1,000 tokens of context each may collapse to a fraction of that if those contexts grow to 32,000 tokens — the same GPU, the same model, the same compute, but the cache pool drained far faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Zuz4_SAajbdP1joHQeXwA.png" /><figcaption>Context Length vs Concurrency</figcaption></figure><p>This is why “how many users can this GPU handle?” has no answer without a context length attached. And it is why the engine’s memory management — how the KV cache is allocated, blocked and reclaimed — has such a direct effect on serving capacity.</p><p><em>Those mechanics get their own article; see Related reading.</em></p><h3>12. Measuring what users actually feel</h3><p>Because generation is streamed, a single latency number is misleading. Three metrics matter.</p><p><strong>TTFT — Time To First Token.</strong> How long before anything appears on screen. Driven by queueing, admission and prefill. This is what determines whether the interface feels responsive.</p><p><strong>TPOT — Time Per Output Token.</strong> The average time to produce each subsequent token. Driven by decode. This determines how fast the answer reads out.</p><p><strong>ITL — Inter-Token Latency.</strong> The gap between individual streamed tokens. Closely related to TPOT, but it exposes <em>variance</em> — the stutters caused by other requests’ prefill work — which an average hides.</p><pre>Request<br>   │<br>   └───────────────┐ first token<br>                   ↑<br>                 TTFT<br><br>token → token → token → token<br>         ↑<br>     TPOT / ITL</pre><p>Alongside these sit the throughput measures: <strong>output tokens per second</strong> across the whole server, and <strong>requests per second</strong>.</p><p>The metric that ties everything together is <strong>goodput</strong>: the rate of requests completed <em>while meeting your latency targets</em>. A server reporting excellent token throughput while every user waits eight seconds for a first token has high throughput and terrible goodput. Goodput is the number worth optimizing.</p><h3>13. The knob: throughput versus latency</h3><p>Almost every tuning decision in an inference server reduces to one trade-off, controlled by how many sequences the scheduler allows to run concurrently.</p><p><strong>Larger batches</strong> → more tokens per weight read → higher throughput, lower cost per token. But more sequences share each pass, so per-user TPOT rises, and more requests compete for KV cache, so queueing grows and TTFT rises.</p><p><strong>Smaller batches</strong> → snappier individual responses, worse GPU economics.</p><pre>throughput<br>    │           ╭──────────  saturation<br>    │       ╭───╯<br>    │   ╭───╯<br>    │ ╭─╯<br>    └─────────────────────── batch size<br>      ↑ latency stays flat   ↑ latency climbs</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nETC-mvl11VZ1QuYdz7XdA.png" /><figcaption>Throughput and Latency Knee Analysis</figcaption></figure><p>The useful operating point is the knee: the largest batch size that still meets your TTFT and TPOT targets. That is a business decision as much as a technical one — an interactive chat product and an overnight batch summarisation job sit at opposite ends of this curve, on identical hardware.</p><h3>14. The knobs in practice</h3><p>Most engines expose the same handful of controls. In vLLM’s naming:</p><p><strong>GPU memory utilization</strong> — what fraction of VRAM the engine may claim. Higher means a bigger KV cache pool and more concurrency; too high and you risk out-of-memory errors.</p><p><strong>Max sequences</strong> — the ceiling on concurrently running requests. Your direct handle on the trade-off in Section 13.</p><p><strong>Max model length</strong> — the longest context accepted. Caps how much cache any single request can consume, which protects everyone else.</p><p><strong>Chunked prefill size</strong> — how finely long prompts are split, trading a new request’s TTFT against existing users’ inter-token smoothness.</p><p>There is no default configuration that is correct for all workloads, because the right answer depends on your prompt lengths, output lengths and latency targets.</p><h3>15. Why GPU utilization lies</h3><p>One warning before closing. nvidia-smi reporting 95% utilization does not mean the GPU is doing 95% of the useful work it could.</p><p>That figure reports whether <em>any</em> kernel was executing — not whether the compute units were doing anything worthwhile. A GPU in a pure decode loop, stalled on memory reads with most of its arithmetic capacity idle, will happily report near-100% utilization.</p><p>That is exactly the condition batching is meant to fix. So a server showing high utilization and low token throughput is not a healthy server — it is a bandwidth-starved one, and the fix is a larger batch, not a bigger GPU. Judge an inference deployment on tokens per second and goodput, not on utilization.</p><h3>16. Putting it together</h3><p>Two hundred users are talking to one GPU. What is actually happening?</p><p>The model weights sit in VRAM, loaded once. The remaining memory is a KV cache pool, sliced among the active conversations.</p><p>Requests arrive continuously. Each one is tokenized and queued. Before every forward pass the scheduler looks at what is waiting and what is running, admits what fits in memory, and assembles a batch — typically some decode steps plus a chunk of someone’s prefill.</p><p>The GPU runs that pass. The weights are read once and produce one token for each of the sequences in the batch. Those tokens stream out to their respective users. Sequences that just finished release their cache slices; requests that were waiting take their place.</p><p>Then it does it again, tens of times per second.</p><blockquote>No user has a GPU. No user has a copy of the model. Every user has a slice of memory and a seat in a batch that is rebuilt thousands of times a second.</blockquote><h3>The next problem</h3><p>Everything in this article eventually came back to the same resource.</p><p>Batching works because weights are shared. Concurrency is capped not by compute but by cache memory. The scheduler’s admission decision is a memory decision. Preemption happens when memory runs out. Capacity depends on context length, because context length is what consumes memory. The number of users a GPU can serve is, more than anything else, a question of how much KV cache fits and how efficiently it is used.</p><p>Which means we have been treating the most important resource in the system as a black box. We have said the KV cache exists, that it grows with context, and that it is what runs out. We have not said how it is stored, how it is allocated when every sequence is a different length, what happens to the space when a request finishes, or which state gets thrown away first when the pool is full.</p><p>Those answers determine, very directly, how many users fit on your hardware:</p><p><strong>How does an inference engine actually store, allocate and reclaim attention state — and why does that determine serving capacity?</strong></p><p>That takes us inside the KV cache itself.</p><h3>The series</h3><p><strong>1. </strong><a href="https://pub.towardsai.net/what-really-happens-when-you-click-send-on-chatgpt-2307e3ede14e"><strong>What really happens when you click ‘Send’ on ChatGPT</strong></a> — A journey through modern AI Infrastructure</p><p><strong>2. </strong><a href="https://pub.towardsai.net/what-do-you-do-with-a-model-thats-too-big-for-your-gpu-443bbbe70d4d"><strong>What Do You Do With a Model That’s Too Big for Your GPU?</strong></a> — Quantization, Sharding and Parallelism Explained</p><p><strong>3. </strong><a href="https://pub.towardsai.net/how-does-one-gpu-serve-hundreds-of-users-at-the-same-time-b999e60a440e"><strong>How Does One GPU Serve Hundreds of Users at the Same Time?</strong> </a>— Inside an LLM inference server</p><p><strong>4. </strong><a href="https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7"><strong>The KV Cache Explained: Why Long Conversations Get Expensive</strong></a> — How LLMs remember context without recomputing everything<br>(Next Article)</p><p><strong>5. </strong><a href="https://pub.towardsai.net/why-is-your-llm-recomputing-the-same-prompt-1-000-times-a-day-675e0cdab4b3"><strong>Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?</strong></a> — Prefix caching, radix trees and block hashing explained</p><p><strong>6. </strong><a href="https://pub.towardsai.net/why-traditional-load-balancing-breaks-for-llms-40311f7b29e2"><strong>Why Traditional Load Balancing Breaks for LLMs</strong></a> — Building an LLM-aware router</p><p><strong>7. </strong><a href="https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c"><strong>Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster</strong></a></p><p><strong>8. </strong><a href="https://pub.towardsai.net/llm-d-explained-69f7d0b2b035"><strong>LLM-D Explained</strong></a> — How modern AI infrastructure routes, schedules and scales LLM inference</p><p><strong>9. </strong><a href="https://pub.towardsai.net/inside-a-modern-ai-inference-platform-8846434bb14b"><strong>Inside a Modern AI Inference Platform</strong></a> — The full stack end-to-end</p><p><strong>Sources</strong></p><ul><li><a href="https://docs.vllm.ai/">vLLM documentation</a></li><li><a href="https://www.usenix.org/conference/osdi22/presentation/yu">Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI ‘22)</a></li><li><a href="https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/">NVIDIA — Mastering LLM Techniques: Inference Optimization</a></li><li><a href="https://handbook.modular.com/inference-optimization/static-dynamic-continuous-batching/">Modular — Static, dynamic and continuous batching</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b999e60a440e" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/how-does-one-gpu-serve-hundreds-of-users-at-the-same-time-b999e60a440e">How Does One GPU Serve Hundreds of Users at the Same Time?</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The KV Cache Explained: Why Long Conversations Get Expensive]]></title>
            <link>https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7?source=rss-82d3a738e07a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bd8f77dd7e7</guid>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[kv-cache]]></category>
            <category><![CDATA[llm-context-window]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Rinit Jain]]></dc:creator>
            <pubDate>Mon, 07 Sep 2026 16:31:00 GMT</pubDate>
            <atom:updated>2026-09-15T13:50:41.955Z</atom:updated>
            <content:encoded><![CDATA[<h4>How LLMs remember context without recomputing everything</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BzHV2hgJuS-5Sy2EW95Xnw.png" /></figure><blockquote><strong><em>TL;DR</em></strong></blockquote><blockquote>Generating each new token requires attention over every previous token. Without caching, the model would redo that work on every single step.</blockquote><blockquote>The keys and values of past tokens <strong>never change</strong>, so they are computed once and stored. That store is the KV cache.</blockquote><blockquote>Its size is fixed per token by the model’s architecture, and it scales with <strong>context length × concurrent users</strong>. It routinely exceeds the size of the model weights.</blockquote><blockquote>Reserving memory per sequence up front wastes most of it, because you never know how long a response will be.</blockquote><blockquote><strong>PagedAttention</strong> stores the cache in small fixed-size blocks instead, allocated on demand — which is where a large share of modern serving throughput comes from.</blockquote><blockquote>When the pool fills, the engine evicts running requests. Cache memory is what actually caps concurrency.</blockquote><p>Ask an LLM a question and the answer arrives one token at a time. To produce each token, the model attends to everything before it — the prompt and every token it has already generated.</p><p>Done naively, that is a staggering amount of repeated work. Generating token 500 would mean reprocessing 499 tokens that were already processed to generate token 499, which had already reprocessed the 498 before it.</p><p>The <strong>KV cache</strong> is what stops that. It is a simple idea with outsized consequences: it is usually the largest dynamic consumer of GPU memory in a serving system, and it — not compute — is what decides how many users a GPU can hold.</p><p>This article covers what it stores, what it costs, how it is managed, and what happens when it runs out.</p><h3>1. What attention needs</h3><p>A short primer, only as much as we need.</p><p>For every token it processes, a transformer produces three vectors:</p><p><strong>Query (Q)</strong> — what this token is looking for. <strong>Key (K)</strong> — what this token offers, so other tokens can find it. <strong>Value (V)</strong> — the actual information this token contributes.</p><p>To compute the output for a given token, the model compares that token’s query against the keys of all preceding tokens, and uses the resulting weights to blend their values.</p><p>So generating a new token requires <strong>the K and V of every token before it.</strong></p><h3>2. The observation that makes caching possible</h3><p>Here is the property everything depends on.</p><p>When the model generates token 501, it needs the keys and values of tokens 1 through 500. Those keys and values were already computed when those tokens were processed — and <strong>they do not change.</strong> Token 200’s key is a function of token 200 and the tokens before it. Adding new tokens to the end of the sequence does not alter it.</p><p>The queries are different: each new token needs a fresh query. But queries are needed only for the token being generated right now, and then discarded.</p><pre>Generating a new token needs:<br><br>  Q  →  only for the current token   →  compute fresh, throw away<br>  K  →  for every previous token     →  same as last step<br>  V  →  for every previous token     →  same as last step</pre><p>So the keys and values get computed once and kept. That store is the <strong>KV cache</strong>.</p><p>Without it, every generation step reprocesses the entire context. With it, each step processes exactly one new token and reads the rest from memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OFoBDZv63i341P7samnaww.png" /></figure><h3>3. What it costs</h3><p>The cache is not free, and its size is fixed by the model’s architecture. Per token:</p><pre>bytes per token = 2 × layers × KV heads × head dimension × bytes per value</pre><p>The leading 2 is for K and V. Every layer keeps its own, because attention happens at every layer.</p><p>For a 70B-class model with 80 layers, 8 KV heads, a head dimension of 128, in 16-bit precision:</p><pre>2 × 80 × 8 × 128 × 2 bytes ≈ 320 KB per token</pre><p>That looks small. Now scale it:</p><pre>8,000 tokens of context  ≈   2.5 GB   for one user<br>   100 users at 8,000 tokens  ≈   250 GB</pre><p>Compare that with the model itself, which needs roughly 140 GB of weights at the same precision. <strong>The cache for a hundred moderately-long conversations is larger than the model.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G53dfDRPwqy3vjIjSCCoOA.png" /></figure><p>This is the fact that surprises people. Weights are the number everyone sizes hardware on, and in a busy deployment they are frequently the smaller half of the memory budget.</p><h3>4. Two axes, not one</h3><p>The cache grows along two dimensions at once, and both are outside your control.</p><p><strong>Context length.</strong> Every token added to a conversation adds its slice. A conversation that has been running for a while costs more than a fresh one, even from the same user.</p><p><strong>Concurrency.</strong> Every simultaneous request has its own cache. They do not share anything.</p><p>Multiply them and you get the real number. This is why a deployment can be perfectly stable at a hundred short conversations and fall over at twenty long ones, on identical hardware with identical compute.</p><p>It also means capacity questions are unanswerable in the abstract. “How many users can this GPU serve?” has no answer without a context length attached to it.</p><h3>5. The allocation problem</h3><p>So we know what to store. The harder question is <em>where</em> — and this is where naive implementations lose most of their memory.</p><p>The difficulty: <strong>you do not know how long a sequence will be.</strong> A request arrives with a 200-token prompt. Will the answer be 5 tokens or 5,000? Nobody knows, including the model.</p><p>Early serving systems handled this by reserving a contiguous region large enough for the maximum supported length, for every sequence. If the model supports 32,000 tokens, every request reserves 32,000 tokens’ worth of cache the moment it is admitted.</p><p>The waste is severe, and it comes in three forms:</p><p><strong>Over-reservation.</strong> A request that generates 100 tokens holds a 32,000-token reservation for its entire life. Over 99% of that memory is untouched but unavailable to anyone else.</p><p><strong>Internal fragmentation.</strong> Even when a sequence does grow, the unused tail of its reservation sits idle until it gets there — which it may never do.</p><p><strong>External fragmentation.</strong> Requests finish in a different order than they started, leaving gaps of free memory between live reservations. The gaps may add up to plenty of space, but if none is large enough to hold a new contiguous reservation, a new request cannot be admitted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LtONvX8b7uKxQaX0sr6jSA.png" /></figure><p>The vLLM paper that introduced PagedAttention reported that systems of this kind wasted the large majority of their KV cache memory. Which meant the practical limit on concurrency was not the hardware — it was the allocator.</p><h3>6. PagedAttention: stop reserving, start allocating</h3><p>The fix is to abandon the requirement that a sequence’s cache be contiguous.</p><p><strong>PagedAttention</strong> splits the cache into small fixed-size <strong>blocks</strong>, each holding a fixed number of tokens (16 is a common default). A sequence is allocated blocks as it grows, one at a time, and those blocks can sit anywhere in the pool. A per-sequence <strong>block table</strong> records which physical blocks hold which part of the sequence, and the attention kernel follows that table when it reads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CJFDgC7uM0f80Gphw3UuTg.png" /><figcaption>PagedAttention Memory</figcaption></figure><p>The technique is borrowed from operating system virtual memory, which is where the name comes from — pages, page tables, and a layer of indirection between logical and physical addresses.</p><p>Three things change immediately:</p><p><strong>No over-reservation.</strong> A sequence holds only the blocks it has actually filled. A 100-token answer uses seven blocks, not two thousand.</p><p><strong>No external fragmentation.</strong> Every block is the same size, so any free block fits any sequence. Free memory is never unusable because of its shape.</p><p><strong>Waste is bounded.</strong> The only unused space is the partially-filled last block of each sequence — at most 15 tokens’ worth. Compared to reserving thousands, this is nothing.</p><p>The practical effect is that far more sequences fit in the same VRAM. Since concurrency is what drives batching, and batching is what drives throughput, better cache allocation translates fairly directly into more tokens per second from the same GPU.</p><h3>7. When the pool fills up</h3><p>Blocks are allocated on demand, which raises an obvious question: what happens when a running sequence needs a new block and none is free?</p><p>The engine has to take memory from someone. It <strong>preempts</strong> a running request, freeing its blocks, and resumes it later when space is available. Two ways to do that:</p><p><strong>Recompute.</strong> Discard the sequence’s cache entirely. When it resumes, run prefill again over its tokens to rebuild the state. Cheap to evict, expensive to restore.</p><p><strong>Swap.</strong> Copy the blocks out to CPU memory and copy them back on resume. Preserves the work, but pays the cost of moving data across the PCIe link in both directions.</p><p>Either way, from the user’s side this appears as a response that stalls mid-generation and then continues. From the operator’s side, frequent preemption is a clear signal that the deployment is running past its memory limits — usually the point to reduce concurrency, shorten the maximum context, or add capacity.</p><p>This is also the mechanism behind a claim that sounds odd until you know where it comes from: <strong>concurrency in an LLM server is capped by cache memory, not by compute.</strong> The GPU is rarely out of arithmetic. It is out of blocks.</p><h3>8. Making the cache smaller</h3><p>Since cache size is set by architecture and context, there are only a few levers.</p><p><strong>Grouped-query attention (GQA)</strong> is the big one, and it is why the number in Section 3 was as small as it was. In classic multi-head attention, every attention head keeps its own K and V. GQA has several query heads share one set, cutting the KV heads dramatically. That 70B example used 8 KV heads; with 64 it would have been eight times larger — 2.5 MB per token instead of 320 KB. Nearly every recent model uses GQA or a variant, and it is largely a serving-cost decision.</p><p><strong>KV cache quantization</strong> stores keys and values in 8-bit instead of 16-bit, halving the cache at some cost to quality. Independent of weight quantization — you can do either, or both.</p><p><strong>Context limits.</strong> Capping the maximum accepted context bounds what any single request can consume, protecting everyone else’s capacity. Blunt, but effective and entirely under your control.</p><p><strong>Longer-term architectural work</strong> — sliding-window attention, sparse and linear attention, cross-layer sharing — aims at changing the growth curve rather than its constant. This is an active area and worth watching if long context matters to your workload.</p><h3>9. The mental model</h3><p>Four sentences hold most of it:</p><p>The weights are static, shared by everyone, and sized once. The KV cache is dynamic, private to each request, and grows with both context length and concurrency. Whatever VRAM the weights do not occupy becomes the cache pool. <strong>How efficiently that pool is managed determines how many users the GPU can serve.</strong></p><p>That is why memory management is not a detail of LLM serving. It is most of it.</p><h3>The next problem</h3><p>There is one thing we have quietly accepted throughout.</p><p>When a request finishes, its cache is released. The blocks return to the pool and their contents are gone. That is correct — the conversation is over, the state is not needed.</p><p>But look at what actually flows through a production deployment. Every request from the same application begins with the same system prompt, often thousands of tokens of instructions and policy. Every follow-up turn in a chat re-sends the entire conversation so far. Users in the same product ask questions against the same documents.</p><p>The prefixes are identical. And we compute them from scratch, every single time, throwing the result away at the end of each request.</p><p>That is a lot of work being repeated — the very thing the KV cache was invented to avoid, reappearing one level up. Within a request we already refuse to recompute. Across requests, we do nothing.</p><p><strong>If two requests start with the same tokens, why compute the same cache twice?</strong></p><p>That question turns out to have a genuinely interesting answer.</p><h3>The series</h3><p><strong>1. </strong><a href="https://pub.towardsai.net/what-really-happens-when-you-click-send-on-chatgpt-2307e3ede14e"><strong>What really happens when you click ‘Send’ on ChatGPT</strong></a> — A journey through modern AI Infrastructure</p><p><strong>2. </strong><a href="https://pub.towardsai.net/what-do-you-do-with-a-model-thats-too-big-for-your-gpu-443bbbe70d4d"><strong>What Do You Do With a Model That’s Too Big for Your GPU?</strong></a> — Quantization, Sharding and Parallelism Explained</p><p><strong>3. </strong><a href="https://pub.towardsai.net/how-does-one-gpu-serve-hundreds-of-users-at-the-same-time-b999e60a440e"><strong>How Does One GPU Serve Hundreds of Users at the Same Time?</strong> </a>— Inside an LLM inference server</p><p><strong>4. </strong><a href="https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7"><strong>The KV Cache Explained: Why Long Conversations Get Expensive</strong></a> — How LLMs remember context without recomputing everything</p><p><strong>5. </strong><a href="https://pub.towardsai.net/why-is-your-llm-recomputing-the-same-prompt-1-000-times-a-day-675e0cdab4b3"><strong>Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?</strong></a> — Prefix caching, radix trees and block hashing explained<br>(Next Article)</p><p><strong>6. </strong><a href="https://pub.towardsai.net/why-traditional-load-balancing-breaks-for-llms-40311f7b29e2"><strong>Why Traditional Load Balancing Breaks for LLMs</strong></a> — Building an LLM-aware router</p><p><strong>7. </strong><a href="https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c"><strong>Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster</strong></a></p><p><strong>8. </strong><a href="https://pub.towardsai.net/llm-d-explained-69f7d0b2b035"><strong>LLM-D Explained</strong></a> — How modern AI infrastructure routes, schedules and scales LLM inference</p><p><strong>9. </strong><a href="https://pub.towardsai.net/inside-a-modern-ai-inference-platform-8846434bb14b"><strong>Inside a Modern AI Inference Platform</strong></a> — The full stack end-to-end</p><p><strong>Sources</strong></p><ul><li><a href="https://arxiv.org/abs/2309.06180">Efficient Memory Management for Large Language Model Serving with PagedAttention</a></li><li><a href="https://docs.vllm.ai/">vLLM documentation</a></li><li><a href="https://arxiv.org/abs/2305.13245">GQA: Training Generalized Multi-Query Transformer Models</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bd8f77dd7e7" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7">The KV Cache Explained: Why Long Conversations Get Expensive</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Do You Do With a Model That’s Too Big for Your GPU?]]></title>
            <link>https://pub.towardsai.net/what-do-you-do-with-a-model-thats-too-big-for-your-gpu-443bbbe70d4d?source=rss-82d3a738e07a------2</link>
            <guid isPermaLink="false">https://medium.com/p/443bbbe70d4d</guid>
            <category><![CDATA[sharding]]></category>
            <category><![CDATA[quantization]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[parallelism]]></category>
            <dc:creator><![CDATA[Rinit Jain]]></dc:creator>
            <pubDate>Sun, 06 Sep 2026 15:01:01 GMT</pubDate>
            <atom:updated>2026-09-15T13:49:47.018Z</atom:updated>
            <content:encoded><![CDATA[<h4>Quantization, Sharding and Parallelism Explained</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jgRv697O9AwJD3aHrSw_PQ.png" /></figure><blockquote><strong>TL;DR</strong></blockquote><blockquote>Model weights need roughly <strong>1 GB per billion parameters at 8-bit</strong>, double that at 16-bit. A 70B model in FP16 is ~140 GB of weights before anything else.</blockquote><blockquote><strong>Quantization</strong> shrinks the model. <strong>Offloading</strong> moves parts of it off the GPU. Both trade something away.</blockquote><blockquote><strong>Sharding</strong> splits one model across GPUs. <strong>Replication</strong> makes copies. These solve different problems and are constantly confused.</blockquote><blockquote><strong>Tensor parallelism</strong> splits inside layers and needs a fast interconnect. <strong>Pipeline parallelism</strong> splits by depth and tolerates slower links. <strong>Expert parallelism</strong> applies only to MoE models.</blockquote><blockquote>Once a model is distributed, <strong>communication bandwidth</strong> becomes a first-class constraint. More GPUs is not automatically more performance.</blockquote><blockquote>Weights are not the only thing in VRAM. The KV cache competes for the same memory.</blockquote><p>A modern LLM can contain billions or even hundreds of billions of parameters.</p><p>The problem is straightforward:</p><p><strong>GPU memory is finite.</strong></p><p>If a model requires more memory than one GPU provides, you cannot simply load the entire model onto that GPU and start serving requests.</p><p>So what do inference systems do?</p><p>They have several options:</p><ul><li>Make the model smaller through <strong>quantization</strong></li><li>Move some data outside GPU memory through <strong>offloading</strong></li><li>Split the model across multiple GPUs through <strong>sharding</strong></li><li>Replicate the model across GPUs when memory allows and more throughput is needed</li><li>Combine different forms of <strong>parallelism</strong> to distribute the computation</li></ul><p>These techniques solve different problems.</p><p>Understanding that distinction is important because “use more GPUs” is not a single strategy. There are several fundamentally different ways to use those GPUs.</p><p>This article starts from the memory problem and works upward to the distributed inference architecture.</p><h3>1. Start with the memory problem</h3><p>An LLM is a collection of learned parameters called <strong>weights</strong>. During inference, those weights must be available to the hardware doing the computation. That is the first constraint: <strong>the weights have to fit somewhere.</strong></p><p>A simplified estimate:</p><pre>weight memory ≈ number of parameters × bytes per parameter</pre><p>A 70-billion-parameter model at 16-bit precision:</p><pre>16 bits = 2 bytes<br>70,000,000,000 × 2 ≈ 140 GB</pre><p>A rule of thumb worth memorising: <strong>about 1 GB per billion parameters at 8-bit precision</strong>, halve it for 4-bit, double it for 16-bit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*efCU9CY5X8k922IUfFSilA.png" /></figure><p>But that is <strong>only the weights</strong>. Real inference also needs memory for the KV cache, activations and temporary buffers, CUDA and runtime overhead, and communication buffers.</p><p>So a 140 GB weight estimate does not mean a 140 GB GPU is enough. The actual requirement is meaningfully larger, and the gap widens with long context and high concurrency, because the KV cache scales with both.</p><h3>2. The First Question: Can We Make the Model Smaller?</h3><p>Before distributing a model across GPUs, there is an obvious question:</p><p><strong>Do we actually need all those bits?</strong></p><p>This is where <strong>quantization</strong> enters.</p><h3>Quantization</h3><p>Quantization represents model values using fewer bits.</p><p>For example:</p><p><strong>FP32</strong><br> ████████████████████████████████ 280 GB</p><p><strong>FP16 / BF16</strong><br> ████████████████ 140 GB</p><p><strong>INT8</strong><br> ████████ 70 GB</p><p><strong>INT4</strong><br> ████ 35 GB</p><p>These are simplified raw-weight estimates. Real implementations have additional metadata and overhead.</p><p>The important idea is the relationship:</p><p><strong>Fewer bits → smaller weights → lower memory requirement</strong></p><p>Hugging Face documents quantization as a way to reduce both memory footprint and computational cost, with 8-bit and 4-bit approaches commonly used for large models.</p><h3>But quantization is not free</h3><p>Reducing precision can introduce accuracy loss, and the performance impact depends on the model, hardware and quantization method.</p><p>Modern inference stacks support several approaches, including GPTQ, AWQ, bitsandbytes and other quantization schemes.</p><p>So quantization is usually the first question:</p><blockquote><strong><em>Can we reduce the model enough to fit on fewer GPUs without unacceptable quality or performance loss?</em></strong></blockquote><p>If yes, the problem may become much easier.</p><p>If not, we need to distribute the model.</p><h3>3. What About CPU Offloading?</h3><p>There is another possibility:</p><p><strong>Keep some of the model outside GPU memory.</strong></p><p>The basic architecture becomes:</p><p><strong>CPU RAM → GPU VRAM → computation</strong></p><p>Instead of keeping everything permanently in VRAM, some data can be transferred between system memory and GPU memory when needed.</p><p>CUDA’s memory system supports mechanisms that allow applications to work with data larger than the physical memory of an individual processor, although actual performance depends heavily on the hardware and access pattern.</p><p>This can help when memory capacity is the immediate problem.</p><p>But there is a fundamental trade-off.</p><p>GPU memory is extremely fast.</p><p>Moving data across the CPU-GPU boundary is much slower than accessing data already resident in GPU memory.</p><p>If the inference workload constantly needs to move model weights back and forth, the GPU can spend more time waiting for data than performing computation.</p><p>So CPU offloading is generally a <strong>memory-capacity technique</strong>, not a magical way of turning CPU RAM into equivalent GPU VRAM.</p><p>For high-performance production inference, keeping the active model state in GPU memory is usually preferable when possible.</p><h3>4. The More Scalable Answer: Sharding</h3><p>Suppose the model still does not fit.</p><p>You have four GPUs.</p><p>Instead of putting the entire model on one GPU:</p><pre>GPU 0 → entire model</pre><p>we can divide the model:</p><pre>GPU 0 → part of model<br>GPU 1 → part of model<br>GPU 2 → part of model<br>GPU 3 → part of model</pre><p>This is <strong>model sharding</strong>.</p><p>The model is still logically one model.</p><p>Physically, however, its parameters and computation are distributed across multiple devices.</p><p>But there is an important consequence.</p><p>The GPUs now have to communicate.</p><p>That communication becomes one of the central engineering constraints of distributed inference.</p><h3>5. Communication is now a first-class constraint</h3><p>Four resources govern inference performance:</p><p><strong>compute capacity</strong> (how much arithmetic the GPU can do), <br><strong>memory capacity</strong> (how much fits in VRAM), <br><strong>memory bandwidth</strong> (how fast data moves between VRAM and the compute units), and <br><strong>communication bandwidth</strong> (how fast GPUs exchange data).</p><p>The fourth one only starts to matter once you shard, and then it dominates the design. A model that fits neatly across four GPUs is not automatically efficient — those GPUs may spend a large fraction of their time exchanging intermediate results.</p><p>The scale of the problem is easier to see with numbers. Approximate figures for a current-generation datacenter GPU:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ILA5NyOImCeDM5UcJxyHFg.png" /></figure><p>Each step down is roughly an order of magnitude. That single hierarchy explains most distributed inference design decisions — including, as we will see, why some parallelism strategies work well inside a server and badly across a network.</p><blockquote>More GPUs give you more memory and more compute, but they introduce communication overhead. Distributed inference is a balance, not a multiplier</blockquote><p>That leads to the real question: <strong>how exactly should we split the model?</strong></p><h3>6. Tensor parallelism: split each layer</h3><p>The first major approach is <strong>tensor parallelism</strong> (TP). Rather than giving entire layers to different GPUs, it splits the large tensor operations <em>inside</em> the layers.</p><p>Take a matrix multiplication:</p><pre>Y = X × W</pre><p>Instead of storing the complete weight matrix W on every GPU, we divide it:</p><pre>GPU 0 → W₀<br>GPU 1 → W₁<br>GPU 2 → W₂<br>GPU 3 → W₃</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1e86itFN8G_tEfYwkQPp1A.png" /></figure><p>Each GPU performs part of the multiplication, and the partial results are combined. NVIDIA describes tensor parallelism as <a href="https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/">sharding individual Transformer layers across devices</a>.</p><p><strong>Why use it?</strong> Every GPU participates in computing the same layer. That reduces memory per GPU and can give low latency when the GPUs have a fast link. vLLM supports distributed tensor-parallel inference and recommends it for multi-GPU setups where a model is too large for one device.</p><p><strong>The cost.</strong> The GPUs must synchronise intermediate results, so collective operations like all-reduce and all-gather sit on the critical path. The faster the interconnect, the less this hurts — which is why TP is most attractive across GPUs linked by NVLink inside a single server.</p><p><strong>A practical constraint people hit immediately:</strong> the tensor-parallel degree generally has to divide the number of attention heads evenly. You cannot scale TP to an arbitrary GPU count. With grouped-query attention, the relevant number is often the KV head count, which is smaller still.</p><h3>7. What does GPU communication actually look like?</h3><p>It is easy to say TP requires GPUs to communicate. What does that mean concretely?</p><p>Split W across two GPUs:</p><pre>GPU 0: Y₀ = X × W₀<br>GPU 1: Y₁ = X × W₁</pre><p>Neither GPU has the complete result — each has its own portion. If the split was column-wise, the full result is formed by concatenating:</p><pre>Y = [Y₀ | Y₁]</pre><p>Crucially, <strong>the GPUs are not sending the model to each other.</strong> They exchange only the intermediate data needed to continue.</p><p>For other partitionings the combination is a sum instead. Say the two GPUs hold partial results:</p><pre>GPU 0: [3, 5]<br>GPU 1: [7, 2]<br>        ↓<br>   communication<br>        ↓<br>both GPUs hold: [10, 7]</pre><p>That is an <strong>all-reduce</strong>. Which collective is needed depends on how the layer was partitioned; all-reduce, all-gather and reduce-scatter are the common ones.</p><p>So the GPUs run a continuous cycle:</p><pre>compute → communicate → compute → communicate → …</pre><p>If communication is fast, distributing the computation pays off. If it is slow, the GPUs spend their time waiting for each other. When we say a model is “split across GPUs”, we do not mean each GPU independently runs an isolated piece — they participate in one coordinated computation.</p><h3>8. Pipeline parallelism: split the model by depth</h3><p><strong>Pipeline parallelism</strong> (PP) takes the opposite approach. Suppose the model has 80 Transformer layers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J5p4XyI23IqXSrw0F7-V5w.png" /><figcaption>Pipeline Parallelism</figcaption></figure><p>Each GPU owns a different section. Activations produced by one stage pass to the next, so communication happens only between adjacent stages rather than inside every layer. That is dramatically less traffic than TP.</p><p>NVIDIA’s <a href="https://nvidia.github.io/TensorRT-LLM/performance/performance-tuning-guide/deciding-model-sharding-strategy.html">model sharding guidance</a> draws the distinction directly: pipeline parallelism places contiguous groups of layers on different GPUs, while tensor parallelism distributes each layer and requires far more frequent collective communication.</p><p><strong>The trade-off is pipeline bubbles.</strong> At any moment some stages may be idle waiting for work from an earlier stage. Training systems address this by splitting work into micro-batches; for inference, the scheduler has to account for requests moving through stages sequentially.</p><p>So PP reduces communication but introduces scheduling constraints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZST4mfzN7eiFhFgQ" /></figure><h3>9. Tensor vs pipeline parallelism</h3><p>Both distribute a model across GPUs. They divide the work differently.</p><p><strong>Tensor parallelism</strong> splits computation <em>within</em> a layer, so multiple GPUs work on the same layer simultaneously. High communication, low latency, wants NVLink.</p><p><strong>Pipeline parallelism</strong> splits the model into groups of layers, each GPU owning a stage. Low communication, tolerates slower links, introduces bubbles.</p><p>In practice they combine. A model might use TP across the GPUs inside a server and PP across servers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UNjoW1SYW0mr75T3BhJPYA.png" /><figcaption>Tensor Parallelism VS Pipeline Parallelism</figcaption></figure><p>This lets very large models span machines while still exploiting fast GPU-to-GPU links within each one. It follows directly from the bandwidth hierarchy in Section 5: use the chatty strategy where bandwidth is cheap, the quiet one where it is expensive.</p><h3>10. Data parallelism is different</h3><p><strong>Data parallelism</strong> does not split the model. It replicates it:</p><pre>GPU 0 → full model<br>GPU 1 → full model<br>GPU 2 → full model<br>GPU 3 → full model</pre><p>Each GPU processes different requests. Excellent for throughput — and useless for the original problem. If the model needs 100 GB and each GPU has 80 GB, four copies do not help. You still cannot load one complete copy anywhere.</p><blockquote>Sharding solves model capacity. Replication solves workload capacity.</blockquote><h3>11. Sharding vs replication</h3><p>Consider four GPUs. There are two fundamentally different ways to use them.</p><p><strong>Option A — sharding.</strong> All four GPUs hold one model replica.</p><pre>GPU 0 ┐<br>GPU 1 ├─→ one model replica<br>GPU 2 │<br>GPU 3 ┘</pre><p>Use this when the model is too large to fit on fewer GPUs.</p><p><strong>Option B — replication.</strong> Two replicas of two GPUs each.</p><pre>GPU 0 ┐→ replica A<br>GPU 1 ┘<br><br>GPU 2 ┐→ replica B<br>GPU 3 ┘</pre><p>Use this when the model already fits in two GPUs and you want more serving capacity.</p><p><strong>And you combine them.</strong> Suppose a model requires four GPUs and you have sixteen:</p><pre>Replica 1 → GPU 0–3<br>Replica 2 → GPU 4–7<br>Replica 3 → GPU 8–11<br>Replica 4 → GPU 12–15</pre><p>Each replica internally uses tensor or pipeline parallelism; the replicas together provide data-parallel scaling. This is how a production fleet grows beyond a single distributed model instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_eMedWtpiE4FmD8NS649bw.png" /><figcaption>Sharding VS Replication</figcaption></figure><h3>12. Expert parallelism: a different kind of model</h3><p>So far we have assumed a <strong>dense</strong> model, where essentially every parameter participates for every token.</p><p>Many modern LLMs use <strong>Mixture-of-Experts</strong> (MoE) instead. An MoE model contains multiple expert networks, and a router selects which experts process each token:</p><pre>Token<br>  ↓<br>Router<br>  ↓<br>Expert 1 / Expert 4 / Expert 7<br>  ↓<br>Combine</pre><p>The key property: not every expert processes every token. That creates a distribution opportunity.</p><p><strong>Expert parallelism</strong> distributes experts across GPUs instead of replicating them all everywhere:</p><pre>GPU 0 → experts  1–4<br>GPU 1 → experts  5–8<br>GPU 2 → experts  9–12<br>GPU 3 → experts 13–16</pre><p>The router sends tokens toward the GPUs holding the required experts. This substantially reduces expert weight memory per GPU — but introduces a new communication pattern, because now the <em>tokens themselves</em> move between GPUs. <a href="https://developer.nvidia.com/cuda/cuda-x-libraries/megatron-core">Megatron Core</a> describes expert parallelism this way, and current vLLM deployments can combine data, tensor and expert parallelism depending on the architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LrfrQAeQwHjpwyGkv7Ht3A.png" /><figcaption>Expert Parallelism (MoE)</figcaption></figure><p>So EP is not “another version of tensor parallelism.” It applies specifically when the architecture contains independently routable experts.</p><h3>13. The parallelism stack</h3><p>We now have five ways to distribute a model:</p><p><strong>Tensor parallelism</strong> — split the computation inside layers. <strong>Pipeline parallelism</strong> — split the model by depth. <strong>Data parallelism</strong> — replicate the model and split requests. <strong>Expert parallelism</strong> — distribute MoE experts across GPUs. <strong>Context parallelism</strong> — distribute the context dimension itself, for very long sequences.</p><p>These compose. A large production model might run:</p><pre>Data Parallelism × Tensor Parallelism × Pipeline Parallelism</pre><p>or, for an MoE model:</p><pre>Data Parallelism × Tensor Parallelism × Expert Parallelism</pre><p>Megatron Core supports combinations of these strategies, and vLLM and TensorRT-LLM both support tensor, pipeline, data and expert-parallel deployment configurations.</p><p>The right configuration depends on model architecture, model size, GPU memory, interconnect, GPUs per node, node count, and your latency and throughput targets. <strong>There is no universally optimal parallelism configuration.</strong></p><h3>14. The model is not the only thing consuming VRAM</h3><p>Getting weights onto multiple GPUs solves one problem. It does not mean the remaining memory is free for serving requests.</p><p>During generation, an LLM maintains a <strong>KV cache</strong> — previously computed key and value tensors reused when generating subsequent tokens. Every active sequence occupies some, and it grows as that sequence gets longer. So GPU memory is shared:</p><pre>VRAM = model weights + KV cache + runtime memory</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DnyeBatB_Dj_r4fyjc-79g.png" /></figure><p>Sharding changes the shape of this budget in a way that catches people out. Splitting a model across four GPUs cuts the <strong>weight</strong> footprint per GPU by roughly four. It does not shrink the KV cache the same way — with tensor parallelism the cache is partitioned across GPUs, but total cache demand still scales with concurrency and context length, and under some attention configurations parts of it are replicated on every device.</p><blockquote>Fitting the model is a capacity problem. Serving many concurrent requests is a separate KV-cache capacity problem.</blockquote><p>The KV cache’s memory behaviour has large enough consequences that it gets its own article — see Related reading.</p><h3>15. A model replica becomes a small distributed system</h3><p>This changes how to think about a model server. With one GPU:</p><pre>Request → Model Server → GPU</pre><p>With a distributed model:</p><pre>Request<br>   ↓<br>Model Server<br>   ↓<br>Distributed inference runtime<br>   ↓<br>GPU 0 ↔ GPU 1 ↔ GPU 2 ↔ GPU 3</pre><p>From the outside it still looks like one logical service. Internally it coordinates multiple workers.</p><h3>16. Leader and worker architecture</h3><p>A simplified distributed model server looks like this:</p><pre>        API / Model Server<br>                │<br>                ▼<br>       Coordinator / Leader<br>                │<br>   ┌──────┬─────┴─────┬──────┐<br>   ▼      ▼           ▼      ▼<br> GPU    GPU         GPU    GPU<br>worker  worker    worker  worker<br>   0      1           2      3</pre><p>The coordinator handles orchestration; the workers perform GPU-side computation. The exact architecture differs between engines, but the general idea holds:</p><blockquote>One logical model-serving endpoint can represent a coordinated group of GPU processes.</blockquote><p>vLLM’s architecture, for example, includes an API server, engine core, scheduler, GPU workers and distributed coordination components. This matters even more once the model spans machines.</p><h3>17. One model can span multiple machines</h3><p>Suppose each server has eight GPUs and the model needs sixteen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cyXhbX_WV-v2ZQb_w3J-mA.png" /></figure><p>Now the model spans the network between machines, and the communication hierarchy has three tiers:</p><pre>GPU memory              fastest<br>    ↓<br>GPU ↔ GPU (in node)<br>    ↓<br>Node ↔ Node             slowest</pre><p>The farther communication travels, the more carefully the system must be designed. This is precisely why tensor parallelism is attractive <em>within</em> a node while pipeline parallelism becomes useful <em>across</em> nodes. vLLM’s guidance describes exactly this: single-node tensor parallelism, and multi-node combinations of tensor and pipeline parallelism.</p><h3>18. So how do we choose?</h3><p>A practical sequence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9xDG019kTNvizTjmvaaaxg.png" /><figcaption>How to choose parallelism strategy for LLM inference</figcaption></figure><p><strong>The model fits on one GPU</strong> → use single-GPU inference. Don’t shard without a reason.</p><p><strong>It does not fit, but fits across GPUs in one node</strong> → tensor parallelism. The fast interconnect makes frequent communication practical.</p><p><strong>It does not fit in one node</strong> → tensor parallelism within nodes, pipeline parallelism across them.</p><p><strong>It fits on one distributed replica, but throughput is insufficient</strong> → add data-parallel replicas.</p><p><strong>It is a Mixture-of-Experts model</strong> → expert parallelism, usually combined with TP and DP.</p><p><strong>It barely misses the memory limit</strong> → try quantization before adding hardware, assuming the resulting quality is acceptable.</p><p>This is not a rigid decision tree, but it is a useful starting point.</p><h3>19. The real production equation</h3><p>Serving a large LLM is not:</p><pre>model size &lt; GPU memory</pre><p>It is closer to:</p><pre>model weights + KV cache + runtime memory<br>+ communication buffers + concurrency<br><br>must fit within the available distributed GPU memory</pre><p>and then:</p><pre>compute + memory bandwidth + interconnect bandwidth<br><br>must be fast enough to meet latency and throughput targets</pre><p>That is why two systems with identical GPU counts can have very different inference performance. The hardware matters — but so does how the model is distributed across it.</p><h3>20. The bigger picture</h3><p>We started with one problem: the model does not fit on one GPU. There are six responses, and they combine.</p><p><strong>Quantization</strong> makes the model smaller. <strong>Offloading</strong> moves data outside GPU memory. <strong>Tensor parallelism</strong> splits layers across GPUs. <strong>Pipeline parallelism</strong> splits the model into stages. <strong>Data parallelism</strong> replicates it to serve more requests. <strong>Expert parallelism</strong> distributes experts in an MoE model.</p><p>A real deployment might end up looking like:</p><pre>USERS<br>                │<br>          Load Balancer<br>                │<br>      ┌─────────┴─────────┐<br>      ▼                   ▼<br> MODEL REPLICA A     MODEL REPLICA B<br>   GPU 0–3              GPU 4–7</pre><p>Each replica internally using tensor or pipeline parallelism. This is the point where a model stops being a single GPU process and becomes a distributed serving system.</p><h3>The next problem</h3><p>The model now fits. It is quantized, sharded, spread across GPUs that can talk to each other quickly enough to be useful.</p><p>But look at what we actually sized for: <strong>weights</strong>. Every calculation in this article was about making a fixed, read-only block of parameters fit into memory. And Section 14 already hinted at the catch — the weights are not the only thing in VRAM, and they are not the part that grows.</p><p>Now put the replica into production. Hundreds of users start sending requests to it simultaneously. They are not each getting their own copy of the model; there is only one, and they all share it. What each of them <em>does</em> get is a slice of the memory left over after the weights — and those slices grow as their conversations get longer.</p><p>So the question changes shape entirely. Not “does the model fit?”, which we have answered, but:</p><p><strong>How does a single GPU serve hundreds of users at once, and what actually decides how many it can hold?</strong></p><p>That means going inside the inference server itself — the queues, the scheduler, and the batching machinery that lets one set of weights produce tokens for many people at the same time.</p><h3>The series</h3><p><strong>1. </strong><a href="https://pub.towardsai.net/what-really-happens-when-you-click-send-on-chatgpt-2307e3ede14e"><strong>What really happens when you click ‘Send’ on ChatGPT</strong></a> — A journey through modern AI Infrastructure</p><p><strong>2. </strong><a href="https://pub.towardsai.net/what-do-you-do-with-a-model-thats-too-big-for-your-gpu-443bbbe70d4d"><strong>What Do You Do With a Model That’s Too Big for Your GPU?</strong></a> — Quantization, Sharding and Parallelism Explained</p><p><strong>3. </strong><a href="https://pub.towardsai.net/how-does-one-gpu-serve-hundreds-of-users-at-the-same-time-b999e60a440e"><strong>How Does One GPU Serve Hundreds of Users at the Same Time?</strong> </a>— Inside an LLM inference server<br>(Next Article)</p><p><strong>4. </strong><a href="https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7"><strong>The KV Cache Explained: Why Long Conversations Get Expensive</strong></a> — How LLMs remember context without recomputing everything</p><p><strong>5. </strong><a href="https://pub.towardsai.net/why-is-your-llm-recomputing-the-same-prompt-1-000-times-a-day-675e0cdab4b3"><strong>Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?</strong></a> — Prefix caching, radix trees and block hashing explained</p><p><strong>6. </strong><a href="https://pub.towardsai.net/why-traditional-load-balancing-breaks-for-llms-40311f7b29e2"><strong>Why Traditional Load Balancing Breaks for LLMs</strong></a> — Building an LLM-aware router</p><p><strong>7. </strong><a href="https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c"><strong>Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster</strong></a></p><p><strong>8. </strong><a href="https://pub.towardsai.net/llm-d-explained-69f7d0b2b035"><strong>LLM-D Explained</strong></a> — How modern AI infrastructure routes, schedules and scales LLM inference</p><p><strong>9. </strong><a href="https://pub.towardsai.net/inside-a-modern-ai-inference-platform-8846434bb14b"><strong>Inside a Modern AI Inference Platform</strong></a> — The full stack end-to-end</p><p><strong>Sources</strong></p><ul><li><a href="https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/">NVIDIA — Mastering LLM Techniques: Inference Optimization</a></li><li><a href="https://nvidia.github.io/TensorRT-LLM/performance/performance-tuning-guide/deciding-model-sharding-strategy.html">TensorRT-LLM — Deciding Model Sharding Strategy</a></li><li><a href="https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/parallel-strategy.md">TensorRT-LLM — Parallelism strategies</a></li><li><a href="https://huggingface.co/docs/transformers/quantization/overview">Hugging Face — Quantization</a></li><li><a href="https://docs.vllm.ai/en/latest/serving/distributed_serving.html">vLLM — Distributed inference and serving</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=443bbbe70d4d" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/what-do-you-do-with-a-model-thats-too-big-for-your-gpu-443bbbe70d4d">What Do You Do With a Model That’s Too Big for Your GPU?</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What really happens when you click ‘Send’ on ChatGPT]]></title>
            <link>https://pub.towardsai.net/what-really-happens-when-you-click-send-on-chatgpt-2307e3ede14e?source=rss-82d3a738e07a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2307e3ede14e</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Rinit Jain]]></dc:creator>
            <pubDate>Sat, 05 Sep 2026 17:01:02 GMT</pubDate>
            <atom:updated>2026-09-15T13:43:45.099Z</atom:updated>
            <content:encoded><![CDATA[<h4>A journey through modern AI Infrastructure — from one GPU to a Kubernetes cluster</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJAL2_9Bm9cV7QNRSWoFQw.png" /></figure><p>You type a question into an AI application and press Send. A moment later, the response starts appearing. That interaction looks simple. The infrastructure behind it is not.</p><blockquote><strong><em>TL;DR</em></strong></blockquote><blockquote>An LLM is a large block of numbers that has to physically fit in GPU memory. Everything else follows from that.</blockquote><blockquote>Your text becomes tokens, and tokens — not messages — are what the infrastructure schedules, caches and bills.</blockquote><blockquote>A request runs in two phases with completely different hardware behaviour: <strong>prefill</strong> (reads the prompt, compute-heavy) and <strong>decode</strong> (generates one token at a time, memory-bandwidth-heavy).</blockquote><blockquote>One GPU serves many users through <strong>batching</strong>. The limit is usually not compute, it’s the <strong>KV cache</strong> consuming VRAM.</blockquote><blockquote>When a model is too big for one GPU it gets <strong>sharded</strong>; when traffic is too high it gets <strong>replicated</strong>. These are different problems.</blockquote><blockquote>Because requests cost wildly different amounts, normal load balancing breaks and you need <strong>LLM-aware routing</strong>.</blockquote><blockquote>Once there are hundreds of GPUs, <strong>Kubernetes</strong> operates the fleet.</blockquote><p>A single request can pass through an API gateway, a load balancer, a model server, an inference engine, a GPU scheduler, several layers of memory and — at scale — a Kubernetes cluster containing hundreds or thousands of GPUs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Vy-O0bgZb5xYLeqK" /><figcaption>The AI Infrastructure</figcaption></figure><p>The interesting part is that every layer exists for a reason.</p><p>A model needs memory.</p><p>A GPU needs work.</p><p>Many users need to share the same GPU.</p><p>Large models need multiple GPUs.</p><p>Multiple model instances need intelligent routing.</p><p>And once there are hundreds of machines, something has to operate the whole fleet.</p><p>This article builds that stack from the bottom up, starting with <strong>one model and one GPU</strong>, and gradually turning it into a production LLM platform.</p><h3>The Stack at a Glance</h3><p>Before diving in, here is the system we are going to build:</p><blockquote><strong><em>User → API → Load Balancer → Model Server → Inference Engine → GPU → Prefill → KV Cache → Decode → Tokens back to User</em></strong></blockquote><p>At larger scale, that becomes:</p><blockquote><strong><em>User → Gateway → LLM-aware Router → Kubernetes → Model Server Fleet → GPU Cluster</em></strong></blockquote><p>The rest of the article is essentially an explanation of every box in those two lines.</p><h3>1. Start With the Model</h3><p>An LLM is not a magical object living somewhere in the cloud.</p><p>At its core, it is a neural network architecture together with a very large collection of learned numerical parameters called <strong>weights</strong>.</p><p>A model might contain 7 billion, 13 billion, 70 billion, or hundreds of billions of parameters.</p><p>A useful first approximation is:</p><blockquote><strong><em>Model weight memory ≈ Number of parameters × Bytes per parameter</em></strong></blockquote><p>For example, a 70-billion-parameter model represented using 16-bit values requires roughly:</p><blockquote><em>70 billion × 2 bytes ≈ </em><strong><em>140 GB</em></strong></blockquote><p>That number immediately creates a problem.</p><p>A model this large cannot simply be loaded into the VRAM of a small GPU.</p><p>And this is where the physical infrastructure starts to matter.</p><h3>2. CPU, RAM, GPU and VRAM</h3><p>A server running an LLM usually contains both a CPU and one or more GPUs. They are optimized for very different kinds of work.</p><h3>CPU</h3><p>The <strong>CPU</strong> is the general-purpose processor.</p><p>It handles tasks such as:</p><ul><li>network communication</li><li>tokenization</li><li>request processing</li><li>scheduling</li><li>application logic</li><li>operating-system tasks</li><li>coordination between different components</li></ul><h3>RAM</h3><p><strong>RAM</strong> is the machine’s main memory.</p><p>It is used by the operating system and applications and can hold model files, request data, buffers, and other runtime state.</p><h3>GPU</h3><p>The <strong>GPU</strong> is the accelerator that performs the majority of the neural-network computation.</p><p>LLMs perform enormous numbers of operations such as matrix multiplications. These operations contain large amounts of parallel work, which GPUs are designed to exploit.</p><h3>VRAM</h3><p>The GPU has its own high-bandwidth memory, commonly called <strong>VRAM</strong>. (Video Random Access Memory) (<em>on datacenter GPUs this is HBM — high-bandwidth memory — stacked next to the die)</em>.</p><p>This is where the GPU keeps the data it needs to work on quickly, including:</p><ul><li>model weights</li><li>intermediate data</li><li>runtime buffers</li><li>KV cache</li></ul><p>A useful mental model is:</p><blockquote><strong><em>RAM is the server’s main working memory. VRAM is the high-speed working memory directly attached to the GPU.</em></strong></blockquote><p>They are not interchangeable.</p><p>Having 256 GB of system RAM does not turn a 24 GB GPU into a 256 GB GPU.</p><h3>The Four Resources That Matter</h3><p>When evaluating infrastructure for LLM inference, four resource dimensions matter enormously:</p><p><strong>1. Compute capacity</strong><br>How much arithmetic the accelerator can perform.</p><p><strong>2. Memory capacity</strong><br>How much data can fit into GPU memory (VRAM).</p><p><strong>3. Memory bandwidth</strong><br>How quickly data can move between GPU memory and the GPU’s compute units.</p><p><strong>4. Network bandwidth</strong><br>How quickly data can move between GPUs or between machines.</p><p>These are separate constraints.</p><p>A GPU can have enormous compute capacity but still perform poorly if its memory cannot feed the compute units fast enough. Similarly, a model distributed across multiple GPUs can be limited by the communication between those GPUs, even when each individual GPU has sufficient compute and memory.</p><blockquote><strong>Think of compute as the number of workers, memory capacity as the size of the workspace, memory bandwidth as how quickly materials reach those workers, and network bandwidth as how quickly different factories can exchange materials.</strong></blockquote><p>For a single-GPU workload, the first three are particularly important. Once inference becomes distributed, communication bandwidth becomes another critical part of the performance equation.</p><h3>3. Getting the Model Onto the GPU</h3><p>A simplified path looks like this:</p><pre>Model Repository / Storage<br>            ↓<br>         RAM<br>            ↓<br>       GPU VRAM<br>            ↓<br>      GPU Compute</pre><p>The model files may initially be stored on local SSDs, network storage, or an object store.</p><p>When the inference server starts, the model is loaded into memory and eventually placed in the GPU’s working memory.</p><p>Once loaded, the model can process requests.</p><p>But the GPU still does not know what to do with a sentence such as:</p><blockquote><em>“Explain how a nuclear reactor works.”</em></blockquote><p>The text first has to become something the model understands.</p><h3>4. Text Becomes Tokens</h3><p>LLMs do not directly operate on words.</p><p>They operate on <strong>tokens</strong>.</p><p>A tokenizer converts text into token IDs.</p><p>For example, a sentence might conceptually become:</p><pre>&quot;Explain how tokenization works&quot;<br>        ↓<br>[&quot;Explain&quot;, &quot; how&quot;, &quot; token&quot;, &quot;ization&quot;, &quot; works&quot;]<br>        ↓<br>[849, 1268, 4037, 2065, 4375]</pre><p>The exact tokenization depends on the model and tokenizer. A token is roughly ¾ of an English word on average, but that ratio collapses for code, JSON, and non-Latin scripts.</p><p>Those IDs are then fed into the model.</p><p>This distinction becomes important later because inference systems do not primarily reason about “messages”.</p><p>They reason about <strong>tokens</strong>.</p><p>Token count affects:</p><ul><li>computation</li><li>memory consumption</li><li>latency</li><li>KV-cache usage</li><li>throughput</li><li>cost</li></ul><h3>5. The Model Server</h3><p>Now we need a program that can accept requests and run the model.</p><p>That program is the <strong>model server</strong>.</p><p>A model server typically handles:</p><pre>Request<br>   ↓<br>Tokenization<br>   ↓<br>Scheduling<br>   ↓<br>Model execution<br>   ↓<br>Generated tokens<br>   ↓<br>Response</pre><p>Without a model server, every application would need to implement model loading, GPU execution, batching, request management, streaming, and concurrency itself.</p><p>The model server turns the model into a service.</p><p>For example:</p><pre>POST /generate<br>{<br>  &quot;prompt&quot;: &quot;Explain how a nuclear reactor works&quot;<br>}</pre><p>The server receives that request, executes the model and returns generated tokens.</p><p>One important distinction:</p><blockquote><strong><em>The model is the thing being executed. The model server is the software that manages that execution and exposes it as a service.</em></strong></blockquote><h3>6. Where Does vLLM Fit?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*CRWUUk0vxw5pFNl2" /></figure><p>Several inference engines exist today. <strong>vLLM</strong> is one of the most widely used open-source engines for serving LLMs, and it is designed around the realities of the workload: large models, many concurrent requests, expensive GPU memory, dynamic request lengths and heavy cache management.</p><p>Its documented feature set includes PagedAttention, continuous batching, prefix caching, quantization and disaggregated prefill/decode — all of which we will touch on below.</p><blockquote><em>vLLM does not replace the GPU or the model. It is the inference engine that manages how requests use them.</em></blockquote><h3>7. A Request Has Two Very Different Phases</h3><p>Once the request reaches the model server, inference is not simply prompt → answer. It runs in two phases, and understanding the difference is the single most important idea in LLM infrastructure.</p><pre>Prompt<br>  ↓<br>PREFILL<br>  ↓<br>First output token<br>  ↓<br>DECODE<br>  ↓<br>Next token<br>  ↓<br>Next token<br>  ↓<br>Next token<br>  ↓<br>...</pre><p><strong>Prefill</strong> processes the input tokens and builds the internal state needed for generation. Because all prompt tokens can be processed together, prefill exposes a lot of parallelism and tends to be <strong>compute-bound</strong> — it can saturate the GPU’s arithmetic units.</p><p><strong>Decode</strong> generates the answer one token at a time, and each new token depends on everything generated so far. There is very little parallel work per step, but the model’s weights must be read from VRAM on every single step. Decode is therefore <strong>memory-bandwidth-bound</strong> — the GPU spends most of its time waiting for data, not calculating.</p><p>That one difference explains most of what follows: why batching works, why GPU utilisation is hard, why latency has two separate metrics, and eventually why some systems run prefill and decode on entirely different machines.</p><p>Now attach a 10,000-token document to that same prompt. Prefill has 10,000 tokens to chew through before a single output token appears, while decode is unchanged. This is why a request with a 20,000-token prompt behaves nothing like a request with a 100-token prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f2BJZhOoQq2Qzi95-MP2TQ.png" /></figure><h3>9. Measuring the user experience</h3><p>Because the two phases are different, latency needs more than one number. Three metrics do most of the work:</p><p><strong>TTFT — Time To First Token</strong> How long the user waits before anything appears on screen. Driven by queueing and prefill.</p><p><strong>TPOT — Time Per Output Token</strong> The average time to produce each subsequent token. Driven by decode.</p><p><strong>ITL — Inter-Token Latency</strong> The delay between individual streamed tokens. Driven by decode and scheduling.</p><p>Mapped onto the request:</p><pre>Request<br>   │<br>   └────────────────┐ first token<br>                    ↑<br>                  TTFT<br><br>   token → token → token → token<br>          ↑<br>      TPOT / ITL</pre><p>A system can have a perfectly reasonable total completion time and still feel slow because TTFT is high. It can also produce the first token instantly and then stream the rest sluggishly. In production, both matter.</p><h3>10. Now add a second user</h3><p>So far this has been easy: one user, one request. Production services have hundreds or thousands of requests in flight.</p><p>We clearly do not want one GPU per user — that would require an absurd number of GPUs. Instead, a single model instance serves many users. The same weights stay loaded in VRAM while the inference engine continuously schedules work from multiple requests.</p><p>This is <strong>batching</strong>. Instead of processing requests one at a time, the engine combines work from multiple sequences and executes it together, which suits a highly parallel device far better than feeding it one request’s worth of work.</p><p>Modern engines go further with <strong>continuous batching</strong>: rather than forming a fixed batch and waiting for every request in it to finish, requests enter and leave the active workload dynamically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CeceKB-3ua5KxJVzPj0tiQ.png" /></figure><h3>11. Batching creates a memory problem</h3><p>Batching improves GPU efficiency, but every active request consumes memory. The weights already occupy VRAM. Now every active sequence also needs runtime state — and the largest part of that state is the <strong>KV cache</strong>.</p><p>Transformer attention produces queries, keys and values for each token. During generation, the keys and values from previous tokens get reused on every step, so instead of recomputing them the engine stores them. That stored state is the KV cache.</p><p>Without it, generating each new token would mean reprocessing the entire context from scratch. With it, the system keeps the useful intermediate state and reuses it.</p><h3>12. Why the KV cache changes the memory equation</h3><p>Model weights are static. The KV cache is dynamic — it grows with the number of active users <em>and</em> with how long each of their contexts becomes.</p><p>So VRAM has to accommodate something closer to:</p><pre>VRAM ≈ model weights + KV cache + runtime buffers + overhead<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mw_xTFJMb0wAT_o4jQQ3sw.png" /></figure><p>This produces an important consequence:</p><blockquote><em>A model can fit on a GPU and still not leave enough free memory to serve a realistic number of concurrent long-context requests.</em></blockquote><p>Capacity is not determined by model weights alone. This is also why memory allocation strategy matters: sequences have wildly different lengths, and reserving one large contiguous region per sequence wastes space. vLLM’s <strong>PagedAttention</strong> instead organises the KV cache into fixed-size blocks that can live anywhere in GPU memory, which removes most of that waste.</p><p><em>The mechanics of KV cache growth, block management, eviction and PagedAttention get their own article — see Related reading.</em></p><h3>13. A second cache appears: prefix caching</h3><p>Imagine an application where every request begins with the same 8,000-token system prompt. Without any caching, the system performs the same prefill computation over and over.</p><p><strong>Prefix caching</strong> reuses the KV cache blocks already computed for a shared prefix:</p><pre>Request A: [Shared prefix][Question A]  → computed and cached<br>Request B: [Shared prefix][Question B]  → reuses cached prefix</pre><p>That removes a large amount of repeated prefill work. But it introduces a question that becomes central later: <strong>which server already holds the useful cached state?</strong></p><h3>14. What if the model does not fit on one GPU?</h3><p>Back to the 70B model needing roughly 140 GB for its weights. A single 24 GB GPU cannot hold it. There are four responses:</p><p><strong>Quantization</strong> — represent the model with fewer bits (FP16 → INT8 → INT4). Fewer bits means less memory per parameter, at some cost to accuracy.</p><p><strong>A larger GPU</strong> — use an accelerator with enough VRAM.</p><p><strong>Offloading</strong> — keep some state in CPU memory and move it when needed. This makes otherwise impossible configurations fit, but data movement is slow relative to VRAM.</p><p><strong>Sharding</strong> — split the model across multiple GPUs.</p><p>That last option opens up distributed inference, and there is more than one way to split a model:</p><p><strong>Tensor parallelism</strong> — splits the individual operations <em>inside</em> each layer across GPUs. Used when a model is too large for one GPU and the GPUs have a fast interconnect.</p><p><strong>Pipeline parallelism</strong> — splits the model by depth, giving each GPU a contiguous group of layers. Used when the model has to span multiple machines.</p><p><strong>Data parallelism</strong> — splits nothing. The model is replicated, and each copy handles different requests. Used for throughput, not capacity.</p><p><strong>Expert parallelism</strong> — distributes the experts of a Mixture-of-Experts model across GPUs. Only applies to MoE architectures.</p><p><strong>Context parallelism</strong> — distributes the context dimension itself. Used for very long context.</p><p>The trade-off across all of them is the same: more GPUs reduce the memory burden per GPU, but the GPUs now have to communicate, and that communication becomes its own performance constraint.</p><p><em>How each strategy actually works, and how to choose between them, is a full article of its own — see Related reading.</em></p><h3>15. Sharding vs replication</h3><p>These two are often confused. They solve different problems.</p><p><strong>Replication</strong> creates multiple complete copies of the model, each able to serve requests independently. It is a capacity and availability strategy.</p><p><strong>Sharding</strong> splits one model across multiple GPUs that together execute a single model instance. It is a model-size and execution strategy.</p><blockquote>Replication creates more copies. Sharding creates one distributed copy.</blockquote><h3>16. One model becomes a fleet</h3><p>Suppose one model server handles 50 concurrent requests, and traffic rises. Now we deploy several:</p><pre>              Load Balancer<br>             /      |      \<br>      Server A  Server B  Server C<br>          |         |         |<br>       GPU(s)    GPU(s)    GPU(s)</pre><p>Each server holds its own copy of the weights. This buys throughput, redundancy, fault tolerance and headroom for traffic spikes — and raises a new question: <strong>which server should receive each request?</strong></p><h3>17. Why traditional load balancing breaks</h3><p>A normal web service might use round robin: request 1 to A, request 2 to B, request 3 to C. That works when requests are broadly similar.</p><p>LLM requests are not. Consider:</p><pre>                  Request A      Request B<br>Input tokens           500         20,000<br>Output tokens           50          8,000</pre><p>Both count as “one request” to a normal load balancer. Their compute and memory requirements are nowhere close to equal.</p><p>An LLM-aware router instead cares about active sequences, queue depth, token workload, GPU utilisation, KV cache occupancy, prefix-cache locality, model identity and available capacity. That is a fundamentally richer problem than counting HTTP connections.</p><p>Prefix caching makes it richer still. If Server A already holds the cached prefix for a request while Server B merely has more free capacity, sending it to B throws away real work.</p><blockquote>The best server is not necessarily the one with the most spare compute. It may be the one that already has the right state.</blockquote><p>This is the idea behind LLM-aware routing systems such as <strong>LLM-D</strong>, a Kubernetes-native stack for distributed LLM inference. Rather than treating model servers as interchangeable HTTP backends, it makes routing decisions using inference-specific signals, and supports concepts like inference pools, cache-aware scheduling and prefill/decode disaggregation.</p><p><em>Routing algorithms and LLM-D each get their own article — see Related reading.</em></p><h3>18. Kubernetes enters the picture</h3><p>We now have many models, many model servers, many GPUs, replicas, distributed shards, networking, health checks, autoscaling and failures. Managing that manually is impractical.</p><p><strong>Kubernetes</strong> is an orchestration system for running containerised workloads across a cluster of machines. It does not perform the neural network computation — it manages the infrastructure around it.</p><p><strong>Pods and nodes.</strong> A Pod is Kubernetes’ smallest deployable unit, containing one or more containers. For inference, a Pod typically runs a model-serving container such as vLLM, placed on a GPU node.</p><p><strong>Scheduling.</strong> Clusters contain a mix of CPU-only nodes, different GPU generations, and machines with different interconnects. GPUs are exposed to Kubernetes through vendor device plugins, so a Pod can request them as schedulable resources. For sharded inference, GPU <em>topology</em> matters as much as GPU count — four GPUs in one tightly connected server behave very differently from four GPUs scattered across machines.</p><p><strong>Self-healing.</strong> Pods are replaceable. A Deployment maintains a desired number of replicas, and if one dies Kubernetes creates a replacement, assuming the cluster has room. Because Pods come and go, clients talk to a <strong>Service</strong>, which provides a stable network endpoint over a changing set of Pods.</p><p><strong>Autoscaling — with a catch.</strong> Kubernetes can run more replicas as traffic grows. But creating an LLM-serving Pod means scheduling a GPU node, pulling a large container image, loading model files, and moving weights into VRAM. That can take minutes, not seconds. As a result LLM autoscaling cannot rely on CPU utilisation; useful signals include request queueing, token throughput, active sequences, latency and KV cache utilisation.</p><p><em>Kubernetes for AI workloads is a full article on its own — see Related reading.</em></p><h3>19. The full request</h3><p>We can now follow one request end to end. The user sends:</p><blockquote><em>“Explain how nuclear reactor” — with a 10,000-token document attached.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LtNVDb6Sn7YryEDDtpFzBQ.png" /></figure><p>Step by step:</p><ol><li><strong>The request arrives</strong> at the API gateway.</li><li><strong>Routing happens.</strong> The platform picks a model and a serving instance.</li><li><strong>Tokenization.</strong> The text becomes token IDs.</li><li><strong>The request enters the inference scheduler</strong>, which decides when this sequence gets GPU time.</li><li><strong>Prefill runs</strong> over the 10,000 prompt tokens.</li><li><strong>KV state is produced</strong> and stored in the cache.</li><li><strong>Decode begins</strong> and the first output token is generated.</li><li><strong>Decode continues</strong>, reusing the KV cache on every step.</li><li><strong>Other users run simultaneously</strong>, their work continuously batched alongside yours.</li><li><strong>Tokens stream back.</strong> Rather than waiting for the full answer, the server pushes each token to the client as it is produced — usually over a streaming HTTP connection such as server-sent events. That is why the response appears word by word instead of arriving all at once.</li></ol><h3>20. The five layers</h3><p>It helps to separate the stack into five layers, top to bottom:</p><p><strong>Application</strong> — chat app, API, authentication. What the user actually interacts with.</p><p><strong>Orchestration</strong> — Kubernetes, Pods, Services, autoscaling. Operates the fleet.</p><p><strong>Distributed inference</strong> — sharding, replicas, inter-GPU communication, routing. Determines how the model is spread and how requests are distributed.</p><p><strong>Inference</strong> — model server, scheduler, batching, prefill/decode, KV cache. Determines how requests use the hardware.</p><p><strong>Hardware</strong> — CPU, RAM, GPU, VRAM, network. Provides the physical resources.</p><p>The platform is the combination of all five.</p><h3>21. A debugging framework</h3><p>Return to the four resources from Section 3. When an LLM deployment is slow, the useful question is: <strong>which resource is actually limiting it?</strong></p><ul><li>If GPU memory is full, adding CPU cores will not help.</li><li>If GPU compute is saturated, enlarging the KV cache will not help.</li><li>If distributed workers spend their time waiting on each other, adding more compute just creates more communication.</li><li>If decode feels slow while prefill is fine, look at memory bandwidth, not FLOPs.</li></ul><p>Optimization starts by identifying the actual bottleneck, not by adding hardware.</p><h3>22. Revision: How One Request Became an Infrastructure Stack</h3><p>We started with:</p><blockquote><em>“</em>Explain how a nuclear reactor works.<em>”</em></blockquote><p>That innocent sentence led us through:</p><pre>Text<br> ↓<br>Tokens<br> ↓<br>API<br> ↓<br>Model Server<br> ↓<br>vLLM<br> ↓<br>Scheduler<br> ↓<br>Prefill<br> ↓<br>KV Cache<br> ↓<br>Decode<br> ↓<br>Batching<br> ↓<br>GPU</pre><p>Then one GPU became insufficient:</p><pre>One GPU<br>   ↓<br>Sharding<br>   ↓<br>Multiple GPUs</pre><p>Then one model server became insufficient:</p><pre>One Replica<br>   ↓<br>Multiple Replicas<br>   ↓<br>Load Balancing</pre><p>Then normal load balancing became insufficient:</p><pre>Load Balancing<br>   ↓<br>Cache-aware / LLM-aware Routing<br>   ↓<br>LLM-D</pre><p>And finally, the number of machines became large enough that we needed a system to operate them:</p><pre>Many GPU Nodes<br>       ↓<br>    Kubernetes<br>       ↓<br>Pods + Services + Scheduling<br>       ↓<br>Autoscaling + Networking + Recovery</pre><p>That is modern LLM infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A4XGoiLxyxcNsR27" /><figcaption>Kubernetes based LLM Inference Stack</figcaption></figure><h3>A Problem</h3><p>The architecture now looks complete. The model is in VRAM, the engine schedules requests, prefill reads the prompt, decode generates tokens, the KV cache holds useful state, batching shares the GPU, and Kubernetes manages the fleet.</p><p>But we have conveniently avoided one thing: <strong>what happens when the model genuinely cannot fit on a single GPU?</strong></p><p>A model may need hundreds of gigabytes while an individual GPU has far less. Even when it does fit, serving many users may require more GPUs than one machine provides.</p><p>So a choice has to be made. Do we make the model smaller? Split it across GPUs? Replicate it? And if we split it, how do those GPUs cooperate without spending all their time talking to each other?</p><p>That is the next layer of the stack: quantization, sharding, replication and parallelism.</p><h3>The series</h3><p><strong>1. </strong><a href="https://pub.towardsai.net/what-really-happens-when-you-click-send-on-chatgpt-2307e3ede14e"><strong>What really happens when you click ‘Send’ on ChatGPT</strong></a> — A journey through modern AI Infrastructure</p><p><strong>2. </strong><a href="https://pub.towardsai.net/what-do-you-do-with-a-model-thats-too-big-for-your-gpu-443bbbe70d4d"><strong>What Do You Do With a Model That’s Too Big for Your GPU?</strong></a> — Quantization, Sharding and Parallelism Explained <br>(Next Article)</p><p><strong>3. </strong><a href="https://pub.towardsai.net/how-does-one-gpu-serve-hundreds-of-users-at-the-same-time-b999e60a440e"><strong>How Does One GPU Serve Hundreds of Users at the Same Time?</strong> </a>— Inside an LLM inference server</p><p><strong>4. </strong><a href="https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7"><strong>The KV Cache Explained: Why Long Conversations Get Expensive</strong></a> — How LLMs remember context without recomputing everything</p><p><strong>5. </strong><a href="https://pub.towardsai.net/why-is-your-llm-recomputing-the-same-prompt-1-000-times-a-day-675e0cdab4b3"><strong>Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?</strong></a> — Prefix caching, radix trees and block hashing explained</p><p><strong>6. </strong><a href="https://pub.towardsai.net/why-traditional-load-balancing-breaks-for-llms-40311f7b29e2"><strong>Why Traditional Load Balancing Breaks for LLMs</strong></a> — Building an LLM-aware router</p><p><strong>7. </strong><a href="https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c"><strong>Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster</strong></a></p><p><strong>8. </strong><a href="https://pub.towardsai.net/llm-d-explained-69f7d0b2b035"><strong>LLM-D Explained</strong></a> — How modern AI infrastructure routes, schedules and scales LLM inference</p><p><strong>9. </strong><a href="https://pub.towardsai.net/inside-a-modern-ai-inference-platform-8846434bb14b"><strong>Inside a Modern AI Inference Platform</strong></a> — The full stack end-to-end</p><p><strong>Sources</strong></p><ul><li><a href="https://docs.vllm.ai/">vLLM documentation</a></li><li><a href="https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/">NVIDIA — Mastering LLM Techniques: Inference Optimization</a></li><li><a href="https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/">Kubernetes — Schedule GPUs</a></li><li><a href="https://llm-d.ai/">LLM-D documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2307e3ede14e" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/what-really-happens-when-you-click-send-on-chatgpt-2307e3ede14e">What really happens when you click ‘Send’ on ChatGPT</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Craziest 36 Hours of My Life: A Tale of Research, Exams, and Dance]]></title>
            <link>https://medium.com/@rinitjain9/the-craziest-36-hours-of-my-life-a-tale-of-research-exams-and-dance-06d403656cce?source=rss-82d3a738e07a------2</link>
            <guid isPermaLink="false">https://medium.com/p/06d403656cce</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[student-life]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[time-management]]></category>
            <category><![CDATA[research-paper]]></category>
            <dc:creator><![CDATA[Rinit Jain]]></dc:creator>
            <pubDate>Thu, 15 May 2025 11:05:59 GMT</pubDate>
            <atom:updated>2025-05-15T11:05:59.040Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>“Balancing research presentations, final exams, and a national dance competition — and somehow surviving it all.”</em></blockquote><p>It was December 20, 2024. Midnight at the Chennai International Airport. The air was humid, my mind buzzing, and my phone screen glowing:<br> <strong>“AI &amp; ML PPT — SPPU End Semester Exam.”</strong><br> My final exam was in 7 hours.<br> I was some 1,200 kilometers away from home.</p><p>This is the story of probably the craziest 36 hours of my life.</p><h3><strong>ACT 1: The Chaos</strong></h3><p>It all began a few weeks back. My team received an email from the <a href="https://www.au-kbc.org/icon2024/"><em>International Conference on Natural Language Processing (ICON)</em></a> — our research paper had been selected for presentation!<br> It was a huge moment. My team — <a href="https://www.linkedin.com/in/aastha-shah-6827b0285?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAEVZ4sIBGwXh95KyDXPwl8yJ6uGJpWa3T1Q&amp;lipi=urn%3Ali%3Apage%3Ad_flagship3_search_srp_all%3BO5GROtPST%2F%2BjKyuK8exKDQ%3D%3D">Aastha Shah</a>, <a href="https://www.linkedin.com/in/ansh-vora-61464121a?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAADdEdjABwe4wWdjmx6EstZ_0waqGOMXpe40&amp;lipi=urn%3Ali%3Apage%3Ad_flagship3_search_srp_all%3BO5GROtPST%2F%2BjKyuK8exKDQ%3D%3D">Ansh Vora</a>, our mentor <a href="https://www.linkedin.com/in/sheetal-sonawane-6b94b01b?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAQqnu0Br6e8LlB52siYSKMtaiaMsDR4XtI&amp;lipi=urn%3Ali%3Apage%3Ad_flagship3_search_srp_all%3BO5GROtPST%2F%2BjKyuK8exKDQ%3D%3D">Dr. Sheetal Sonawane</a>, and I — had poured months of hard work into our paper titled:<br><a href="https://aclanthology.org/2024.icon-1.54/"> <strong>“Extractive Summarization using Extended TextRank Algorithm.”</strong></a></p><p>Our paper focused on improving the popular <strong>TextRank algorithm</strong> used for extractive text summarization. We introduced two key upgrades: a <strong>dynamic damping factor</strong> to improve scoring, and <strong>topic modeling with LDA</strong> to help the algorithm better understand context. These changes led to more accurate and meaningful summaries — and it felt amazing to see our months of hard work recognized on such a big stage.</p><figure><img alt="The Research Paper Team: Dr. Sheetal Sonawane, Aastha Shah, Ansh Vora, Rinit Jain" src="https://cdn-images-1.medium.com/max/958/1*ekcQs5QAhRW2Gfy94Wz9dw.jpeg" /><figcaption>The Research Paper Team: Dr. Sheetal Sonawane, Aastha Shah, Ansh Vora, Rinit Jain</figcaption></figure><p>We were overjoyed! But there was one major conflict:<br> The conference was scheduled for <strong>December 19–21</strong> — right in the middle of our university’s end semester exams. After a lot of back-and-forth, a situation arose where only one of us could make the trip. And that person turned out to be me.</p><p>So now, not only was I heading to present our research at an international conference during finals — I was doing it <em>alone</em>.</p><p>And if that wasn’t enough, we had just returned from another conference the previous week — <a href="https://www.daiict.ac.in/events/6535"><em>FIRE 2024</em> at DA-IICT, Gandhinagar.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2iycJTTouLE0Ojnhy3AuUw.jpeg" /><figcaption>FIRE Conference 2024</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lxGkyJMaedNLy3-pzEQO2Q.jpeg" /><figcaption>The Team at DA-IICT</figcaption></figure><p>We gave our CNS exam on the 14th, flew to Ahmedabad the same night, attended the conference on the 15th, flew back on the 16th, and sat for another exam on the 17th.<br>A whirlwind, to say the least.<br>And now, ICON was waiting for us next.</p><p>Oh, but it gets better.</p><p>I’m also part of the <a href="https://www.instagram.com/pictartcircle/"><strong>Art Circle</strong></a>, our college’s cultural society, and for the past 3 months, we’d been preparing tirelessly for <a href="https://moodi.org/"><strong>Mood Indigo</strong></a>, IIT Bombay’s prestigious inter-college competition.<br> The dates? December 23rd onwards — just two days after ICON.</p><h3>ACT 2: The Dilemma</h3><p>So, if everything went perfectly, this is what my schedule looked like:</p><ul><li>Exam on Dec 19</li><li>Fly to Chennai same night</li><li>Present at ICON on Dec 20</li><li>Fly back early morning Dec 21 (3 am)</li><li>Exam on the same day Dec 21 (10 am)</li><li>Final practice with Art Circle on Dec 22</li><li>Leave for IIT Bombay on Dec 23</li><li>Elimination &amp; Semis on Dec 24–25</li></ul><p>I looked at the calendar and laughed. This seemed <em>impossible</em>.</p><p>The night of December 18, I broke down. I spoke to <em>The Setters</em> — my Art Circle team.<br>An intense conversation with my friend and teammate Mira turned into a midnight argument in the college lobby.<br>I felt torn between passion and responsibility.<br>But even through the chaos, my team stood by me.</p><p>I received calls, messages, and unwavering support from Purti, Aditya, Shravani — people who believed in me.</p><p>Still, the guilt of leaving a team of 20+ just before our biggest stage felt like carrying a storm inside my chest.</p><p>That night, I cried in front of my parents — the weight of everything sinking in. And like always, their words carried strength:<br><strong>“Don’t worry. Do what you feel is right. We’ve got your back.”</strong></p><p>My dance team didn’t miss a beat either:<br><strong>“Tu bindass ja yaar! Make us proud. Baaki team hum sambhaal lenge!”</strong></p><p>My friends Anshi and Purti literally were ready to <em>teach me the syllabus</em> overnight so I wouldn’t fail my upcoming exam.<br>“<strong>Focus on the conference, we’ve got the books. <em>Tujhe hum padha denge!</em></strong>”</p><p>And my research teammates stayed up late finalizing the poster, just so I could squeeze in a few hours of sleep before my flight.<br><strong>“Tu so ja Rinit. By the time you reach the airport, we’ll have everything ready.”</strong></p><p>I’ve never felt so supported.<br>And I knew — I couldn’t let any of them down.</p><h3>ACT 3: The 36-Hour Madness</h3><p>I boarded the flight.<br>Landed in Chennai. Welcomed by sticky air, pouring rain, and language barriers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*0zOC6VWoXnjw_gCTBo27Zw.jpeg" /><figcaption>Humid morning at the Chennai International Airport</figcaption></figure><p>Google Translate was my best friend.<br>After navigating local chaos, I managed to print our research poster — mission one, complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*VfSYz5O583vFU-qIoU6uTg.jpeg" /><figcaption>ICON 2024 Poster</figcaption></figure><p>Then came the <strong>conference</strong> — and it was phenomenal.<br>From inspiring keynote sessions by <a href="https://www.linkedin.com/in/rada-mihalcea-b93521?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAATq18BGiBlP3cBw1GYngIxC-214sziDY0&amp;lipi=urn%3Ali%3Apage%3Ad_flagship3_search_srp_all%3BO5GROtPST%2F%2BjKyuK8exKDQ%3D%3D"><strong>Dr. Rada Mihalcea</strong></a> (yes, the creator of TextRank!) and <a href="https://www.linkedin.com/in/monojit-choudhury-54225898?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAABSupgIBbAAxegvH9DNXia47c8kcZ0KdTug&amp;lipi=urn%3Ali%3Apage%3Ad_flagship3_search_srp_all%3BO5GROtPST%2F%2BjKyuK8exKDQ%3D%3D"><strong>Dr. Monojit Choudhury</strong></a>(Professor of NLP at MBZUAI), to a deep discussion with <a href="https://www.linkedin.com/in/manishrivastava?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAF7JkgBGZm7VDLGoQx-zDZ4FBShO9HlYvE&amp;lipi=urn%3Ali%3Apage%3Ad_flagship3_search_srp_all%3BO5GROtPST%2F%2BjKyuK8exKDQ%3D%3D"><strong>Prof. Manish Shrivastava</strong></a> from IIIT Hyderabad — I was in absolute awe.<br>Learning, networking, presenting — ICON was everything I had imagined and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*5W7z9zbs1q756c9_-JxspQ.jpeg" /><figcaption>Paper and Poster Presentation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_u-0REtXH1hqRtMUAVRsKA.jpeg" /><figcaption>Discussions with Prof. Manish Shrivastava</figcaption></figure><p>Sadly, I had to skip the <strong>conference banquet</strong> due to my early morning flight.<br>But fate had one last surprise — I bumped into <strong>Shourya</strong>, an old school friend.<br>Turns out, we were on the same flight.<br>After a long day alone, I finally had company to laugh, eat, and kill time with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*l7a1JcQr7q78vjNotdBLqQ.jpeg" /></figure><p>Still, the anxiety loomed.<br>The exam was in 7 hours.<br>I was flipping between naps and scrolling through that cursed “AI &amp; ML ppt.”<br>One flight delay, and I’d be staring at an<strong> ‘<em>F’</em> </strong>on my marksheet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*3s1iKOJGoZ0GJSiYRjyKsA.jpeg" /><figcaption>Trying to get in some sleep</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MCCoRwJaUV_pkL2GpvwX1w.jpeg" /><figcaption>Bidding Farewell to Chennai</figcaption></figure><p>Thankfully, the universe was kind.</p><p>I reached Pune.<br>My dad was waiting at the airport before sunrise, just to make sure I didn’t waste a single minute.<br>Straight home. Quick shower. Off to the college library. My friends were already there — waiting for me with a plan titled: <strong>“How to Help Rinit Pass!”</strong></p><h3><strong>The Encore</strong></h3><p>I passed the exam.<br>The next day, I was back in the dance studio, greeted with smiles, hugs, and encouragement. The Final practice was on!</p><p>We left for Mumbai.<br>Slept in dorms.<br>Woke up to the buzz of Mood Indigo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*cJFvG8yYoB2hr9Q1gCFlZQ.jpeg" /><figcaption>Team SETTERS</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xnRT59PcsxA0FUZBZ7o5lw.jpeg" /><figcaption>Setters on the Stage!</figcaption></figure><p>The moment the results were announced and we realized we’d made it to the semifinals — the cheer from our team was unforgettable. Later that night, we screamed our lungs out at the <strong>Armaan Malik concert</strong>, completely losing ourselves in the moment. By the end of it all, we were exhausted, yet glowing with pride.</p><p>No, we didn’t win — but I’ve never been prouder of my team.</p><p>And I’ve never been more grateful for the people around me — <br>My research partners,<br>My dancers,<br>My friends,<br>My parents.</p><p>None of this would’ve been possible without them.</p><h3>What This Journey Taught Me</h3><p>These 7 days — from Gandhinagar to Pune to Chennai to Mumbai — were a crash course in life.</p><p>They represent my college experience in a nutshell:<br><strong>Juggling academics, chasing dreams, leaning on friendships, and never giving up on passion.</strong><br><em>Where there’s a will, there truly is a way.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*OOhw2XD6xqhXzIzp5Sx0Xw.jpeg" /></figure><p>Writing this blog brought back every emotion I felt during those chaotic, unforgettable days. It’s reminded me that each tough decision, every sleepless night, and every breakdown I survived — it was worth it.</p><p>Through it all, I’ve learned that growth doesn’t happen in comfort. It happens in chaos — in those messy, uncertain moments where you’re forced to show up, dig deep, and keep going anyway.</p><p>I realized that you don’t always have to choose between passion and responsibility, between your dreams and your duties. Sometimes, with enough heart and hustle, you can do it all — even if it nearly breaks you.</p><p>So if you’re ever stuck, torn between two paths, or feeling like it’s all too much — remember this: you are more capable than you think. You don’t have to wait for the perfect time. Sometimes, the <em>craziest</em> plans make for the most unforgettable stories.</p><blockquote><strong>Trust yourself. Take the flight. Show up. And if you’re lucky — you just might surprise yourself.</strong></blockquote><p>Remembering those tough times or writing about it gives you a sense of validation for your hard work. It motivates you to tackle more and more problems in the future. Because each problem you solve gives you the confidence of you being of the right track. Writing this blog has done me the same and hence felt like sharing this ‘unique’ experience with all of you!</p><p>Thank you for reading this.<br>This was me, trying to write from the heart.<br>Hope it made you smile, and maybe, inspired you a little.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=06d403656cce" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>