<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manish Immadisetty on Medium]]></title>
        <description><![CDATA[Stories by Manish Immadisetty on Medium]]></description>
        <link>https://medium.com/@manishimmi2k3?source=rss-db29cf523fed------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ThSMPQZrdvYwbOGl</url>
            <title>Stories by Manish Immadisetty on Medium</title>
            <link>https://medium.com/@manishimmi2k3?source=rss-db29cf523fed------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:52:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manishimmi2k3/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Yet Another Inference Engineering Blog: A Tour of the Whole Serving Stack]]></title>
            <link>https://medium.com/@manishimmi2k3/yet-another-inference-engineering-blog-a-tour-of-the-whole-serving-stack-0511434a3fd2?source=rss-db29cf523fed------2</link>
            <guid isPermaLink="false">https://medium.com/p/0511434a3fd2</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[llm-inference]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Manish Immadisetty]]></dc:creator>
            <pubDate>Thu, 05 Nov 2026 13:15:37 GMT</pubDate>
            <atom:updated>2026-11-05T13:54:12.393Z</atom:updated>
            <content:encoded><![CDATA[<p>Yes, I know I know one more blog about AI, but my only purpose for the next 20 minutes is to get you excited about something maybe hopefully.</p><p>Or you can use the power of AI and understand it <a href="https://chatgpt.com/share/6aec8a1c-5f24-83ec-9523-be3a29daa7e7">here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uQOvBpHtZjp36KFJwOT5NQ.png" /><figcaption>Inspired by O’Riely Books idk why they use animals as cover images for engineering books</figcaption></figure><p>This entire blog is about the entire engineering that goes on to build the Inference stack sitting behind a simple LLM API Call. We start at the high level architecture, work our way through the cache and the worker runtime, and end with the part I find most interesting, which is how bytes actually move between kernels and between GPUs.</p><p>The way to get there is to look at how real systems are put together, and then build a smaller version that is simple enough to measure, understand, and reason about.</p><p>For our running example, we’ll use a hybrid model that combines Mamba-style layers with attention: Falcon H1 (0.5B) running across two A100 GPUs. It’s an interesting choice because its cache behaves differently from that of a conventional Transformer, and that difference ends up influencing several architectural decisions.</p><p>The code for this is available <a href="https://github.com/AlterHoodie/poor_mans_mamba">here</a>, if you’d like to take a look. I will be using it as reference, extending it where necessary to explore the architecture and understand the underlying systems not covered by the code.</p><p>One humble request before we start. Treat each section as a piece of a map that you redraw in your own head, and by the end you should hopefully be itching to move the pieces around. That is the whole point of building a small system. Its fun only when you get to change it.</p><h3>Control Plane vs Data Plane</h3><p>Follow a single request for a moment. It arrives from a user, someone has to decide where it should run, and someone has to actually run the model on it. Every serving system, once you squint, splits that journey between two kinds of component. The first is a scheduler, a program that accepts requests, keeps track of what is running where and decides what happens next. The second is a worker, a process that owns one or more local GPUs and runs the model on them.</p><p>The scheduler owns metadata and policy. It knows which session lives on which worker, which worker has free room, and which worker should get the next request. What it never does is touch model bytes. Those belong to the workers, which hold the weights, the runtime and the cache.</p><p>These two halves are usually called planes. The control plane is the thinking part and the data plane is the heavy lifting part. The control plane can afford to be slow and simple, and simple is exactly what you want from something that makes decisions. The data plane has to be fast and sit close to the hardware. Keeping them apart means each one can be designed, and each one can fail, independently of the other.</p><blockquote>First Rabbithole of the day: You will often find control plane software written in Go for its simplicity, while data plane components are typically built in C++ or Rust because they are fast and have no garbage collector to pause things at the wrong moment. The reasoning behind these choices is an interesting rabbit hole to dig into. <a href="https://chatgpt.com/share/6aec1795-a26c-83e8-bfa6-710432c966f3">[Chat here to understand more]</a></blockquote><p><a href="https://github.com/ray-project/ray">Ray</a> is a wonderful example of this. A Ray cluster has a head node that runs a global control store, which keeps cluster level metadata like which nodes are alive and where each actor lives. The actual work happens in worker processes, and the actual data sits in object stores on each node and moves from node to node directly. It does not flow through the control store. Serving stacks built for language models follow the same shape. <a href="https://github.com/kvcache-ai/Mooncake">Mooncake</a>, <a href="https://github.com/ai-dynamo/dynamo">Dynamo</a> and <a href="https://github.com/vllm-project/vllm">vLLM</a> each split a thin brain on top from heavy muscle underneath, in their own ways. Only the names change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZwTbHpIS1w0eqHzgX9xspw.png" /><figcaption>Baseline Architecture Diagram (looks cute know)</figcaption></figure><p>This is the map for the rest of the blog. In the control plane we look at how a request gets in, what the scheduler remembers, how it decides where to send things, and how it walks a request through its life, from high level design choices to low level byte movements.</p><h3>Control Plane</h3><p>A request walks the control plane in this order: the front door, the metadata that gets created, the policies that read that metadata, and the state machine that acts on the decisions.</p><h4>The front door</h4><p>A request is not a session yet. When it shows up, nothing has been promised to it. The scheduler first runs some simple middleware style checks, like a known model, a prompt that fits the context length and a sane token limit. After that it has three choices. It can admit the request, queue it, or reject it.</p><p>Admit means there is room somewhere, so the request becomes a session and moves on to placement. Queue means there is no room right now, so the request gets a row in the session table in a queued phase and waits its turn. Reject means the request is bad or the queue is full, and the caller hears about it right away.</p><p>Our rule is deliberately boring. Admit when some worker has free room, queue when none has, reject when the params are invalid or the queue is full. Real systems pile a lot more in here, like per tenant limits, priorities and deadlines, which is mostly business logic. What matters is the position of this step. Admission is the first state of the machine, before any worker is involved.</p><p>Where this front door sits in the wider system decides how it is built. If end users or apps talk to the scheduler directly, it is usually plain HTTP, with server sent events to stream tokens back. If it is one service among many, called by a router, a gateway or some other backend, gRPC streams are common. Or you can roll your own TCP framing, if you enjoy that sort of thing. Either way, from the outside the front door is just a request going in and a stream of token events coming back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uTrVTo6reyMwK3-61R801A.png" /><figcaption>Architecture Diagram</figcaption></figure><h4>What the scheduler remembers</h4><p>At a high level the control plane has to remember two things. One is the state of every request it is serving, which we call session state. The other is the state of every worker, which we call worker state (duh). Everything else it needs, it can ask the workers about.</p><p>Session state is a table with one row per request. Each row records the phase the session is in, which worker holds it, where its cache lives, and how far along it is, such as the prompt length and how many tokens have been generated. This is what the scheduler uses to route the next command for a session and to answer the client.</p><p>Worker state is much smaller. For each worker the scheduler keeps a capacity, which is how many sessions it can hold at once, and an in flight count. Free room is just the difference. That is the only view of worker load the scheduler needs, which keeps everything built on top of it easy to write and easy to reason about.</p><p>Let’s also be clear about what the scheduler does not know. It has no idea what is inside a session’s cache, what the GPU memory looks like, or what shapes the model layers have. It records the address the worker reported and never looks through it. That is the data plane’s business.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g6Kg4vXHDOhUJjgi6e5fuA.png" /><figcaption>Scheduler State</figcaption></figure><p>Now the obvious question. Where should this state live? Many systems keep it in a replicated key value store such as etcd, or a purpose built store, which makes the control plane durable at the cost of an extra moving part on the hot path. For simplicity the project uses a plain old in memory table. The price is that if the scheduler dies, every session is gone.</p><p>Anyone who wants to harden it has options: a hot standby fed from an append only log or a master backed by an external store, or a fully distributed design where workers own their session state and the scheduler is stateless. That last one is a fun weekend project, though not necessarily the best way to build a serving system.</p><h4>Placement policy</h4><p>The control plane needs a way to decide who gets what. A policy is a pure function. It reads the workers metadata view and returns a decision. It never sends a command and never touches a worker, which is what makes policies easy to swap and easy to test.</p><p>The first policy we need is placement, which decides which worker gets a new request. Real routers have rich and complicated versions that look at queue depth, how much of the prompt is already cached on each worker, expected latency, the cost of the model itself and blah blah. The simplest sensible one is placement by free room. A new request arrives, and the scheduler picks the worker with the most free capacity. It is a small function that reads the worker table and returns a worker id.</p><p>At the end of the day, it is just another load balancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lRTQTFgglRfZjgYvMD3oIw.png" /><figcaption>Scheduler Placement Policy</figcaption></figure><h4>The worker interface</h4><p>Before we wire the scheduler to the workers, let’s see what a worker offers. The two obvious operations, prefill and decode, and we add a third called release, which simply frees the memory a session was using. That is the whole interface. The worker looks like a tiny machine with three operations, and every other detail of the model stays hidden behind them.</p><p>The scheduler and the workers talk in two kinds of messages. A command travels from the scheduler to a worker, for example “run this prompt” or “advance this session by one token”. An event travels the other way, for example “here is the token you asked for”. Protobuf format works well here because the schema is explicit, smaller size (owing to binary serialization), and gives the scheduler one consistent way of talking to every worker. [I mean if I use JSON for interservice communication I think I’ll get killed by the microservice gods]</p><h4>The simple state machine</h4><p>Now let’s see how these pieces work together. The metadata, the policy and the worker interface are all fixed. What is left is the machine that turns worker events into the next command. We start with the simplest one. The normal life of a request goes like this.</p><ol><li>The caller submits a request. The front door validates it, and the scheduler admits it, queues it or rejects it. An admitted request gets a row in the session table, in the queued phase.</li><li>The scheduler picks a worker using the placement policy, moves the session to prefilling, counts it against that worker’s capacity and sends a prefill command.</li><li>The worker replies with a prefill event carrying the first token and the location of the cache [bit more on this later]. The scheduler stores that, moves the session to decoding and sends the first decode command.</li><li>Each decode event brings back one token. If the session is not finished, the scheduler sends the next decode command from inside the handler of that very event.</li><li>When the model produces the stop token, or the token limit is reached, the session becomes done and the scheduler sends a release command. A failure at any point sends the session to failed and also releases the memory. Once the release is confirmed, the worker’s in-flight count goes down and the next queued request can get in.</li></ol><p>If the cache runs out of room, that is not a crash. The session simply ends as done with a partial result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D1mAGeSA0h5iQkSWjOOB2g.png" /><figcaption>Inference State Machine</figcaption></figure><p>Look at step 4 again, because it carries the main design idea. The scheduler is inside the token loop. The next decode command is only ever sent as a reaction to the previous decode event, so each session has at most one decode in flight. That makes every single token a decision point, a moment where the scheduler could do something different if it wanted to. We call that moment the decode boundary, and the rest of the blog leans on it.</p><h4>How the loop is built</h4><p>Connecting all the above components and state machinery is the scheduler’s event loop. The project uses a single loop that keeps popping from a queue. Each item is an event, a new client request or a message from a worker, and the loop runs that event’s callback, which updates the tables and sends out whatever command comes next. Nothing else touches the tables, so there are no locks to reason about.</p><p>The only real question is what the loop does when the queue is empty. Most event loops, such as asyncio, libuv and tokio, block in the operating system’s epoll (or its cousins) until some socket or queue has something to read. The loop costs nothing while idle and wakes up in microseconds. The alternative is to busy poll, burning a core to shave off wakeup latency. That is a reasonable trade for a loop that sits next to a GPU, and we will meet it again on the worker side, but the scheduler is not latency critical enough to justify it, so it blocks.</p><blockquote>Another interesting Rabbithole: is understanding epoll from the ground up. How it works internally, when and why to use it, how the Linux kernel puts a thread to sleep without wasting CPU cycles, and how it wakes the thread when an event is ready. <a href="https://chatgpt.com/share/6aec195d-6510-83ee-9343-fded84f60ddc">[Chat here to understand more]</a></blockquote><p>Others make a similar single owner choice. vLLM’s engine core is a busy loop in its own process that owns the request state, advances a batch one step at a time and talks to the API server over ZMQ sockets. Actor runtimes drain a mailbox one message at a time.</p><p>For the channel between scheduler and workers, the choice depends on where the workers sit. If they are processes on the same machine, a local channel such as a Unix socket or a shared memory queue is plenty, with no ports or addresses to manage. If they live on different nodes, you need a real network protocol and a way to discover each other, and gRPC is the common pick since it is network ready from day one.</p><p>In a master and worker design a central node issues commands and workers report back. In a peer to peer design nodes share state among themselves, often by gossip. Everything in our system runs on one machine, so each worker gets a dedicated local channel, and the only thing on it is protobuf commands and events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vsjIC2izEHGGjCvVXNHjUQ.png" /><figcaption>Scheduler Opened Up</figcaption></figure><h3>Where this design hurts</h3><p>This design is easy to reason about, because a clean boundary exists at every token and keeping everything consistent is simple. But it has a price, and the price has three parts.</p><p>First, every token costs a round trip through the scheduler. The control plane, which we said could afford to be slow, is now sitting on the hot path of every single token. We will see exactly where that bites when we get to moving bytes.</p><p>Second, each decode command asks for one token for one session. A decode step has to read all the model weights from memory to produce that token, and doing that for a single session leaves most of the GPU bottlenecked on memory. The design is easy to understand but it is a bad way to keep an expensive GPU busy. Two well known ideas attack this, and they change the machine in interesting ways.</p><h4><em>Continuous batching</em></h4><p>In our machine, the unit of work is a session. In a continuous batching engine, it is a step [Running a one big batch at a time]. The scheduler maintains a queue of waiting requests and a set of running ones. At every step, it decides which requests enter the next batch, subject to token and cache memory budgets. The engine then runs a single forward pass over the entire batch, where some requests contribute a single decode token while others process a chunk of their prompt.</p><p>The fundamental difference is that our machine operates multiple independent streams of individual requests, whereas continuous batching operates one large stream of requests grouped into a shared batch. The per-request state machine becomes much simpler: waiting, running and finished, with preemption allowing a running request to be pushed back into the waiting queue under memory pressure. There are no per-session commands in flight, instead, a single loop runs continuously, making scheduling decisions once per step for the entire batch.</p><p>This is much better for GPU utilisation because multiple sessions share the same forward pass over the model weights. However, stopping, migrating or otherwise manipulating an individual session becomes harder because its execution is now intertwined with a shared batch.</p><h4><em>Speculative decoding</em></h4><p>This one attacks the one token per round trip part. A small draft model guesses the next k tokens, and the large model checks all k in a single pass. That works because a transformer can read many tokens in parallel, exactly like prefill does. The scheduler keeps the longest prefix that matches what the large model would have said, and if every guess passes, the check yields one bonus token for free. So one round trip brings back anywhere from one to k plus one tokens. The decode boundary moves from every token to every verified chunk, and the session row only needs one new field, the count of accepted tokens.</p><blockquote>Here you go another Rabbithole: understanding speculative decoding, the math behind DSpark’s approach, how it works internally, and how it’s implemented in vLLM. <a href="https://chatgpt.com/share/6aec19b3-c3e0-83ee-a84e-8fea42c70ea4">[Chat here to understand more]</a></blockquote><p>Other options sit between these. A pipelined design sends several decodes ahead, which loses the clean boundary. A worker driven design lets the worker keep decoding until told to stop, and the scheduler only steps in now and then. Each trades a bit of control for a bit of speed. We keep the simple machine because it is the easiest to understand, and because the third problem is the one we care about.</p><p>Third, the session lives in one place. Its state sits in the memory of one GPU on one worker, and the scheduler only stores a pointer to it. The moment you want that session to run on a different GPU, the state has to move there, and the machine as drawn has no move operation at all. Adding one is where the rest of the blog goes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OwsA_j-M0D8weUer1pI-Bg.png" /><figcaption>Architecture Diagram 3</figcaption></figure><h4>A new operation: migration</h4><p>Migration is one new call on the scheduler and one new pair of commands on the worker. It also makes the rest of this blog interesting, because it forces every box in the system to cooperate.</p><p>The call asks for a session to move to a destination worker. It requests, it does not actually perform the moving. The scheduler only checks that the request makes sense. The session has to be active and not already moving, the destination has to differ from the source, and the destination needs a free slot. If all is well, the scheduler reserves room on the destination right away by bumping its in flight count, and sets a pending flag on the session. Nothing is copied yet. The reservation exists so the destination does not hand its last free slot to a new request in the meantime. The flip side is that any failure has to give the reservation back, without which it slowly starts filling up even when empty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9MjSvIsTPh8JWhDLu2eSQg.png" /><figcaption>Migration State Machine</figcaption></figure><h4>When to move</h4><p>Something has to decide when to call it. That is the rebalance policy, a cousin of the placement policy. After each token the scheduler builds a light view of its sessions and hands it to the policy along with the worker stats. The policy answers with either nothing, or a session to move and a destination.</p><p>A simple one looks at the gap in free room between workers, and when the gap passes a threshold it moves the longest decoding session from the busy worker to the quiet one. That session has the most work invested, and probably the most work left. Short horizon moves can cost more than they save, so any real policy has to weigh the stall against the number of tokens remaining.</p><p>Placement and rebalance together are simple load balancing, one at admission and one during decoding. Both read the same views and both just return different decisions.</p><h4>The simple migration</h4><p>The real work happens at the next decode boundary [waiting for the current decode event to complete]. When the next event for that session arrives and the scheduler sees the pending flag, the session switches to a migrating phase.</p><p>Inside migrating, the scheduler does the following.</p><ol><li>It sends the destination a receive command and the source a send command (what exactly rides in them depends on the transport, we’ll visit this in sometime).</li><li>It waits for both sides to report back for transfer completion.</li><li>It sends a release command to the source.</li><li>When the source confirms the release, it commits. The session now belongs to the destination, with the slot address the destination reported, and the phase goes back to decoding.</li></ol><p>The next decode command goes to the new worker, and the session carries on as if nothing happened. The order of those last steps is deliberate. The source is only freed after both sides have finished, and the commit only happens after the release is confirmed, so there is never a moment when no worker holds a valid copy. If a worker reports failure, the session is marked failed and the memory is released on both sides.</p><p>We wait for the boundary because the state is being rewritten as the model runs, and copying mid step would tear it. Whether you really have to wait is a question about the cache, and we will come back to it there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*3HaYDIKjgYOIb-oeAhsQ1A.png" /></figure><p>These three stretches of time are the three places a migration can be slow, and together they are what the user feels. Wait for boundary is the time between asking for the move and the move beginning, because the session is finishing its current step. Copy is the time from the transfer being posted until it is done. Commit ack is the time from the copy finishing until the scheduler commits. We also track excess stall, which is how much longer than normal the gap between two tokens was when a migration happened inside it. Hold on to these three names, because we will measure each of them later.</p><h4>The same move, a different trigger: disaggregated serving</h4><p>Prefill reads the whole prompt at once and is limited by compute. Decode reads the whole state to make one token and is limited by memory bandwidth. Disaggregated serving runs the two on separate pools of workers so each can be optimized for its own bottleneck. The handoff between them is a state move, which is exactly a migration with a different trigger.</p><p>The only change is who sets the pending flag. Instead of the rebalance policy, the prefill event itself does, and the destination is chosen from the decode pool. Everything after that is the machine from the previous section. It is not built here, but it needs no new machinery.</p><p>A move like this is a clean story on the control side. The scheduler asks, the destination reserves, the source and destination agree, the scheduler commits. What we have skipped is the middle, the blue part of the picture. A slot of about a hundred MiB has to get from one GPU’s memory into another’s, and to understand how, we need to leave the control plane.</p><h3>Data Plane</h3><h4>Inside a worker</h4><p>A worker is a loop wrapped around those three boxes.</p><ul><li>A cache that holds the model state of every session.</li><li>A transfer agent that knows how to ship bytes to another worker.</li><li>A model runtime that knows how to run prefill and decode.</li></ul><p>There are a few ways to build that loop. A batching engine like vLLM advances a whole batch of requests on every iteration. Other designs give compute and communication their own threads, so that neither blocks the other. To keep things simple we use a single loop that reacts to events. Unlike the scheduler’s loop, it busy polls instead of blocking, because it sits next to a GPU and has several things to check on every pass.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PM2GPF-ROlNBgJBf3BFrrQ.png" /><figcaption><em>A worker opened up.</em></figcaption></figure><p>Before a worker tells the scheduler it is ready, it goes through a fixed startup.</p><ol><li>It starts up and binds to its GPU.</li><li>It loads the model weights and builds the runtime.</li><li>It allocates the whole cache pool in one go and registers it with the transfer agent, once.</li><li>It sets up its transport, which depends on the communication backend and is covered shortly.</li><li>It sends the ready event. The scheduler waits for every worker to say so before it accepts any work.</li></ol><p>After that the worker spins in a loop that does three things in order:</p><ul><li>Checks which pieces of GPU Decode/Prefill work have finished and reports them back.</li><li>Checks whether any transfers have finished.</li><li>Reads the next control message from the scheduler.</li></ul><p>When there is nothing to do, it sleeps for about a millisecond. That polling floor is worth remembering, because any timing we quote later cannot be read more precisely than about a millisecond.</p><p>What happens when a worker dies? All the sessions running on it die as well. This is a deliberate choice. A worker’s cache is ephemeral, so anything the scheduler has not committed is considered unrecoverable. The scheduler’s record is the single source of truth. Recovery means starting a new worker with empty state and rebuilding the session from what the scheduler has, which is the prompt and every token it has accepted so far. That avoids the pain of reconciling a scheduler and a worker that disagree, which is rarely worth the effort.</p><h3>The cache</h3><h4>What is in the cache</h4><p>In a plain transformer, every new token appends a key and a value at every attention layer. That growing pile is the KV cache, and it gets more expensive the longer the session runs. Alongside it, a recurrent layer keeps a second kind of state: a fixed size summary of everything seen so far. Each token rewrites that summary, and it stays the same size however long the context gets. A state space layer such as Mamba works this way. A hybrid model holds both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Btoe9I1IbTKKnJVrCpINw.png" /><figcaption>Transformer Cache</figcaption></figure><p>Take Kimi K3 as the large version of a model that holds both. It has 93 layers and a million-token context, laid out as three Kimi Delta Attention layers followed by one gated multi-head latent attention (MLA) layer, repeated. That is 69 linear layers and 24 attention layers. The linear layers carry the recurrent state. Their update rule differs from Mamba’s, and the cache is still a constant per session.</p><p>The attention layers are where the cache grows, and how fast it grows is a choice. A full key and value on all 93 layers, 96 heads of dimension 128 in 2 byte elements, is 2 x 93 x 96 x 128 x 2 bytes. That is 4.36 MiB per token, about 4.2 TiB for a million tokens.</p><p>But K3 stores a latent token of 576 values instead of a key and a value per head for each of the 24 attention layers. The growing cache comes down to 24 x 576 x 2 bytes, which is 27 KiB per token, about 26 GiB for the full window. Once the weights are loaded, that cache is what limits how many sessions fit and how long their context can be, and it is the state a migration has to carry.</p><blockquote>Damn, so many Rabbitholes: Try understanding the math behind Multi-Head Latent Attention (MLA), how it compresses the KV cache, and its impact on GPU memory consumption, serving capacity, and context length. <a href="https://chatgpt.com/share/6aec1a31-0a74-83e8-b2b3-04b8f3e400af">[Chat here to understand more]</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*smKF1izS12n_a9riINXtgg.png" /><figcaption>Hybrid Cache</figcaption></figure><p>It also explains why sessions are sticky. The state lives in one GPU’s memory, so a session is pinned to the GPU that holds it. Moving the session means moving that state [which can get costly very quickly].</p><p>Our hybrid is the small version of that split. Every one of Falcon H1 0.5B’s 36 layers runs a Mamba block and an attention block side by side, so every layer owns four regions of cache.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*2JOX1yoSVQUx0kUqKioUVA.png" /><figcaption>Falcon H1 Cache Sizes</figcaption></figure><p>This cache layout can be arranged and changed to suit multiple optimizations solving different problems, few of them being</p><h4>Paged attention</h4><p>Reserving one big contiguous block of memory per request for the worst case wastes a lot. Space inside the block sits unused, and space between blocks is too fragmented for anyone else to use. Paged attention fixes this by splitting the KV cache into small fixed size blocks. This is how vLLM does it. At startup it carves the memory set aside for cache into a pool of blocks, each holding the keys and values for a fixed number of tokens. A block pool tracks which are free, which are used, and how many requests are using each through a reference count. Each request has a block table that maps its logical blocks to physical ones, filled left to right as tokens arrive. Blocks come off the free queue on demand and go back when the last user is done, and the attention kernel reads through the table. The pool is created once, so no allocation cost lands on the hot path. The result is much better memory utilisation and easy sharing between requests. [This is exactly how your RAM pages work]</p><h4>Prefix caching</h4><p>Many requests start with the same text, such as a system prompt, a few examples, or the start of a long conversation. Recomputing that prefix each time is wasteful, so systems remember which cached blocks belong to which prefix. There are two common shapes. SGLang keeps a radix tree keyed by token ids, where a new request walks down the tree and reuses whatever matches. vLLM keeps a hash per full block, computed from the block’s own tokens and the hash of the block before it, so a matching hash vouches for the whole prefix up to that point. A new request hashes its prompt block by block, reuses every match and bumps its reference count. In both cases an eviction rule such as least recently used decides what to drop, and only full blocks are shared. The key assumption underneath is that cached KV blocks never change once written, which is what makes sharing them safe. Remember that assumption, because the next part breaks it.</p><h4>Why this hurts for a hybrid model</h4><p>Falcon H1 keeps two kinds of state per session. There is a fixed size convolution and state space state, which is rewritten on every token. And there is attention KV, which grows with the context but remains stable. This causes three problems.</p><p>First, paging only helps the KV cache. The fixed-size state doesn’t need paging, and the two can differ significantly in size. A single layer’s recurrent state can be much larger than the KV cache for one token, while the KV cache keeps growing as tokens arrive. This makes it difficult to use the same block size for both. vLLM’s hybrid KV cache manager uses one memory pool and one block size for all layers, so it has to increase the attention block size to accommodate the larger state. This can result in blocks holding hundreds of tokens (672 in one documented example), which leads to lower cache hits and higher internal fragmentation. Recent work separates the block size used by the cache manager from the one used by attention kernels, helping reduce this problem.</p><p>Second, sharing prefixes is tricky when the state keeps getting overwritten. Unlike KV cache blocks, recurrent state is updated with every new token, so you can’t simply share it between requests. Instead, vLLM has a mode that stores the state only at selected token positions aligned to block boundaries, with the snapshot interval as the knob. If snapshots are too frequent, they waste memory. If they’re too far apart, more tokens need to be recomputed.</p><p>Third, moving a live session has little to copy ahead of time, and this is the one that matters most for us. Systems built on append only KV have a neat trick. Llumnix, which migrates live requests between vLLM instances, overlaps memory movement with compute. Old KV pages are stable while they move, because decoding only touches the newest page. So it copies the bulk of the cache while decoding carries on, then pauses briefly to send the small delta written in the meantime. The copy hides behind the ongoing decode, and the visible pause shrinks to the last small delta meaning the decode boundary could effectively merge with the actual copy time reducing overall transfer time.</p><p>But a hybrid gets much less out of this. The conv and ssm regions are rewritten on every token, so a copy running alongside decode would pick up a mix of old and new state. Only K and V are stable, and the recurrent state is a fixed 27.7 MiB across the 36 layers, which is more than all the KV pairs a session has produced until its context passes about 790 tokens. So for short and medium contexts, the delta left at the end is most of the slot, and there is little compute left to hide it behind. The copy cannot overlap with the decode boundary effectively.</p><h4>The simple choice: one fixed slot per session</h4><p>Notice how much machinery goes into making a growing KV cache and a fixed size state share one pool. So to keep everything easy to follow, our cache pool is a fixed number of slots, and each session leases exactly one. A slot holds the convolution and state space state for the recurrent layers plus the KV for the attention layers, all sized for the maximum sequence length, packed into a single contiguous block of memory. The layers sit one after another, and inside each layer the four regions sit back to back in the order conv, ssm, K, V. Because the pool is preallocated, a slot is the same size no matter how much of the context is used.</p><p>The costs are honest ones. Unused KV capacity is reserved anyway, which is a kind of internal fragmentation. And a migration copies the whole slot, not just the part in use.</p><p>This simplifies the scheduler for us by a lot. One address and one size describe the entire session state. That is why the scheduler can get away with storing a single pointer per session, and why moving a session boils down to asking someone to transfer this address. The same property gives us the whole lifecycle: lease a slot, decode into it, migrate it, release it, and release the source slot only after the destination has taken over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*a8vUVSgsof-8V1moM-mDXg.png" /><figcaption>Fixed = conv+ssm across 36 layers. KV = K+V for<br>the full window.</figcaption></figure><p>The first 27.7 MiB is the same in every row, which is the recurrent state that never grows. Only the rest scales with the context length. Most of this blog uses the 2048 row, a slot of about 100 MiB.</p><blockquote>You know what, another rabbit hole: understanding how vLLM and other inference engines design their cache layouts and how CUDA kernels access them. Then try designing your own architecture for managing hybrid caches. <a href="https://chatgpt.com/share/6aec1a9b-b4f4-83ec-b68b-b418b5f60c77">[Chat here to understand more]</a></blockquote><h3>Moving bytes: the transfer agent</h3><p>Moving a slot means moving bytes between two GPUs, which is the transfer agent’s job. Every backend plugs into the same small interface: register a memory region once, post a transfer, poll the handle, pass along any control messages, and shut down (more on these later). The worker itself does not care which backend sits behind that interface, the transfer agent abstracts each backend’s control traffic and failure modes away, allowing the migration state machine above to keep the same shape.</p><p>Now we take two backends in turn below, asking only who says what to whom and how that changes the scheduler’s migration protocol. How the hardware actually executes the transfer comes later.</p><h4>NCCL</h4><p>NCCL is the collective communication library most people meet through training. It was built for operations like all reduce, where many GPUs combine their data. For a single direct mem to mem transfer it works as a matched pair of calls. One side sends, the other receives, and both sides have to post their half. A send does not complete until the matching receive is posted, and vice versa.</p><p>That changes the state machine in three ways. First, NCCL needs a communicator that links the two workers, so startup gains a bootstrap phase. Before ready, the scheduler hands each worker a shared id and its rank, each worker sets up its communicator, and each replies when done. Second, nothing needs announcing at move time. The source posts a send and the destination posts a receive, with a small header carrying the sequence length followed by the slot itself. The two sides only have to post matching calls. Third, the failure shape is a bit tricky to handle. If the two sides do not match, or one peer is dead, the pair hangs. There is no clean way to cancel just one side other than using a timeout.</p><p>On an idle system, with a slot of roughly 100 MiB, the copy takes about 1.1 ms. Read that as roughly a millisecond, given the polling floor of 1ms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*PUiYeMeTkGHm0GQDiRPcWw.png" /><figcaption><em>NCCL Migration State Machine</em></figcaption></figure><h4>NIXL</h4><p>NIXL is a transfer library built for exactly this kind of job, moving large chunks of memory between agents, and it is used by several serving stacks. It offers one-sided reads and writes between registered memory regions. One sided means one party can read or write the memory of the other without that other party taking part in each transfer. That is the big difference from NCCL’s matched pair, because here only one worker acts. The source never posts anything, and there is nothing to hang on if it is busy.</p><p>The state machine changes in a different way. Each worker has an agent, and each agent has metadata describing it. Before anything can move, agents need each other’s metadata. So at startup every worker publishes its metadata, the scheduler fans it out to the others, and then tells everyone that the peers are final. After that, a transfer only needs addresses for the registered memory. [Similar to NCCL bootstrap but different words]</p><p>The idle copy is about 1.15 ms, essentially the same as NCCL. The first move is a different story. It takes about 34 to 40 ms, because the remote agent is loaded lazily the first time it is needed. The obvious fix is to load it eagerly after the peers are final and run one warm transfer at startup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*oC9vCU48Y3s0HwJKH-4xFg.png" /><figcaption>NIXL Migration State Machine</figcaption></figure><h4>Letting the scheduler carry the address</h4><p>Here is the most interesting change in the whole project. It is about how the destination learns where the bytes are. It is also where the control plane comes back to bite us.</p><p>The first NIXL version had the sender write into the destination, and a NIXL write needs the destination’s address. So every migration paid for an announce round trip.</p><ol><li>The destination leases a slot and tells the scheduler where it is.</li><li>The scheduler forwards that address to the source.</li><li>The source posts the write and notifies the destination when it is done.</li></ol><p>Worse, the announce travelled through the same single loop as decode commands, so it queued behind decode work. With nothing else running, the copy took about 3.2 ms. With 32 sessions decoding in the background, it took about 322 ms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FrtbC6-bDHfT2mGyyj8-Qg.png" /><figcaption><em>Announce per transfer. The red announce goes through the scheduler and the forward waits behind decode commands.</em></figcaption></figure><p>The fix needed one small addition to the session metadata, a new field holding the address of the cache slot. The worker writes it when it returns the first prefill event for that session, basically the address piggybacks on that event. A migration reads the slot address from the cluster’s metadata, so there is no announce round trip to learn where the bytes are.</p><p>So the design flipped from a NIXL write [from src to dst] to a one sided dst read. That address now goes straight into the destination’s receive command, and the destination reads from the source. NIXL agent metadata is still exchanged once at startup, and nothing is announced per transfer. The source is out of the data path completely and only waits to be told to release. Idle, the copy dropped from ~3.2 ms to ~1.2 ms. With the same 32 sessions decoding in the background, on the same slot of about 100 MiB, it dropped from ~322 ms to ~8.3 ms.</p><p>Notice the role the scheduler plays here. It is a shared directory. It holds addresses and fans out agent metadata, which is the global control store in Ray also does. Ray runs a NIXL transport that works the same way, receiver driven, with descriptors and agent metadata travelling over its control plane as object metadata while the sender stays out of the data path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W7-25qcX_E16NWRDrKvDfg.png" /><figcaption><em>Address piggybacked. The source address rides inside the receive command, so there is no announce and the source stays out of the data path.</em></figcaption></figure><h4>Where the time goes</h4><p>Now we raise the temperature and measure the three stretches from the migration section. Background load means other sessions busy decoding on the peer worker while the migration happens. We sweep it from 4 background sessions to 64, with NIXL and NCCL side by side, on a slot of about 100 MiB.</p><p>The headline result is that stall scales with background load and not with the wire/copy time. With 4 background sessions, the stall a session feels is about 11 ms. With 64, it is about 290 ms. Meanwhile the copy itself stays small, a few milliseconds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NlawfcbQvoh9RKVkxgpWtQ.png" /><figcaption>Cache migration performance across NCCL and NIXL backends under increasing background decode loads on 2× A100 SXM GPUs using Falcon-H1 0.5B.</figcaption></figure><p>So the copy/xfer is a few percent of what the user feels, and the time goes into the other two stretches. Both are protocol costs that scale with how long a round of decode takes. The migration command has to wait for the worker’s single threaded loop to come around, and the commit handshake rides the same loop. More sessions on the worker means a longer round. Swapping backends barely moves any of this, since both share the same wait and commit costs, so a faster wire does not help much. What the user feels is the stall.</p><h4>Why this is a skill issue</h4><p>It’s worth being precise about what this result actually tells us. This is a consequence of our simple, single-loop worker architecture and exposes the other half of the problem we hinted at earlier, where the scheduler itself sits on the hot path.</p><p>A migration command is just another message, and the worker only reads messages once per round, after it has issued and checked a full round of decode work. The commit acknowledgement waits in the same line on the way back. Neither is slow because the work is hard. They are slow because they queue behind work that does not care about them.</p><p>That makes it a design choice we can undo, and the remedies are fairly obvious. Move transport progress onto its own thread, so a finished copy is noticed and reported at once. Give control messages priority over decode work. Handle migration commands outside the decode round. We have not built or measured any of these, so treat them as the first experiments in the levers at the end. Until then, the numbers above are what the simple design costs, not what migration has to cost.</p><h4>What the hardware does with the bytes</h4><p>So far the wire has not been the villain. But we still owe an answer to a question hanging since the NCCL section. On the same NVLink path, why do two libraries behave so differently inside the GPU? Everything comes down to the fact that a GPU has two different kinds of machinery that can move or touch memory.</p><p>The compute cores run kernels, which is where matrix multiplies and attention happen. The copy engines are separate, dedicated DMA hardware whose only job is moving bytes from one memory to another. A copy that runs on a copy engine does not take time away from the compute cores. A copy that is written as a kernel does.</p><p>NCCL was built for collectives. Operations like all reduce do arithmetic on the data as it moves, so NCCL fuses the reduction logic and the movement logic into the same kernels. That is great for collectives, because it means no extra pass over memory. The side effect is that even a plain point to point copy runs as a kernel on the compute cores, which compete with the decode kernels. NIXL is built for point to point transfers and has no reduction to fuse, so it can hand the copy to the dedicated copy engines, and the compute cores stay free for decode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uE6nsNtQGnjcaH-eRllSpA.png" /><figcaption>Very high level Abstract view of a Nvidia GPU</figcaption></figure><p>A quick look through a profiler makes the difference visible. NCCL’s copy is a kernel (ncclDevKernel_SendRecv) that holds compute cores for about 16 ms per slot of roughly 100 MiB. NIXL’s copy is a peer to peer memory copy on the copy engine that finishes in about half a millisecond.</p><p>The observed copy time tells the same story once there is something else running on the GPU. Idle, the two backends are almost the same: about 1.1 ms for NCCL and about 1.2 ms for NIXL on a slot of about 172 MiB, mostly a fixed launch cost plus the worker’s roughly 1 ms poll tick.</p><p>Under contention the paths split. With 64 background decode sessions, NCCL’s copy climbs to about 18.7 ms, while NIXL stays near 5.3 ms. NCCL’s SendRecv kernel has to wait for free compute cores the same way any other launched kernel would and starts contending for SM resources with the decode kernels. NIXL’s copy engine does not share that queue, so its copy barely moves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*RkkpjMTGqoKF-WxW7sKwSg.png" /><figcaption>NCCL vs NIXL Copy/Xfer Time at different background decode sessions</figcaption></figure><p>One final distinction. This doesn’t mean NCCL is bad, it means it is built for a different communication pattern. Tensor and pipeline parallelism often involve frequent collective operations such as AllReduce, where data movement is combined with reduction across multiple GPUs. NCCL is designed to handle these patterns efficiently.</p><p>Cache migration is fundamentally different. We’re moving an entire cache slot, essentially one large contiguous region of memory from one GPU to another. There is no reduction or aggregation involved, just data movement. So the comparison isn’t about one library being better than the other. They’re optimized for fundamentally different communication patterns.</p><blockquote>hehe one more rabbithole: Try to understand how NCCL actually implements collective operations, which algorithms it uses for each collective, and how it adapts to different GPU interconnect topologies to optimize communication. (If you want to learn about Distributed Training that is) <a href="https://chatgpt.com/share/6aec1ae4-017c-83e8-9ad8-3e76b2e9b718">[Chat here to understand more]</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LvZDjjnbDNMKr4t-VA-log.png" /><figcaption>Architecture Diagram 4 or 3 I lost count</figcaption></figure><h3>Topology: which roads exist</h3><p>Whether a copy can use the copy engine at all depends on the roads between the two GPUs, so let’s draw that picture properly. We go from inside one machine, to between machines, to the library that chooses among the paths, and finally break one path on purpose to see what it costs.</p><h4>Inside a node</h4><p>Before the paths, the two junctions inside one machine.</p><p>PCIe is a high-speed interconnect that connects the CPU to GPUs, network cards and storage devices through slots on the motherboard. It consists of multiple lanes that carry data between devices. PCIe is full-duplex, meaning data can be sent and received simultaneously. The number of lanes (x4, x8, x16) determines how much bandwidth is available.</p><p>NVLink is a high-speed connection that allows GPUs to communicate directly with each other without routing data through host memory. One GPU can read and write data directly to another GPU’s memory. On these A100s, NVLink provides around 600 GB/s of combined bandwidth, much faster than PCIe. In larger systems, NVSwitch connects multiple GPUs, allowing them to communicate with each other at high bandwidth.</p><p>Now the three paths between two GPUs in the same node that are worth knowing, and each burns something different.</p><ol><li>Direct GPU-to-GPU transfer: Data moves directly between GPU memories over NVLink, handled by the GPU’s copy engine without involving the compute cores. (NIXL CUDA IPC : Fastest)</li><li>Kernel-based GPU-to-GPU transfer: Data moves directly between GPU memories over NVLink, but the transfer is performed by kernels running on the compute cores. (NCCL P2P : Fast, Depends on usecase)</li><li>Host-staged transfer: Data moves from GPU A to host memory over PCIe, across a TCP loopback connection, and then from host memory to GPU B over PCIe. This involves multiple copies through host memory. (Slowest)</li></ol><h4>Between nodes</h4><p>Across machines, NVLink is no longer available, and the network card becomes the new piece of hardware handling data movement. This is where RDMA (Remote Direct Memory Access) comes in. It allows one machine to read from or write directly to memory on another machine. The NIC handles the transfer, breaks the data into packets, sends them across the network, and places them in the destination machine’s memory.</p><p>Two conditions make RDMA possible.</p><ul><li>Memory registration: Memory must be registered before the NIC can access it. Registration pins the memory pages and provides a key that authorizes remote access. Any remote read or write must provide the memory address and this key. In our case, the worker registers its cache pool once when handing it to the transfer agent. The setup cost is paid upfront, subsequent transfers only need an address, size and key.</li><li>Kernel bypass: The kernel handles the initial setup, such as opening the NIC and registering memory, but stays out of the actual transfer path. The application submits read or write requests directly to queues monitored by the NIC and polls a completion queue to know when the transfer finishes. This keeps the CPU out of the data path, avoiding per-transfer kernel processing and reducing CPU overhead and latency.</li></ul><p>The packets still need a fabric [the network outside the two machines] to ride, and RDMA has two options.</p><p>InfiniBand is its own network stack, with its own protocols and its own switches specifically designed for RDMA. A read or a write is a native packet on that fabric, and the switches hand out credits to senders to keep those packets from being dropped.</p><p>RoCE keeps the same reads and writes but sends them over traditional Ethernet. The version used in clusters wraps each operation in IP and UDP, and ordinary Ethernet switches forward the packets. The NICs still speak RDMA, the rest of the fabric just speaks plain Ethernet.</p><p>With that hardware in place, three paths between GPUs on different machines are worth knowing.</p><ol><li>GPUDirect RDMA, on InfiniBand or on RoCE. The NIC reads and writes GPU memory directly over PCIe. The bytes go GPU to NIC to wire to NIC to GPU, host memory and CPU are both skipped. [Fastest]</li><li>RDMA through host memory. The card can only see registered host pages, so the source GPU copies the slot into a pinned host buffer, the card moves that buffer across the fabric, and the destination copies it into its GPU. The network hop is still RDMA, but host memory is back on the path, twice. [Fast but bottlenecked by host memory copies]</li><li>Plain Ethernet, with no RDMA. The bytes take the host staged route from inside the node, with a real network standing in for the loopback socket: device to host over PCIe, through the kernel, across the wire, through the kernel again, and host to device. Both CPUs sit in the middle of every packet. [Slowest]</li></ol><h4>UCX</h4><p>With so many possible paths, picking the right one by hand for every pair of peers would get annoying fast. UCX is a communication framework built to take that off your plate. It exposes one simple API and handles choosing the fastest transport underneath, based on what hardware topology actually connects the two sides. That transport can be shared memory, TCP, CUDA copy, CUDA IPC, or across machines, InfiniBand verbs or RoCE. NIXL sits on top of UCX and exposes one sided reads and writes on registered memory. The list of allowed transports is controlled by a setting, and the difference between the good path and the bad path can be a single entry in it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lgBrExnYuPMcvBbFTyCuyA.png" /><figcaption>GPU Cluster Communication Map containing all the different paths one can take</figcaption></figure><h4>NIXL on CUDA IPC vs NIXL host staged</h4><p>For NIXL specifically, that choice comes down to two paths. The CUDA IPC path needs the two GPUs to reach each other’s memory directly, over NVLink or PCIe peer access. When they can, it’s one hop on the copy engine, and the host is never involved. The host staged path needs almost nothing by comparison, just PCIe, a pinned buffer and a socket, which is why it works on any machine, but it pays for that generality with four hops and two trips through host memory.</p><p>The fastest way to see how much that matters is to break it. Take NIXL and remove CUDA IPC from the list of transports UCX is allowed to use. UCX now has no direct GPU to GPU path left, so every transfer bounces through host memory. Same slot, same machines, and the copy now runs at about 0.23 to 0.29 GB/s and takes roughly 358 to 458 ms. Compare that with the healthy paths, which run at about 11 to 50 GB/s depending on load. Disable peer access in NCCL and it falls back to going through host memory too, at about 3.6 to 9.6 GB/s.</p><p>The lesson is that a misconfigured path costs far more than any choice between protocols. And it is a misconfiguration and not a weakness of NIXL. It is a UCX setting that removes the one transport that avoids the host round trip. The practical advice is to validate the path you think you are using. Record the transport list, the UCX version, and whether NCCL used peer to peer or shared memory, because a silent fallback looks exactly like a slow protocol.</p><h3>What lives below GPU memory</h3><p>Everything so far lives in GPU memory, and that is the smallest and most expensive place to keep model state. Most sessions are idle most of the time, like a chat between turns or an agent waiting on a tool call. If every session must sit in GPU memory, concurrency is capped by memory and not by compute. Cold state could live somewhere cheaper and come back when needed. This section is where the slot design points next.</p><h4>The ladder, and who decides</h4><p>The usual ladder runs from GPU memory down to pinned host memory, then local SSD, then spinning disk, then object storage like S3. Each step down is larger, cheaper and slower. A fixed slot is already the right unit to move around such a ladder. It turns offloading into the same primitive as migration, which is move this slot, except the destination is a storage tier and not another GPU.</p><p>The decision about where a session lives is another policy, the third kind beside placement and rebalance, and it is a pure function over the session view like the other two. The scheduler tracks a last active time for every session, a new field next to the slot handle, and the policy turns idle time into a destination.</p><ol><li>Actively decoding: stays in a GPU slot.</li><li>Idle for a few mins: like a chat between turns or an agent waiting on a tool call: demote to pinned host memory, where a restore costs one PCIe hop.</li><li>Idle for hours: demote to local NVMe. The state is now tied to that node, so a restore has to land on a worker that can reach the disk.</li><li>Idle for days, or the user is gone: write it to object storage. It is shared and durable, so the session can resume on any worker, and it even survives a worker dying, which partly answers the failure gap from the worker section. But first byte latency will be too high.</li></ol><blockquote>Are you tired of the Rabbitholes: Explore how parallel chunked transfers and prefetching can reduce first-byte latency. How can you stream data in chunks and overlap transfers with computation? <a href="https://chatgpt.com/share/6aec1b8a-a630-83e8-85ed-3e978efdcf46">[Chat here to understand more]</a></blockquote><p>That same durability is also where the real money gets saved. Because any worker can restore a session’s model state from object storage, the GPU fleet itself can scale with demand instead of sitting at peak size around the clock. If a worker shuts down during a quiet period or loses its spot instance, its sessions aren’t lost. They’re stored in S3 until another worker picks them up.</p><blockquote>Bruh one more Rabbithole: <a href="https://developer.nvidia.com/blog/nvidia-dynamo-adds-support-for-aws-services-to-deliver-cost-efficient-inference-at-scale/">NVIDIA Dynamo’s S3 backed KV offload</a> exists for exactly this, letting new replicas spin up on demand without recomputing anything, and <a href="https://www.devzero.io/lp/inference">DevZero’s checkpoint restore</a> uses the same trick to survive spot preemption, restoring a session in tens of seconds instead of paying for a GPU that sat idle the whole time. <a href="https://chatgpt.com/share/6aec1bda-019c-83ee-b2fc-fdcf218debec">[Chat here to understand more]</a></blockquote><p>The thresholds are the interesting part. A restore is only worth it if it beats recomputing the prompt, and for recurrent state that is a real question, since the state can only be rebuilt by replaying the whole sequence. You can also imagine adding more business logic here, like per tenant time to live, priority users, and expiring or deleting cold sessions for privacy and cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gdz1V5ytsgnxXKKB8L4ZuA.png" /><figcaption>Close up of the Data Plane</figcaption></figure><h4>Keeping the host out of the hot path</h4><p>The hardware section told us what each step costs, and the same ideas show up again. A naive offload bounces everything through host memory, with the CPU driving each copy. The better designs go device to device, reusing the same RDMA and Infiniband Technology.</p><ol><li>GPUDirect Storage is the storage cousin of GPUDirect RDMA. A DMA engine near the drive moves blocks straight between local NVMe, or NVMe over remote machines, and GPU memory, with no bounce buffer in host RAM. NVIDIA describes it in <a href="https://developer.nvidia.com/blog/gpudirect-storage/">this post</a>.</li><li>SPDK often gets mentioned in the same breath, but it is worth being precise. It is a user space storage stack, and it includes direct memory transfers between NVMe over remote machines, which makes it the efficient storage side of a design like <a href="https://www.seagate.com/content/dam/seagate/assets/resources/enabling-inference-at-massive-scale-with-hybrid-storage-for-kv-cache-offloading/files/kv-caching-white-paper.pdf">this Dynamo based one</a>. It does not by itself move bytes straight into GPU memory. That part is GPUDirect’s job.</li></ol><h3>Kernels, streams and graphs</h3><p>This is where the runtime finally comes in, and it also lets us cash in two promises from earlier.</p><p>A stream is a queue of GPU work that runs in the order it was pushed. The host enqueues kernels and copies onto it, and the GPU drains that queue front to back, one operation finishing before the next on the same stream starts. That ordering guarantee stops at the stream’s own boundary. Two different streams have no ordering promise between them, so if the hardware has a free compute core or a free copy engine, those two streams can run side by side. Everything defaults to a single implicit stream unless the program asks for more, which is why extra streams exist at all: they are how you tell the GPU which pieces of work are allowed to overlap. Every launch, onto any stream, still costs a little host time to issue.</p><p>In my design, each session in the cache pool gets its own execution context built on that model. It consists of a CUDA stream, a cuBLAS handle so that concurrent streams do not step on shared state, and an event, which is a marker that tells the host when the stream’s work is done. A separate stream handles transfers. Because every session has its own stream, one session can pause while the others keep decoding keeping the GPU busy.</p><p>This is where the decode boundary from the control plane becomes concrete. Stopping at the boundary really means waiting for that one session’s stream to finish its current step. It also ties back to the hardware section. A transfer on its own stream still competes for the compute cores if it is a kernel and does not if it runs on a copy engine.</p><p>Beyond streams, there is a whole ladder for how the compute itself gets expressed.</p><ol><li>Fused single operations like Flash Attention, which cut D2H memory traffic and launches.</li><li>Generated kernels from tools like Triton, which make fusing less painful than writing CUDA by hand.</li><li><strong>CUDA Graphs:</strong> Capture a sequence of GPU operations once and replay them with a single launch instead of launching each operation individually. Since decode steps are small, CPU launch overhead can become a bottleneck. Graph replay reduces this overhead and keeps the GPU busy.</li><li>Megakernels at the top, which fuse communication and compute together.</li></ol><blockquote>Sorry but here is another Rabbithole: look into how people write compilers but for CUDA kernels, how these compilers optimize them, and how they automatically fuse operations to reduce memory traffic and launch overhead. <a href="https://chatgpt.com/share/6aec1c81-75f4-83ee-8b1f-3960e8d245d7">[Chat here to understand more]</a></blockquote><h3>Levers to pull</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBxeH6e82nCqi6R6XgHRZw.png" /><figcaption>The Entire Final Architecture</figcaption></figure><p>We have now been from a request at the front door down to bytes crossing a wire. Here is where you take over. Every layer we visited has levers, and each one is an invitation to change something and measure what happens. Everything below is a summary of every idea we have discussed in this absurdly long blog.</p><ul><li>Hardware and topology: NVLink and NVSwitch layouts, GPUDirect RDMA and Storage, faster interconnects. The cheapest experiment is to validate the path you think you are using.</li><li>Compute: Fused kernels, Triton, CUDA graphs, megakernels, allocator hygiene, and copy engine versus compute core transfers.</li><li>Communication: Eager metadata exchange and warm transports.</li><li>Policy: Placement and rebalance thresholds, migrating only when the stall is small against the remaining work, splitting prefill and decode, and the tiering policy. None of these need a new data plane, only a new function in the control plane.</li><li>The model itself: The model architecture directly shapes the inference system. MLA changes how KV cache is stored and accessed. Hybrid models introduce different cache layouts for attention and recurrent state. Mixture of Experts introduces expert routing and additional GPU communication traffic. Each architecture brings different memory, compute and communication requirements, which influence cache management, scheduling, GPU utilisation and how sessions can be migrated between workers.</li></ul><p>Everything ultimately comes down to the economics of running GPUs. An A100 80GB costs around $1.79 per hour on <a href="https://www.runpod.io/pricing">RunPod</a> (at the time of writing), and you pay that amount whether the GPU is busy or sitting idle. Every optimization we discussed is an attempt to squeeze more tokens out of that fixed hourly cost. Batching keeps the GPU busy. Better scheduling ensures sessions use resources efficiently. Faster transports prevent migrations from stalling users. Cheaper storage tiers let more sessions remain alive without occupying expensive GPU memory.</p><p>The business objective is simple. Generate as many tokens per GPU-hour as possible and maximise the revenue generated from every dollar spent renting that GPU. If you can generate X times more tokens for the same GPU-hour cost, you fundamentally change the economics of your inference business. That is what drives the company. None of these optimizations are particularly glamorous, but they are measurable, and that is what makes building a small system of your own so valuable.</p><h4>[Typical cheesy way to end a blog]</h4><p>You now have the whole stack, from a request at the front door to the bytes crossing a wire. Redraw the map in your own way, find the box you care about the most, and go change it.</p><h3>References &amp; Further Reading</h3><h4>1. Inference Engines &amp; Serving</h4><ul><li><a href="https://github.com/vllm-project/vllm">vLLM</a> : High-throughput LLM serving, continuous batching and PagedAttention.</li><li><a href="https://github.com/sgl-project/sglang">SGLang</a> : RadixAttention, prefix caching, scheduling and efficient serving.</li><li><a href="https://github.com/ai-dynamo/dynamo">NVIDIA Dynamo</a> : Distributed inference serving and disaggregated serving architecture.</li><li><a href="https://github.com/kvcache-ai/Mooncake">Mooncake</a> : KV cache-centric disaggregated serving architecture.</li><li><a href="https://github.com/LMCache/LMCache">LMCache</a> : KV cache reuse, storage and offloading.</li></ul><h4>2. Distributed Systems &amp; GPU Communication</h4><ul><li><a href="https://github.com/ray-project/ray">Ray Project</a> : Distributed computing framework.</li><li><a href="https://www.anyscale.com/blog/rdt-ray-direct-transport-fast-easy-weight-syncing-for-rl-reinforcement-learning">Ray Direct Transport (RDT)</a> : Direct GPU-to-GPU data transfer for efficient weight synchronisation.</li><li><a href="https://github.com/ai-dynamo/nixl">NIXL</a> : GPU-aware data transfer and communication library.</li><li><a href="https://github.com/openucx/ucx">UCX</a> : Unified communication framework for high-performance networking.</li><li><a href="https://github.com/NVIDIA/nccl">NCCL</a> : Collective communication primitives for multi-GPU and multi-node workloads.</li><li><a href="https://github.com/deepseek-ai/DeepEP">DeepEP</a> : Expert-parallel communication library for efficient MoE dispatch and combine.</li><li><a href="https://github.com/NVIDIA/Megatron-LM">Megatron-LM</a> : Tensor, pipeline and expert parallelism for large-scale model training and execution.</li></ul><h4>3. GPU Kernels &amp; Model Architectures</h4><ul><li><a href="https://github.com/triton-lang/triton">Triton</a> : GPU programming language and compiler for writing efficient kernels.</li><li><a href="https://github.com/Dao-AILab/flash-attention">FlashAttention</a> : IO-aware attention kernels using tiling and GPU memory hierarchy optimisations.</li><li><a href="https://github.com/flashinfer-ai/flashinfer">FlashInfer</a> : Attention kernels designed around paged KV caches and dynamic batching.</li><li><a href="https://github.com/deepseek-ai/DeepSpec">DeepSpec</a> : DeepSeek’s speculative decoding implementation.</li><li><a href="https://cursor.com/blog/mixture-of-kittens">Mixture of Kittens</a> : Exploring GPU kernel optimisation and efficient execution of mixture-of-experts workloads.</li><li><a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/">CUDA Programming Guide</a> : GPU execution model, memory hierarchy, streams, asynchronous execution and CUDA graphs.</li></ul><h4>4. Disaggregated Serving &amp; Scheduling</h4><ul><li><a href="https://github.com/LLMServe/DistServe">DistServe</a> : Disaggregating prefill and decode across GPUs.</li><li><a href="https://arxiv.org/abs/2311.18677">Splitwise</a> : Separating prompt processing and token generation to improve resource utilisation.</li><li><a href="https://github.com/microsoft/sarathi-serve">Sarathi-Serve</a> : Chunked prefills and scheduling to reduce interference between prefill and decode.</li></ul><h4>5. Foundational Papers</h4><ul><li><a href="https://arxiv.org/abs/2309.06180">PagedAttention</a> : The foundational paper behind vLLM’s paged KV cache design.</li><li><a href="https://www.usenix.org/conference/osdi22/presentation/yu">Orca</a> : Iteration-level scheduling and continuous batching.</li><li><a href="https://arxiv.org/abs/2205.14135">FlashAttention</a> : IO complexity and attention kernel design.</li><li><a href="https://arxiv.org/abs/2405.04434">DeepSeek-V2</a> : Multi-head Latent Attention (MLA) and its implications for KV cache memory.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0511434a3fd2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an HTTP server three times, from the socket down to the wire]]></title>
            <link>https://medium.com/@manishimmi2k3/building-an-http-server-three-times-from-the-socket-down-to-the-wire-780366f827b3?source=rss-db29cf523fed------2</link>
            <guid isPermaLink="false">https://medium.com/p/780366f827b3</guid>
            <category><![CDATA[computer-networking]]></category>
            <category><![CDATA[dpdk]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[https]]></category>
            <category><![CDATA[epoll]]></category>
            <dc:creator><![CDATA[Manish Immadisetty]]></dc:creator>
            <pubDate>Wed, 01 Jul 2026 17:52:30 GMT</pubDate>
            <atom:updated>2026-07-01T18:26:53.064Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eBRnpNwx8huOj7W4ZhFIng.png" /></figure><h3>What this actually is</h3><p>This is not a project about building “<strong>yet another HTTP server.</strong>” Anyone can call <strong>listen()</strong> and serve a response in twenty lines. What this project is actually about is taking the same tiny HTTP server and implementing the <strong>ground underneath it three separate times</strong>, each time peeling back one more layer of the stack that the kernel normally hides from you.</p><p>The same server gets implemented at three different layers of the stack:</p><ul><li><strong>Layer 4, plain sockets:</strong> the kernel handles Ethernet, ARP, IP, and TCP; you only touch bytes once they’ve already been turned into a clean stream.</li><li><strong>Layer 2, kernel still in the way:</strong> reading and writing raw Ethernet frames over a virtual network device, managing ARP, ICMP, TCP, and UDP yourself.</li><li><strong>Layer 2, full kernel bypass:</strong> talking to the network card directly from userspace, with no syscalls anywhere near the hot path.</li></ul><p>Same HTTP server, three different distances from the metal. That’s the whole point: to see, in CPU time and in flamegraphs, exactly what the kernel was doing for you all along, and what it costs (or saves) to do it yourself.</p><p>TL;DR: If you only want numbers and flamegraph takeaways, skip to <a href="https://medium.com/p/780366f827b3#9535">Benchmarks &amp; AWS results</a> at the bottom.</p><h3>What’s being implemented, and why each piece exists</h3><p>Once you go below the socket API, you have to build everything that a socket normally gives you for free. Here’s what that means, piece by piece.</p><p><strong>ARP (Address Resolution Protocol):</strong> before you can send anyone an IP packet, you need their MAC address, because Ethernet doesn’t know what an IP address is. ARP is the <strong>“who has this IP, tell me your MAC”</strong> broadcast. The server keeps an ARP cache so it doesn’t have to ask every single packet.</p><p><strong>ICMP (Internet Control Message Protocol):</strong> this is the protocol behind <strong>ping </strong>, and also the protocol routers use to tell you <strong>“I can’t reach that”</strong> or <strong>“that’s too big, fragment it.”</strong> It is not a transport protocol carrying application data, it’s diagnostics and control riding directly on top of IP.</p><p><strong>TCP (Transmission Control Protocol):</strong> the reliable, ordered, connection oriented protocol that HTTP normally rides on. This is the expensive one to implement correctly: sequence numbers, acknowledgements, retransmission, and a connection lifecycle that has to be tracked by hand instead of getting it for free from the kernel.</p><p>Every TCP connection starts with the <strong>three way handshake</strong>, and it’s worth drawing out because almost every weird latency number later in this post traces back to it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*v8brtjKiH2mc2u0bkCS7ZQ.png" /><figcaption>TCP — Three Way Handshake</figcaption></figure><p>Three packets, minimum one full round trip, before a single byte of your HTTP request has even been sent. That round trip is <strong>“free”</strong> when the kernel does it because it overlaps with everything else the kernel is doing, but when you’re managing TCP yourself,<strong> you feel every microsecond of it.</strong></p><p><strong>UDP (User Datagram Protocol):</strong> the opposite philosophy from TCP. No handshake, no acknowledgement, no ordering guarantee, just fire a packet at an address and hope. It exists in this project specifically to measure raw packets per second without the overhead of connection state, which is a very different kind of load than HTTP traffic.</p><p><strong>HTTP with keep alive:</strong> HTTP itself is just text on top of whatever transport carries it (TCP here), but the interesting part is the connection lifecycle the server has to track per request. Every connection moves through a small state machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*KSvk9qVnzrJLfpr5PpmY3g.png" /><figcaption>HTTP State Machine</figcaption></figure><p>That state machine, step by step:</p><ul><li><strong>READING_HEADERS:</strong> the connection is waiting for the HTTP headers to fully stream in.</li><li><strong>READING_BODY: </strong>once the headers arrive, the `Content-Length` header tells us how big the body is, so we buffer until it’s fully received.</li><li><strong>PROCESSING:</strong> with headers and body in hand, the request is routed to its endpoint handler, which processes it and returns a result.</li><li><strong>WRITING:</strong> that result gets wrapped in an HTTP response and sent back on the connection.</li><li><strong>Back to READING_HEADERS, or CLOSED:</strong> on keep-alive, loop back to `READING_HEADERS` for the next request; otherwise, close the connection.</li></ul><p>Without keep alive, every single request pays for a fresh three way handshake and then a teardown right after. With keep alive, the state machine just loops back to <strong>READING_HEADERS</strong> on the same TCP connection instead of closing it, so the handshake cost is paid once and amortized across every request that connection ever serves. This sounds like a small detail. It is, in practice, one of the biggest levers in the whole project.</p><h3>What’s actually being tested</h3><p>Two very different kinds of load get thrown at each implementation, and they stress completely different things.</p><p>The first is concurrency, using <strong>wrk</strong> at increasing connection counts (<strong>c=200</strong>, <strong>c=1000</strong>). This tests how well the server handles many simultaneous TCP connections, each going through the full HTTP state machine, each one a slightly different combination of <strong>“reading”</strong>, <strong>“processing”</strong>, and <strong>“writing”</strong> at any given instant. This is the realistic web traffic shape.</p><p>The second is raw <strong>packets per second</strong>, using a <strong>UDP flood</strong>. There’s no connection state, no handshake, no HTTP parsing, just <strong>“how many discrete packets can this thing receive and process per second before something falls over.” </strong>This isolates the I/O path itself: NIC, driver, and the hand rolled protocol headers, with almost none of the application logic concurrency brings.</p><p>Together they answer two different questions. Concurrency answers <strong>“how does this scale as users pile on.”</strong> Raw pps answers <strong>“how fast can this thing move packets, full stop.”</strong></p><h3>The stack is boring, the implementation is where it gets fun</h3><p>Here’s an honest admission: <strong>ARP</strong>, <strong>ICMP</strong>, <strong>TCP</strong>, <strong>UDP</strong>, and <strong>HTTP</strong> are not exciting once you know what they do. They’re well documented, decades old, and every networking textbook draws the same diagrams. The actual engineering, the part that’s fun to dig into, is in how each version of this server talks to the outside world to make those protocols happen at all. That’s what the rest of this post is about.</p><h3>Implementation A: plain sockets and epoll</h3><p>This is the version most of us have written some flavor of before: a TCP server built entirely on top of the kernel’s socket API.</p><h4>The basic API, quickly</h4><ul><li><strong>socket(AF_INET, SOCK_STREAM, 0):</strong> asks the kernel for a TCP socket.</li><li><strong>bind():</strong> claims a port.</li><li><strong>listen(): </strong>tells the kernel “start queueing incoming connections for me, up to this backlog size.”</li><li><strong>accept():</strong> pulls a fully formed, already handshaken connection off that queue and hands you back a new file descriptor representing that one client. What makes each accepted connection unique is the file descriptor plus the 4-tuple of <strong>(server IP, server port, client IP, client port).</strong></li><li><strong>read() / write()</strong>: (or their socket specific cousins) move bytes in and out on that per client file descriptor.</li></ul><p>Every one of these calls is a kernel boundary crossing, but the actual TCP state machine, sequence numbers, retransmissions, all of it, is the kernel’s problem from this point on, not yours.</p><h4>epoll, properly explained</h4><p>Most of what a server does is I/O bound, not CPU bound: it spends the overwhelming majority of its time blocked on <strong>read()</strong>, waiting for bytes that haven’t arrived yet, with the CPU sitting completely idle for that connection in the meantime.</p><p>The naive way to serve many clients is one thread per connection, each one blocked on its own <strong>read()</strong>. That falls over fast once you have thousands of connections, because thousands of blocked threads are thousands of stacks and thousands of context switches, even though every one of those threads is doing nothing but waiting.</p><p><strong>epoll</strong> solves this by allowing a single thread to monitor thousands of file descriptors simultaneously, waking it only when one or more become ready for I/O. Instead of dedicating an idle thread to every idle connection, a small number of threads can efficiently manage a massive number of concurrent connections.</p><p>The application registers its interest in a set of file descriptors with epoll and then continues processing other work instead of blocking on any single connection. When data arrives or a socket becomes ready for reading or writing, epoll notifies the thread, which handles the event and moves on. While one connection is idle, the same thread remains free to service any other connection that is ready, maximizing CPU utilization and minimizing wasted resources.</p><p>The mechanics, concretely, from the <strong>tcp_server()</strong> loop: you create an epoll instance with <strong>epoll_create1()</strong>, register file descriptors you care about with <strong>epoll_ctl(EPOLL_CTL_ADD, …)</strong>, specifying which events you want (<strong>EPOLLIN</strong> for data arrived<strong>,</strong> <strong>EPOLLOUT </strong>for you can write now), and then you call <strong>epoll_wait(epfd, events, max_events, timeout)</strong>in a loop.</p><pre>server_fd = socket()<br>bind(server_fd, ...)<br>listen(server_fd, ...)<br><br>epfd = epoll_create1()<br><br>// Register the listening socket to receive notifications for new connections<br>epoll_ctl(epfd, EPOLL_CTL_ADD, server_fd, EPOLLIN)<br><br>while (true) {<br>    // Sleep until one or more file descriptors become ready<br>    n = epoll_wait(epfd, events, MAX_EVENTS, -1)<br><br>    for event in events[0..n]:<br><br>        if event.fd == server_fd:<br>            while (accept() succeeds):<br>                // Register the new client socket for read events<br>                epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, EPOLLIN)<br><br>        else if event.readable:<br>            read into connection buffer<br><br>            if request complete:<br>                prepare response<br><br>                // Change interest from read events to write events<br>                epoll_ctl(epfd, EPOLL_CTL_MOD, client_fd, EPOLLOUT)<br><br>        else if event.writable:<br>            write response bytes<br><br>            if response fully sent:<br>                if keep-alive:<br>                    // Switch back to waiting for the next request<br>                    epoll_ctl(epfd, EPOLL_CTL_MOD, client_fd, EPOLLIN)<br>                else:<br>                    close(client_fd)</pre><p>Here’s the part worth sitting with: when this server calls <strong>epoll_wait(epfd, events, 4096, -1)</strong> with a timeout of <strong>-1</strong>, the thread does not spin. It does not poll. It is suspended by the kernel scheduler entirely, consuming zero CPU, until at least one watched file descriptor has something to report. The CPU is, quite literally, asleep.</p><p>You can see this for yourself: run the server binary under <strong>gdb</strong>, single step with <strong>next</strong> until you land on the <strong>epoll_wait</strong> call, and step into it. The process refuses to go to the next instruction until a new request actually arrives, it just sits parked inside the syscall, unmoving, for as long as no client sends anything.</p><p>The moment a packet for one of the watched sockets arrives, the kernel wakes the thread, <strong>epoll_wait</strong> returns with the list of ready file descriptors, and only then does any of your code run again. This is the entire reason epoll based servers can hold open tens of thousands of idle connections for free: idle connections cost nothing while they’re idle.</p><h4>The request path</h4><p>Tracing one request through this implementation, end to end:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/1*rRuS91YScdebJxkSoPOnwQ.png" /><figcaption>Socket Request Path</figcaption></figure><p>A packet lands on the NIC, the kernel’s own network driver and TCP/IP stack absorb it completely (handshake, ACKs, reassembly, all invisible to this program), and only once there’s actual application data ready does <strong>epoll_wait</strong> wake up and hand this program a readable file descriptor.</p><p>The program reads into a per connection buffer, runs it through the <strong>READING_HEADERS</strong> to <strong>WRITING</strong> state machine described earlier, and once a response is built, flips that connection’s epoll registration to <strong>EPOLLOUT</strong> so the next wakeup is <strong>“you can write now.”</strong></p><p>After the write completes, if the connection is keep alive, the state resets to <strong>READING_HEADERS</strong> on the same fd and the cycle repeats with no new handshake. The kernel handles getting bytes onto the actual wire; this program only ever sees buffers.</p><h4>Flamegraph: concurrency at c=1000</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vUHsg6LlehXzaGqLdjg9XQ.png" /><figcaption>L4 flamegraph, c=1000</figcaption></figure><p><strong>pump_connection</strong> dominates the profile (roughly 60 to 70 percent depending on the run), which makes sense, it’s the function doing the header parsing, body buffering, routing, and response building all in one place.</p><p>Underneath it, <strong>parse_request</strong> accounts for a large chunk on its own, somewhere around a quarter of total time, which is the cost of re-parsing HTTP headers as text on every request.</p><p><strong>__libc_write</strong>, which drops into the kernel’s <strong>tcp_sendmsg</strong>, shows up at roughly 13 to 16 percent, that’s the real cost of handing bytes to the kernel for transmission.</p><p><strong>epoll_wait</strong> itself barely registers, a few percent at most, because at this concurrency the thread is almost always busy servicing ready connections rather than sleeping.</p><h4>Flamegraph: raw UDP</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p-iimu_maZ8OQSLsdcslmw.png" /><figcaption>L4 flamegraph, UDP</figcaption></figure><p>This one looks completely different, and it’s a great illustration of how load shape changes where time goes. <strong>__libc_recvfrom</strong> accounts for around half of total time here, and <strong>epoll_wait</strong> accounts for nearly as much, roughly 48 percent.</p><p>There’s no HTTP parsing, no routing, no response building to speak of, so the entire profile collapses into <strong>“wait for a packet”</strong> and <strong>“read the packet.”</strong></p><p>On a local network, the client (laptop) simply cannot push UDP packets fast enough to keep this server (home server) constantly busy, so it spends a large fraction of its time legitimately asleep in <strong>epoll_wait</strong> waking up, draining a few packets, and going back to sleep.</p><h3>Implementation B: raw Ethernet frames over TAP</h3><p>The socket API is gone here. Where <strong>implementation A</strong> only ever operated at layer 4 (TCP) and above, this one strip away two layers of kernel abstraction at once,<strong> layer 4 (transport)</strong> and <strong>layer 3 (network/IP)</strong>, and drops down to layer 2, reading and writing raw Ethernet frames directly. Everything the kernel’s TCP/IP stack used to do above that layer now has to be implemented by hand.</p><h4>What’s extra here</h4><p>On top of what <strong>implementation A</strong> needed, this version now manages ARP, ICMP, a TCP state machine, UDP, and the same HTTP layer sitting on top. What matters here is that this program is now responsible for parsing a raw frame’s destination MAC, deciding it’s an IPv4 packet, deciding it’s a TCP segment, and walking it all the way up to <strong>“is this a complete HTTP request”</strong> entirely in userspace code that this project owns.</p><h4>TAP devices, properly explained</h4><p>A TAP device is a V<strong>irtual Ethernet Interface</strong> that exposes raw Ethernet frames to a userspace program through a file descriptor. Reading from the file descriptor provides incoming Ethernet frames, while writing to it injects outgoing Ethernet frames, which the kernel can then forward through the underlying physical network interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*MANu7qxHHHZG5GSL3BPPTA.png" /><figcaption>TAP Setup</figcaption></figure><p>This is the exact mechanism that makes VM networking work. When you run a virtual machine, the hypervisor typically creates a TAP device on the host, bridges it to the host’s real network interface (or routes it), and the guest OS believes it has a normal network card, when in reality every frame the guest sends is just bytes being written into a TAP fd that the hypervisor process reads on the host side. Tools like QEMU, libvirt, and most container networking setups under the hood all lean on this same primitive. The sibling device, TUN, does the same thing one layer up, at IP packets instead of full Ethernet frames.</p><p>Concretely, the TAP device just hands us a different kind of file descriptor to read and write. Instead of a clean TCP byte stream like implementation one got from <strong>read()/write() </strong>on a socket, every <strong>read()</strong> here hands back one full raw Ethernet frame at a time, and it’s this program’s job to make sense of it from there.</p><h4>Paired with epoll again</h4><p>The TAP device’s file descriptor is just a <strong>regular fd</strong>, so it can be registered with <strong>epoll</strong> exactly the same way a client <strong>socket fd</strong> would be in Implementation A. No new explanation needed here, it’s the same <strong>epoll_wait</strong> sleep-until-event mechanism, just watching a virtual network interface instead of a client connection.</p><pre>while (true) {<br>    // sleeps until the TAP fd is readable<br>    n = epoll_wait(epfd, events, MAX_EVENTS, -1)   <br>    for event in events[0..n]:<br>        if event.fd == tap_fd:<br>            // one full raw Ethernet frame<br>            frame = read(tap_fd) <br>            // ARP -&gt; ARPHandler, IPv4 -&gt; IPHandler -&gt; TCP/UDP -&gt; HTTP         <br>            dispatch_by_ethertype(frame)  <br>}</pre><h4>The request path</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*3bE045wGoNszNfIrbsGoqQ.png" /></figure><p>Compared to <strong>Implementation A</strong>, the starting point changes. Instead of the kernel giving the application a ready-to-read socket, the program receives a raw Ethernet frame. It then determines whether the frame is ARP (<strong>0x0806</strong>) or IPv4 (<strong>0x0800</strong>), processes the appropriate transport protocol (TCP or UDP), and finally passes the payload to the same HTTP state machine used in Implementation A. The HTTP logic remains unchanged; the difference is that this implementation handles all the lower networking layers itself.</p><h4>Flamegraph: concurrency at c=1000</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ArzCyVWIYMaQDSnHxAo9tw.png" /><figcaption>L2 flamegraph, c=1000</figcaption></figure><p>This profile shows us that TAP is not spending much time in the HTTP layer at all. <strong>epoll_wait</strong> is about 45 percent of the profile, while <strong>pump_connection</strong> is under 2 percent and <strong>parse_request</strong> is under 1 percent. Most of the time is either waiting for the next frame to arrive on the <strong>TAP fd</strong>, or doing the userspace TCP bookkeeping needed before that frame can become bytes for the shared HTTP code.</p><h4>Flamegraph: UDP flood</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Or2u8Wol3Vy410cMumecVQ.png" /><figcaption>L2 flamegraph, UDP</figcaption></figure><p>As with <strong>implementation A</strong>’s UDP profile, raw packet load with no HTTP semantics collapses the profile down toward the I/O path, except this I/O path is now <strong>“read a frame off the TAP fd and walk it through hand rolled Ethernet and IP parsing”</strong> rather than <strong>“ask the kernel’s UDP stack for a datagram.”</strong></p><h3>Implementation C: DPDK, kernel bypass</h3><p>The kernel is no longer involved in moving packets at all.</p><p><strong>DPDK (the Data Plane Development Kit)</strong> is a set of userspace libraries and drivers that let a program talk directly to a network card, <strong>bypassing the kernel’s networking stack entirely</strong>.</p><p>Normally, a packet arriving on a NIC triggers an interrupt, the kernel driver handles it, copies it into kernel memory, and eventually a syscall like <strong>read()</strong> copies it again into your program’s memory. DPDK removes essentially all of that: the NIC’s driver runs in userspace (a <strong>“PMD”,</strong> poll mode driver), DMA’s packets directly into preallocated userspace memory, and your program polls for new packets in a tight loop instead of waiting for an interrupt.</p><p>This is the technology behind most high throughput network appliances: virtual switches, NFV (network function virtualization) platforms, load balancers, and a lot of high frequency trading infrastructure, anywhere the cost of a syscall and a kernel copy is large enough relative to the work being done that it’s worth giving up the kernel’s conveniences entirely.</p><h4>Setup: single core today, multi core would be better</h4><p>In this implementation, everything happens on one core: a single <strong>while(true)</strong> loop calls <strong>rte_eth_rx_burst </strong>to pull packets off the NIC, runs every one of them through the full Ethernet/ARP/IP/TCP/HTTP stack synchronously, and calls <strong>rte_eth_tx_burst</strong> to send responses, all on the same call stack, all on the same core. This works, but it means receiving packets, processing them, and transmitting responses are all competing for the same CPU core’s cycles.</p><pre>rte_eal_init(argc, argv)<br><br>// Reserve hugepage memory and initialize DPDK subsystems<br>configure hugepages<br>create mbuf pool<br><br>// Configure the NIC<br>rte_eth_dev_configure(port)<br>setup RX queue<br>setup TX queue<br>rte_eth_dev_start(port)<br><br>while (true) {<br>    // Poll the NIC (no interrupts)<br>    n = rte_eth_rx_burst(port, queue, pkts, BURST_SIZE)<br><br>    for pkt in pkts[0..n]:<br>        // Ethernet -&gt; ARP/IP/ICMP -&gt; TCP/UDP -&gt; HTTP<br>        handle_packet(pkt)<br><br>    // Transmit any packets generated while processing<br>    rte_eth_tx_burst(port, queue, tx_pkts, tx_count)<br>}</pre><p>The better design, an <strong>“ideal pipeline”</strong> would split this across at least three cores: one core dedicated purely to RX, busy polling <strong>rte_eth_rx_burst</strong> and nothing else; one or more application cores doing the actual protocol and HTTP work; and one core dedicated purely to TX, busy polling <strong>rte_eth_tx_burst</strong>. Each stage would hand work to the next over a lockless ring buffer rather than a shared function call stack, so a slow request being processed never stalls the core that’s draining the NIC’s RX queue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKHak0_LQpU05mck45l4VA.jpeg" /><figcaption>Ideal DPDK Pipeline</figcaption></figure><h4>The deeper technicalities</h4><p>A few specific pieces make DPDK’s whole model possible.</p><ul><li><strong>The memory pool:</strong> before any packets can be received, a large pool of fixed size packet buffers gets preallocated up front, typically backed by hugepages . This is the foundational trick: no allocation happens on the packet receive path, ever. A buffer is grabbed from the pool, filled by DMA from the NIC, handed through the stack, and returned to the pool, with the pool itself never growing or shrinking at runtime. That’s what it saves you: no malloc/free churn, no allocator lock contention, no unpredictable pauses under load, just constant time buffer reuse no matter how much traffic is flowing.</li><li><strong>Fixed size headers:</strong> because every packet’s buffer comes from that same pool with the same fixed size and layout, parsing Ethernet, IP, and TCP headers becomes a matter of fixed offset arithmetic rather than variable length, bounds checked, allocation heavy parsing. That’s what it saves you: the framing cost is paid once, by the pool, instead of being repeated on every protocol layer for every packet, which is part of why application logic on top of DPDK ends up so cheap once packets are actually arriving.</li><li><strong>Burst processing:</strong> rather than asking for one packet at a time, the NIC is polled for a whole batch of packets in a single call, and the receive/transmit rings are sized generously so the NIC has room to land bursts before the program has to drain them. That’s what it saves you: the fixed cost of touching the NIC’s ring gets amortized across many packets at once instead of being paid per packet.</li><li><strong>rte_ring:</strong> this is DPDK’s lockless, single producer single consumer (or multi producer/consumer, configurable) circular buffer, living in the same hugepage backed memory as everything else, that two different cores can push and pop from without taking a <strong>lock</strong> or making a <strong>syscall</strong>. This is the piece the <strong>“ideal pipeline”</strong> design above depends on: an RX core pushing parsed packets into an inbound <strong>rte_ring</strong>, and an app core popping them off, with no kernel pipe, no mutex, nothing but atomic pointer arithmetic between them.</li></ul><h4>The request path</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/1*VzXw7AtlygWxzFUUm8slJA.png" /><figcaption>DPDK Request Path</figcaption></figure><p>There is no kernel anywhere in this diagram. The core never sleeps in an interrupt handler, never crosses into kernel mode for I/O, never copies a packet through a kernel buffer. It polls the NIC directly, gets a burst of raw frames back, and runs every single one through the exact same Ethernet to HTTP pipeline conceptually described in implementation two, except now the <strong>“NIC”</strong> at the start of the diagram is a real physical card and the <strong>“frame”</strong> handed to the program came straight off it via DMA.</p><h4>Flamegraph: concurrency at c=1000</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iKc5_FtquKsLvmZJMNzlYA.png" /><figcaption>DPDK flamegraph, local c=1000</figcaption></figure><p>Locally, this profile is dominated by the NIC driver: <strong>rtl_recv_pkts</strong>, the Realtek RTL8169 driver’s packet drain routine, alone accounts for over half of total CPU time (around 57 percent, most of which is the CPU just waiting for packets), with the poll loop on top of that at around a quarter. The application’s own TCP and HTTP logic barely register, around 2 percent. The bottleneck here is the cheap consumer NIC and the modest local client traffic, not the application.</p><h4>Flamegraph: UDP flood</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bdqKKmx4NNKV4ohI_rn-7A.png" /><figcaption>DPDK flamegraph, local UDP</figcaption></figure><p>Same story, intensified: this local setup only manages to push around 72,000 packets per second, nowhere near enough to keep the CPU busy, so almost all of the measured time sits inside the RTL8169 driver’s poll loop just waiting for the next packet, and the application’s own protocol parsing is a rounding error, well under one percent.</p><h3>Local results: wait, why do these all look so similar</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*FfhROucbhDPsW5cuD39lIQ.png" /><figcaption>Local Rig Results</figcaption></figure><p>Here’s the thing that should genuinely surprise you the first time you line these three implementations up locally. Plain sockets, hand rolled TCP over a TAP device, and full DPDK kernel bypass all land in roughly the same throughput ballpark on this hardware: somewhere in the 8,000 to 15,000 requests per second range for standard HTTP benchmarks, depending on the exact run.</p><p>Wait, what? You’d expect kernel bypass to crush plain sockets. It doesn’t, here, and the flamegraphs already told you why: in every single local profile, the bottleneck is not the implementation, <strong>it’s the test rig</strong>.</p><p>A cheap consumer NIC and a laptop generating load over Wi-Fi simply cannot produce enough packets per second to make any of these three implementations sweat. DPDK’s local flamegraph spending 57 percent of its time inside the RTL8169 driver isn’t DPDK being slow, it’s DPDK waiting, extremely efficiently, for a NIC that can’t keep up. All three implementations were, client and hardware bound rather than implementation bound, which is exactly why they all look so similar.</p><h3>Changing the setup: into an AWS VPC</h3><p>To actually separate these implementations by their own merits, the test setup moved into an AWS VPC with two purpose built instances. The server under test ran on a <strong>c6i instance</strong>, a compute optimized type with a modern <strong>Elastic Network Adapter (ENA) NIC</strong> capable of genuinely high packet rates, chosen specifically so the network interface itself would stop being the limiting factor.</p><p>The load was generated from a separate <strong>c7n instance</strong>, a network optimized type, chosen specifically so the load generator itself could never become the bottleneck and silently cap what the server was capable of receiving, the exact same mistake a home Wi-Fi laptop made in the local setup. Putting both instances in the same VPC, same availability zone, also removed the unpredictable, jittery path that real internet or home network routing would otherwise add.</p><h4>TAP got dropped here, and that’s worth being honest about</h4><p>The TAP based implementation does not appear anywhere in the AWS comparisons, and the reason is unglamorous: AWS’s virtualized networking, built on the <strong>Nitro hypervisor</strong> and <strong>ENA</strong>, does not give you the kind of straightforward host bridge that makes spinning up a TAP device and wiring it to real traffic simple.</p><p>After enough fighting with AWS’s networking model, with no profile output to show for it, that path was sadly abandoned in favor of just comparing the two implementations that did work cleanly on AWS: plain kernel sockets and DPDK.</p><p>I just got tired boss.</p><h4>AWS results: DPDK versus epoll sockets, for real this time</h4><p>With the load generator no longer the bottleneck, the gap between implementations finally shows up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*OFdc2ZyebWogOqPFURtOYg.png" /><figcaption>HTTP Results — AWS Setup</figcaption></figure><p>At concurrency 1000, the plain socket implementation pushed around 54,000 requests per second on AWS, with a p50 latency around 12 to 16 milliseconds and a p99 that ranged from a reasonable 35 milliseconds up to occasional spikes past a full second.</p><p>DPDK, on the same hardware, pushed roughly 120,000 requests per second, more than double, with a p50 around 6.6 to 8.4 milliseconds and, strikingly, a p99 that stayed in the same 6.8 to 8.7 millisecond range as its own p50.</p><p>That last part is the headline.<strong> DPDK’s tail latency is almost perfectly flat:</strong> at concurrency 1000, <strong>p99 sits within roughly 15 percent of p50</strong>. The plain socket implementation’s tail, by comparison, can stretch to ten times its own median or worse under the same load.</p><p>The reason traces directly back to what each implementation depends on to make that progress. The socket based server depends on the kernel scheduler waking its thread out of <strong>epoll_wait</strong>, depends on the kernel’s own TCP stack doing acknowledgement and retransmission bookkeeping, and depends on <strong>write()</strong> eventually handing off to <strong>tcp_sendmsg</strong> inside the kernel, every one of which is a place where the <strong>kernel can insert a few extra milliseconds of scheduling delay under load</strong>, especially as thousands of connections compete for the same kernel data structures.</p><p>None of that exists in the DPDK implementation. There’s no epoll wakeup to be delayed, no kernel scheduler involved at all on the data path, just a tight userspace loop continuously polling the NIC and continuously doing the same fixed amount of per packet work. A saturated, steady userspace pipeline doesn’t have anywhere for a multi millisecond gap to hide. That flatness is what kernel bypass actually buys you: not just raw throughput, but predictability.</p><h4>Why packets per second capped out around one million</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*0uF9M1x6D9Pb1fIgauXV5w.png" /><figcaption>UDP Results — AWS Setup</figcaption></figure><p>On the raw UDP flood test, the plain socket server received around 700,000 packets per second on AWS before kernel level loss set in (<strong>recvfrom</strong> alone consuming 92 percent of CPU time, with the kernel’s own UDP stack adding another 32 percent on top, well over 100 percent because softirq driver time overlaps in profiling). DPDK, on the same hardware, received right around one million packets per second and essentially plateaued there.</p><p>That plateau is not this program running out of steam. The client was actually sending faster, around 1.4 million packets per second, and the loss rate stayed essentially identical whether the server was being asked to handle 1.4 million or artificially throttled lower, which is the signature of a hard external limit rather than an application bottleneck.</p><p>That limit comes from AWS, not the application. AWS’s network virtualization layer enforces a <strong>maximum packets-per-second (PPS) limit for each EC2 instance type</strong>. For the instance used here, that limit is around one million packets per second, so the application cannot receive packets any faster regardless of how efficient the userspace code is.</p><p>Kernel bypass bought <strong>pps</strong> right up to the point where AWS’s own infrastructure, not this code, said no further.</p><h3>The flamegraph bottleneck flips upside down</h3><p>This is the single most satisfying result I had in this whole journey, and it’s a direct inversion of the local story.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iKc5_FtquKsLvmZJMNzlYA.png" /><figcaption>DPDK flamegraph, local c=1000</figcaption></figure><p>Locally, DPDK’s flamegraph showed the RTL8169 driver eating 55 to 57 percent of CPU time, with the application’s own TCP and HTTP handling sitting at a mere 2 percent. The bottleneck was the cheap NIC and the underpowered local client; the application was basically idle, waiting on packets that weren’t arriving fast enough.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uuJoEYK2peEz5siCu904Kw.png" /><figcaption>DPDK flamegraph, AWS c=1000</figcaption></figure><p>On AWS, you can literally see the application code expand significantly in the flame graph, indicating that most CPU cycles are now spent executing the application rather than waiting on the networking stack.</p><p>The ENA driver now accounts for only around 8 to 9 percent of CPU time in the HTTP benchmark, while this project’s own TCP handler, specifically <strong>TCPHandler::handle_packet</strong>, jumps to roughly 50 percent, and the deque management and segment transmission logic inside the hand rolled TCP stack accounts for around 47 percent on top of that.</p><p>The NIC is no longer the limiting factor at all, it’s feeding packets faster than this program’s own TCP stack can acknowledge and transmit responses for them. <strong>The bottleneck moved from the hardware all the way up into application code,</strong> which is exactly the inversion you’d hope to see once you give kernel bypass a NIC fast enough to actually expose it.</p><p>Put simply: locally, every implementation here was waiting on hardware that couldn’t keep up. On AWS, with a NIC capable of actually saturating the path, DPDK’s bottleneck moved to exactly where it should, into the one piece of this whole project that’s entirely homegrown:<strong> My handwritten TCP Stack</strong>.</p><h3>Closing thought</h3><p>None of this makes Kernel/epoll or DPDK the<strong> “better” </strong>approach in some absolute sense. They’re built for different load shapes, and picking one over the other is really a bet on what kind of traffic you expect.</p><p><strong>Kernel/epoll</strong> is the right call when load is<strong> idle or bursty</strong>, which describes most day to day application servers: long stretches of near silence punctuated by real requests. That’s exactly the shape single threaded systems like<strong> Redis</strong> are built around too, one thread doing real work only when there’s real work to do, asleep the rest of the time. Continuous polling in that world would just burn CPU cycles for no reason.</p><p><strong>DPDK</strong> earns its complexity at the other extreme: sustained, high, continuous load, where the NIC is essentially never idle and every microsecond of syscall and kernel copy overhead adds up across millions of packets a second. That’s a much narrower set of problems, but for that set of problems it’s a categorically different tool, not just a faster version of the same one.</p><p>The logical stack is identical in all three implementations. As you move closer to the hardware, each layer gets faster and the bottleneck doesn’t disappear, it simply moves. Eventually, until the most honest bottleneck remains: <strong>your own code.</strong></p><h3>Read more</h3><ul><li><a href="https://copyconstruct.medium.com/the-method-to-epolls-madness-d9d2d6378642">The method to epoll’s madness</a>: An interesting walk through of how epoll works and its api’s.</li><li><a href="https://embedx.medium.com/introduction-to-dpdk-architecture-and-principles-40db9a61a6f5">Introduction to DPDK: Architecture and Principles</a>: How DPDK bypasses the Linux network stack with poll mode drivers, hugepages, and userspace packet processing from NIC to application.</li><li><a href="https://medium.com/@aytul.shad2002/the-secret-weapon-why-aws-replaced-xen-with-nitro-to-achieve-near-bare-metal-cloud-performance-213a23a4fc05">The Secret Weapon: Why AWS Replaced Xen with Nitro</a>: Why AWS moved I/O off the Xen Dom0 bottleneck onto dedicated Nitro hardware, and what that buys you in throughput and predictable performance</li><li><a href="https://github.com/AlterHoodie/poor_mans_http_server">poor_mans_http_server</a>: The full source for all three implementations, benchmarks, flamegraphs, and table generation scripts.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=780366f827b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Built an LLM Inference Engine on a 15 year old GPU and the Math was the Easy Part]]></title>
            <link>https://medium.com/@manishimmi2k3/i-built-an-llm-inference-engine-on-a-15-year-old-gpu-and-the-math-was-the-easy-part-592f06c6cd28?source=rss-db29cf523fed------2</link>
            <guid isPermaLink="false">https://medium.com/p/592f06c6cd28</guid>
            <category><![CDATA[cpp]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[llama-cpp]]></category>
            <dc:creator><![CDATA[Manish Immadisetty]]></dc:creator>
            <pubDate>Wed, 10 Jun 2026 18:58:37 GMT</pubDate>
            <atom:updated>2026-06-11T17:49:19.643Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hF2b2O9tjhIuBj2iALcgHA.png" /></figure><p>There’s a fun lie that gets talked about <strong>AI engineering</strong>, that the hard part is the machine learning. The math. The architecture. The loss curves.</p><p>It isn’t. The hard part is making it <em>fast</em>. And fast is a systems problem.</p><p>I spent a few weeks building an LLM inference engine from scratch, no PyTorch, no llama.cpp, no cuBLAS. Just good old C++, a bit of CUDA, an i3–7100 running at 3.9GHz, and a GTX 750 Ti with 2GB of VRAM. What I ended up learning had almost nothing to do with transformers. It had everything to do with memory, cache lines, and why your GPU is doing a lot of nothing most of the time.</p><p>Before we get into any of that, here’s the entire operation surface of the engine, every mathematical primitive the transformer calls:</p><pre>{<br>    matmul,<br>    add,<br>    mul,<br>    rms_norm,<br>    silu,<br>    rope,<br>    scale,<br>    dot,<br>    softmax,<br>    argmax<br>}</pre><p>That’s it. Ten functions.</p><p>A 1.1 billion parameter language model, in its entirety. The intelligence is in the weights, hundreds of megabytes of learned numbers nobody fully understands. The operations are textbook. The interesting work is making it fast enough that users don’t end up doomscrolling Instagram between requests (and let’s be honest, we’ve all done it).</p><h4><strong>What You Should Know Going In</strong></h4><p>This isn’t an intro to transformers. I’ll assume you know roughly what a decoder-only LLM is: a stack of layers, each with an attention block and an MLP, that takes in a token sequence and predicts the next token.</p><p>If you want a solid primer, Andrej Karpathy’s <a href="https://www.youtube.com/watch?v=kCc8FmEb1nY">Let’s build GPT</a> is the best one out there.</p><p>The rest you’ll pick up as we go.</p><h3>First, You Need to Open the Model</h3><p>Before any math happens, you need to get the weights off disk and into memory. And how you do that is dictated entirely by how the model’s laid out on disk in the first place.</p><p>There are many formats a model can ship in, safetensors, AWQ, GPTQ, GGUF, and others. Each with its own trade-offs around quantization, tooling, and how the bytes are arranged. Throughout this project we work with a <strong>GGUF TinyLlama 1.1B model</strong>.</p><p><strong>GGUF</strong> format, a binary file format created by Georgi Gerganov for the llama.cpp project. To understand why it’s designed the way it is, it helps to look at what’s actually inside one.</p><p>A GGUF file has three regions, laid out sequentially on disk:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eWrC2ADH8HMgf-yn5_teLg.png" /><figcaption>The Official GGUF Specification can be found <a href="https://github.com/ggml-org/ggml/blob/master/docs/gguf.md">here</a></figcaption></figure><p>The <strong>file header</strong> is just a sanity check: a 4-byte magic number GGUF and a version number. If those don&#39;t match what you expect, you throw exception.</p><p>The <strong>key/value metadata </strong>section (marked as green) is where the model’s metadata lives. Every hyperparameter the engine needs is stored here as a named key: llama.context_length tells you the context length (4096 for TinyLlama 1.1B), tokenizer.ggml.model gives you the name of the tokenizer used, llama.block_count gives you the number of layers (22). You read these once at startup and use them to construct the model.</p><p>The <strong>tensor metadata</strong> section (marked as purple) is a table, one entry per weight matrix in the model. Each entry gives us metadata about each tensor parameter weight: here is a tensor named blk.0.ffn_gate.weight, it has shape [4096, 32000], it&#39;s stored in Q2_K format, and its raw bytes start at byte offset X in the file.</p><p>The <strong>raw tensor data</strong> at the bottom is just packed bytes. No framing, no separators. The offsets from the metadata table are the only way to find where one tensor ends and the next begins.</p><p>Why this format over something like <strong>HDF5</strong> or <strong>Parquet</strong>? The key reason is that GGUF is designed to be mmap-ed. Instead of reading 700MB into RAM with read(), you call mmap() and the OS maps the file into your virtual address space.<strong> The tensor data is never actually copied</strong>, the OS pages it in from disk on demand, and the kernel&#39;s page cache handles everything. <strong>Each weight matrix is just a pointer into that mapping</strong>.</p><h4>Tensor</h4><p>Once the file is parsed, every weight becomes a tensor, a lightweight struct that describes where data lives and how it&#39;s shaped:</p><pre>struct tensor {<br>    // pointer into the mmap-ed file (or GPU memory later)<br>    float* data;   <br>    uint32_t n_dimensions;<br><br>    // shape: e.g. {2048,2048} for a square weight<br>    vector&lt;uint64_t&gt; dimensions;  <br>    // how many elements to skip per step in each dim<br>    vector&lt;uint64_t&gt; strides;     <br>};</pre><p>The dimensions field is the shape you’d recognize from NumPy: a 2048×2048 matrix has dimensions = {2048,2048}.</p><p>The strides field records how far apart consecutive elements sit in memory along each axis, a recipe for turning a logical (row, col) position into a flat array index. The payoff is that operations like transposes, slices, and broadcasts become free: instead of copying and reshuffling the data, you change the strides and the same underlying bytes get reinterpreted in a new layout.</p><p>At startup, the engine walks that tensor metadata table and, for each entry, builds one of these tensor structs: it reads the <strong>shape </strong>and <strong>dtype </strong>from the metadata and points data at the entry’s byte offset inside the <strong>mmap region</strong>. Each struct is then wired into the model by name, blk.0.attn_q.weight becomes layer 0’s query projection, blk.0.attn_k.weight its key projection, and so on for all 22 layers. By the time model construction finishes, every layer holds its Q, K, V, and output projection weights, but nothing has been copied. <strong>They’re all just pointers into the memory-mapped file.</strong></p><h4>Nothing loads until you touch it</h4><p>The first time you actually touch a weight during a forward pass, the OS fetches the relevant page from disk — <strong>page fault</strong>. This isn’t an error, it’s the normal demand-paging mechanism. The OS loads the 4KB page containing that data, maps it into your process, and resumes execution. After that, the page lives in the kernel’s page cache and future accesses are fast.</p><p>You can see this happen directly. Running perf stat on the baseline forward pass reports over <strong>1 million page faults at roughly 44,000 per second</strong>. That&#39;s the OS spending a non-trivial portion of inference just pulling weight pages off disk on demand.</p><p>The model is not a data structure you load. It’s a description of how to interpret a region of memory that the OS manages for you.</p><h3>Two Numbers That Tell the Whole Story</h3><p>Before looking at a single optimization, you need to internalize two concepts that frame everything else. They also explain the two metrics we’ll be tracking throughout.</p><p><strong>TTFT</strong> (Time to First Token) is how long it takes to process the input prompt and produce the very first output token. During this phase, called the <strong>prefill</strong>, you’re running the full forward pass over all input tokens at once. <strong>It’s compute-heavy.</strong></p><p><strong>Decode latency</strong> (ms per token) is how long each subsequent token takes. During decode, you process exactly one new token at a time using the KV cache. <strong>It’s memory-heavy.</strong></p><p>That difference matters because of two fundamental limits every processor has:</p><ul><li><strong>Arithmetic intensity</strong> is how many floating-point operations you perform per byte of memory you touch. Think of it as: how hard are you working with the data you fetch?</li><li><strong>The roofline model</strong> says every processor has two ceilings ,a<strong> peak compute throughput </strong>(FLOP/s) and a <strong>peak memory bandwidth </strong>(bytes/s). Their ratio is the machine’s <em>ridge point/knee</em>. If your workload’s arithmetic intensity is below that ridge point, you’re <strong>memory-bound</strong>: the compute units sit idle waiting for data to arrive. Adding faster cores won’t help. If it’s above, you’re <strong>compute-bound</strong>: the memory is fine, you just need faster cores for more FLOPS.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*amLhcgrU5JESYU_g-Kx7eA.png" /><figcaption>Roofline Model — Courtesy of <a href="https://brrrviz.com/">BrrViz</a></figcaption></figure><p><strong>For LLM decode (one token at a time), arithmetic intensity is tiny</strong>. To see why, take one of the FFN weight matrices: it’s 5632x2048 floats, about 11 million values. To multiply one token’s activations (a vector of 2048 floats) against that matrix, you do roughly 11 million <strong>multiply-accumulate</strong> operations. But you also had to load all 11 million weights from memory to do it. That’s roughly <strong>1 FLOP per byte</strong>, far below the ridge point of any modern processor. The compute units finish their work and then sit idle, waiting for the next batch of weights to arrive from RAM.</p><pre>Decode: 1 token × weight matrix<br>  memory loaded  : 11M floats  (~44MB)<br>  FLOPs done     : ~11M<br>  arithmetic intensity ≈ 1 FLOP/byte   ← deep in memory-bound territory</pre><p><strong>For prefill (processing the full input prompt at once)</strong>, the picture changes. Now you’re multiplying a matrix of token activations — say 64 tokens ,against the same weight matrix. You still load the weights once, but now you do 64× more math with the same bytes.</p><pre>Prefill: 64 tokens × weight matrix<br>  memory loaded  : 11M floats  (~44MB)   ← same as decode<br>  FLOPs done     : ~0.7B<br>  arithmetic intensity ≈ 64FLOPs/byte  ← much closer to compute ceiling</pre><p>Same weights, same memory cost. But the compute is now spread across 64 rows instead of 1. That’s why TTFT and decode respond so differently to the same optimization — they’re hitting different parts of the roofline.</p><h3>Quantization: The Model Doesn’t Run on INT4</h3><p>There are many quantization formats out there: <strong>Q4_0</strong>, <strong>Q4_1</strong>, <strong>Q5_K</strong>, <strong>Q8_0</strong>, and more. They all make the same basic trade-off: fewer bits per weight means smaller file, less RAM, faster memory transfers, and some loss of precision. Tiny Llama 1.1B uses <strong>Q4_K</strong> for most weight matrices and <strong>Q6_K</strong> for a smaller set of more sensitive ones (the attention output and FFN down projections), where the extra precision is worth the cost.</p><p>Now here’s something that surprises people: the weights are stored in 4-bit format, but the model doesn’t actually <em>compute</em> in 4-bit.</p><p>To understand why, you need to know what quantization actually does to a number.</p><p>A 32-bit float can represent values across an enormous range with high precision. A 4-bit integer can only hold 16 distinct values: 0 through 15. So how do you squash a float into 4 bits without losing everything?</p><p>You divide the weights into small blocks say, 32 values at a time and for each block you find the minimum and maximum value. Those two numbers define the range the block needs to represent. Every weight in the block is then expressed as an integer from 0 to 15, mapping linearly across that range.</p><pre>One block (32 weights on disk):<br>┌────────────────────────────────────────────────────┐<br>│  min: -0.5   max: 0.8   (stored once per block)    │<br>│  scale = (max - min) / 15                          │<br>├──────┬──────┬──────┬──────┬──────┬─────  ...  ─────┤<br>│  9   │  14  │  2   │  15  │  10  │       ...       │<br>│ 4bit │ 4bit │ 4bit │ 4bit │ 4bit │                 │<br>└──────┴──────┴──────┴──────┴──────┴─────────────────┘</pre><p>Each 4-bit integer is meaningless on its own. It only becomes a usable float when you apply the block’s scale and min at compute time:</p><pre>dequantized = q * scale + min</pre><p>Concretely, say a block spans min = -0.5 to max = 0.8, and one of the weights is 0.3:</p><pre>scale     = (0.8 - (-0.5)) / 15 = 0.0867<br>quantized = round((0.3 - (-0.5)) / 0.0867) = round(9.23) = 9</pre><p>That single weight is now stored as the integer 9 , 4 bits instead of 32. The scale (0.0867) and min (-0.5) are stored once per block. To recover the original value at compute time:</p><pre>dequantized = 9 * 0.0867 + (-0.5) = 0.28</pre><p>Not exactly 0.3, but close. The error is the price you pay for the compression.</p><p>This also handles negative weights cleanly. A weight of -0.5 maps to 0, and 0.8 maps to 15. The full signed range of the block fits into 16 buckets.</p><p><strong>Q4_K</strong> and <strong>Q6_K </strong>are more sophisticated variants of this, they use a two-level scaling scheme (a scale-of-scales) for better accuracy, but the core idea is identical.</p><p>At a high level, it’s the same principle as any compression algorithm:</p><pre>FP32 weights  →  compress (quantize)  →  4-bit storage<br>4-bit storage →  decompress (dequantize)  →  FP32 for compute</pre><p>The difference from general-purpose compression is that this is <em>lossy</em> . You don’t get the exact original value back. But for neural network weights, a small amount of precision loss turns out to matter very little in practice.</p><p>This is why quantization is a <strong>memory</strong> optimization, not a compute optimization. You fit a larger model in RAM and transfer fewer bytes from memory to the compute units. But the compute itself still happens in floating point.</p><p>Every <strong>production matmul GEMM </strong>dequantizes each block on the fly, right before accumulating the products. Now that you understand the roofline, you can see exactly why this matters: if we’re memory-bound, anything that reduces bytes in flight directly raises throughput.</p><h3>Mental Model</h3><p>So far we’ve covered the scaffolding: how the model is <strong>laid out on disk</strong>, how it gets <strong>mapped into memory</strong>, how the<strong> weights are quantized</strong>, and the two metrics, <strong>TTFT</strong> and <strong>decode</strong> — that every optimization will be judged against. Now we’ll put it all together and build a mental model of exactly what happens during inference — just the structure, no details yet.</p><p>There are roughly three steps:</p><pre>// step 1: load weights into memory<br>transformer();<br>prompt = &quot;Migrate entire codebase from typescript to assembly&quot;;<br><br>// step 2: prefill — process the entire prompt in one forward pass<br>tokens = tranfsormer.tokenize(prompt);<br>first_token = transformer.forward(tokens);<br>tokens.append(first_token);<br><br>// step 3: decode — generate one token at a time<br>while tokens.last != END_TOKEN:<br>    next_token = transformer.forward(tokens);<br>    tokens.append(next_token);</pre><p>Every transformer.forward call is the same thing: run each of the 22 decoder blocks serially. Each block applies a series of mathematical operations:rms_norm, matmul, softmax, silu — using that layer’s weights, and passes its output to the next block. Where the costliest operation inside every block is matmul, by a large margin.</p><p>This means there are two knobs that control how fast inference runs:</p><ul><li><strong>How fast weights load:</strong> fewer bytes to move means less time blocked on memory</li><li><strong>How fast matmul runs:</strong> it dominates the compute inside every block</li></ul><p>Every optimization in this article tries to play around with these two knobs.</p><h3>The Baseline</h3><p>Now let’s see what happens when we keep everything as simple as possible.</p><p><strong>Weights </strong>are loaded and<strong> dequantized eagerly</strong> at startup: every <strong>Q4_K/Q6_K </strong>matrix is unpacked into a plain FP32 array before the first token is ever generated. The 700MBmodel balloons to ~2.8GBin RAM. Simple — but the CPU now streams 4× more data through cache than necessary on every forward pass and model now occupies a lot of space in memory.</p><p><strong>Matmul</strong> is just a simple naive triple loop:</p><pre>for i in 0..M:<br>    for j in 0..N:<br>        acc = 0<br>        for k in 0..K:<br>            acc += A[k][i] * B[k][j]<br>        C[i][j] = acc</pre><p>The inner loop over k accesses B[k][j] by stepping through memory in a pattern that jumps N elements forward each time — which is not how the data is laid out. With K=2048 and N=5632, <strong>you&#39;re skipping thousands of floats between each access</strong>. The CPU&#39;s prefetcher can&#39;t predict it. Cache lines get loaded, used for one value, and evicted before the next column access can reuse them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*G0i-4NVQFPNi6C7QVmSgiQ.gif" /><figcaption>Courtesy of <a href="https://medium.com/data-science-collective/learning-triton-one-kernel-at-a-time-matrix-multiplication-44851b4146dd">Ryan Pégoud</a></figcaption></figure><p>Running this on TinyLlama 1.1B with a 512-token prompt:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*ZmJcxqo6TYLKnI5Qsst6YA.png" /></figure><p>Profiling show that the CPU is running at 0.98 instructions per cycle. Almost exactly 1. A well-optimized loop should be 3 to 4 (CPU Pipelining). The processor is executing one instruction and then stalling, waiting for memory just cause the data isn’t in cache.</p><h3>Optimization 1: Cache Tiling</h3><p>To understand what tiling fixes, you need a concrete picture of what a cache miss actually costs.</p><p>Your CPU has three levels of cache sitting between it and main memory:</p><pre>L1 cache :  32 KB  —  ~4 cycles to access<br>L2 cache : 256 KB  —  ~12 cycles to access<br>L3 cache :   3 MB  —  ~40 cycles to access<br>DRAM     :   8 GB  —  ~200 cycles to access</pre><p>When the CPU needs a float that isn’t in L1, it checks L2, then L3, then finally goes to DRAM. That last step, a full cache miss to DRAM — costs 200 cycles. For one float. While the CPU is waiting, the pipeline stalls. This is exactly what was happening: 0.98 IPC, one instruction executed, then a ~200 cycle stall while the next chunk of the weight matrix arrives from RAM.</p><p>The naive matmul makes this inevitable. Ideally, we’d like to load each segment of data only once and perform all the operations in which they are used before dropping them from memory.</p><p><strong>Tiling</strong> does exactly this(also called blocking). Instead of iterating over the full K dimension at once, you break A, B, and C into small square tiles that fit comfortably in L1 or L2 cache. You work on one tile at a time: load a tile of A, a tile of B, accumulate into a tile of C, then move to the next. The key insight is that within a tile, you reuse the same data across many multiply-accumulate operations — so the cost of fetching it from memory gets amortized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*v1_hv5z2Lpe86o5VtMfPYQ.gif" /><figcaption>Courtesy of <a href="https://medium.com/data-science-collective/learning-triton-one-kernel-at-a-time-matrix-multiplication-44851b4146dd">Ryan Pégoud</a></figcaption></figure><p>The code changes the loop structure to walk through tiles first:</p><pre>for ii in 0..M step TILE:<br>    for jj in 0..N step TILE:<br>        for kk in 0..K step TILE:          // tile over K<br>            for i in ii..ii+TILE:<br>                for j in jj..jj+TILE:<br>                    acc = C[i][j]<br>                    for k in kk..kk+TILE:  // inner loop now fits in cache<br>                        acc += A[k][i] * B[k][j]<br>                    C[i][j] = acc</pre><p>Same six loops. Same arithmetic. But now the inner k loop only walks TILE_SIZE elements in B&#39;s column — a distance that fits in L1. The cache lines fetched for this tile get reused across all the i and j iterations within the tile, instead of being evicted before the next access.</p><p>I honestly didn’t expect this to make such a large difference. It’s a loop reorder — no new math, no new hardware, no vectorization. Just a different traversal order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*Hr1i5QdgW9w9bogWRTbucw.png" /></figure><p><strong>2× faster. Purely from keeping data in cache.</strong></p><p><em>the algorithm’s memory access pattern matters more than the algorithm itself.</em></p><h3>Optimization 2: SIMD — A Lesson in Expectations</h3><p>After tiling cut runtime in half, the natural next step seemed obvious: vectorize the inner loop with SIMD.</p><p><strong>SIMD</strong> (Single Instruction, Multiple Data) lets the CPU batch a chunk of data into <strong>wide registers</strong> and <strong>apply the same operation across all of it in a single step</strong>, instead of one number at a time. <strong>AVX2</strong> is the instruction set (intrinsics) that exposes this on the i3–7100: its 256-bit registers hold 8 floats, so one instruction multiplies all 8 at once. In theory, the inner loop could run 8× faster.</p><pre>// each output row of C<br>for (int i = 0; i &lt; M; i++) {<br>    // contiguous reads, no big jumps                             <br>    pack_panel(A, i, packedA); <br>    // 8 columns at a time                           <br>    for (int j = 0; j &lt; N; j += 8) {    <br>        // 8 accumulators set to 0                  <br>        __m256 acc_vec = _mm256_setzero_ps();             <br>        for (int k = 0; k &lt; K; k++) {<br>            // copy A[k] into all 8 lanes<br>            __m256 a_vec = _mm256_broadcast_ss(&amp;packedA[k]); <br>            // load B[k][j .. j+8] <br>            __m256 b_vec = _mm256_loadu_ps(&amp;B[k][j]);   <br>            // multiply + add, 8 lanes, 1 instruction      <br>            acc_vec = _mm256_fmadd_ps(a_vec, b_vec, acc_vec); <br>        }<br>        // write 8 results back<br>        _mm256_storeu_ps(&amp;C[i][j], acc_vec);               <br>    }<br>}<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*T1vMTThqgFqwroXAlIrpeQ.gif" /><figcaption>SIMD — Step By Step Visualization</figcaption></figure><p>Clean implementation. AVX2 FMA instructions firing. Should be fast.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*HQ0m68R9S1cU6_4l8Y2Czg.png" /></figure><p>3%. Essentially nothing.</p><p>The roofline model predicted this exactly. The i3–7100’s memory bandwidth is ~37 GB/s. Its peak AVX2 throughput is ~230 GFLOP/s. The ridge point, bandwidth divided by peak compute — is around 6 FLOP/byte. Our matmul is running at roughly 1 FLOP/byte. We are nowhere near the compute ceiling.</p><p>This means that the ALUs are already sitting idle doing nothing, just waiting for memory to load. Making them process 8 floats simultaneously when they were already idle changes nothing — the bottleneck is still how fast data arrives from RAM.</p><p>This is a brutal but important result I had to internalize after spending an entire day writing beautiful vectorized code, profiling it, watching it execute — <strong>and it makes no measurable difference</strong>. Because the true bottleneck is not what I optimized.</p><p><em>Measure first. Optimize second.</em></p><h3>Optimization 3: KV Cache</h3><p>Let’s optimize our true bottleneck. Here’s a good analogy for what was happening without a KV cache.</p><p>Imagine you’re reading a book, and every time you encounter a new word, you go back and re-read the entire book from page one before continuing. That’s what the baseline decoder was doing. For each new token it generated, it re-projected every previous token through the K and V weight matrices, effectively re-reading the entire context from scratch — even though those projections hadn’t changed at all.</p><p>The KV cache eliminates this. At startup, you pre-allocate a single contiguous arena in memory sized for the maximum sequence length:</p><pre>KV arena layout:<br>┌──────────────────────────────────────────────────────────────┐<br>│  Layer 0  K  [ head_dim × kv_heads × max_seq_len ]          │<br>│  Layer 0  V  [ head_dim × kv_heads × max_seq_len ]          │<br>├──────────────────────────────────────────────────────────────┤<br>│  Layer 1  K  ...                                             │<br>│  Layer 1  V  ...                                             │<br>├──────────────────────────────────────────────────────────────┤<br>│  ...  (22 layers total)                                      │<br>└──────────────────────────────────────────────────────────────┘</pre><p>During prefill, K and V for every token and every layer are computed and written into their slots. On each subsequent decode step, <strong>you only compute K and V for the single new token</strong>, <strong>append it into the next slot</strong>, and <strong>attend over the full cached sequence</strong>. The projection work for everything that came before is free.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*5KHnpyDPLXVQzZ64g8hB_g.png" /></figure><p><strong>Decode</strong> improves by <strong>10.6×</strong>. <strong>TTFT</strong> barely changes and that’s expected. The KV cache only helps decode. During prefill, the cache is cold: there’s nothing stored yet, so every token’s K and V still has to be computed from scratch. It’s not that the cache failed during prefill, it’s that it simply hasn’t accumulated anything yet. The first token is always the most expensive.</p><p>This is the single biggest behavioral change in the whole series. Tiling and SIMD were trying to speed up each matmul — they were fighting a memory-bandwidth wall, making already-idle ALUs work slightly harder. This optimization basically reduces the amount of redundant work the ALUs were doing by simply caching the results.</p><p>Real production systems like<strong> </strong><a href="https://vllm.ai/"><strong>vLLM</strong></a><strong> </strong>go further with <strong>paged attention</strong> — managing the KV cache in fixed-size pages instead of a single contiguous arena, similar to how an OS handles virtual memory. This avoids fragmentation when serving many users with different context lengths simultaneously. But that’s a story for another time.</p><h3>Optimization 4: Quantized Kernels</h3><p>Remember what the baseline did at startup: it walked every weight tensor in the GGUF file and eagerly dequantized all of it, every <strong>Q4_K </strong>and <strong>Q6_K </strong>block — into full FP32 float arrays, allocating fresh memory to hold the result. By the time main() returned from the constructor, the 700MB model file had been unpacked into roughly ~2.8GB of floats sitting in RAM. Every matmul then operated on those floats directly.</p><p>This made the matmul simple.But it also meant longer startup times and larger memory bandwidth between memory and the CPU.</p><p>The fix is to stop dequantizing upfront and instead fuse dequantization directly into the matmul, <strong>basically make it lazy</strong>. The weights stay in their compact Q4_K/Q6_K format in memory, exactly as they appear in the file. The matmul kernel reads one block at a time, unpacks just those 32 values into a small scratch buffer, and immediately uses them for the dot product accumulation:</p><pre>// Quantized matmul (one output row m, over all K blocks):<br>for each block b in 0..K/32:<br>    scratch[0..31] = dequantize(W_row_m[b])   // unpack 32 × 4-bit → 32 floats<br>    for each column n:<br>        C[m][n] += dot(scratch, X[b*32..(b+1)*32][n])</pre><p>The scratch buffer is 32 floats, 128 Bytes, fits entirely in L1 cache. You unpack a block, consume it immediately across all output columns, and move on. The 4-bit weights are loaded once and never written to a large intermediate float array. No upfront allocation. No ~2.8GB in memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*syWQimWh8xjASgWkarNMDg.png" /></figure><p>A <strong>4× improvement on TTFT, </strong>the biggest single jump in the whole series.</p><p>The reason is straightforward now. The FFN up/down projection matrices are 5632x2048 weights each. In FP32, one matrix is ~44MB. In Q4_K, the same matrix is ~11MB. Every time the matmul sweeps through those weights, it now moves 4xfewer bytes across the memory bus. We’re still memory-bound, but we’ve moved up towards the roofline ceiling, the same bandwidth now delivers 4x more weights per second.</p><p>This is the point where quantization stops being just a storage trick and starts being a genuine runtime optimization. Same arithmetic, fewer bytes in flight.</p><h3>Optimization 5: CUDA — Moving to the GPU</h3><p>Everything so far has been CPU-only. Now we move compute to the GPU.</p><h4>A quick GPU mental model</h4><p>A GPU runs thousands of tiny threads at once, organized into thread <strong>blocks </strong>where each block is a group of threads (here, 256) that run together. Each block is arranged in a <strong>grid </strong>like fashion all executing in parallel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GeDaBJMbRfbK4SgnoZgLCA.png" /><figcaption>Courtesy of <a href="https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/programming-model.html">Nvidia</a></figcaption></figure><p>Each thread block can share a small patch of fast on-chip <strong>shared memory </strong>(analogous to the CPU L1 cache). Everything else lives in <strong>global memory</strong>: GPU’s VRAM (analogous to CPU RAM), large by comparatively slow. The game is the same one we’ve been playing on the CPU all along, keep hot data in fast memory and minimize trips to slow memory.</p><p>The <strong>CPU</strong> and <strong>GPU </strong>are separate chips with separate memory. They communicate over a PCIe bus and that link is the first thing that can go badly wrong.</p><h4><strong>The memory movement problem</strong></h4><p>The 750 Ti sits on a PCIe 3.0 x16 slot. The theoretical bandwidth of that bus is ~16 GB/sin each direction. The GPU’s own GDDR5 memory runs at ~86 GB/s internally. The moment you need to move data between CPU and GPU, you’re bottlenecked by PCIe. A 5× slowdown compared to what the GPU can do on its own memory.</p><p>A naive GPU port, upload activations to the GPU, run cublasSgemm (A CUDA Matmul), download the result, repeat per layer, would be catastrophically slow. The decode path runs ~100matmuls per token across 22 layers. Each one would require two PCIe transfers: one to upload the input, one to download the output. At even a few MB per transfer, PCIe would dominate the runtime completely.</p><pre>Naive (wrong) approach:<br>for each layer:<br>    cudaMemcpy(x_gpu ← x_cpu)          // PCIe upload<br>    cublasSgemm(w_gpu, x_gpu → y_gpu)   // GPU compute<br>    cudaMemcpy(y_cpu ← y_gpu)          // PCIe download<br>    ... repeat 100+ times per token</pre><p>The fix is to never come back. Everything stays on the GPU:</p><pre>Correct approach:<br>startup:  <br>  Step 1: upload all weights to GPU VRAM once<br>  Step 2: allocate KV cache in GPU VRAM<br>  Step 3: allocate activation buffers in GPU VRAM<br>per token: <br>  Step 1: run entire forward pass on GPU<br>  Step 2: cudaMemcpy just the final logits back to CPU (32,000 floats, ~128KB)</pre><p>Weights are uploaded once, a one-time cost at startup, not on every token. The KV cache lives in VRAM. Activations are computed and consumed entirely on-device. The only PCIe transfer per token is 128KB of logits at the very end.</p><h4><strong>Why a plain matmul kernel isn’t enough</strong></h4><p>Even with data resident on the GPU, you can’t just use cublasSgemm for the quantized weights. cuBLAS operates on FP32 or FP16 matrices. Your weights are in Q4_K format, 4 bits per value with block-level scales. You need a custom kernel that understands the quantization format, something similar to the CPU quantized kernel we built.</p><h4>The <strong>Cuda Quantized Kernel</strong></h4><p>So we write our own. A group of 256 GPU threads takes one compressed weight block, and each thread unpacks a single value into fast on-chip shared memory. As soon as the block is unpacked, the same threads read from the shared memory and use those values directly in the dot product — the unpacked weights never get written back out to slow global memory:</p><pre>Each thread block handles one output tile:<br>  load weight block (packed bytes) from global memory<br>  256 threads cooperate: each thread dequantizes its element into shared mem<br>  accumulate: each thread computes its contribution to the dot product<br>  write result to output in global memory</pre><p>No intermediate FP32 weight matrix is ever materialized. The dequantization and multiplication happen inside the same register file, on the same data, without additional memory traffic.</p><h4><strong>The Fused Attention Kernel</strong></h4><p>The attention forward pass has a similar problem. The standard approach writes a full [seq_len × seq_len] score matrix to global memory, then reads it back for the softmax, then reads it again for the weighted sum of V. Three passes over a matrix that can be large.</p><p>The fused kernel eliminates the score matrix entirely using online softmax , an algorithm that computes the numerically stable softmax and weighted V accumulation in a single pass, maintaining a running max and sum:</p><pre>// Grid : (q_heads, seq_len)  — one block per (head, query position)<br>// Block: head_dim threads    — one thread per element in the head<br>Each block:<br>  load Q[h, q_pos] → shared memory<br>  single pass over key positions 0..q_pos:<br>      score = scale × dot(Q, K[k])   // QK^T<br>      update running max, softmax denominator<br>      accumulate weighted V[k]<br>  write output → global memory<br>  (score matrix never touches global memory at all)</pre><h4><strong>What nvprof actually shows</strong></h4><p>Running nvprof (Nvidia Benchmarking Tool) on the full inference gives a precise breakdown of where GPU time goes:</p><pre>GPU activities:<br>  80.4%  4.89s   matmul_q4k_kernel    1206 calls  avg 4.1ms<br>  17.2%  1.05s   matmul_q6k_kernel     189 calls  avg 5.5ms<br>   2.0%  124ms   CUDA memcpy HtoD      232 copies  avg 534µs<br>   0.15%  9.3ms  fused_attn_kernel     198 calls  avg 47µs<br>   0.03%  1.9ms  rms_norm_kernel       405 calls  avg 4.6µs<br>   0.02%  933µs  add_kernel            396 calls<br>   0.01%  754µs  silu_kernel           198 calls<br>   0.01%  627µs  CUDA memcpy DtoH       40 copies<br>API calls:<br>  95.9%  5.93s   cudaFree             2404 calls  avg 2.5ms  ← !!<br>   2.4%  151ms   cudaMemcpy           1856 calls<br>   1.0%   64ms   cudaMalloc           2605 calls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*2rmrwlCMmYoVEG9oDGcb1g.png" /></figure><p>Decode is a clear win, 2.8x faster — but TTFT moved the <em>wrong</em> way: 3.93s versus 2.76s on CPU. The nvprof breakdown above explains where the time actually goes.</p><p>The quantized matmul kernels dominate GPU activity, 97.6% of GPU time combined — which is exactly right. That’s the real work.</p><p>The fused attention is remarkable: 198 calls across all decode steps averaging 47µs each. Total attention time: 9.3ms. On the CPU it was a meaningful fraction of runtime; here it’s noise.</p><p>So why is TTFT slower on GPU when it’s faster at everything else? It’s not a GPU limitation, it’s two avoidable costs in my own implementation stacking up.</p><p><strong>Cost 1: the embedding lookup still runs on the CPU.</strong> This happens right at the start of every forward pass. It’s the very first step, before any of the 22 decoder layers run, turning the raw token IDs into the input vectors the layers operate on. The <strong>embedding lookup</strong> itself is simple: each token is just an integer ID, and the model stores a big table with one learned vector per vocabulary entry. The lookup grabs that token’s row and uses it as the token’s starting vector. In this model the rows are stored quantized, and that dequantization was never ported to a GPU kernel, so for every prompt token the data bounces across the PCIe bus:</p><pre>// weights live on GPU including the embedding table...<br>transformer.device(cuda)               <br>tokens = transformer.tokenize(prompt)<br><br>// ...but embed() was never ported, so every token round-trips across PCIe:<br>for each token in tokens:<br>    // PCIe: GPU → CPU<br>    cudaMemcpy(row_cpu ← emb_table_gpu[token])  <br>    // CPU unpacks<br>    row_vec = dequantize(row_cpu)               <br>    // PCIe: CPU → GPU<br>    cudaMemcpy(input_vecs_gpu ← row_vec)        <br>    // 2 PCIe transfers per token<br>    input_vecs.append(row)  <br>               <br>first_token = transformer.forward_(input_vecs)   // GPU finally starts</pre><p><strong>Cost 2: memory is allocated and freed on every layer.</strong> Each decoder layer calls cudaMalloc/cudaFree for its scratch buffers, and this is a <strong>big systems no no — never allocate and free on a hot path</strong>, cause allocating and deallocating is costly memory operation.</p><p>Both costs are easy to fix, move the embedding lookup onto the GPU, and allocate a <strong>scratch memory arena</strong> like we did for the KV Cache, once instead of freeing it every layer. But left as-is, it’s a textbook example of where hidden PCIe round-trips and allocator stalls quietly eat your time.</p><h3>What This Whole Journey Actually Looks Like</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*Moj546RFD73SWzib8LEVcg.png" /></figure><p>From 22.67s per token to 190ms. Same model, same machine. Every bit of that speedup came from how the code touches memory.</p><h3>Where This Goes Next</h3><p>Everything here is running on a single device with a single sequence. The real frontier is different:</p><p><a href="https://vllm.ai/blog"><strong>Paged Attention (vLLM)</strong></a>: a virtual-memory-like approach to the KV cache paged, reclaimed, and shared across requests.</p><p><a href="https://medium.com/data-science-collective/learning-triton-one-kernel-at-a-time-matrix-multiplication-44851b4146dd"><strong>Triton Matmul Kernels</strong></a>: our hand-written CUDA matmul kernels dominated the profile, 97.6% of GPU time. This blog builds the same matmul as a Triton kernel instead of raw CUDA, less boilerplate, and it autotunes for you.</p><p><a href="https://medium.com/@mandeep0405/distributed-training-overview-scaling-pytorch-across-multiple-gpus-b698bc22c81a"><strong>Multi-GPU inference and distributed training</strong></a>: Explains how large scale frontier models are trained, covers data and model parallelism across devices.</p><p>And if you want to <em>see</em> the memory movement of a GPU, <a href="https://brrrviz.com/"><strong>BrrrViz</strong></a> is an interactive visualizer for how data moves through the GPU Memory Hierarchy, gives a better understanding of the compute and memory arguments discussed here.</p><p>The crossover of ML and systems is far more interesting than it gets credit for. Most of the attention goes to the models, the architectures, the benchmarks, but the layer underneath all of that, the part that decides whether a model runs in two seconds or two minutes, is its own quiet, fascinating world. The math got solved years ago. The systems are what actually puts the model in your hands.</p><p>This project is available on<a href="https://github.com/AlterHoodie/poor_mans_vllm"> <strong>GitHub</strong></a>. Each optimization lives on its own branch, building directly on top of the one before it: NAIVE → TILING → SIMD → KV_SIMD → KV_QUANTIZED → KV_CUDA.</p><p>So you can check out any stage in isolation and see exactly what changed from the previous step. Each branch also carries its own benchmarks, profiling data, and detailed stats, so you can dig into the numbers behind every speedup yourself rather than taking mine at face value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=592f06c6cd28" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>