<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by AIwithMaha on Medium]]></title>
        <description><![CDATA[Stories by AIwithMaha on Medium]]></description>
        <link>https://medium.com/@mahareddyroja247?source=rss-e80d331b9776------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VsyO7LdokTZZz8WujhXxEg.jpeg</url>
            <title>Stories by AIwithMaha on Medium</title>
            <link>https://medium.com/@mahareddyroja247?source=rss-e80d331b9776------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 17:26:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mahareddyroja247/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How vLLM Serves Thousands of Requests: PagedAttention and Continuous Batching]]></title>
            <link>https://medium.com/@mahareddyroja247/how-vllm-serves-thousands-of-requests-pagedattention-and-continuous-batching-39b86e0bceb3?source=rss-e80d331b9776------2</link>
            <guid isPermaLink="false">https://medium.com/p/39b86e0bceb3</guid>
            <category><![CDATA[pagedattention]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[continuous-batching]]></category>
            <category><![CDATA[kv-cache]]></category>
            <category><![CDATA[vllm-serving]]></category>
            <dc:creator><![CDATA[AIwithMaha]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 19:15:22 GMT</pubDate>
            <atom:updated>2026-09-21T19:15:22.453Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is for beginners. If you’ve read my KV cache post, you already know half of what you need. If you haven’t, don’t worry, I’ll recap the one idea everything here builds on. So your model answers one person’s question just fine. Now imagine 1,000 people open your app at the same time and all start chatting with it.</p><p>What breaks first? It’s not the model’s intelligence. It’s memory and waiting. This post is about the two ideas that fix both: <strong>PagedAttention</strong> and <strong>continuous batching</strong>. Together, they are the engine inside vLLM, the open-source serving system that made fast, cheap LLM serving normal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*doWjYBjrbg0s4ETIyKBhYg.png" /></figure><p>The distinction is simple: PagedAttention addresses KV-cache memory management, while continuous batching addresses request scheduling and GPU utilization.</p><p><strong>1. the KV cache</strong></p><p>Quick recap from my earlier post. When a model generates text, it does it one token at a time. To produce a new token, the model needs the keys and values of the previous tokens. Without a KV cache, those previous keys and values would have to be recomputed repeatedly. The KV cache stores them so each new token can reuse what was already computed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s5IDoza7dZ971Dox9-RdtA.png" /></figure><p>Here’s the part that matters for serving: every active request needs its own KV cache, and it grows as the request generates tokens. A request that ends up 2,000 tokens long needs room for 2,000 tokens’ worth of keys and values, for every layer of the model. KV-cache memory is often a major constraint on how many requests you can serve concurrently.</p><p><strong>2. one at a time, or waiting for the slowest</strong></p><p>The simplest server handles one request at a time. It works and the GPU spends most of its time underused, because generating one token at a time barely uses the GPU’s compute it’s stuck waiting on memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HUGkiz8rZpI7KWWT3JbdSw.png" /></figure><p>The obvious fix is <strong>batching,</strong> process several requests together, one token per request per step. Better GPU usage. But naive batching has an embarrassing problem, requests finish at different times. If you batch 8 requests and 7 finish quickly, the 8th the long one holds the whole batch hostage. The GPU keeps running just for that one straggler while 7 slots sit empty.</p><p><strong>3. paying for seats nobody sits in</strong></p><p>There’s a second problem, and it’s worse. Before a request starts, the server has to reserve memory for its KV cache. But it doesn’t know how long the request will be. In a naive contiguous-allocation approach, the server may reserve a large contiguous region for each request’s KV cache, often based on a maximum or anticipated sequence length.</p><p>Most requests are much shorter. A 100-token reply sits in a 2,048 token reservation. The rest is empty, reserved, unusable memory. Worse, the empty space gets fragmented: free memory scattered in odd-sized chunks that no new request quite fits into, like a parking lot full of half-empty spaces no car can use.</p><p>So the naive server wastes memory twice, reserving too much per request, and fragmenting what’s left.</p><p><strong>4. PagedAttention: the idea from operating systems</strong></p><p>When programs run, they don’t get one big contiguous chunk of RAM. The OS splits memory into fixed-size pages (usually 4KB), hands pages out on demand, and keeps a page table<strong> </strong>mapping each program’s logical addresses to wherever the physical pages actually are. Programs think they have smooth, continuous memory. Physically, it’s scattered everywhere and nobody cares, because the page table translates.</p><p>PagedAttention does exactly this for the KV cache they split each request’s KV cache into fixed-size blocks (vLLM uses 16 tokens per block). Allocate a new block only when the request actually needs it, when its tokens spill past the current blocks. Keep a block table per request: logical position → physical block.</p><p>Attention doesn’t care that block 3 of your cache lives far away from block 2 in GPU memory. It just looks up the block table and gathers the right keys and values.</p><p>Walk through it: block size 16, and your request has generated 40 tokens. You hold 3 blocks (16 + 16 + 8 used). A new token arrives still fits in block 3, no allocation. Eight tokens later so block 3 fills up, and the system grabs one fresh block from the free pool. You never reserved space for 2,048 tokens. You hold exactly what you use, plus at most 15 tokens of slack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zohmjppABeVoePSf5KQMQA.png" /></figure><p>External fragmentation is greatly reduced. Every free block is the same size, so any available block can be assigned to a request. The remaining internal waste is bounded by the unused space in the final partially filled block, and the same GPU memory can therefore fit far more concurrent requests.</p><p><strong>5. Continuous batching: never wait for the slowest eater</strong></p><p>Now fix the waiting problem. Instead of forming a batch and waiting for every request in it to finish, schedule at the granularity of a single generation step: At each step, the scheduler looks at the running requests. So once Finished? Evicted immediately, slot freed.</p><p>New request arrived? It joins at the next step. No waiting for the current batch to drain. Back to the restaurant, this is seating new guests the moment a chair frees up, instead of waiting for the whole table to leave. The GPU can stay better utilized as completed requests are replaced by waiting requests.</p><p>There’s one subtlety. A brand-new request needs a <strong>prefill</strong> step, while running requests need <strong>decode</strong> steps. The scheduler handles both in the same step, it can run a chunk of some request’s prefill alongside everyone else’s decode. (vLLM calls this <em>chunked prefill</em> — splitting long prompts into pieces so one giant prompt doesn’t stall the whole system.)</p><p>This technique is often described as iteration-level scheduling. The Orca system introduced this approach by scheduling inference at iteration granularity rather than waiting for an entire request to finish.</p><p><strong>6. three requests walk in</strong></p><p>Let’s see a tiny example. We’ll use a block size of 16 and assume, for illustration, that the GPU has enough KV-cache memory for four active requests.</p><p><strong>t=0: </strong>Requests A and B arrive. Both prefill, both start decoding. 2 slots used.</p><p><strong>t=1: </strong>Request C arrives. It prefills and joins at the next decode step A and B don’t wait for anything. 3 slots used.</p><p><strong>t=2: </strong>A finishes (“The answer is 42.”). Its blocks return to the free pool instantly. Request D arrives and takes the slot at the very next step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EMTJPK8-fKxEki_93wrQ6g.png" /></figure><p>At no point did the new requests have to wait for the slowest request in the previous batch. And with block-based KV-cache allocation, memory does not have to be reserved for a maximum sequence length up front. That’s the basic idea: make scheduling and memory-allocation decisions at every generation iteration.</p><p><strong>7. To conclude</strong></p><p>PagedAttention and continuous batching are core ideas behind vLLM , an open-source LLM serving and inference engine. In the original vLLM benchmarks, vLLM achieved up to 24× the throughput of Hugging Face Transformers under the tested workloads and hardware configurations. The improvement came from more efficient KV-cache memory management and serving, rather than from using a larger GPU.</p><p>Related ideas such as paged or block-based KV-cache management and dynamic scheduling are now used across several modern LLM serving systems, including TensorRT-LLM and SGLang.</p><p>And notice the pattern across this series, tiling was about using memory movement well, the KV cache was about avoiding recomputation, PagedAttention is about using memory <em>capacity</em> well. Fast AI keeps coming back to one thing managing memory intelligently.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39b86e0bceb3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FP32, FP16, BF16, and Quantization]]></title>
            <link>https://medium.com/@mahareddyroja247/fp32-fp16-bf16-and-quantization-e0344f17e812?source=rss-e80d331b9776------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0344f17e812</guid>
            <category><![CDATA[int8]]></category>
            <category><![CDATA[bf16]]></category>
            <category><![CDATA[fp32]]></category>
            <category><![CDATA[fp16]]></category>
            <category><![CDATA[quantization]]></category>
            <dc:creator><![CDATA[AIwithMaha]]></dc:creator>
            <pubDate>Fri, 11 Sep 2026 17:10:49 GMT</pubDate>
            <atom:updated>2026-09-12T17:13:28.493Z</atom:updated>
            <content:encoded><![CDATA[<p>How many bits does a number need? That question is the whole idea behind fp32, fp16, bf16, and quantization. More bits make a number more exact but slower and heavier to store. Fewer bits make it lighter and faster, but less exact. This blog walks through what each format actually does.</p><p>Here is an example. When your internet is fast, Netflix streams video in full, high quality. When your internet is slow, it switches to a lower quality version automatically. The video is much smaller and keeps playing smoothly instead of buffering, and it still looks good to our eyes. AI number formats work the same way, just with numbers inside a model, not video quality on a screen.</p><h3>FP32</h3><p>FP32 (short for “32-bit floating point”) is the normal, standard format. It uses 32 bits for each number :</p><ul><li>1 bit for the sign (is it positive or negative?)</li><li>8 bits for the “exponent” (how big or how small the number is)</li><li>23 bits for the fraction — the actual digits that make the number precise (this field is sometimes called the “mantissa,” but “fraction” is the more accurate name)</li></ul><p>This gives about 7 correct decimal digits, and it can hold numbers that are very big or very small. It is the all-purpose format think of it as the full-quality Netflix stream of number formats.</p><h3>FP16</h3><p>FP16 (short for “half precision”) uses only 16 bits (half of FP32). Split roughly like this:</p><ul><li>1 bit for sign</li><li>5 bits for exponent</li><li>10 bits for the fraction</li></ul><p>So each number takes half the memory, and the computer can often handle twice as many numbers per second, since twice as many now fit in the same space.</p><p>But there is a problem. With only 5 bits for the exponent, FP16 can only hold numbers within a fairly small range. If a number in your model becomes too big or too small, FP16 cannot hold it anymore it “breaks,” turning into infinity or zero. Picture a small measuring cup. It is great for measuring a cup of flour, but the moment you try to a massive sack of flour, it just cannot do it. That small range is FP16’s biggest weakness. And during AI training, numbers really do jump between very big and very small, so training with FP16 can go wrong.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_lENPaI3oN7dULYPvskM6g.jpeg" /></figure><h3>BF16</h3><p>BF16 (short for “bfloat16,” from “Brain Floating Point,” made by Google Brain) is also 16 bits in total but it splits those bits in a different way:</p><ul><li>1 bit for sign</li><li>8 bits for exponent (same as FP32!)</li><li>7 bits for the fraction</li></ul><p>Because BF16 keeps the same 8-bit exponent as FP32, it can hold the same wide range of numbers as FP32, so it does not “break” like FP16 does. The trade off is that it only has 7 fraction bits instead of FP16’s 10, so it is a little less exact about the fine details of a number.</p><p>In practice, this is a great trade for deep learning. Neural networks care much more about not “breaking” than about having every tiny decimal perfectly correct, small rounding differences tend to cancel out across millions of numbers. This is why BF16 has become the normal choice for training big models on modern hardware.</p><p>So in short, FP16 keeps more fine detail but has a small range. BF16 keeps FP32’s full range but gives up some fine detail. Same number of bits, different choices.</p><h3>Quantization</h3><p>Quantization usually shrinks an already-trained model down into a much smaller format, like <strong>int8</strong> or <strong>int4</strong>.</p><p>Think of shoe sizes feet come in endless exact lengths, but a shoe store only stocks a few fixed sizes and your foot just gets rounded to the closest one, quantization does the same to a models numbers:</p><ul><li>Look at the real range of values.</li><li>Split that range into a small number of fixed steps, like shoe sizes.</li><li>Round every number to its nearest step, and store only that step number, a tiny number instead of a long decimal.</li><li>Keep one shared number (a <strong>scale factor</strong>) to convert a step back into an approximate real value when needed.</li></ul><p>int8 uses 256 steps, int4 uses just 16. Fewer steps means a smaller, faster model, at a small cost to accuracy.</p><p>So here is the key difference to remember: FP16/BF16 are usually about training and running a model faster using a smaller floating point format. quantization is usually about shrinking an already trained model down into integers for cheap and fast use.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0344f17e812" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KV Cache From Scratch: Prefill and Decode]]></title>
            <link>https://medium.com/@mahareddyroja247/kv-cache-from-scratch-prefill-and-decode-e34794e3f191?source=rss-e80d331b9776------2</link>
            <guid isPermaLink="false">https://medium.com/p/e34794e3f191</guid>
            <category><![CDATA[prefill-decode]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[kv-cache]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[AIwithMaha]]></dc:creator>
            <pubDate>Mon, 17 Aug 2026 00:34:59 GMT</pubDate>
            <atom:updated>2026-08-17T00:42:22.520Z</atom:updated>
            <content:encoded><![CDATA[<p>llms write one word at a time so to pick each new word, the model looks back at everything written so far. So the model recomputes a huge amount of the same work at every single step work it already did a moment ago.</p><p>the KV cache is the fix. It is one of the most important optimizations in LLM inference, and a big reason systems like vLLM exist. This blog builds it from scratch in PyTorch. So we will write attention, generate text the naive way and then add the cache and measure the difference. Everything runs on a laptop, no GPU is needed here.</p><h3>Step 1: words are just numbers</h3><p>computer can’t do anything with the word “cat.” right, it only understands the numbers, so the very first thing that happens inside any language model is that each word gets turned into a list of numbers. You may now ask how many numbers per word? That count is called the hidden dimension, usually written d. It&#39;s a design choice:</p><ul><li>Small models: 128, 256, 512</li><li>GPT-2: 768</li><li>Llama-3: 4096</li></ul><p>so in Llama-3, the word “cat” is represented by a list of 4096 numbers. Here’s a tiny version with d = 4:</p><pre>&quot;cat&quot;  →  [0.2, -0.5, 0.8, 0.1]<br>&quot;dog&quot;  →  [0.3, -0.4, 0.7, 0.2]<br>&quot;car&quot;  →  [-0.9, 0.1, 0.0, 0.6]</pre><p>it’s the whole list that matters, not any single number. 0.2 on its own means nothing, it’s the four numbers together that represent &quot;cat&quot; and the same way a GPS coordinate needs both latitude and longitude. One number alone can&#39;t place you on a map.</p><p>similar words get similar numbers. If you look at the above example “cat” and “dog” have close values (both are animals), while “car” is way off. The model learns these numbers during training, and it arranges them so related words land near each other. You can think of each word as a point in a big “meaning space,” and <em>d</em> is how many coordinates that space has.</p><pre>&quot;the&quot;  →  [0.1, 0.2, 0.3, 0.4]<br>&quot;cat&quot;  →  [0.2, -0.5, 0.8, 0.1]<br>&quot;sat&quot;  →  [0.6, 0.1, -0.2, 0.9]</pre><p>that grid has shape [3, 4] — 3 words (rows), 4 numbers each (columns). In general, a sequence is [number_of_words, d]. Getting your understanding very good at shapes is very important.</p><h3><strong>Step 2: what attention is doing</strong></h3><p>attention is how each word figures out which other words matter to it. In “the animal didn’t cross the street because it was tired,” the word “it” needs to know it refers to “animal.” Attention is the machinery that lets “it” look around and pull meaning from the right place. To do this, every word produces three vectors: a <strong>Query</strong>, a <strong>Key</strong>, and a <strong>Value — </strong>Q, K, V.</p><ul><li><strong>Query (Q):</strong> what I’m looking for. (“I’m the word ‘it’ what do I refer to?”)</li><li><strong>Key (K):</strong> what each word advertises about itself. (“I’m ‘animal’, a noun, a possible subject.”)</li><li><strong>Value (V):</strong> the actual content a word hands over if it turns out to be relevant.</li></ul><p>Q and K are the <em>matching</em> part — they decide who’s relevant to whom. V is the <em>content</em> part — it’s what actually gets passed along once relevance is decided.</p><p>here do Q, K, V come from? Each is made by multiplying the word’s numbers by a learned weight matrix<em> W_q, W_k, W_v</em>. Same input word, three different transformations and also three different &quot;views&quot; of it.</p><p>o the formula for Attention goes by,</p><blockquote><strong>Attention(Q, K, V) = softmax( QKᵀ / √d ) · V</strong></blockquote><p>1. Q @ K.Tcompare everything to everything<strong>.</strong> and this multiplies every Query against every Key, producing a grid of scores. A high score means &quot;this query and this key match well.&quot; If there are 3 words, you get a 3×3 grid: how much each word relates to each other word.</p><p>2. √d scale the scores down. Those scores come from adding up <em>d</em> products. The bigger <em>d</em> is, the bigger the scores get, just from having more terms. If the scores get too large, the next step (softmax) breaks it gets so confident it puts basically all the weight on one word and ignores everything else. Dividing by √d keeps the numbers in a sane range. (If d=4, then √d=2)</p><p>3. softmax turn scores into weights<strong>.</strong> Softmax takes a row of raw scores and turns it into a set of weights that are all positive and add up to 1. So a row like [2.0, 0.0, 1.0] becomes something like [0.6, 0.1, 0.3] clean &quot;spend 60% of my attention here, 10% there, 30% there.&quot;</p><p><strong>4. @ V </strong>blend the Values. Finally, use those weights to take a weighted average of the Value vectors. If a word gives 60% of its attention to word 1, its output is 60% word-1&#39;s Value mixed with the rest. That blended result is the word&#39;s output now enriched with information from the words it cared about.</p><p>Here it is in PyTorch:</p><pre>import torch<br>import torch.nn.functional as F<br>import math</pre><pre>torch.manual_seed(0)<br>d = 8</pre><pre>W_q = torch.randn(d, d)<br>W_k = torch.randn(d, d)<br>W_v = torch.randn(d, d)</pre><pre>def attention(x):<br>    &quot;&quot;&quot;Full attention over all tokens in x. x shape: [seq_len, d]&quot;&quot;&quot;<br>    Q = x @ W_q                                    #turn words into Queries<br>    K = x @ W_k                                    #Keys<br>    V = x @ W_v                                    #Values<br>    scores = Q @ K.T / math.sqrt(d)                #compare + scale<br>    seq = x.shape[0]<br>    mask = torch.triu(torch.ones(seq, seq), diagonal=1).bool()  # causal mask<br>    scores = scores.masked_fill(mask, float(&#39;-inf&#39;))<br>    weights = F.softmax(scores, dim=-1)            #scores → weights<br>    return weights @ V                             #blend the Values</pre><p>Let’s now see the shapes flow:</p><pre>Q = torch.tensor([[1., 0., 1., 0.],<br>                  [0., 1., 0., 1.],<br>                  [1., 1., 0., 0.]])<br>K = Q.clone()<br>V = torch.tensor([[10., 0.],<br>                  [0., 10.],<br>                  [5.,  5.]])</pre><pre>scores = Q @ K.T                 # shape [3, 3]<br># tensor([[2., 0., 1.],<br>#         [0., 2., 1.],<br>#         [1., 1., 2.]])</pre><p>the diagonal is highest (each word matches itself best). Word 1 barely relates to word 2 (score 0) but somewhat to word 3 (score 1) and now let’s scale and softmax it:</p><pre>weights = F.softmax(scores / math.sqrt(4), dim=-1)   # √4 = 2<br># tensor([[0.51, 0.19, 0.31],<br>#         [0.19, 0.51, 0.31],<br>#         [0.27, 0.27, 0.45]])</pre><p>every row adds up to 1. Row 1 says word 1 spends 51% of its attention on itself, 19% on word 2, 31% on word 3. Finally blend the Values</p><pre>output = weights @ V             # shape [3, 2]<br># tensor([[6.60, 3.40],<br>#         [3.40, 6.60],<br>#         [5.00, 5.00]])</pre><p>word 1 attended mostly to itself (whose Value is [10, 0] ), so its output [6.6, 3.4] leans that way but gets pulled partway by the others. The output has the same shape as V three words in, three enriched words out. That&#39;s a full attention pass.</p><p>that mask line in the code, I skipped past it. It&#39;s small, but it&#39;s the whole reason the KV cache works, so let&#39;s go there next.</p><h3>Step 3: the one rule that makes caching possible</h3><p>that mask is a causal mask. So it basically stops each word from looking at words that come after it. so word 1 can look at: word 1, word 2 can look at: words 1 and word 3 can look at: words 1, 2, 3</p><p>each word sees itself and everything to its left, but nothing to its right. In the code, torch.triu(…, diagonal=1) picks out the &quot;future&quot; cells and masked_fill(…, -inf) sets them to negative infinity, so after softmax they become zero weight. The future gets blocked out.</p><p>why block the future? Because when the model is generating text, the future words don’t exist yet it hasn’t written them. So each word genuinely has nothing to its right to look at. And here’s the payoff, the single fact the KV cache is built on, Because a word only depends on itself and the words before it, its Key and Value never change when new words are added later.</p><p>when the model processes “cat” in “the cat sat,” it computes a Key and Value for “cat.” Now suppose the sentence keeps going for another thousand words. Does “cat”’s Key or Value change? No, those new words are all in “cat”’s future, and “cat” can’t see its future. “cat” was frozen the instant it was processed.</p><p>.</p><h3>Step 4: generating text — the naive, wasteful way</h3><p>llms generate autoregressively: predict a word, append it, predict the next, append it, and repeat. The most obvious way to code this is to just run full attention over the entire sequence every time a new word is needed:</p><pre>x = torch.randn(1, d)   # start with one token<br>for step in range(5):<br>    out = attention(x)                       # attention over ALL tokens, every step<br>    next_token = out[-1:].detach()           # take the last word&#39;s output as the next token<br>    x = torch.cat([x, next_token], dim=0)    # append it, sequence grows by 1<br>    print(f&quot;step {step}: sequence length is now {x.shape[0]}&quot;)</pre><p>run that and you’ll see the sequence grow: 1 → 2 → 3 → 4 → 5 → 6.</p><p>it works. But look closely at what attention(x) does on each pass. It recomputes Q, K, and V for <strong>every token in the sequence </strong>including all the old ones. And we just established that the old ones never change. So at step 5, it recomputes tokens 1 through 5 from scratch, redoing the exact same work it already did at step 4, and step 3, and step 2. At token 100, every new word triggers a recompute of ~100 words. At token 1000, ~1000 words. All of it redundant.</p><h3>Step 5: adding the KV cache</h3><p>the fix comes straight out of the causal rule and old tokens’ K and V never change, so compute them once and save them. Then each step, only compute K and V for the <em>new</em> token, append them to a running store (the cache), and attend over the whole store.</p><pre>def attention_cached(x_new, k_cache, v_cache):<br>    q = x_new @ W_q          # [1, d]<br>    k = x_new @ W_k          # [1, d] — only the NEW token&#39;s K<br>    v = x_new @ W_v          # [1, d] — only the NEW token&#39;s V</pre><pre>    if k_cache is None:                              # first token<br>        k_cache, v_cache = k, v<br>    else:<br>        k_cache = torch.cat([k_cache, k], dim=0)     # append new K<br>        v_cache = torch.cat([v_cache, v], dim=0)     # append new V</pre><pre>    scores = q @ k_cache.T / math.sqrt(d)   # new token <br>    weights = F.softmax(scores, dim=-1)     # no mask needed here<br>    out = weights @ v_cache<br>    return out, k_cache, v_cache</pre><p>let’s run it and watch the cache grow:</p><pre>x_new = torch.randn(1, d)<br>k_cache, v_cache = None, None<br>for step in range(6):<br>    out, k_cache, v_cache = attention_cached(x_new, k_cache, v_cache)<br>    x_new = out.detach()<br>    print(f&quot;step {step}: cache holds {k_cache.shape[0]} tokens&quot;)<br># step 0: cache holds 1 tokens<br># step 1: cache holds 2 tokens<br># ...<br># step 5: cache holds 6 tokens</pre><p>so for Naive at every step recomputes K and V for <em>all</em> tokens and for Cached, every step computes K and V for one new token, then appends it with torch.catThe old ones just sit in the cache, reused.</p><p>that torch.cat is the heart of it, it&#39;s literally &quot;glue the new token&#39;s K and V onto the bottom of the cache.&quot; One new row per step.</p><p>why no mask this time? In the naive version we needed a causal mask to stop words seeing the future. Here there’s no future to hide — the new token is the newest thing in the sequence, and it’s supposed to attend to all the tokens before it. There’s nothing ahead of it to mask out. So the mask just disappears.</p><h3>Step 6: prefill and decode</h3><p>so it naturally has two phases and these two words come up constantly in inference work, so they’re worth naming.</p><p><strong>Prefill</strong> is the model reading your prompt. It takes all the prompt’s words at once, computes their K and V in one shot, and fills the cache and out comes the first word of the answer. It’s like reading the entire exam question before writing anything. This phase is compute-heavy but parallel, so it’s fast.</p><p><strong>Decode</strong> is the model writing its answer, one word at a time. Each step computes just the new word’s K and V, appends them to the cache, and produces the next word. It’s like writing the answer word by word. Each individual step is cheap, precisely because the cache is doing the remembering.</p><p>prefill governs TTFT — time to first token (how long before the answer starts).</p><p>decode governs TBT — time between tokens (how fast words stream out after that).</p><p>common source of confusion isn’t this just “encoder / decoder”? No and the shared word “decode” makes it worse. Encoder/decoder describes how a model is built an encoder understands input, a decoder generates output, and modern LLMs like GPT and Llama are decoder-only. Prefill/decode describes two runtime phases that one decoder-only model goes through every time it runs. One is what the model is; the other is what it’s doing.</p><h3>Step 7: does it actually help? Let’s measure</h3><p>let’s time both versions across increasing sequence lengths:</p><pre>import time</pre><pre>def generate_naive(n_tokens, d):<br>    x = torch.randn(1, d)<br>    for _ in range(n_tokens):<br>        out = attention(x)<br>        x = torch.cat([x, out[-1:].detach()], dim=0)<br>    return x</pre><pre>def generate_cached(n_tokens, d):<br>    x_new = torch.randn(1, d)<br>    k_cache, v_cache = None, None<br>    for _ in range(n_tokens):<br>        out, k_cache, v_cache = attention_cached(x_new, k_cache, v_cache)<br>        x_new = out.detach()<br>    return x_new</pre><pre>for n in [100, 200, 400, 800, 1600]:<br>    t0 = time.time(); generate_naive(n, d);  t_naive = time.time() - t0<br>    t0 = time.time(); generate_cached(n, d); t_cached = time.time() - t0<br>    print(f&quot;{n:&gt;4} tokens: naive {t_naive:.4f}s  cached {t_cached:.4f}s  ({t_naive/t_cached:.1f}x)&quot;)</pre><p>the results: tokens naive (s) cached (s) speedup 100 0.0109 0.0044 2.5× 200 0.0421 0.0089 4.7× 400 0.1846 0.0206 9.0× 800 1.5787 0.0658 24.0× 1600 12.4764 0.1247 100.1×</p><p>the speedup keeps climbing as the text gets longer, and <em>that</em> is the real story — not just that caching is faster, but that it pulls away faster and faster.The naive version’s total work is quadratic. At each step it redoes attention over the whole sequence so far, so generating N tokens costs roughly 1 + 2 + 3 + … + N, which grows like N². Double the length and you roughly quadruple the work.</p><p>The cached version is linear. Each step does a fixed amount of work (one new token), so N tokens costs about N. Double the length, double the work. As N grows, N² leaves N in the dust — so the speedup climbs from single digits into the triple digits. And real LLMs generate long responses (hundreds or thousands of tokens), so this isn’t a minor tune-up. It’s the difference between something usable and something that grinds to a halt.</p><h3>Step 8: the catch — the cache isn’t free</h3><p>there’s a cost, and it’s worth being honest about: storing all those keys and values takes memory and the cache grows along several dimensions at once sequence length, number of layers, number of attention heads, and the size of each head. Multiply those together and it adds up fast. For a long conversation, or for many users being served at once, the KV cache can swallow a huge amount of GPU memory single long chat can run into gigabytes.</p><p>that’s exactly the problem PagedAttention (the trick behind vLLM) was built to solve, managing all this cache memory efficiently so a GPU can serve lots of requests without wasting space. So to conclude this KV cache blog, we built attention from scratch, saw how generating text the naive way wastes an enormous amount of effort re-deriving a past that never changes, and fixed it by simply saving that past and reusing it.</p><p>and it all comes down to one idea worth remembering: because attention is causal, a word’s Key and Value never change once they’re computed so you can cache them and stop redoing the work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e34794e3f191" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tiled Matrix Multiplication in CUDA]]></title>
            <link>https://medium.com/@mahareddyroja247/tiled-matrix-multiplication-in-cuda-7fd907dd6b3a?source=rss-e80d331b9776------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fd907dd6b3a</guid>
            <category><![CDATA[matrixmultiply]]></category>
            <category><![CDATA[shared-memory]]></category>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[ncu]]></category>
            <category><![CDATA[tiled-matrix]]></category>
            <dc:creator><![CDATA[AIwithMaha]]></dc:creator>
            <pubDate>Wed, 10 Jun 2026 21:02:57 GMT</pubDate>
            <atom:updated>2026-06-11T20:10:09.241Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post is for beginners. If you already know more advanced GPU techniques, you can skip it. But if you are new to GPU programming, start here, tiling is the one idea everything else is built on.</em></p><p>Why does tiling matter so much? Because almost all the heavy work in modern deep learning is really just matrix multiplication. Attention is matrix multiplication. The layers inside a Mixture-of-Experts (MoE) model are matrix multiplication. Even fast attention kernels like FlashAttention are, at their core, the same matrix-multiply work arranged cleverly. If you ever want to read those advanced kernels and truly understand why they are fast, you first need to understand tiling. It is the foundation. So we will learn it slowly and clearly.</p><p>Now, if you come from CPU programming C or Python, your first instinct for this is probably nested loops a loop over rows, a loop over columns, and an inner loop to add things up. That works, and on a CPU it is often the best you can do, because a CPU has only a handful of cores to spread the work over. A GPU is different. It has thousands of cores, so the number of cores is no longer the thing holding you back. The question becomes, how do we use all of them well?</p><p>That is where this article goes. The first time you write matrix multiplication on a GPU, it works but it is slow. Very slow, for a chip with thousands of cores. The fix is not a faster GPU or more threads. The fix is tiling, a trick that works with the way GPU memory moves data, instead of against it.</p><p>If you have never written CUDA, you can follow every step. We will cover what matrix multiplication is, how a matrix sits in memory, how the GPU splits up the work, and then the simple (naive) kernel and the tiled one.</p><h3>1. What matrix multiplication actually does</h3><p>If you multiply two matrices A and B to get C, each single element of C is a dot product: take one row from A, one column from B, multiply them element by element, and add it all up.</p><pre>C[i][j] = Σ ( A[i][k] × B[k][j] )   for k = 0 .. width-1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*mVdRk78UK0jUAD9Zuwy9qg.gif" /></figure><p><em>One output cell consumes an entire row of A and an entire column of B.</em></p><p>Two things to remember. First, every cell of C is independent. C[0][0] does not need C[0][1]. That is why GPUs are great at this: many workers can compute many cells at the same time. Second, cells next to each other use a lot of the same data. Every cell in row <em>i</em> needs the same row <em>i</em> of A. Keep that in mind — it is the whole reason tiling exists.</p><h3>2. How a matrix lives in memory: flat indexing</h3><p>We draw a matrix as a grid. But computer memory is not a grid. It is one long line of slots, numbered from 0. So the rows are stored one after another, end to end. To find a cell, you turn its (row, col) position into one number, with one formula:</p><pre>index = row × width + col</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*YIq4CTH3HFcbJB2OpluFEw.gif" /></figure><p>Think of an apartment building where all the doors are numbered in one long line. Each floor has width doors. To reach floor <em>i</em>, door <em>j</em>, you pass i × width doors on the floors below you, then walk j doors down your own hallway. That is all row × width + col means. The first part jumps to your floor. The second part walks to your door.</p><p>This is called row-major order, and it is not special to CUDA. Memory is always one line, so languages that store 2D arrays in one contiguous block flatten them this way. NumPy (by default) and C both compute i * width + j behind the scenes. CUDA just makes you write it yourself. (Fortran and MATLAB use the opposite order, column-major, where the formula flips to j * height + i — but everything in this article is row-major.)</p><p>This one formula shows up everywhere in the kernel. When you see A[row * width + k], it&#39;s just flat indexing into A at (row, k).</p><h3>3. The CUDA hierarchy: thread, block, grid</h3><p>A GPU does not run one program that loops over every cell. It starts thousands of cores, running millions of small workers called threads, and they all run the same function at the same time. The threads are grouped in a simple structure. Learn these four words first the code will make sense after.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MvMLRwoE9W9YQjLArmjcww.png" /></figure><p><em>Threads sit inside blocks,blocks sit inside the grid. Each block maps to one tile of the output.</em></p><ul><li><strong>Thread</strong> — the smallest worker. One thread runs the kernel once. Here, one thread computes one cell of C.</li><li><strong>Block</strong> — a group of threads. Key point: threads <em>in the same block</em> can share fast memory and wait for each other. Threads in different blocks cannot. Think of a block as a team at one table. In tiling, one block computes one tile of C.</li><li><strong>Grid</strong> — all the blocks together for one launch. If a block is a table, the grid is the whole room of tables.</li><li><strong>Kernel</strong> — the function (marked __global__) that every thread runs. You write it once. The GPU runs thousands of copies.</li></ul><h3>What the launch line means</h3><p>Everything starts with one line. It looks like a normal function call, plus strange triple angle brackets:</p><pre>MatrixMulCUDA&lt;&lt;&lt;gridSize, blockSize&gt;&gt;&gt;(C, A, B, width);</pre><p>The &lt;&lt;&lt; &gt;&gt;&gt; part exists only in CUDA. It is called the execution configuration. It answers a different question than the parentheses:</p><ul><li>The angle brackets say <em>how many workers, in what groups</em>: launch gridSize blocks, each with blockSize threads.</li><li>The parentheses pass the <em>data</em>: pointers to C, A, B, and the width. Same as any function call.</li></ul><p>Read the line as: “run MatrixMulCUDA as gridSize blocks of blockSize threads, and give them C, A, B, width.&quot; After this line, all the threads start at once. There is no for loop over cells in your CPU code. The many threads <em>are</em> the loop.</p><h3>4. How a thread finds its cell</h3><p>Every thread runs the same kernel code. So the first job of each thread is to find out, which cell of C is mine? It uses three built-in values:</p><ul><li>threadIdx — my seat inside my block (my place at the table).</li><li>blockIdx — which block I am in (which table in the room).</li><li>blockDim — the size of each block (how many seats per table).</li></ul><p>Put them together and a thread finds its global position. It is the same “skip, then walk” idea as flat indexing:</p><pre>int row = blockIdx.y * blockDim.y + threadIdx.y;<br>int col = blockIdx.x * blockDim.x + threadIdx.x;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*r1txsPlZUNCOO7zp1YIROQ.gif" /></figure><p>This is the most common beginner bug, so let’s be very clear. A row is a vertical question. So every part of row must measure something vertical (.y):</p><ul><li>blockIdx.y — how many blocks are above me.</li><li>blockDim.y — how many rows tall each block is (the size of one jump).</li><li>threadIdx.y — my row inside my block.</li></ul><p>Multiply the first two: that skips all the rows in the blocks above you. Add the third: that walks down to your own row. Every piece is vertical, so the answer is a real row.</p><p>Now imagine mixing axes, like blockIdx.x * blockDim.y. That multiplies a <em>sideways</em> position by a <em>vertical</em> height. The result means nothing — like finding your apartment by multiplying which column of the building you are in by how many floors it has. It only looks right when the grid and blocks are perfect squares, because then the x and y numbers happen to match. For most blocks, it gives the wrong row.</p><h3>5. The naive kernel, and why it’s slow</h3><p>Now we can read the simplest kernel. One thread per output cell. Each thread does its own dot product:</p><pre>__global__ void matmulNaive(float* C, float* A, float* B, int width) {<br>    int row = blockIdx.y * blockDim.y + threadIdx.y;   // all .y → vertical<br>    int col = blockIdx.x * blockDim.x + threadIdx.x;   // all .x → horizontal<br><br>    float sum = 0.0f;<br>    for (int k = 0; k &lt; width; ++k) {<br>        sum += A[row * width + k] * B[k * width + col];<br>    }<br>    C[row * width + col] = sum;<br>}</pre><h3>One counter, two directions</h3><p>Look at A[row * width + k] and B[k * width + col]. The counter k sits in a <em>different spot</em> in each:</p><ul><li>In A, row stays fixed and k is the column. So A is read across a row.</li><li>In B, col stays fixed and k is the row. So B is read down a column.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*rptZ8nJlt2B_364CX27opw.gif" /></figure><p>That is why B[k * width + col] is right and B[col * width + k] is wrong. The wrong one walks <em>across</em> a row of B. Row times row is not matrix multiplication. And B[row * width + col] is wrong too: it has no k, so it points at the same single cell on every loop step. A quick self-check: inside the k loop, look at each index and ask, &quot;where is the k?&quot; If k is missing, that read never moves — and it is almost surely a bug. row and col hold still. k moves.</p><h3>Why this kernel wastes the GPU</h3><p>The kernel gives the right answer. But it spends most of its time waiting for memory, not computing. Every thread reads its whole row of A and whole column of B from global memory the GPU’s big, slow main memory. Each read costs hundreds of clock cycles. And here is the waste the thread for C[1][2] and the thread for C[1][3] both read the <em>same row 1 of A</em>, separately, from that slow memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*C3W32RaKXxf_OsvxG9AxNA.gif" /></figure><p>For a matrix of width N, each value of A gets read N times from global memory once for every output column. Same for B. That is a huge amount of repeated, slow traffic. The kernel is memory bound, faster math would not help, because the waiting is the problem.</p><h3>6. The idea behind tiling</h3><p>The GPU has a small, fast scratchpad next to each block, called shared memory. It is about 100× lower latency than global memory, but tiny . Whole matrices do not fit in it. They do not need to. Tiling works like this: load one small square tile of A and one tile of B into shared memory. Let the whole block compute with those fast copies. Then slide to the next tile and keep adding. Each value is loaded from slow memory once per block, then reused many times from fast memory. With a 16×16 tile, every loaded value is reused 16 times, so the number of requests to slow memory drops by about 16×. The math does not get cheaper the waiting does.</p><h3>7. How tiled matmul works, line by line</h3><p>Lets pick a tile width say 16×16 (we draw it as 2×2 here so it is easy to see). Each block computes one tile of C. The work happens in phases, load a tile, wait, compute a partial dot product, wait, slide to the next tile. First the big picture: watch the tiles march across the matrices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*yKDOBm1US3A8S9OUUEhL1Q.gif" /><figcaption><em>A’s tiles march right, B’s tiles march down; the same output tile accumulates across phases.</em></figcaption></figure><h3>Zooming in: what actually happens inside one block</h3><p>The march above hides the real action. Each phase is not just a “slide”. It is a copy from slow global memory into the block’s fast shared memory, then a tiny matrix multiply <em>inside</em> that fast memory. This animation zooms into one block. Watch the data land in the shared tiles, then watch the block multiply them and add the result to its running total.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*tMBYT_Mz9_baEcPUTkMUvw.gif" /></figure><p>This is the heart of tiling. The slow trip to global memory happens <strong>once per tile</strong>. After that, every thread in the block reads those values from shared memory, which is about 100× faster. And remember: while this block works, <em>every other block is doing the same thing on its own tile, at the same time</em>.</p><p>Here’s the kernel. Each part of the code matches one of the four steps.</p><pre>#define TILE_WIDTH 16<br><br>__global__ void matmulTiled(float* C, float* A, float* B, int width) {<br>    __shared__ float tileA[TILE_WIDTH][TILE_WIDTH];   // fast scratchpad<br>    __shared__ float tileB[TILE_WIDTH][TILE_WIDTH];<br><br>    int tx = threadIdx.x, ty = threadIdx.y;<br>    int row = blockIdx.y * TILE_WIDTH + ty;<br>    int col = blockIdx.x * TILE_WIDTH + tx;<br><br>    float sum = 0.0f;<br><br>    for (int phase = 0; phase &lt; width / TILE_WIDTH; ++phase) {<br>        // 1. Load: my row fixed, column slides with phase (A walks right)<br>        tileA[ty][tx] = A[row * width + (phase * TILE_WIDTH + tx)];<br>        // my col fixed, row slides with phase (B walks down)<br>        tileB[ty][tx] = B[(phase * TILE_WIDTH + ty) * width + col];<br><br>        __syncthreads();   // 2. wait: whole tile loaded before anyone reads<br><br>        // 3. Partial dot product over the tile in fast shared memory<br>        for (int k = 0; k &lt; TILE_WIDTH; ++k)<br>            sum += tileA[ty][k] * tileB[k][tx];<br><br>        __syncthreads();   // 4. wait: everyone done before tile is overwritten<br>    }<br><br>    C[row * width + col] = sum;<br>}</pre><h3>The two loops, and why phase joins k</h3><p>Here is the one truly new thing tiling adds: <em>two</em> loops instead of one. The naive kernel had one k loop over the full width. Tiling wraps it in an outer <strong>phase</strong> loop. The phase picks <em>which tile</em> to load (the load lines move over by phase * TILE_WIDTH each time). The inner k loop then runs <em>inside</em> that tile, reading from fast shared memory. Phases pick the tile. k moves inside it.</p><h3>Why there are two barriers</h3><p>The two __syncthreads() calls are required. Each one prevents a different mistake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8WoWrMF-D6yT64f9zzULsA.png" /></figure><ul><li><strong>After loading, before computing:</strong> if a fast thread starts reading the tile before a slow thread finishes writing its piece, it reads garbage. The first barrier makes everyone wait until the tile is fully loaded.</li><li><strong>After computing, before the next load:</strong> if a fast thread starts loading the next tile while a slow thread is still using the current one, it overwrites data that is still needed. The second barrier makes everyone finish before the tile is replaced.</li></ul><p>Both barriers protect shared memory: never read it before it is written, and never overwrite it before everyone is done with it.</p><h3>8. Takeaways, and where to go next</h3><p>Tiling is not a CUDA trick. It is a memory idea that shows up everywhere CPU caches, deep learning kernels, databases. The core idea: move data to fast memory once, then reuse it many times.</p><p>The naive kernel and the tiled kernel give the exact same answer. The only difference is working with the way the hardware moves data<em> </em>and that difference is everything in high-performance computing. If you started this article new to CUDA, you now have the whole chain: a matrix stored as one line in memory, a grid of threads that each find their cell, a dot product that walks across a row and down a column, and a tiling trick that swaps repeated slow reads for fast reuse.</p><p>Shared-memory tiling is where every fast matmul starts, but a production kernel (like the ones inside cuBLAS) stacks several more ideas on top. Each one applies the same “reuse data in even faster storage” principle, just one level deeper in the hardware.</p><p>The best next step is to profile your own kernel and let the hardware tell you which rung to climb next. Pick a tool (NVIDIA’s Nsight Compute is the standard one), measure where your time actually goes, and optimize the bottleneck the profiler points at.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fd907dd6b3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Waiting to Feel Motivated]]></title>
            <link>https://medium.com/@mahareddyroja247/stop-waiting-to-feel-motivated-cd16c92267cf?source=rss-e80d331b9776------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd16c92267cf</guid>
            <category><![CDATA[hard-work]]></category>
            <category><![CDATA[discipline]]></category>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[motivation]]></category>
            <dc:creator><![CDATA[AIwithMaha]]></dc:creator>
            <pubDate>Wed, 13 May 2026 17:47:01 GMT</pubDate>
            <atom:updated>2026-05-13T17:47:01.037Z</atom:updated>
            <content:encoded><![CDATA[<p>Most people are waiting to feel motivated before they start. They want to learn something new, build a habit, or finish a project. But they keep waiting for the right feeling before they begin. That feeling does not always come, and when it does, it goes away fast.</p><p><strong>We Have It Wrong</strong></p><p>The common idea is, first you feel motivated, then you do the work. That is how it looks on social media. Someone wakes up early, feels amazing, and gets everything done. But I feel that is not how it works in real life.</p><p>Action comes first. Motivation comes after. You do not wait until you feel ready. You start, and somewhere along the way, you start to feel like continuing. That feeling is motivation. It showed up because you started, and not the other way around.</p><p><strong>Why Motivation Is Overrated</strong></p><p>Motivation is just a feeling. And feelings change all the time. You feel less motivated when you are tired, stressed, or having a bad day. You cannot build good habits around a feeling that comes and goes.</p><p>People who do good work consistently are not always motivated. They are just disciplined. They show up and do the work even when they do not feel like it. Motivation, when it shows up, is a bonus. It is not what keeps things moving.</p><p><strong>So What Actually Works</strong></p><p>Discipline means doing the same thing every day without waiting for a good mood. It means opening your laptop when you have no ideas. It means doing the task even when it feels boring. Hard work done without motivation still counts. It still adds up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5rb1-oPLJctGPq55XSa9MQ.jpeg" /></figure><p>If you only work when you feel motivated, you will not get much done. Because most days, you will not feel motivated. And the gap between where you are and where you want to be keeps growing. The simple answer is this, decide what needs to be done, and do it whether you feel like it or not. Do it slowly on hard days. Do it badly if you have to. Just do it. After some time, you will notice you actually want to keep going. That is motivation. It came late, after you already started working. That is where it belongs.</p><p><strong>And Start Anyway</strong></p><p>The first hour of doing something you do not feel like doing is always the hardest. It is slow and uncomfortable. But the first hour leads to the second. The second is usually better. By the third, you are in the flow. Motivation does not show you those first hard hours. It only shows you the end result. So stop waiting and start now, even if you are not ready.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd16c92267cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top 5 Mistakes Developers Make When Building LLM Apps]]></title>
            <link>https://medium.com/@mahareddyroja247/top-5-mistakes-developers-make-when-building-llm-apps-fd10b5eb9b7c?source=rss-e80d331b9776------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd10b5eb9b7c</guid>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[AIwithMaha]]></dc:creator>
            <pubDate>Thu, 16 Apr 2026 22:56:12 GMT</pubDate>
            <atom:updated>2026-04-16T22:56:12.170Z</atom:updated>
            <content:encoded><![CDATA[<p>You have built a few LLM projects maybe a chatbot for a college assignment, a summarizer for a hackathon, or a side project you pushed to GitHub. You know how to write a prompt, call an API, and get a response. But when you try to build something that works consistently, things start breaking in ways that are difficult to reproduce and even more difficult to fix.</p><p>These are not beginner mistakes. These are the mistakes that come after you have crossed the basics the ones that block your project for days or ruin your demo at the worst possible time.</p><h3>1. Treating LLM output as deterministic</h3><p>When you first build with an LLM, you directly parse the model’s response as JSON. It works in your local tests. You submit the project. Then someone else runs it, or you run it on a slightly different input, and it crashes because the model returned JSON inside a markdown code block, or added one extra sentence before the object, or changed the format slightly.</p><p>This happens because LLM output is not fixed. The same prompt does not always return the same structure. A small change to your system prompt, a model update from the provider, or a different input can change the output format without any warning. Directly parsing the raw response as JSON without any validation is one of the most common reasons LLM projects break during demos.</p><p><strong>Try doing:</strong></p><p>Use structured outputs when the provider supports them. Anthropic’s API lets you define a tool with a JSON schema and force the model to call it. This way, the response always comes back as a proper dict matching your schema not a raw string you have to clean and parse manually.</p><p>If structured output is not available, validate and retry. Write a small wrapper that catches parse failures, logs what the model actually returned, and retries with a corrected prompt before showing an error. This one change alone will save your demo multiple times.</p><p>Do not assume the model will behave the same way across different calls, model versions, or prompt changes. It will not.</p><h3>2. Iterating prompts without an eval set</h3><p>You change a prompt, test it on the three examples that were failing, see that they pass now, and move on. Two days later something else has broken. You revert the prompt. Now the original examples fail again. You spend a full week going back and forth like this.</p><p>This is exactly what happens when you do prompt development without an evaluation set. You have no way to know if a change is actually better or if it just fixed one thing and broke five others.</p><p><strong>Try doing:</strong></p><p>Before you start changing prompts, build a small eval set 20 to 50 representative inputs with expected outputs, or expected output properties when an exact match does not make sense. Run every prompt change against this full set, not just the examples you are currently looking at.</p><p>For classification tasks, track accuracy, precision, and recall. For open-ended generation, combine automated checks does the output have the required fields, is it within the expected length with a small number of manually reviewed samples.</p><p>You do not need any special framework for this. A simple script that runs your prompt against a JSON file of test cases and prints a pass rate is enough to stop the guesswork. Every bug you find goes into the eval set. By the time you are presenting your project, that set will give you more confidence than any amount of manual testing.</p><h3>3. Ignoring context window management</h3><p>In your early builds, inputs are small and controlled. You send the full conversation history, paste in a document, add some retrieved chunks — everything works fine because your test data is short. But when you test with real data, or when other people start using your app, inputs get longer and you start seeing errors you have never seen before.</p><p>Going over the context limit will throw an error. Getting close to the limit is actually worse the model starts giving less attention to the earlier parts of the context, and your API costs keep increasing with every extra token you are sending unnecessarily.</p><p><strong>Try doing:</strong></p><p>Count tokens before sending the request. Anthropic’s SDK has a token counting method on the messages client that tells you the exact input token count without actually sending the request — use this to trim your input before sending, instead of reacting to errors after the fact.</p><p>For conversation history, use a rolling window or summarization approach. Summarize older turns and keep only the recent exchange as it is. Do not keep adding to the history without any limit.</p><p>For document inputs, split on meaningful boundaries like paragraphs and sections, not on fixed character counts. In RAG pipelines, rank your retrieved chunks and include only the top-k results, not everything that crosses a similarity threshold.</p><h3>4. Using RAG without measuring retrieval quality</h3><p>RAG is one of the first things developers try when they need to give an LLM access to their own data a PDF, a knowledge base, a set of documents. It works well. But most implementations only check whether the final answer looks correct, and completely skip checking whether the retrieval step is actually finding the right content.</p><p>Retrieval failures are not obvious. The model does not tell you it got the wrong chunks it just answers based on whatever it received. You retrieve chunks using cosine similarity, the top results look reasonable, but they are missing the specific information the user asked about. The model either hallucinates or gives a confident answer using the wrong source. Your tests pass because you are testing questions where the retrieval happened to work correctly.</p><p><strong>Try doing:</strong></p><p>Test retrieval separately from generation. For a set of test queries where you know the correct source document, measure two things Recall@k, which tells you whether the correct chunk is present in the top-k results, and MRR (mean reciprocal rank), which measures the average reciprocal of the correct chunk’s rank across all your test queries. Together, these tell you if your retrieval is working before the model even sees the results.</p><p>Fix retrieval issues directly. Useful improvements include splitting documents on meaningful boundaries instead of character counts, adding a reranker like Cohere Rerank or a cross-encoder to sort results before sending to the model, and hybrid search combining dense vector retrieval with BM25 keyword search to cover both semantic and exact-match cases.</p><p>Do not try to fix bad retrieval by changing your generation prompt. Fix the retrieval pipeline directly.</p><h3>5. No error handling for LLM API calls</h3><p>Most LLM projects have no error handling at all around API calls. The API rate limits you during a demo, or a request times out, and the whole application crashes. Or you catch the error but do nothing with it, so the user just sees a blank screen with no explanation.</p><p>LLM APIs fail in ways that are mostly predictable and mostly recoverable but only if your code is written to handle failure.</p><p><strong>Try doing:</strong></p><p>Use exponential backoff for errors that are temporary. Rate limit errors, overload errors, timeouts, and connection failures are all safe to retry. Server errors can usually be retried as well, but not always a 500 error caused by a bad request will keep failing no matter how many times you retry, so make sure you log what you sent when this happens.</p><p>The Anthropic Python SDK retries automatically by default. If you are writing your own retry logic, first disable the SDK’s built-in retries, otherwise both layers will run together and you will lose track of how many times a request is actually being attempted.</p><p>Decide in advance what should happen when a call fails after all retries. Return a cached result, show an error message, or degrade the response gracefully whichever fits your app. The worst thing that can happen is an unhandled exception crashing your entire session in the middle of a presentation.</p><p>Also set request timeouts. Streaming LLM calls can hang for a very long time on connection drops. Make sure timeouts and network errors are handled and retried, not only HTTP status code failures.</p><h3>So Final Conclusion</h3><p>All five of these mistakes come from the same wrong assumption that the LLM works like a simple function call that always returns the same output from a server that never fails.</p><p>It does not work like that. The output is probabilistic, the API is an external service that can fail, and the context window has a hard limit. Once you keep these three things in mind and validate output, measure what actually matters, manage context carefully, and handle errors properly your projects will stop breaking at the wrong moments.</p><p>None of this is LLM-specific. It is the same basic engineering you would apply to any external API.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd10b5eb9b7c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Using To-Do Lists. Put Everything on Your Calendar Instead]]></title>
            <link>https://medium.com/@mahareddyroja247/stop-using-to-do-lists-put-everything-on-your-calendar-instead-26081d8c6e98?source=rss-e80d331b9776------2</link>
            <guid isPermaLink="false">https://medium.com/p/26081d8c6e98</guid>
            <category><![CDATA[time-management]]></category>
            <category><![CDATA[calendar]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[to-do-list]]></category>
            <dc:creator><![CDATA[AIwithMaha]]></dc:creator>
            <pubDate>Sat, 11 Apr 2026 08:33:42 GMT</pubDate>
            <atom:updated>2026-04-11T08:33:42.127Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PUSFylpWzZmoYksJMqYLug.jpeg" /></figure><p>We all have a to-do list. Some of us have three.And every night, we look at that list and feel the same thing, guilt. Half the things didn’t get done. They just rolled over to tomorrow. And then tomorrow and then next week. Until eventually you stop looking at the list altogether because it feels like a record of your failures, not a plan for your day. Here’s the uncomfortable truth, the to-do list isn’t the solution. It’s the problem.</p><h3>What’s wrong with to-do lists?</h3><p>A to-do list has no concept of time. It just sits there, growing longer, with no sense of when anything is actually going to happen. Writing something down feels productive, but it isn’t. Nothing has been scheduled. Nothing has been committed to. You’ve just moved the decision to a later version of yourself who is equally busy and equally overwhelmed.</p><p>Think about it this way. If a meeting is in your calendar, you show up. You don’t reschedule it five times. You don’t forget about it. It happens because it has a time and a place. Your tasks deserve the same treatment.</p><h3>What to do instead</h3><p>Take everything on your to-do list and put it on your calendar as a block of time. Not just meetings, everything. If you need to reply to emails, block 30 minutes at 9am. If you need to finish a report, block two hours on Thursday afternoon. If you’ve been meaning to go for a run, block an hour on Tuesday morning. Give every task a time slot, just like you would a meeting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*MZkk-YmNP-dyjcpPedjM9A.jpeg" /></figure><p>When it’s on the calendar, three things happen. First, you can see exactly how much time you actually have. Most people think they have more time than they do the calendar shows you the truth. Second, you stop making the same decision over and over. The task isn’t “should I do this today?” anymore. It’s already decided. You just follow the plan. Third, when the time comes, you know exactly what you’re supposed to be doing. No more staring at a list wondering where to start.</p><h3>You will also see what doesn’t fit</h3><p>This is the part nobody talks about. When you try to put everything on a calendar, some things won’t fit. And that’s not a problem that’s the point.</p><p>A to-do list lets you pretend you can do 20 things in a day. A calendar shows you the truth, you have maybe 6 to 8 hours of real working time, and most of it is already spoken for. When something doesn’t fit, you have to make a real decision push it to next week, drop it entirely, or ask someone else to handle it. That’s a much healthier relationship with your workload than a never-ending list.</p><h3>A few simple rules to make it work</h3><p>Start each week by looking at what you need to get done and blocking time for each thing before the week begins. Be realistic about how long things take most tasks take longer than you think, so give yourself more time than you think you need. Leave some empty space in your calendar for things that come up unexpectedly, because they always do. And when something new lands on your plate, don’t add it to a list find a spot on the calendar immediately or decide it’s not happening this week.</p><h3>It works because it respects time</h3><p>When you write something into a time slot, you’re saying, this matters enough to deserve a real piece of my day. That small shift changes how you feel about the task. It’s not floating out there in a vague someday. It has a home. You know when it’s happening. And when that time comes, you show up just like you would for any other appointment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26081d8c6e98" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Anthropic Just Made It Way Easier to Build AI Agents]]></title>
            <link>https://medium.com/@mahareddyroja247/anthropic-just-made-it-way-easier-to-build-ai-agents-4960cbd40361?source=rss-e80d331b9776------2</link>
            <guid isPermaLink="false">https://medium.com/p/4960cbd40361</guid>
            <category><![CDATA[managed-agent]]></category>
            <category><![CDATA[production]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[claude]]></category>
            <dc:creator><![CDATA[AIwithMaha]]></dc:creator>
            <pubDate>Sat, 11 Apr 2026 08:07:30 GMT</pubDate>
            <atom:updated>2026-04-11T08:07:30.799Z</atom:updated>
            <content:encoded><![CDATA[<p>Building an AI agent sounds exciting. But anyone who has actually tried to ship one knows the truth most of the work has nothing to do with AI. When you spend weeks setting up containers, managing memory, handling crashes, securing credentials, and debugging why the agent went off the rails. By the time you’ve done all of that, you’ve barely touched the actual product. Anthropic launched <strong>Claude Managed Agents</strong> to fix exactly that.</p><h3>What does it actually do?</h3><p>So you tell Claude what job you want it to do, what tools it can use, and what rules it should follow. Anthropic handles everything else the servers, the memory, the security, the infrastructure.</p><p>You focus on the “what.” They handle the “how it runs.” Before this, building a production ready AI agent meant months of setup work before a single user could try it. With Managed Agents, that same setup can be done in days.</p><h3>Why does this matter?</h3><p>Most AI demos look great. Most AI products break in production. The gap between the two is infrastructure. Can the agent remember what it did two hours ago? What happens when the task takes three hours, not three minutes? Can it safely browse the web or call an API? If something goes wrong mid task, does it recover or start over from scratch?</p><p>Managed Agents handles all of this out of the box. It’s not magic but it removes a huge amount of friction that was slowing teams down.</p><h3>What can you build with it?</h3><p>Companies are already using it for real things. Notion lets users hand off complex tasks like generating slides or writing code without leaving their workspace. Sentry paired their existing bug-detection tool (Seer) with a Claude agent, Seer finds the bug, then Claude writes the fix and opens a pull request, with no one touching it. Asana built what they call AI Teammates, agents that work alongside humans inside project management workflows, picking up tasks and delivering finished work.</p><p>The pattern is the same across all of them, give the agent a goal, let it run, come back to a finished result.</p><h3>Two experimental features worth watching</h3><p><strong>1. Agents that spawn other agents</strong> For big complex tasks, one agent can now spin up other agents to work in parallel. Like a manager assigning subtasks to a team. This is hard to build from scratch having it built into the infrastructure is a real timesaver.</p><p><strong>2. Automatic quality improvement</strong> The system can automatically refine how it responds to get better results. In Anthropic’s own tests, this improved structured file generation success rates by up to 10 points compared to a standard setup. That’s significant.</p><h3>If you use Claude Pro or Max</h3><p>Separately, Anthropic made a change on April 4, if you use third-party agent tools like OpenClaw with your Claude subscription, that no longer works on the flat monthly plan. You’ll need to use pay-as-you-go API access instead.</p><p>Why? Because AI agents run non-stop. A $200/month subscription was being used to run thousands of dollars worth of compute. That math doesn’t work. This change reflects a broader shift subscriptions were designed for humans, and humans sleep. Agents don’t.</p><p>Anthropic is quietly building a full platform not just a model. Claude Code, computer use, Cowork, and now Managed Agents. Each piece makes Claude more capable of doing real work autonomously, not just answering questions.</p><p>For anyone building AI products, this matters. The barrier to shipping a working agent just got a lot lower.</p><p><em>I write about AI, LLM inference, and GPU programming. Follow along if you’re interested in these topics.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4960cbd40361" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[20 CUDA Concepts Explained Simply]]></title>
            <link>https://medium.com/@mahareddyroja247/20-cuda-concepts-explained-simply-b23b89918ba0?source=rss-e80d331b9776------2</link>
            <guid isPermaLink="false">https://medium.com/p/b23b89918ba0</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[cuda-concepts]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[AIwithMaha]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 14:21:57 GMT</pubDate>
            <atom:updated>2026-04-01T14:22:49.870Z</atom:updated>
            <content:encoded><![CDATA[<p>You train a model, move it to device = “cuda” , and suddenly everything runs faster. It works. But most of us don’t really know what actually changed when we did that.</p><p>I used CUDA for a while before I properly understood what was happening behind the scenes. Tutorials usually tell you what to do, but not what the GPU is actually doing when your code runs.</p><p>If you use PyTorch, TensorFlow, or run models on a GPU, this will help you understand what is really happening when you press run.</p><h3>1. What CUDA actually is</h3><p>CUDA stands for Compute Unified Device Architecture. What it actually is, a way to write code that runs on your GPU instead of your CPU. Nvidia built it, which is why CUDA only works on Nvidia cards. When people say “GPU programming,” they usually mean CUDA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*KZBi3OtiRJcXA0oUFV04og.avif" /></figure><p>Think of CUDA as a language your GPU understands like how you speak to a contractor in their own terms to actually get work done. Without it, your GPU is just sitting there, expensive and bored.</p><h3>2. CPU vs GPU the real difference</h3><p>Your CPU is a few very smart workers. It can handle complicated tasks, make decisions, deal with surprises. But there are only a handful of them typically between 8 and 32 cores on a modern machine.</p><p>Your GPU is thousands of very simple workers. Each one isn’t particularly clever, but when you have 10,000 of them doing the same thing at the same time, you get things done fast.</p><h3>3. The kernel</h3><p>In CUDA, a kernel is just a function that runs on the GPU.</p><p>You write it in C++ with a special tag (__global__), call it from your regular code, and it runs on the GPU instead of the CPU. The word &quot;kernel&quot; sounds fancy but it just means &quot;a GPU function.&quot;</p><p>Think of it like a memo you send to thousands of workers at once same instructions, everyone gets a copy, everyone starts working. The memo is the kernel.</p><p>In frameworks like PyTorch, you usually don’t see kernels directly every tensor operation like addition, multiplication, or convolution launches CUDA kernels under the hood.</p><h3>4. Threads</h3><p>When a kernel runs, it doesn’t run once. It runs thousands of times simultaneously each run is called a thread.</p><p>Think of a thread as one worker doing one small piece of the job. If you’re adding two arrays together, each thread adds one pair of numbers. Ten thousand threads, ten thousand additions, all at once.</p><p>Like a factory assembly line where every worker has exactly one bolt to tighten nobody waits for anyone else, and the whole car gets built at once.</p><pre>x = torch.randn(10000, device=&quot;cuda&quot;)<br>y = torch.randn(10000, device=&quot;cuda&quot;)<br><br>z = x + y</pre><p>This simple addition launches thousands of CUDA threads behind the scenes. Each thread is responsible for adding one element of the tensors.</p><h3>5. Blocks</h3><p>Threads are organized into groups called blocks.</p><p>Why not just have one giant pile of threads? Because the GPU hardware has limits on how many threads can share resources. Blocks are how CUDA manages that. Each block runs on one chunk of the GPU and the threads inside it can talk to each other. Threads in different blocks can’t.</p><h3>6. The grid</h3><p>A grid is the full collection of blocks running your kernel.</p><p>So the structure is grid → blocks → threads. Your kernel launches a grid. The grid contains blocks. Each block contains threads. The whole grid runs your kernel, just divided up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*XC78pgG5lIVwcPQoqTX6CQ.png" /></figure><p>Think of it like a city. The grid is the whole city, blocks are the neighbourhoods, and threads are the individual houses. Everything is organised, everyone has an address, nothing overlaps.</p><h3>7. Thread IDs</h3><p>If every thread is running the same function, how does each one know which piece of data to work on?</p><p>It uses its own ID. Every thread knows its position its block number and its position within that block. Using those numbers, each thread figures out which element it’s responsible for. Thread 0 handles element 0, thread 1 handles element 1, and so on.</p><h3>8. GPU memory (device memory)</h3><p>Your GPU has its own memory, completely separate from your regular RAM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*h-k_hmYRV94E12JaiXUdjg.png" /></figure><p>Before your GPU can work on data, that data has to be copied from host memory to device memory. When it’s done, you copy results back. Forgetting this step or doing it inefficiently is one of the most common reasons GPU code runs slow.</p><h3>9. Shared memory</h3><p>Inside each block, threads can access a small, very fast piece of memory called shared memory.</p><p>Think of it like a whiteboard in a room full of workers. Any worker in that room can read and write on it. It’s much faster than going to the main storage room (device memory). If your threads need to share results or reuse data, putting it in shared memory first can make a big difference.</p><p>It’s the difference between a team writing answers on a shared whiteboard versus each person running to the filing cabinet every time they need a number. Same information, very different speed.</p><h3>10. Global memory</h3><p>Global memory is the main storage on your GPU the device memory everyone can access.</p><p>It’s big but slow compared to shared memory. When your GPU reads data from global memory and it’s not in a good pattern, it wastes a lot of time. Good CUDA programming is often about minimizing how much you touch global memory.</p><h3>11. Memory coalescing</h3><p>When threads read from global memory, the GPU doesn’t fetch one piece of data at a time. It fetches a whole chunk. If your threads are reading memory in a nice, sequential pattern thread 0 reads element 0, thread 1 reads element 1 the GPU can grab it all in one go. That’s coalesced access, and it’s fast. If your threads are reading scattered memory locations, the GPU makes many separate trips. Very slow.</p><h3>12. Warps</h3><p>Your GPU doesn’t run every thread independently it groups them into sets of exactly 32 called warps. All 32 threads in a warp execute the same instruction at the same time, they move in lockstep. This is why branching (if/else statements) in GPU code is expensive. If half your warp goes one way and half goes the other, the GPU has to run both paths and ignore half the results each time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*cOsfyLfeOoAW-6-unRkCSw.png" /></figure><p>This is one reason why vectorized tensor operations are preferred over Python loops in PyTorch they produce more uniform GPU work and better warp efficiency.</p><h3>13. Occupancy</h3><p>Occupancy is how busy your GPU actually is.If you’re only using half the threads available, your GPU is half idle. Higher occupancy means more threads running at once, which usually means faster code up to a point.</p><p>Occupancy isn’t everything, but if yours is very low, that’s worth investigating.</p><h3>14. Memory transfer bottleneck</h3><p>Moving data between your CPU and GPU takes time. A lot of time, relatively speaking. If your GPU kernel finishes in 1 millisecond but moving data back and forth takes 10 milliseconds, your “GPU speedup” is actually a slowdown. Good GPU code tries to move data as infrequently as possible and do as much work as possible once it’s on the GPU.</p><p>This is why ML frameworks try to keep tensors on the GPU as long as possible instead of moving them back and forth every operation. Accidentally calling .cpu() or mixing CPU and CUDA tensors is one of the most common real-world performance bugs.</p><p>Profiling often shows GPU kernels taking microseconds while memory transfers take milliseconds.</p><h3>15. Parallelism vs concurrency</h3><p>These sound similar but they’re different. Concurrency is when multiple tasks make progress at the same time by taking turns. Parallelism is when multiple tasks literally run at the exact same moment. Your GPU does real parallelism thousands of threads running simultaneously, not taking turns. That’s what makes it powerful for the right jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w6vU_9iblwQDa6FfZvxLlw.png" /></figure><p>Concurrency is one chef jumping between three dishes on the stove. Parallelism is three chefs, each with their own stove, cooking simultaneously. The food gets done much faster but only if you have three chefs.</p><h3>16. CUDA streams</h3><p>By default, CUDA operations happen one after another copy data, run kernel, copy results back.</p><p>Streams let you overlap these steps. While one batch of data is being processed by the GPU, you can be copying the next batch over. While results are being sent back to the CPU, you can already be running the next kernel. Done well, streams keep everything busy and cut idle time significantly.</p><h3>17. Synchronization</h3><p>When you launch a kernel, your CPU doesn’t wait for it to finish. It just kicks it off and moves on.</p><p>That’s usually fine. But sometimes you need to make sure the GPU is done before the CPU continues for example, before reading results. That’s called synchronization. In PyTorch you’ll see torch.cuda.synchronize(). It&#39;s basically the CPU saying &quot;hang on, let me wait for the GPU to catch up.&quot;</p><p>Think of it like texting someone to do a task, then calling them before they’ve replied to ask if it’s done. Synchronization is waiting for the reply before you pick up the phone.</p><h3>18. FLOPS</h3><p>FLOPS stands for floating point operations per second. It’s how GPU performance is measured.</p><p>One FLOP is one simple math operation an addition, a multiplication. A modern GPU can do hundreds of teraFLOPS — hundreds of trillions of operations per second. When people compare GPUs, FLOPS is one of the main numbers they argue about.</p><h3>19. Tensor cores</h3><p>Regular GPU cores can do one multiplication at a time. Tensor cores found in newer Nvidia GPUs can do a small matrix multiplication in a single operation. Since neural networks are essentially endless matrix multiplications, tensor cores made a massive difference. When you use mixed precision training (float16 instead of float32), you’re often doing it to take advantage of tensor cores specifically.</p><p>Think of regular cores as a calculator one sum at a time. Tensor cores are a spreadsheet that fills in an entire table in one keystroke. Same math, wildly different speed.</p><p>When you enable automatic mixed precision (torch.cuda.amp), PyTorch is often trying to route operations to tensor cores automatically.</p><h3>20. cuDNN</h3><p>CUDA is the foundation, but most people never write raw CUDA code.</p><p>cuDNN is Nvidia’s library of pre-built, highly optimized GPU operations for deep learning convolutions, activations, pooling, and more. When PyTorch runs a convolution on your GPU, it’s almost certainly calling cuDNN under the hood.</p><p>Most of the speed you experience in PyTorch comes not from CUDA alone, but from the years of optimization baked into cuDNN. The speed isn’t magic its accumulated engineering.</p><p>You now know more about what’s happening inside your GPU. None of this requires you to write CUDA code yourself but knowing the concepts changes how you read error messages, understand performance issues, and think about what your model is actually doing.</p><p>Next time you write <strong>model.to(“cuda”) </strong>, you&#39;re launching thousands of threads, scheduling warps, moving memory across buses, and invoking libraries Nvidia engineers spent a decade optimizing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b23b89918ba0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Transformers Faster: GPU Memory Optimization for Matrix Multiplication]]></title>
            <link>https://medium.com/@mahareddyroja247/making-transformers-faster-gpu-memory-optimization-for-matrix-multiplication-48736c9de1a4?source=rss-e80d331b9776------2</link>
            <guid isPermaLink="false">https://medium.com/p/48736c9de1a4</guid>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[memory-optimization]]></category>
            <category><![CDATA[matrix-multiplication]]></category>
            <dc:creator><![CDATA[AIwithMaha]]></dc:creator>
            <pubDate>Thu, 26 Mar 2026 22:54:48 GMT</pubDate>
            <atom:updated>2026-03-26T22:54:48.682Z</atom:updated>
            <content:encoded><![CDATA[<p>Every time a large language model generates a token, thousands of matrix multiplications execute on the GPU. The difference between a model that responds in seconds versus one that takes minutes often comes down to a single factor memory bandwidth utilization.</p><p>This article explores how memory optimization transforms matrix multiplication performance, using techniques that power libraries like cuBLAS and CUTLASS. The journey moves from understanding why naive implementations fail to implementing tiled algorithms that achieve significant speedups. Matrix multiplication provides the perfect case study because it sits at the heart of every neural network.</p><h3>Matrix Multiplication Challenge in AI</h3><p>Matrix multiplication appears everywhere in modern AI systems. Every transformer layer performs multiple matrix multiplies. Attention mechanisms compute queries, keys, and values through matrix operations. Feed forward networks rely entirely on matrix multiplication. Matrix multiplication operations dominate the computational workload in transformer models.</p><p>A 10x improvement in matrix multiply performance means 10x more throughput for the same hardware, or serving 10x more users with the same infrastructure.</p><h3>Memory Bottleneck</h3><p>To appreciate why memory matters more than raw compute power, consider the NVIDIA A100.</p><p>The A100 delivers 19,500 GFLOPS for single precision operations. That’s nearly 20 trillion calculations per second. However, the global memory bandwidth sits at 1,555 GB/second.</p><p>For matrix multiplication, the kernel performs 0.25 operations per byte of data accessed. Given the 1,555 GB/second memory bandwidth and 0.25 FLOP/B ratio, the maximum achievable performance becomes 1,555 × 0.25 = 389 GFLOPS.</p><p>This represents only 2% of the GPU’s peak computational capacity. The GPU cores sit idle most of the time, waiting for data to arrive from slow global memory.</p><h3>CUDA Memory Hierarchy</h3><p>CUDA devices provide several types of memory, each with different characteristics.</p><p><strong>Global Memory</strong> resides in off-chip DRAM. It offers gigabytes of capacity but suffers from high access latency (hundreds of cycles) and limited bandwidth. This is where your input matrices and output results live.</p><p><strong>Registers</strong> provide the fastest memory available. Each thread gets its own private registers with minimal access latency typically just a few cycles. The catch is that registers cannot be shared between threads.</p><p><strong>Shared Memory</strong> occupies the middle ground. It resides on-chip within each SM, providing much lower latency and higher bandwidth than global memory. All threads within a block can access it. Each SM typically has only tens of kilobytes of shared memory (164 KB per SM on the A100).</p><p>The bandwidth difference is striking. Register access achieves at least two orders of magnitude higher bandwidth than global memory. Shared memory falls in between but still provides substantially higher bandwidth than global memory.</p><h3>Matrix Multiplication Approach</h3><p>Here’s the straightforward way to implement matrix multiplication on the GPU:</p><pre>__global__ void matrixMulNaive(float* M, float* N, float* P, int Width) {<br>    int row = blockIdx.y * blockDim.y + threadIdx.y;<br>    int col = blockIdx.x * blockDim.x + threadIdx.x;<br>    <br>    if (row &lt; Width &amp;&amp; col &lt; Width) {<br>        float Pvalue = 0;<br>        for (int k = 0; k &lt; Width; ++k) {<br>            Pvalue += M[row * Width + k] * N[k * Width + col];<br>        }<br>        P[row * Width + col] = Pvalue;<br>    }<br>}</pre><p>Each thread computes one element by iterating through one row of M and one column of N. For a matrix of dimension Width, this requires 2 × Width global memory accesses. Performance maxes out around 389 GFLOPS on the A100, leaving 97% of computational capacity unused.</p><h3>Tiling Breakthrough</h3><p>The solution lies in data reuse through tiling.Consider computing a 2×2 output tile. Notice the overlap: M[0,:] gets used twice (for P[0,0] and P[0,1]). For a 16×16 tile, each element of M loaded into shared memory gets reused 16 times.</p><p>Tiling partitions the input matrices into smaller subsets called tiles. The algorithm loads these tiles into shared memory where threads can access them repeatedly at high bandwidth.</p><p>The process works in phases. In each phase, threads collaboratively load one tile of M and one tile of N into shared memory. All threads then use these shared tiles to compute partial results. After synchronization, the next phase loads new tiles and accumulates more partial results.</p><h3>Implementing Tiled Matrix Multiplication</h3><p>Here’s the tiled kernel:</p><pre>#define TILE_WIDTH 16</pre><pre>__global__ void matrixMulTiled(float* M, float* N, float* P, int Width) {<br>    __shared__ float Mds[TILE_WIDTH][TILE_WIDTH];<br>    __shared__ float Nds[TILE_WIDTH][TILE_WIDTH];<br>    <br>    int bx = blockIdx.x;  int by = blockIdx.y;<br>    int tx = threadIdx.x; int ty = threadIdx.y;<br>    <br>    int row = by * TILE_WIDTH + ty;<br>    int col = bx * TILE_WIDTH + tx;<br>    <br>    float Pvalue = 0;<br>    <br>    for (int ph = 0; ph &lt; Width/TILE_WIDTH; ++ph) {<br>        // Collaborative loading into shared memory<br>        Mds[ty][tx] = M[row * Width + ph*TILE_WIDTH + tx];<br>        Nds[ty][tx] = N[(ph*TILE_WIDTH + ty) * Width + col];<br>        <br>        __syncthreads();<br>        <br>        // Compute using shared memory<br>        for (int k = 0; k &lt; TILE_WIDTH; ++k) {<br>            Pvalue += Mds[ty][k] * Nds[k][tx];<br>        }<br>        <br>        __syncthreads();<br>    }<br>    <br>    P[row * Width + col] = Pvalue;<br>}</pre><p>The kernel operates in phases. Each thread loads one element from M and one from N into shared memory. The first __syncthreads() ensures all threads complete loading before computation begins. The inner loop performs partial dot product calculations using only shared memory accesses. The second __syncthreads() ensures all threads finish using the current tiles before loading new data.</p><h3>Performance Analysis</h3><p>For TILE_WIDTH = 16, tiling reduces global memory traffic by a factor of 16.</p><p>The naive implementation achieved 0.25 FLOP/B. With 16x less memory traffic, the tiled version achieves 0.25 × 16 = 4 FLOP/B.</p><p>For the A100 with 1,555 GB/second memory bandwidth: Maximum Performance = 1,555 × 4 = 6,220 GFLOPS</p><p>This represents 32% of the GPU’s peak throughput a dramatic improvement from the 2% utilization of the naive approach.</p><h3>Real World Impact</h3><p>These techniques aren’t academic exercises. NVIDIA’s cuBLAS library implements highly optimized matrix multiplication using these exact principles. Every deep learning framework (PyTorch, TensorFlow, JAX) relies on cuBLAS for its matrix operations.</p><p>Flash Attention applies similar tiling principles. The algorithm tiles the attention computation across sequence length, loading tiles of queries, keys, and values into shared memory. This significantly reduces memory traffic for the attention operation.</p><p>The fundamental insight is the same: identify data reuse opportunities, partition the computation into tiles that fit in fast memory, and orchestrate data movement to maximize reuse. Flash Attention achieves 2–4x speedups for attention operations by applying these memory optimization principles.</p><h3>Boundary Checks</h3><p>Real world matrices rarely have widths that are perfect multiples of TILE_WIDTH. The solution requires adding conditional checks during the tile loading phase:</p><pre>if (row &lt; Width &amp;&amp; (ph*TILE_WIDTH + tx) &lt; Width) {<br>    Mds[ty][tx] = M[row * Width + ph*TILE_WIDTH + tx];<br>} else {<br>    Mds[ty][tx] = 0.0;<br>}</pre><p>Threads that would access out-of-bounds elements write zero to shared memory instead. This ensures the dot product computation remains correct.</p><h3>Conclusion</h3><p>Matrix multiplication performance fundamentally depends on memory optimization, not raw computational power. Tiling transforms memory bound operations by exploiting data reuse through shared memory.</p><p>For matrix multiplication on the A100 GPU, tiling with 16×16 tiles improves performance from 389 GFLOPS (2% of peak) to 6,220 GFLOPS (32% of peak).</p><p>These principles extend beyond matrix multiplication to nearly all GPU computing. Understanding memory hierarchy, identifying reuse opportunities, and managing data movement forms the core skillset for GPU optimization.</p><h3>References</h3><p>[1] Kirk, D. B., &amp; Hwu, W. W. (2016). <em>Programming Massively Parallel Processors: A Hands-on Approach</em> (3rd ed.). Morgan Kaufmann. Chapter 5: Memory Architecture and Data Locality.</p><p>[2] NVIDIA Corporation. (2020). <em>NVIDIA A100 Tensor Core GPU Architecture</em>. NVIDIA Whitepaper. <a href="https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf">https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf</a></p><p>[3] Dao, T., Fu, D. Y., Ermon, S., Rudra, A., &amp; Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. <em>Advances in Neural Information Processing Systems</em>, 35, 16344–16359.</p><p>[4] NVIDIA Corporation. (2023). <em>CUDA C++ Programming Guide</em>. <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/">https://docs.nvidia.com/cuda/cuda-c-programming-guide/</a></p><p>[5] NVIDIA Corporation. <em>cuBLAS Library User Guide</em>. <a href="https://docs.nvidia.com/cuda/cublas/">https://docs.nvidia.com/cuda/cublas/</a></p><p>[6] NVIDIA Corporation. <em>CUTLASS: CUDA Templates for Linear Algebra Subroutines</em>. <a href="https://github.com/NVIDIA/cutlass">https://github.com/NVIDIA/cutlass</a></p><p><em>Part of the GPU programming journey through “Programming Massively Parallel Processors” by Kirk and Hwu.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48736c9de1a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>