<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aksh Garg on Medium]]></title>
        <description><![CDATA[Stories by Aksh Garg on Medium]]></description>
        <link>https://medium.com/@aksh-garg?source=rss-bafc0a1d440a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OY2kCIbiKHX3y9S-JEZQJg.jpeg</url>
            <title>Stories by Aksh Garg on Medium</title>
            <link>https://medium.com/@aksh-garg?source=rss-bafc0a1d440a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:36:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aksh-garg/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Shard: On the Decentralized Training of Foundation Models]]></title>
            <link>https://aksh-garg.medium.com/shard-on-the-decentralized-training-of-foundation-models-2fd982176724?source=rss-bafc0a1d440a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fd982176724</guid>
            <category><![CDATA[decentralized-training]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[foundation-models]]></category>
            <category><![CDATA[model-parallelism]]></category>
            <category><![CDATA[heterogeneous-computing]]></category>
            <dc:creator><![CDATA[Aksh Garg]]></dc:creator>
            <pubDate>Mon, 20 May 2024 17:33:49 GMT</pubDate>
            <atom:updated>2024-05-20T17:33:49.920Z</atom:updated>
            <content:encoded><![CDATA[<h3>Macro Thesis</h3><p>Within the past year, we have seen major macro shifts in AI, crypto, hiring, and more. Each of these movements has incited different levels of impulse response from the public. Here, I hope to dive into <strong>decentralized training of AI models</strong>, a topic, which I personally think has received a much weaker response than it merits. In the next couple of years as communication and compression mechanisms improve, I see two things happening: (1) either I’m wrong and the world continues developing huge centralized clusters or (2) <strong>the hypothesis ends up being correct, and we have a major unlock in the amount of available compute for training AI models.</strong></p><p>Below, I’ll cover why decentralized AI training makes a lot of sense, discuss some concrete challenges, which currently block it, and then end with a reading</p><h3>Outline</h3><ul><li><strong>Macro Thesis</strong></li><li><strong>Why Decentralized</strong></li><li><strong>What’s holding us back + Exciting Research Shifts</strong></li><li><strong>What’s Next?</strong></li></ul><h3><strong>Why Decentralized?</strong></h3><ol><li><strong>Unlocks a new kind of scale</strong></li></ol><p>It’s clear that we are going to keep wanting to train larger and larger models. And as these models scale, so do the data needs for them. However, I would argue that the world isn’t facing a <strong>GPU supply problem,</strong> rather it’s facing a GPU <strong>under-utilization</strong> <strong>problem</strong>.</p><p>Let’s run a quick thought experiment. Most modern-estimates agree that around around <strong>48,000 A100s</strong> were used to train Llama3 and <strong>25,000 A100s</strong> were used to train GPT4 with estimate for training times range from 3 months-12 years. First things first, these are huge jobs and require very careful systems-level engineering from the OpenAI and Meta/FAIR teams to accomplish.</p><p>Now, let’s consider a different market: the Ethereum blockchain. At its peak GPU usage in 2022, there were <strong>20–40 million GPUs </strong>on the chain, 3 orders of magnitude more than the amount of GPUs used to train Llama 3, one of the largest open source training experiments of all time.</p><p>If 1000x available compute doesn’t excite you, I don’t know what will. But as the final nail on the coffin, imagine a world where you train machine learning jobs on your Mac and iPhone. As of 2023, there were over 2 billion apple devices globally. And with major improvements in their GPUs for ML training with MLX and MPS for example, they are becoming serious contenders for ML training and inference. Even tapping into a tiny fraction of this widely distributed network of computation would be game-changing.</p><p><strong>2. Nicer Margins</strong></p><p>Most people I know like making more and paying less. What if I said decentralization unlocks both?</p><p>First, let’s think about why people make more. There are incredible arbitrage opportunities between crypto and AI when it comes to GPUs. On leading chains a Nvidia 4090 might make you $5/day after electricity costs. For AI workloads, the going price of these GPUs is ~$17/day. So if you’re a miner and looking to make extra cash, my very unbiased opinion might be to not mine at all and instead add your compute to Together.ai or vast.ai and start making <strong>a lot </strong>more money.</p><p>Second, let’s consider how costs decrease. Imagine you wanted to train a job, which needs 70 GB of RAM. Normally, I would go get a A100 to avoid the dreaded OOM errors. However, more recently my approach has been slightly different. 3x RTX 4090s stitched together have a much better FLOP/dollar ratio than training a job on these larger devices. Luckily for them, Nvidia has succeeded in preventing people from easily training on the highly efficient 4090s for business reasons. However, this decentralized mechanism would allow us to erect new, much more efficient compound systems for training.</p><p><strong>3. New Marketplaces</strong></p><p>What does your Mac do when you go to sleep? I guess not much. Now imagine being paid for people in India to train ML models on your computer as you sleep. Maybe a bit scary from a security perspective, but once those kinks are ironed out, I personally am a big fan of earning extra money with zero extra work.</p><p>When we consider that most jobs today are memory and not FLOPs bound, it’s quite interesting to think about how 5 16GB Macs connected with fully-sharded-data-parallel support the same memory model as an A100. And once again, the economics are very, very compelling: we’re looking at ~$2/hr for a A100 vs ~$0.60/hr for a fleet of 5 Macbooks. And if you think $0.12/hr per Macbook might not be enough of an incentive for people to let you train on their computers, that’s over $1,000 per year of passive earnings, enough to pay for your mac in under 2 years.</p><h3>What’s holding us back + Exciting Research Shifts</h3><p>Decentralized training is really, really hard. I’ll try to decompose this into 2 sub-problems: (1) decentralized training are tricky and (2) networking naturally bottlenecks speed.</p><ol><li>Let’s start with <strong>decentralized training methods</strong>.</li></ol><p>Most individual provisioners of GPUs don’t have huge clusters A100s or H100s connected with Nvlink and InfiniBand. Instead, we’re looking at a heterogenous network of 4090s, Macbooks, etc. scattered across the globe. How do distribute a huge model across these small nodes? Enter data/model parallelism methods.</p><p>Given this is meant to be a macro exposition, I’ll keep the technical details brief. However, as a broad overview, you can take a large model and shard into small pieces that can fit on conventional hardware. Below we illustrate how this might work in a fully-shaded data parallel scheme. Essentially, let’s say we have a huge 80GB model. We can chunk the total weights in the model into four sub-pieces each of size 20GB, which can then be scattered across 4 RTX 4090s. Woohoo!! For interested readers, training works by gathering the required weights for a specific layer, executing on it, and then dropping the weights. Thus at any given moment in time, we save on large memory footprints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cZoc3b_ROz1lO8r3Q4bTGw.png" /><figcaption>Simple Representation of Fully Sharded Data Parallel</figcaption></figure><p>Another design called pipeline parallel, is even more intuitive (and some of you might have already seen it on the <a href="https://x.com/AkshGarg03/status/1790824537904554351">Shard demo</a>!). Basically, we just sequentially partition layers of a network onto different devices and they different devices are responsible for different parts of the overall model pipeline — hence the name pipeline parallel. Each device simply communicates to the ones right before and after it in the pipeline, making for an elegant partition overall. Unfortunately, coordinating the bubble and debugging it tends to tricky, so I might suggest starting with fully sharded data parallel, which has more native library support.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ClJz-_KcTja9thPL33N9EA.png" /><figcaption>Representation of pipeline parallel. We partition a large model into subparts that each live on a single device.</figcaption></figure><p>2. <strong>Communication Bottlenecks.</strong> Activations for larger models are on the order of several GBs. This doesn’t matter much when we train on centralized hardware like A100s with internal bandwidths of <strong>2TB/s; </strong>however, in a decentralized space, we are relying upon the internet, which, albeit powerful, is much slower.</p><p>There’s been a ton of really interesting work in this domain such as <a href="https://arxiv.org/abs/2311.08105">Distributed Low-Communication Training (DiLoCo)</a>, <a href="https://arxiv.org/abs/2403.10616">DiPaCo: Distributed Path Composition</a>, <a href="https://openreview.net/pdf?id=w2Vrl0zlzA">Cocktail-SGD</a>, and <a href="https://arxiv.org/pdf/2206.01299">Fine-Tuning Foundation Models with Slow Network Speeds</a>. Most of these efforts revolve around a mixture of gradient and activation compression, quantization, and random sparsification, with the ultimate goal of reducing the amount of information that has to be shared over these weaked bandwidth internet connections. There’s been very promising strides — for example, Cocktail SGD found only a ~1.2 fold reduction in training speed relative to data-center networks with a 500mbps connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*B4jD_RLGY3cBg1ZnHabz7w.png" /><figcaption>Figure from CocktailSGD. We see that with 500Mbps connections, there’s very little reduction in overall throughput.</figcaption></figure><h3>What’s Next?</h3><p>To be honest, the road is a bit unclear. I think decentralized computing is still 2–3 years from being a reality, but that makes it one of those early opportunities to tap into. With Shard, we’re hoping to build out some early tools for easily integrating these solutions and are very excited about the future to come.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fd982176724" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gemma-10M Technical Overview]]></title>
            <link>https://aksh-garg.medium.com/gemma-10m-technical-overview-900adc4fbeeb?source=rss-bafc0a1d440a------2</link>
            <guid isPermaLink="false">https://medium.com/p/900adc4fbeeb</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[inference-optimization]]></category>
            <category><![CDATA[gemma]]></category>
            <category><![CDATA[infini-attention]]></category>
            <dc:creator><![CDATA[Aksh Garg]]></dc:creator>
            <pubDate>Thu, 09 May 2024 15:02:00 GMT</pubDate>
            <atom:updated>2024-05-09T17:55:24.344Z</atom:updated>
            <content:encoded><![CDATA[<h3>Motivation</h3><p>Transformers, although powerful, are very compute-intensive, scaling <strong>O(n²) in time</strong> and <strong>memory</strong> with the number of tokens. This makes scaling context-windows for modern LLMs very challenging. In Gemma-10M, we merge insights from recurrent neural networks with local attention blocks, to capture long-term knowledge retention with <strong>O(1) memory</strong> and <strong>O(n) time. </strong>Thus, our solution allows models to expand to arbitrary context-sizes.</p><p>Check us out on:</p><ul><li><strong>Github</strong>: <a href="https://github.com/mustafaaljadery/gemma-10M-mlx/">https://github.com/mustafaaljadery/gemma-10M-mlx/</a></li><li><strong>HuggingFace</strong>: <a href="https://huggingface.co/mustafaaljadery/gemma-10M-safetensor">https://huggingface.co/mustafaaljadery/gemma-10M-safetensor</a></li></ul><h3>Outline</h3><ol><li>The challenge with standard transformers</li><li>Recurrent Attention</li><li>Infini-Attention</li><li>Incremental context-size training</li></ol><h3>The Challenge with Standard Transformers</h3><p>The biggest bottleneck for expanding is the expanding size of the KV-cache, which involves storing the Key-Value pairs in the attention table from prior tokens before computing the attention on the latest token. Not doing so increases the computational cost cubicly, making it almost necessary for longer-sequences. The GIF below illustrates this idea of a KV-cache.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uyuyOW1VBqmF5Gtv225XHQ.gif" /><figcaption>KV-Cache Avoid Re-computation. GIF from <a href="https://medium.com/@joaolages/kv-caching-explained-276520203249">https://medium.com/@joaolages/kv-caching-explained-276520203249</a></figcaption></figure><p>However, storing this cache is expensive — specifically, quadratically so. Especially, when computing attention on contexts lengths of 1M, we are looking at 1,000,000 x 1,000,000 = 12 trillion entries, which we can’t fit in conventional hardware.</p><h3>Recurrent Local Attention</h3><p>Borrowing insights from Transformer-XL and recurrent neural networks, we instead compute attention on local 2048 x 2048 blocks and pass that into an MLP to store a state vector. This state vector is then passed as an additional parameter into the next local attention block that we compute, with the goal of preserving memory from earlier in the sequence of tokens.</p><p>This schematic is illustrated below. At block 1, our initial state vector is randomly initialized and we compute local attention as normal. This is then passed into an MLP, which computes the next hidden-state vector, retaining memory this layer. As we advance to block 2, we drop block 1 from memory and simply pass its compressed state into the attention module for block 2. At subsequent layers, we repeat this same process, ensuring that only the current attention block and the compressed state from the previous are in memory at any given moment in time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tsjA65aG4kuhqGnW3Vmfmw.png" /><figcaption>Schematic of Recurrent Local Attention. We compress each activation matrix into a state vector, which is passed onto the next attention block. At any given moment in time, only the blocks in green are active in memory and we can discard the rest, which keeps our memory footprint constant.</figcaption></figure><p>This unlocks a lot. First, by dropping previous caches, we have no memory overhead with expanding window sizes. Thus, we can fit arbitrarily large context sizes into consumer hardware — which enabled us to run a Gemma 2B-10M model on just 32GB of RAM!</p><p>Additionally, we save a lot on inference time. A dense attention block involves computing n² entries. On the other hand, our computation trick decomposes the original matrix into a sparse diagonal block matrix, filling which requires much fewer computations. With an attention block of size d, we have n/d blocks and d² * n/d = nd total parameters. Thus, we go from a total execution flops of O(n²) to O(nd). Especially when n &gt;&gt; d, this reduction is massive and can boost inference, with the caveat that we can’t easily parallelize computations across the whole original matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SWa5l60fVtOyrZtny6m7tw.png" /><figcaption>Demonstration of the reduction in total computation with Recurrent-Attention. We go from O(n²) to O(nd) operations.</figcaption></figure><h3>Infini-Attention</h3><p>Having established the practicality of a recurrent state vector, let’s dive into some specifics on how our’s works. Borrowing from Google’s seminal Infini-attention paper, we use compressive memory to store information from previous layers. This allows our model to only need to perform standard dot-product (quadratic attention) on our local block and linearly attend to the compressed memory from the past. We illustrate this in the diagram below, where we have standard self-attention to the current Query and Key-Value pairs, as well as cross-attention to the previously cached compressive memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/0*j3rItBAuZalLF6Is.jpeg" /><figcaption>High-level overview of Infini-Attention. We perform standard dot-product attention on the current block as well as linearly attend to a compressed memory, which stores information from previous layers. Figure from original Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention paper.</figcaption></figure><p>This leaves one major question: how does the compressive memory work? The full mathematical details of this are too complex for this blog, but I’ll try to provide you with some intuition. At each layer, we add in a key-value store to our matrix. Thus, at layer 2, our memory matrix M, might look like M0 + v1 k1 + v2 k2. Now, if our keys are orthogonal to one another, we can retrieve the corresponding value associated with a key via a simple dot-product, similar to how coefficients are restored in Fourier-transforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/0*PfaAfFjI_86On5jQ.png" /></figure><p>But remember, while these simple linear operations generally work, there’s a large graveyard of papers showing that we can’t effectively model softmax(Q K-T) with just a linear matrix multiplications. Therefore, instead of just embedding Q and K into our matrix directly, we instead apply a kernel to both Q and K, with the hope that the softmax operation can be learned as an affine transformation of Q and K in the kernel, σ. This can be a lot to digest, and I find it helpful to think of kernels in terms of how (x+y)² = x² + 2xy + y² are non-linear in both x, y; however, linear in [x², xy, y²]. In this case, our kernel σ. This completes our true update rule for our memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lYwvEYBWxDmncJC-Dp56VA.png" /><figcaption>Memory Update rules for Infini-Attention. From original paper</figcaption></figure><h3><strong>Progressively Growing Context Windows</strong></h3><p>To optimize for training costs and data, we followed the approach in GrowLength, where we gradually increased context size from 32K → 64K → 128K → 256K → 512K → 1M → 2M → 4M → 10M. This allowed us to prioritize pretraining with shorter sequences in the beginning, thereby offering higher utilization rates and allowing us to learn simpler representational embeddings. As the context windows expanded, we then benefited from more powerful initial representational states, which simplified training at larger context-window sizes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*weNk7-edA7fweagyt3kdyQ.png" /><figcaption>Example of how we expand our window sizes. Much like dynamic programming, by building up the length of the context windows, we benefit from being able to assemble smaller windows, which are easier to model.</figcaption></figure><p>Below, we give a simple example of how this incremental window expansion looks like. At the first layer, our model learns to represent context windows of length up to 32k. Having learned this at layer 1, its task of learning windows of length 64k is made easier as the lower-level representations are already rich. In general, by layer N, the patterns learned by layer N-1 allow us to effectively scale instead of needing to learn complex million token representations all at once. From an algorithmic perspective, this bears semblances to merge sort and dynamic programming, where we learn to solve smaller, simpler problems first, eventually working our way towards larger representations.</p><h3>Thanks for reading!</h3><p>If you have any questions, feel free to shoot us an email at</p><p>Mustafa Aljadery: aljadery@usc.edu<br>Aksh Garg: akshgarg@stanford.edu<br>Sid Sharma: sidshr@stanford.edu</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=900adc4fbeeb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>