<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ketaki on Medium]]></title>
        <description><![CDATA[Stories by Ketaki on Medium]]></description>
        <link>https://medium.com/@ketaki.kolhatkar99?source=rss-53ab7f6ac8f1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Wusi3hoefdOZfyRwOXv72A.jpeg</url>
            <title>Stories by Ketaki on Medium</title>
            <link>https://medium.com/@ketaki.kolhatkar99?source=rss-53ab7f6ac8f1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:26:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ketaki.kolhatkar99/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Hype behind Andrej Karpathy’s Claude.md file]]></title>
            <link>https://medium.com/@ketaki.kolhatkar99/the-hype-behind-andrej-karpathys-claude-md-file-17a1f1b0778c?source=rss-53ab7f6ac8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/17a1f1b0778c</guid>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[andrej-karpathy]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Ketaki]]></dc:creator>
            <pubDate>Tue, 16 Jun 2026 12:33:03 GMT</pubDate>
            <atom:updated>2026-06-16T12:33:03.159Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>If you have been following the AI race, you are probably aware that Andrej Karpathy joined Anthropic as a Member of Technical Staff in March, 2026.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BClGCwVXplxSqcvoZLFHIw.png" /><figcaption>credits and a must watch :) <a href="https://x.com/siddsax/status/2058556345566241092">https://x.com/siddsax/status/2058556345566241092</a></figcaption></figure><p>Back in January, Karpathy noted down some points of his experience using Claude Code. As the term “vibe coding” was coined by Andrej himself, he admits to using English as his primary programming language. Moving from an 80% manual coding to an 80% agent coding, as many of us have experienced in the last couple of years. And yet, he says that the models make mistakes:</p><ul><li>Wrong assumptions, hidden confusion, missing tradeoffs</li><li>Overcomplication, bloated abstractions</li><li>Orthogonal edits, touching code you shouldn’t</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HDsd0CgsER_bPm1Ju1QFLg.png" /><figcaption><a href="https://x.com/karpathy/status/2015883857489522876?lang=en">https://x.com/karpathy/status/2015883857489522876?lang=en</a></figcaption></figure><p>Karpathy’s observations resonate with many developers working with LLMs in this AI revolution. I use these models regularly at work and for personal projects, and see an overcomplication of code, the “tenacity” he talks about in the tweet- the model trying really hard to reason without asking for clarifications.</p><blockquote>He writes, “LLMs are exceptionally good at looping until they meet specific goals, and this is where most of the &#39;feel the AGI&#39; magic is to be found”<em>.</em></blockquote><h3>The Claude.md File</h3><p>Here is where the Claude.md file becomes interesting. This file has been derived from the points Karpathy mentioned in the tweet and is to address the coding pitfalls. At its core, Claude.md is a persistent instruction file that provides coding agents with project context, engineering principles, and behavioural constraints.</p><h3>The Four Principles</h3><ol><li><strong>Think Before Coding</strong></li></ol><p>Designed towards wrong assumptions, hidden confusion, and missing trade-offs, this principle forces explicit reasoning:</p><ul><li>State assumptions explicitly: If uncertain, ask rather than guess</li><li>Present multiple interpretations: Don’t pick silently when ambiguity exists</li><li>Push back when warranted: If a simpler approach exists, say so</li><li>Stop when confused: Name what’s unclear and ask for clarification</li></ul><p>2. <strong>Simplicity First</strong></p><p>Addresses overcomplication and bloated abstractions. Working on the underlying objective of “Would a senior engineer say this is overcomplicated?” If yes, simplify.”</p><p>Combat the tendency toward overengineering:</p><ul><li>No features beyond what was asked</li><li>No abstractions for single-use code</li><li>No “flexibility” or “configurability” that wasn’t requested</li><li>No error handling for impossible scenarios</li><li>If 200 lines could be 50, rewrite it</li></ul><p>3. <strong>Surgical Changes</strong></p><p>Based on Karpathy’s observation about Claude making orthogonal changes, this principle addresses avoiding touching code you shouldn’t.</p><p>When editing existing code:</p><ul><li>Don’t “improve” adjacent code, comments, or formatting</li><li>Don’t refactor things that aren’t broken</li><li>Match existing style, even if you’d do it differently</li><li>If you notice unrelated dead code, mention it — don’t delete it</li></ul><p>4. <strong>Goal-Driven Execution</strong></p><p>This principle focuses on the point of defining a success criterion. Loop until verified. Transform imperative tasks into verifiable goals:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GjJI8pSvftd7AdgLQzur3A.png" /></figure><h3>Test Examples</h3><p>The repository has an Examples.md file that walks users through real-world code examples demonstrating the principles and common break points for LLMs and their fixes.</p><p>User Request:</p><pre>&quot;Add a feature to export user data&quot;</pre><p><strong>What LLMs Do (Wrong Assumptions)</strong></p><pre>def export_users(format=&#39;json&#39;):<br>    &quot;&quot;&quot;Export all users to a file.&quot;&quot;&quot;<br>    users = User.query.all()<br>    <br>    if format == &#39;json&#39;:<br>        with open(&#39;users.json&#39;, &#39;w&#39;) as f:<br>            json.dump([u.to_dict() for u in users], f)<br>    elif format == &#39;csv&#39;:<br>        with open(&#39;users.csv&#39;, &#39;w&#39;) as f:<br>            writer = csv.DictWriter(f, fieldnames=[&#39;id&#39;, &#39;email&#39;, &#39;name&#39;])<br>            writer.writeheader()<br>            writer.writerows([u.to_dict() for u in users])<br>    <br>    return f&quot;Exported {len(users)} users&quot;</pre><p>Problems:</p><ul><li>Assumed it should export ALL users (what about pagination? privacy?)</li><li>Assumed file location without asking</li><li>Assumed which fields to include</li><li>Assumed CSV fieldnames without checking the actual data structure</li></ul><p><strong>What Should Happen (Surface Assumptions)</strong></p><pre>Before implementing, I need to clarify:<br><br>1. **Scope**: Export all users or filtered subset? (privacy implications)<br>2. **Format**: You mentioned &quot;export&quot; - did you mean:<br>   - Download file in browser?<br>   - Background job with email notification?<br>   - API endpoint returning data?<br>3. **Fields**: Which user fields? (some might be sensitive)<br>4. **Volume**: How many users typically? (affects approach)<br><br>Simplest approach: Add an API endpoint that returns paginated JSON.<br>Would need more info for file-based exports.<br><br>What&#39;s your preference?</pre><p>Here is the <a href="https://github.com/multica-ai/andrej-karpathy-skills/blob/main/EXAMPLES.md">Examples.md</a> that walks through the core principle of the Claude.md file with a couple of more examples, the bottlenecks/ mistakes to avoid when using these models and optimise workflows.</p><h3>New Principles</h3><p>While this GitHub addresses all the concerns that Karpathy wrote about, there are a few gaps that the repo does not touch on. <a href="https://www.aibuilderclub.com/blog/karpathy-claude-md-rules">This blog</a> consolidates a few of those gaps that the community identified:</p><ul><li>No token budget rule</li><li>No checkpoint rule for multi-step tasks</li><li>No “read before you write” rule</li><li>No “fail loud” rule</li></ul><h3>Installation</h3><p>The file can be installed using a Claude Code Plugin that works across all projects. There is an option to have a file for every project based on the requirements.</p><p>It also comes with a Cursor IDE installation option. The installation steps can be found <a href="https://github.com/multica-ai/andrej-karpathy-skills/tree/main#install">here</a>.</p><h3>Where does this leave us?</h3><p>The GitHub repository has 175k stars and 17.9k forks within just 2 months of its release. Those numbers show us the need for a robust system that directs these models to <em>“meet specific goals”. </em>Many developers using these models face these bottlenecks, which makes refactoring the code written by the model seem like more work than writing it from scratch. Some more intriguing questions from Kartpathy as he concludes his long tweet seem like great insights.</p><p>He talks about LLMs being better at “<em>fill-in-the-blanks ” </em>than<em> “grand strategy”. </em>I tend to agree. Tools like Claude.md improve execution, but they don&#39;t replace architectural thinking. The engineer still defines the objective, constraints, and trade-offs; the agent accelerates the implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ivf1di72khlrkzzPLF4og.png" /></figure><p><strong>References:</strong></p><p>Github Link: <a href="https://github.com/multica-ai/andrej-karpathy-skills">https://github.com/multica-ai/andrej-karpathy-skills</a><br><a href="https://www.aibuilderclub.com/blog/karpathy-claude-md-rules">https://www.aibuilderclub.com/blog/karpathy-claude-md-rules</a></p><p><a href="https://www.robonuggets.blog/p/the-karpathy-claudemd-file-that-43000">https://www.robonuggets.blog/p/the-karpathy-claudemd-file-that-43000</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17a1f1b0778c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLMs as Operating Systems? MemGPT Decoded.]]></title>
            <link>https://medium.com/@ketaki.kolhatkar99/llms-as-operating-systems-memgpt-decoded-70cb97e7cde5?source=rss-53ab7f6ac8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/70cb97e7cde5</guid>
            <category><![CDATA[memgpt]]></category>
            <category><![CDATA[operating-systems]]></category>
            <category><![CDATA[memory-management]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Ketaki]]></dc:creator>
            <pubDate>Mon, 25 May 2026 13:26:07 GMT</pubDate>
            <atom:updated>2026-05-25T13:26:58.509Z</atom:updated>
            <content:encoded><![CDATA[<p>Paper Link: <a href="https://arxiv.org/pdf/2310.08560">https://arxiv.org/pdf/2310.08560</a></p><p>Github: <a href="https://github.com/letta-ai/letta?utm_source=chatgpt.com">https://github.com/letta-ai/letta?utm_source=chatgpt.com</a></p><p>With AI becoming such an integral part of our lives, people turn to models for everything from coding assistance to small life decisions. One thing that still makes the model feel less human is when it <em>forgets</em>. You can tell the model something important, and weeks later, it has no idea what you are talking about! Context is everything, and as the conversation grows too long, the model starts losing details. Modern LLMs are getting better at this by using techniques like retrieval, summarisation, and context management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*D2DgE4pTPptWQz78fKV3Ng.png" /><figcaption>An example of model having context in conversation</figcaption></figure><p>MemGPT takes this idea further by creating an<em> </em><strong><em>illusion of an infinite context window</em></strong><em>. </em>The researchers borrow the concept of virtual context management from <a href="https://en.wikipedia.org/wiki/Memory_hierarchy">hierarchical memory systems</a> in traditional operating systems that use paging between physical memory and disk. Traditional operating systems are designed to utilise limited memory efficiently. They move information between the storage layers based on accessibility and requirement, giving an illusion of infinite memory. “<em>LLMs as Operating Systems”</em> could be misleading- the paper&#39;s idea is to borrow key OS concepts, such as memory management, and apply them to optimise memory utilisation in LLM-based agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BSVFtMk6_SCcJgWrGSxCg.png" /><figcaption>Memory Hierachy</figcaption></figure><p>The architecture operates on two primary memory types- main context and external context. Main context is analogous to RAM; it consists of the LLM prompt tokens currently available to the model. External context is analogous to disk memory; it stores information outside the active context window. The information must flow from the external context to the main context during inference using MemGPT’s function calls.</p><p>The paper breaks down the OS-inspired architecture for MemoryGPT (MemGPT) into three sections:</p><h4><strong>Main Context:</strong></h4><p>MemGPT prompt tokens have three forms-</p><p>a. System Instructions: read-only static data on the model control flow that contains information on different memory levels and instructions on how to use MemGPT functions.</p><p>b. Working Context: unstructured text writable by only MemGPT function calls used to store facts, preferences, and important information about the user to maintain fluency in conversation.</p><p>c. FIFO Queue: a rolling queue of the back-and-forth conversation from the model and the user. The first index of the queue is a system message containing a recursive summary of messages that have been evicted from the queue.</p><h4><strong>Queue Manager</strong></h4><p>The queue manager works with the FIFO queue and the recall storage together to manage messages. The FIFO queue appends incoming messages to the active context, which is then used to trigger LLM inference. Both the output and the input are written to the recall storage, which acts as a message database. When messages in recall storage are retrieved via a MemGPT function call, the queue manager appends them to the back of the queue to reinsert them into the LLM’s context window.</p><h4><strong>Function Executor</strong></h4><p>MemGPT operates through a self-directed memory-editing and retrieval process, enabling automatic updates and searches through its own memory based on the current context. Instructions are provided to the system that guide the LLM on how to interact with the memory systems. The paper talks about having a feedback loop where the output is parsed by MemGPT to ensure correctness, and the function is executed if it is validated. This enables the system to update its memory based on its own actions and adjust future responses accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FCjWLrFQJqPy3CNxldzs_A.png" /><figcaption>Memory management in MemGPT</figcaption></figure><h4>Function Chaining</h4><p>LLM inference can be triggered by different events, including user messages, system messages, and user interactions such as a login alert or a document upload notification. Function chaining allows MemGPT to execute multiple function calls sequentially before returning control to the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*VY63MBUB4XGwLhwOdE5Ufw.png" /><figcaption>An example conversation snippet where MemGPT (left) updates stored information. Here, the information is stored in working context memory (located within the prompt tokens).</figcaption></figure><p>The paper provides further experiments for the two most common LLM use cases- conversational agents and document analysis, where the MemGPT architecture performs better against several SOTA architectures and models.</p><p>MemGPT redefines the context window problem as a memory problem. Instead of trying to fit more tokens into the prompt, it keeps only what is relevant in the active memory and moves out irrelevant data that can be retrieved later. This architecture makes it relevant for long-running agents or assistants.</p><p>This architecture later evolved into Letta, and one of its products is Letta Code, a personalised stateful desktop agent that can act as a coding assistant while retaining memory across interactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EEqrLtsJIf7CQKGpIuL2Og.png" /><figcaption><a href="https://docs.letta.com/letta-code/">https://docs.letta.com/letta-code/</a></figcaption></figure><p>References:</p><p>Letta: <a href="https://github.com/letta-ai/letta?utm_source=chatgpt.com">https://github.com/letta-ai/letta?utm_source=chatgpt.com</a></p><p>Memory Hierarchy in OS: <a href="https://www.youtube.com/watch?v=JogSnkvENr0&amp;t=614s">https://www.youtube.com/watch?v=JogSnkvENr0&amp;t=614s</a></p><p>Memory Hierarchy in OS: <a href="https://en.wikipedia.org/wiki/Memory_hierarchy">https://en.wikipedia.org/wiki/Memory_hierarchy</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70cb97e7cde5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Trust with Machines: Defeating Nondeterminism in LLM Inference]]></title>
            <link>https://medium.com/@ketaki.kolhatkar99/building-trust-with-machines-defeating-nondeterminism-in-llm-inference-b01fef416d87?source=rss-53ab7f6ac8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b01fef416d87</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[matrix]]></category>
            <dc:creator><![CDATA[Ketaki]]></dc:creator>
            <pubDate>Tue, 30 Sep 2025 10:38:45 GMT</pubDate>
            <atom:updated>2025-09-30T10:38:45.065Z</atom:updated>
            <content:encoded><![CDATA[<p>Blog Link: <a href="https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/">https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/</a></p><p>Reproducibility is a bedrock of scientific progress. Imagine using a calculator that gives a different result every time you ask what 2+2 equals. That seems absurd and an untrustworthy system. This is exactly what is happening with language models today. And yet, it is used as a pivotal piece in so many industries. Neither does setting the hyperparameter temperature to 0 (which theoretically maintains a deterministic output), nor using your own hardware, ensure a consistent result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*xyVW6eXpLHjDEq3r" /><figcaption>Building Trust with Machine! 😃</figcaption></figure><p>Thinking Machines Corp., led by the former OpenAI CTO Mira Murati, addresses the very concept of determinism in their first blog. A foundational yet crucial concept when dealing with language models. For long, big tech hypothesised that floating point non-associativity and concurrent execution caused this non-determinism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zYvxTbRpGkTZGCYbQaRHrA.png" /><figcaption>DeepSeek LLM inference process</figcaption></figure><p>The researchers establish the following statements, although contradictory, as being simultaneously true before diving into further analysis:</p><ol><li>Some kernels on GPUs are <strong>nondeterministic</strong>.</li><li>However, all the kernels used in a language model’s forward pass are <strong>deterministic</strong>.</li><li>Moreover, the forward pass of an LLM inference server (like vLLM) can also be claimed to be <strong>deterministic</strong>.</li><li>Nevertheless, from the perspective of anybody using the inference server, the results are <strong>nondeterministic</strong>.</li></ol><h4>The long-accepted culprits: floating-point non-associativity + concurrency</h4><p>Language models are just a bunch of matrix manipulations, outputting the most probable answer. It is assumed these matrix multiplications give the same result every time it is run when run on a GPU with a lot of concurrency. And yet we see some inconsistency on every run. The following equation shows what non-associativity is meant by.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*OzZx7LreYtCBjD_rkGed5A.png" /><figcaption>Non-Associativity</figcaption></figure><p>Floating points are known for their dynamic level of precision. A number, 3450, can be represented as 3.45 * 10³ while 0.486 as 4.86 * 10^-1. But when adding two numbers of different scale exponents, we drop the last two numbers to maintain a consistent 3-digit precision. With this, we have destroyed information!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3QSo-U_5vBoxZXUtJPjluw.png" /></figure><p>The following array produces 102 different results depending on the order of the computation. This is the underlying reason for the undeterminism we see~</p><pre>import random<br><br>vals = [1e-10, 1e-5, 1e-2, 1]<br>vals = vals + [-v for v in vals]<br><br>results = []<br>random.seed(42)<br>for _ in range(10000):<br>    random.shuffle(vals)<br>    results.append(sum(vals))<br><br>results = sorted(set(results))<br>print(f&quot;There are {len(results)} unique results: {results}&quot;)<br><br># Output:<br># There are 102 unique results: [-8.326672684688674e-17, -7.45931094670027e-17, ..., 8.326672684688674e-17]</pre><p><strong>But why are floating points summed in different orders?</strong></p><p>The hypothesis has been that floating-point summation order is determined by the order in which the concurrent threads finish. <em>But even concurrence is not the reason for the indeterminism! </em>Forward pass by nature is concurrence-demanding and has enough parallelism across the batch dimension, hence does not require techniques like atomic adds. Besides, many SOTA libraries handle concurrency efficiently. Hence, the forward pass is <strong>run-to-run deterministic</strong>: simply put, this is not the reason for the variance in output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jot0_CIO2psgAalRosnlPQ.png" /><figcaption>From the perspective of the inference server, it <em>is</em> deterministic. Given the same user requests, it will always provide the same deterministic output.</figcaption></figure><h4><strong>If not concurrency, then what?</strong></h4><p>Consider the following example, theoretically out1 and out2 should be the same. But, in practice, we see there is a mismatch! The reason for the mismatch is that the way GPUs handle matrix-matrix and matrix-vector multiplications differs, while matrices of different sizes are also handled differently in GPUs. This variance is also dependent on the hardware and software versions.</p><pre>import torch<br>torch.set_default_device(&#39;cuda&#39;) <br><br>B = 2048<br>D = 4096<br>a = torch.linspace(-1000, 1000, B*D).reshape(B, D)<br>b = torch.linspace(-1000, 1000, D*D).reshape(D, D)<br># Doing a matrix vector multiplication by taking<br># the first element of the batch<br>out1 = torch.mm(a[:1], b)<br># Doing a matrix matrix multiplication and then taking<br># the first element of the batch<br>out2 = torch.mm(a, b)[:1]<br>print((out1 - out2).abs().max()) # tensor(1669.2500, device=&#39;cuda:0&#39;)</pre><p>Assume that in the matrix shown above, ‘b,’ each row represents one user, and the columns represent the user input tokens. To efficiently utilise LLMs, all the user requests are processed in parallel. Since the number of user requests varies across time, the dimensions of ‘b’ are inconsistent. <strong>This varying property of batch size is the cause of indeterminism!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EUTyb58VJa3O3RaUu4LKJg.png" /><figcaption>Although the inference server itself can be claimed to be “deterministic”, the story is different for an individual user. From the perspective of an individual user, the other concurrent users are not an “input” to the system but rather a nondeterministic property of the system. This makes LLM inference “nondeterministic” from the perspective of each user.</figcaption></figure><h4>How to combat Indeterminism</h4><p>To combat the defined indeterminism, locate kernels in LLM where we see reductions: RMSNorm, matrix multiplication, and attention.</p><p><strong>Batch Invariant RMS Norm</strong></p><p>RMSNorm is a normalisation layer that rescales activations by their root-mean-square (RMS) value, without subtracting the mean. To make it batch-invariant, apply RMSNorm in the feature dimension instead of the batch dimension. This means we need to compute each token’s normalisation independently without splitting or combining across batches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*Yvs1WmyUi0i-1OuiPVwJlA.png" /></figure><p><strong>Batch Invariant Matrix Multiplication</strong></p><p>The same theory of computing batch invariant RMS Norm applies to matrix mults. Each row should be multiplied by the weight vector independently,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*6VD2KZyYmNOxlNetEMtD0w.png" /></figure><p><strong>Batch Invariant Attention</strong></p><p>Compute each query independently, using its own key-value (KV) pairs, so no cross-batch interactions affect results. Always apply the same computation pattern for KV caching and attention output, regardless of batch size or hardware parallelism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*leP_3LLrTErV00AygbNduA.png" /></figure><p>With these strategies laid out in the blog by Thinking Machines Corp, to tackle indeterminism in LLMs, a crucial yet overlooked issue, we are on a path to make AI more trustworthy. It was quite intriguing for us to go into the depths of the indeterminism we quite often question when using the models.</p><p>Excited to see what Mira Murati and her team tackle next!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*axxYHv8ooVvvTPhU.jpg" /><figcaption>Mira Murati</figcaption></figure><p>References:</p><p>[1] <a href="https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/">https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/</a></p><p>[2] <a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.modules.normalization.RMSNorm.html">https://docs.pytorch.org/docs/stable/generated/torch.nn.modules.normalization.RMSNorm.html</a></p><p>[3] <a href="https://jalammar.github.io/illustrated-transformer/">https://jalammar.github.io/illustrated-transformer/</a></p><p>[4] <a href="https://numba.pydata.org/numba-doc/dev/cuda/reduction.html">https://numba.pydata.org/numba-doc/dev/cuda/reduction.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b01fef416d87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NVIDIA’s Iconic Pivot: From Championing Bigger Models to Betting on Small]]></title>
            <link>https://medium.com/@ketaki.kolhatkar99/nvidias-iconic-pivot-from-championing-bigger-models-to-betting-on-small-22d8dad4485a?source=rss-53ab7f6ac8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/22d8dad4485a</guid>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[small-language-model]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ketaki]]></dc:creator>
            <pubDate>Wed, 17 Sep 2025 11:06:55 GMT</pubDate>
            <atom:updated>2025-09-17T11:06:55.171Z</atom:updated>
            <content:encoded><![CDATA[<p>Small Language Models are the Future of Agentic AI: <a href="https://arxiv.org/pdf/2506.02153">https://arxiv.org/pdf/2506.0215</a></p><p>NVIDIA’s strategy of selling shovels in the AI gold rush with GPUs powering the LLM breakthrough has positioned the company at a $3 trillion market cap. Jensen has time and again championed more compute and bigger models, emphasising that scaling is the path to intelligence. From his CNBC interview about Reasoning AI needing 100x more compute than previous models to the TechCrunch article, where he says their systems were progressing faster than Moore’s Law, the NVIDIA CEO has consistently advocated for larger models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IcStHBPTcNtGEEtxqKx19Q.png" /><figcaption>Febraury 26, 2025 at CNBC</figcaption></figure><p>Until their recent publication titled “Small Language Models are the Future of Agentic AI”. It seems like an iconic pivot for NVIDIA, given that the company benefited from the bigger-is-better<em> </em>strategy, is now betting on small models!</p><p>With the growing demand, the agentic AI sector is estimated to be at $200 billion by 2034. Making models smaller and efficient is the need of the hour for sustainable growth. Not only do SLMs guarantee cost-efficiency, but they also come with more security, data privacy and energy efficiency, with models running locally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ooUaBsE4nIz2BJRN7ZywLA.png" /></figure><p>Small Language Models are indeed the future of Agentic AI. But first, what are Agents? <a href="https://www.ibm.com/think/topics/ai-agents">Agents</a> are AI systems that can perceive their environment, make decisions, and take actions autonomously to achieve specific goals. To understand the concept better, imagine planning a vacation and booking a hotel. You provide your preferences, like price, reviews, location, and amenities, to a hotel booking agent. The agent goes to the web, searches relevant hotels, maximises the criteria as much as possible and books the best find. Agents have a limited scope of tasks they need to perform as opposed to the immense capabilities of Large Language Models. Therefore, a full-blown LLM is often unnecessary for building efficient agents.</p><p>Because Agents are focused on a specific task, SLMs fit perfectly given their size, cost and efficiency. By now, we have understood that Small Language Models are, well, <em>smaller </em>and<em> </em>more<em> </em>efficient. But what really are SLMs? <a href="https://huggingface.co/blog/jjokah/small-language-model">Small Language Models</a> (SLMs) are lightweight versions of traditional language models designed to operate efficiently in resource-constrained environments such as smartphones, embedded systems, or low-power computers.</p><p>In this paper, NVIDIA, along with Georgia Tech, contends that Small Language Models can be the future of Agentic AI only if they are:</p><ul><li><strong>V1: principally sufficiently powerful to handle language modeling errands of agentic applications;</strong></li><li><strong>V2: inherently more operationally suitable for use in agentic systems than LLMs;</strong></li><li><strong>V3: necessarily more economical for the vast majority of LM uses in agentic systems than their general-purpose LLM counterparts by virtue of their smaller size;</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-aGZX-Du7P6WIT40mUtosw.png" /><figcaption><a href="https://www.superannotate.com/blog/small-language-models">https://www.superannotate.com/blog/small-language-models</a></figcaption></figure><p>The paper goes over these contentions and provides non-exclusive supporting arguments. Listed a few of them below.</p><p>A1: The scaling curve between model size and capabilities is becoming increasingly steeper, supporting argument V1. This implies that the newer small models are close to and even exceed the older large models’ performance in some cases.</p><p>A2: A few examples of SLMs include the NVIDIA Nemotron-H family, Microsoft Phi series, Huggingface SmolLM2 series, and DeepSeek-R1-Distil series, which are paving the path to smaller models. As discussed in previous blogs, employing techniques like Inference efficiency, Parameter Efficient Fine Tuning (like Low Rank Adaptation), etc., makes SLMs economical while serving great capabilities that reinforce V3.</p><p>A3: Agentic systems are naturally heterogeneous, aligning in contention V2. Language models themselves can be a tool called by another model; similarly, when an agent’s code invokes a language model, it can choose any language model. The appropriate model is called the specific tasks. The figure below shows that an LLM can be used for the model with the root agency, while an SLM could be used for the subordinate LM. The figure on the right showcases that all LMs could, in principle, be specialised SLMs: one for conversationality, another one for carrying out controller-defined language modelling tasks.</p><p>… check the paper for all the arguments! 😃</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dgv5q7uL9YvrOrHCx_m2qw.png" /><figcaption>An illustration of agentic systems with different modes of agency. Left: Language model agency. The language model acts both as the HCI and the orchestrator of tool calls to carry out a task. Right: Code agency. The language model fills the role of the HCI (optionally) while a dedicated controller code orchestrates all interactions.</figcaption></figure><p>With that being said, the authors outline the steps for the LLM-to-SLM Agent Conversion:</p><ol><li>Secure usage data collection: Log agent calls, tool usage, and latencies via encrypted, access-controlled, and anonymised pipelines.</li><li>Data curation and filtering: Clean collected data by removing/masking PII/PHI and obfuscating sensitive details while preserving task relevance.</li><li>Task clustering: Cluster prompts and agent actions to uncover recurring patterns that define candidate specialisation tasks.</li><li>SLM selection: Choose suitable small language models based on task fit, capabilities, benchmarks, licensing, and deployment needs.</li><li>Specialised SLM fine-tuning: Fine-tune selected SLMs on curated task-specific datasets using efficient Parameter Efficient Fine Tuning (PEFT) methods or distillation.</li><li>Iteration and refinement: Continuously retrain and adapt models with fresh data to sustain and improve task performance.</li></ol><p>In our earlier blogs, we went over <a href="https://medium.com/@ketaki.kolhatkar99/gpt-oss-architecture-code-flow-gpu-setup-and-more-a71d06d8bf71">GPT-OSS architecture</a> and <a href="https://medium.com/@ketaki.kolhatkar99/gpt-oss-part-2-huggingface-setup-peft-and-gated-experts-a0e553ee9632">fine-tuning</a> using PEFT and Gated Experts. While GPT-OSS itself is a general-purpose model that comes under the large language model family, working with it highlighted the significance of Small Language Models being efficient with techniques like PEFT. NVIDIA’s paper on Small Language Models for Agentic AI feels like the next step in experimenting with generalists to build lightweight specialists.</p><p>Although this isn’t the first time the company has worked on smaller models like Minitron / Nemotron-Mini models, the Llama Nemotron reasoning family, and more, this seems like a move from what would be a side project to the core of future Agentic AIs. NVIDIA has once again proven that it is a thought leader in the AI space while also building the trust of its customers for delivering the best!</p><p>References:</p><p>[1] <a href="https://www.cnbc.com/2025/02/26/nvidia-ceo-huang-says-next-generation-ai-will-need-more-compute.html">https://www.cnbc.com/2025/02/26/nvidia-ceo-huang-says-next-generation-ai-will-need-more-compute.html</a></p><p>[2] <a href="https://www.superannotate.com/blog/small-language-models">https://www.superannotate.com/blog/small-language-models</a></p><p>[3] <a href="https://arxiv.org/pdf/2506.02153">https://arxiv.org/pdf/2506.0215</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22d8dad4485a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPT OSS Part 2: HuggingFace Setup, PEFT and Gated Experts]]></title>
            <link>https://medium.com/@ketaki.kolhatkar99/gpt-oss-part-2-huggingface-setup-peft-and-gated-experts-a0e553ee9632?source=rss-53ab7f6ac8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0e553ee9632</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[gpt-oss]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Ketaki]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 12:53:01 GMT</pubDate>
            <atom:updated>2025-09-09T13:16:24.398Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 1: GPT OSS Architecture, Code Flow, GPU Setup and More! <a href="https://medium.com/@ketaki.kolhatkar99/gpt-oss-architecture-code-flow-gpu-setup-and-more-a71d06d8bf71">https://medium.com/@ketaki.kolhatkar99/gpt-oss-architecture-code-flow-gpu-setup-and-more-a71d06d8bf71</a></p><p>OpenAI Cookbook: <a href="https://cookbook.openai.com/articles/gpt-oss/fine-tune-transfomers#setup">https://cookbook.openai.com/articles/gpt-oss/fine-tune-transfomers#setup</a></p><p>For those living under a rock, OpenAI released their open-weight models gpt-oss-20b and gpt-oss-120b last month. This marks a significant step forward in democratizing language models, allowing researchers and tech enthusiasts to go a step further — to fine-tune, understand the code flow, and tinker with the architecture too! It seems like the OpenAI CEO didn’t want to be on the wrong side of history for too long, as seen from his Reddit post a few months back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*U6Rn5ZzIkhekwN3BmcgXOQ.png" /><figcaption><a href="https://www.instagram.com/p/DFptw4oMNQw/">https://www.instagram.com/p/DFptw4oMNQw/</a></figcaption></figure><p>We have explored GPT OSS architecture, code flow, and GPU setup in <a href="https://medium.com/@ketaki.kolhatkar99/gpt-oss-architecture-code-flow-gpu-setup-and-more-a71d06d8bf71"><strong>Part 1</strong></a> 😃! This blog dives into the next frontier of fine-tuning and adding new components to the architecture, from parameter-efficient tuning and expert routing techniques. We closely followed the <a href="https://cookbook.openai.com/articles/gpt-oss/fine-tune-transfomers#setup">OpenAI Cookbook</a> for the model setup with the HuggingFace library on a multilingual reasoning dataset. Once the setup is ready, we will move on to the fine-tuning and architectural tweaks we explored!</p><h4>Hugging Setup</h4><ul><li><strong>Hardware Setup: </strong>We have used runpod.io with an NVIDIA H200 GPU (140 GB of memory). Ideally, an H100 should be sufficient for this setup.</li><li><strong>Install dependencies:</strong></li></ul><pre>%pip install torch --index-url https://download.pytorch.org/whl/cu128<br>%pip install &quot;trl&gt;=0.20.0&quot; &quot;peft&gt;=0.17.0&quot; &quot;transformers&gt;=4.55.0&quot; trackio</pre><ul><li><strong>Prepare Dataset</strong>: The data used in this demo is Multilingual-Thinking, a reasoning dataset where the chain-of-thought is translated from English into one of four languages: Spanish, French, Italian, and German. Most language models are trained on English data; hence, their reasoning is in English, which is then translated to the expected language. In this process, the language context is lost. Here, we intend to fine-tune the GPT-oss model to learn to reason in these languages.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*awkU9TGOBv5a4MrusTp17g.png" /><figcaption><a href="https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking/viewer/default/train?views%5B%5D=train">https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking/viewer/default/train?views%5B%5D=train</a></figcaption></figure><p>As seen above, only the analysis column (reasoning steps) is in the target language; the rest of the columns are in English. Most prompts are in English, and using this data, we enable the model to “think” in the target languages, providing contextually relevant results. This makes it accessible to a wider multilingual audience. An important aspect of the dataset is its columns that mirror the OpenAI Responses API discussed in depth in Part 1 blog. The apply_chat_template method converts the messages into a format that the model can understand.</p><pre>from transformers import AutoTokenizer<br>tokenizer = AutoTokenizer.from_pretrained(&quot;openai/gpt-oss-20b&quot;)<br>messages = dataset[0][&quot;messages&quot;]<br>conversation = tokenizer.apply_chat_template(messages, tokenize=False)<br>print(conversation)</pre><ul><li><strong>Prep the Model</strong>: The torch framework is used for the model initialization. An important aspect of the definition is the Mixed Floating-Point 4, quantization scheme that mixes 4-bit + higher precision where needed. Quantization is a technique of storing model weights in fewer bits that reduces memory and speeds up inference. Here we quantize to 4 bits instead of 16/32-bit. The attn_implementation field specifies the attention backend, with eager being stable and easy to debug. The use_cache parameter is set to False, which means previous key/values are not passed into the model during generation. As discussed before, we are using the ‘gpt-oss-20b’ model for the setup.</li></ul><pre>import torch<br>from transformers import AutoModelForCausalLM, Mxfp4Config<br><br>quantization_config = Mxfp4Config(dequantize=True)<br>model_kwargs = dict(<br>    attn_implementation=&quot;eager&quot;,<br>    torch_dtype=torch.bfloat16,<br>    quantization_config=quantization_config,<br>    use_cache=False,<br>    device_map=&quot;auto&quot;,<br>)<br><br>model = AutoModelForCausalLM.from_pretrained(&quot;openai/gpt-oss-20b&quot;, **model_kwargs)</pre><p>Next, we pass in the input to the model in the required format by using the apply_chat_template method, i.e., adding tokens to the input for the model to reason better. The return_tensors field converts the text to Pytorch tensors. The output_ids from the model are then passed into the batch_decode function that converts the output token to natural, human-readable language.</p><pre>messages = [<br>    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;¿Cuál es el capital de Australia?&quot;},<br>]<br><br>input_ids = tokenizer.apply_chat_template(<br>    messages,<br>    add_generation_prompt=True,<br>    return_tensors=&quot;pt&quot;,<br>).to(model.device)<br><br>output_ids = model.generate(input_ids, max_new_tokens=512)<br>response = tokenizer.batch_decode(output_ids)[0]<br>print(response)</pre><p>A Parameter Efficient Fine-Tuning layer, here, Low-Rank Adaptation is defined with rank 8 and applied to the MOE layers at 7, 15, and 23. This allows fine-tuning on new data efficiently while keeping the MOE model frozen. Read our blog on <a href="https://medium.com/@ketaki.kolhatkar99/the-power-of-a-few-lora-fe4dd7b3bde4">LoRA</a> to know the architecture in depth! 😃</p><pre>from peft import LoraConfig, get_peft_model<br><br>peft_config = LoraConfig(<br>    r=8,<br>    lora_alpha=16,<br>    target_modules=&quot;all-linear&quot;,<br>    target_parameters=[<br>        &quot;7.mlp.experts.gate_up_proj&quot;,<br>        &quot;7.mlp.experts.down_proj&quot;,<br>        &quot;15.mlp.experts.gate_up_proj&quot;,<br>        &quot;15.mlp.experts.down_proj&quot;,<br>        &quot;23.mlp.experts.gate_up_proj&quot;,<br>        &quot;23.mlp.experts.down_proj&quot;,<br>    ],<br>)<br>peft_model = get_peft_model(model, peft_config)<br>peft_model.print_trainable_parameters()</pre><ul><li><strong>Fine-Tuning: </strong>TRL’s (Transformer Reinforcement Learning, a HuggingFace library) SFTTrainer is used here for supervised fine-tuning. We set the learning rate at 2e-3, 1 training epoch, and a 2048 max context length, along with more configurations.</li></ul><pre>from trl import SFTConfig<br><br>training_args = SFTConfig(<br>    learning_rate=2e-4,<br>    gradient_checkpointing=True,<br>    num_train_epochs=1,<br>    logging_steps=1,<br>    per_device_train_batch_size=4,<br>    gradient_accumulation_steps=4,<br>    max_length=2048,<br>    warmup_ratio=0.03,<br>    lr_scheduler_type=&quot;cosine_with_min_lr&quot;,<br>    lr_scheduler_kwargs={&quot;min_lr_rate&quot;: 0.1},<br>    output_dir=&quot;gpt-oss-20b-multilingual-reasoner&quot;,<br>    report_to=&quot;trackio&quot;,<br>    push_to_hub=True,<br>)</pre><pre>from trl import SFTTrainer<br><br>trainer = SFTTrainer(<br>    model=peft_model,<br>    args=training_args,<br>    train_dataset=dataset,<br>    processing_class=tokenizer,<br>)<br>trainer.train()</pre><ul><li><strong>Inference: </strong>Finally, before inference, we combine the two defined models, the gpt-oss-20b with our Low-Rank Adaptor to get one efficient model.</li></ul><pre>from transformers import AutoModelForCausalLM, AutoTokenizer<br>from peft import PeftModel<br><br># Load the tokenizer<br>tokenizer = AutoTokenizer.from_pretrained(&quot;openai/gpt-oss-20b&quot;)<br><br># Load the original model first<br>model_kwargs = dict(attn_implementation=&quot;eager&quot;, torch_dtype=&quot;auto&quot;, use_cache=True, device_map=&quot;auto&quot;)<br>base_model = AutoModelForCausalLM.from_pretrained(&quot;openai/gpt-oss-20b&quot;, **model_kwargs).cuda()<br><br># Merge fine-tuned weights with the base model<br>peft_model_id = &quot;gpt-oss-20b-multilingual-reasoner&quot;<br>model = PeftModel.from_pretrained(base_model, peft_model_id)<br>model = model.merge_and_unload()</pre><p>We then test this “gpt-oss-20b-multilingual-reasoner” model by giving it a reasoning in German and a user prompt in Spanish. Here, we are asking the model to reason in German even if the user prompt is in German.</p><pre>REASONING_LANGUAGE = &quot;German&quot;<br>SYSTEM_PROMPT = f&quot;reasoning language: {REASONING_LANGUAGE}&quot;<br>USER_PROMPT = &quot;¿Cuál es el capital de Australia?&quot;  # Spanish for &quot;What is the capital of Australia?&quot;<br><br>messages = [<br>    {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: SYSTEM_PROMPT},<br>    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: USER_PROMPT},<br>]<br><br>input_ids = tokenizer.apply_chat_template(<br>    messages,<br>    add_generation_prompt=True,<br>    return_tensors=&quot;pt&quot;,<br>).to(model.device)<br><br>gen_kwargs = {&quot;max_new_tokens&quot;: 512, &quot;do_sample&quot;: True, &quot;temperature&quot;: 0.6, &quot;top_p&quot;: None, &quot;top_k&quot;: None}<br><br>output_ids = model.generate(input_ids, **gen_kwargs)<br>response = tokenizer.batch_decode(output_ids)[0]<br>print(response)</pre><h4>Architectural Tweaks</h4><p>This Cookbook presents an amazing blueprint for us to extend on it, experiment with different datasets, extend the architecture, and play around with fine-tuning it further. With such great open source work, language models would be the building blocks of intelligent software applications. Think of this concept as the next software revolution, where developers would just need to import LLMs as a library module to address their use case. This is similar to using the pandas library in legacy ML applications. For example, currently, a developer uses pandas to manipulate data, while it is not the central part of their application. Similarly, LLMs would become a module to manipulate and understand data/context.</p><p>We took up this challenge too and built an additional layer of a gated expert layer to choose a weight LoRA and GPT OSS model output based on their expertise. The illustration below shows a gate that takes outputs from both the OpenAI model and the fine-tuned LoRA to output a weighted sum of their results. Through training architecture, we hypothesize that the LoRA model would be weighted more for the tasks specific to the fine-tuned dataset, while the base model weighted more for other tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pU5u-yMRpiSWcrfYSDUZXw.png" /><figcaption>Gated Architecture of the base gpt-oss-20b with the LoRA model resulting in a weighted sum</figcaption></figure><p>Below is the code used for the gating. The GatedLoRAModel function defines the weighted combination of the trainable LoRA and the frozen base model weights. Setting required_grad to false freezes the weight updation of the base model. The gate is like a small neural network in itself. For causal language modelling, we shift predictions by one token.</p><pre>import torch<br>import torch.nn as nn<br>from peft import LoraModel<br>from transformers import AutoModelForCausalLM, AutoTokenizer<br><br>class GatedLoRAModel(nn.Module):<br>    def __init__(self, base_model_name, lora_config):<br>        super().__init__()<br>        # Base frozen GPT OSS model<br>        self.base_model = AutoModelForCausalLM.from_pretrained(base_model_name)<br>        self.base_model.requires_grad_(False)<br><br>        # LoRA adapted GPT OSS model (trainable)<br>        self.lora_model = LoraModel(self.base_model, lora_config, adapter_name=&quot;default&quot;)<br><br>        # Gating network: learns how much weight to give LoRA vs base<br>        hidden_size = self.base_model.config.hidden_size<br>        self.gate = nn.Sequential(<br>            nn.Linear(hidden_size, hidden_size // 2),<br>            nn.ReLU(),<br>            nn.Linear(hidden_size // 2, 1),<br>            nn.Sigmoid()  # outputs value in [0,1]<br>        )<br><br>    def forward(self, input_ids, attention_mask=None, labels=None):<br>        # Hidden states from base (frozen)<br>        with torch.no_grad():<br>            base_out = self.base_model(input_ids, attention_mask=attention_mask, output_hidden_states=True)<br>            hidden_states = base_out.hidden_states  # now this works<br>        base_logits = base_out.logits<br>        hidden = base_out.hidden_states[-1].float()  # final hidden states<br><br>        # LoRA adapted output<br>        lora_out = self.lora_model(input_ids, attention_mask=attention_mask)<br>        lora_logits = lora_out.logits<br><br>        # Gate value per token<br>        gate_vals = self.gate(hidden).squeeze(-1).unsqueeze(-1)  # [batch, seq, 1]<br><br>        # Weighted combination of logits<br>        combined_logits = (1 - gate_vals) * base_logits + gate_vals * lora_logits<br><br>        loss = None<br>        if labels is not None:<br>            loss_fct = nn.CrossEntropyLoss()<br>            # shift for causal LM<br>            loss = loss_fct(<br>                combined_logits[:, :-1, :].reshape(-1, combined_logits.size(-1)),<br>                labels[:, 1:].reshape(-1)<br>            )<br>        return {&quot;loss&quot;: loss, &quot;logits&quot;: combined_logits}</pre><p>The optimizer used here is AdamW: Adaptive Moment Estimation with Weight Decay, which combines adaptive learning rate with decoupled weight decay regularizing the model during training. We choose the linear rate scheduler to control the learning rate through the training steps.</p><pre># Optimizer (only trainable params: LoRA + gate)<br>optimizer = AdamW(filter(lambda p: p.requires_grad, model.parameters()), lr=2e-4)<br># Scheduler<br>num_training_steps = 1000<br>scheduler = get_scheduler(&quot;linear&quot;, optimizer=optimizer, num_warmup_steps=50, num_training_steps=num_training_steps)</pre><p>We see a smooth decay curve for the cross-entropy loss that stabilizes after about 50 epochs. The graph is between steps and loss values. Here, the x-axis is the step count and the y-axis is the loss value. As we can see, we are successfully able to converge the loss function. Though there are multiple hyperparameters that can be adjusted but in this case, we tried to only adjust the learning rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eMVdCfjn5q7cvEMj5NIEEg.png" /><figcaption>Smooth cross-entrop loss curve across the number of steps</figcaption></figure><p>In the inference part, we ask the model about the capital of Australia and ask it to think in German. But the model “hallucinates” and generates gibberish. This implies there is room for improving the current model either by better parameter tuning or providing better fine-tune data. This can also emphasise some underlying problem with the pre-trained baseline model. To better understand model output, we can try increasing the thinking budget to see how the model is concluding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*32ECBnPePWLNvU7Mv7W_aA.png" /><figcaption>Output from our tweaked architecture</figcaption></figure><p>To conclude, we have covered the HuggingFace setup for the GPT-OSS model by using a small use case of a multilingual dataset. Open-source opens many doors to experiments on extending the existing architecture to get better results on custom use-cases and datasets. We have implemented one such example with the gated network and achieved good cross-entropy loss. This is just the beginning, and it opens up these language models to all the capabilities and advancements of deep neural networks.</p><p>Github Link to complete code:<br><a href="https://github.com/ketakiii/GPT-OSS-Experiment">https://github.com/ketakiii/GPT-OSS-Experiment</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0e553ee9632" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPT OSS: Architecture, Code Flow, GPU Setup and More!]]></title>
            <link>https://medium.com/@ketaki.kolhatkar99/gpt-oss-architecture-code-flow-gpu-setup-and-more-a71d06d8bf71?source=rss-53ab7f6ac8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a71d06d8bf71</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[gpt-oss]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Ketaki]]></dc:creator>
            <pubDate>Mon, 25 Aug 2025 14:54:14 GMT</pubDate>
            <atom:updated>2025-08-25T18:14:37.678Z</atom:updated>
            <content:encoded><![CDATA[<p>OpenAI’s Blog on GPT OSS: <a href="https://openai.com/index/introducing-gpt-oss/">https://openai.com/index/introducing-gpt-oss/</a></p><p>GPT OSS Github Link: <a href="https://github.com/openai/gpt-oss/tree/main">https://github.com/openai/gpt-oss/tree/main</a></p><p>Open AI Harmony Github Link: <a href="https://github.com/openai/harmony">https://github.com/openai/harmony</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cph7CgvV34PlkM0sTKhNgA.png" /></figure><p>In an <a href="https://www.youtube.com/watch?v=jvqFAi7vkBc">interview</a> (watch 28:40) last year, when asked about what the “open” in OpenAI means to Sam Altman, he responded that if he had to go back in time, he’d pick a different name! Fast forward to August 2025, OpenAI really did follow through and released their two state-of-the-art open weight models: gpt-oss-120b (120B parameters) and gpt-oss-20b (20B parameters) in six years since GPT-2. This release is exciting as it pushes the frontier of open-weight reasoning models. It allows researchers and developers to go beyond the black box API, study the weights, architecture, build custom strategies and run the model locally!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1tmijwUZ_Av6988MQRwrhg.png" /><figcaption>Lex Fridman Podcast, Mar 18, 2024</figcaption></figure><p>We already have open-source models like DeepSeek, Kimi, Qwen, and with gpt-oss we have a window into how one of the pioneers in the field is thinking about their models in terms of architecture, scaling and design choices.</p><p>These models were trained with a mix of reinforcement learning and other techniques used in the most advanced models like, o3 and other frontier systems. OpenAI claims gpt-oss outperforms similarly sized open models on reasoning tasks while being optimized for efficient deployment on consumer hardware.</p><p><strong>Open-Weight model not Open-Source!</strong></p><p>A key point to note is that gpt-oss is an open-weight model and not a fully open-source model. OpenAI also mentions this as an open-weight model in their blog. What does that mean? The researchers have released the trained weights that the model is optimized for, which means we can run the model locally for inference. However, the training data and the recipe used to train the model are not provided, meaning it is not fully reproducible. OpenAI also refers to this as an open-weight and not an open-source model on their webpage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JVt9wyxPF0wBTjoyO11EUA.png" /><figcaption>OpenAI GPT OSS blog release</figcaption></figure><h3>Architecture</h3><p>With that being said, let’s dive deeper into the architecture! GPT OSS has 2 main building blocks we will be talking about:<br>1. <a href="https://github.com/openai/harmony">OpenAI Harmony</a> response format - gpt oss was trained on the response format for defining conversation structures, reasoning output and function calls. If you are using an API wrapper like Ollama or Hugging Face, these concepts are abstracted. However, when working with the raw model to set up locally or while making changes to the architecture, the Harmony framework plays a pivotal role. It consists of roles and channels.</p><p><strong>Role</strong>: The model assigns every message a role with the following hierarchy. As the names suggest, each role is associated with the kind of response that is expected from the model.</p><pre>system &gt; developer &gt; user &gt; assistant &gt; tool</pre><p>Below is an example of how roles are assigned on a prompt</p><pre>&lt;|system|&gt; You are a helpful AI<br>&lt;|user|&gt; Explain mixture of experts<br>&lt;|assistant|&gt;</pre><p><strong>Channel</strong>: Models outputs through three different channels in the following hierarchy, where analysis is the model’s internal reasoning hidden from the user, followed by commentary that includes tool traces and then the final, which is the polished version given to the user.</p><pre>analysis &gt; commentary &gt; final</pre><p>2. <a href="https://medium.com/@ketaki.kolhatkar99/choose-drop-route-moe-vs-mod-vs-mor-b553d38171c1">Mixture of Experts</a> - First introduced in 2017, MOE is a fundamental concept used in gpt-oss. Mixture of Experts introduces multiple “experts” for handling specific tasks. Hence, the model does not have to use all its parameters for every token, and only the relevant ones are activated. Just like we have different healthcare specialists for different health issues. This architecture choice allows a 120B parameter run on consumer hardware. Read our <a href="https://medium.com/@ketaki.kolhatkar99/choose-drop-route-moe-vs-mod-vs-mor-b553d38171c1">previous blog on MOE</a> to know the in-depth architecture! 😃</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZrbuGy9_afvHfuUztOSCA.png" /><figcaption>GPT OSS Architecture by Jay Alammar</figcaption></figure><h3>Code Flow</h3><p>Now that we have covered the fundamentals of gpt-oss, let’s get our hands dirty, understanding the code flow of the concepts discussed above, along with the GPU setup! The chat.py main method is the entry point to the gpt-oss model, which initializes the framework to use using the input args. The input goes to the Harmony framework to assign the prompt a role. Below is the code flow for Torch, where model.py encapsulates the generator with the Transformer Block. As we are aware, the Transformer Block further contains the AttentionBlock and the MLPBlock while computing the q, k, v vectors, linear layers, and <a href="https://arxiv.org/pdf/2309.00071">Rotary Embedding</a>.</p><p>The generate method takes in the model configs like hidden layers, number of experts, experts per token, vocab size, number of attention layers, and more. This method performs the inferences and provides the output. The output is finally passed through a Harmony parser that decides which channel the prompt should take, for example, if it is analysis (hidden from the user), etc.</p><pre>chat.py : main(*args)   # main function starting point<br>   model = TokenGenerator(checkpoint, context, device)  # model initialization<br>   │<br>   ▼<br>   │<br>   ├──── torch  # torch framework<br>   │        │<br>   │        ▼<br>   │        ├─ init.py    # torch model initialization<br>   │        ├─ model.py      # torch model <br>   │        │    ├── generator = TokenGenerator  # TokenGenerator class<br>   │        │    │     ├── __init__(checkpoint, device)<br>   │        │    │     │<br>   │        │    │     ▼<br>   │        │    │     self.model=Transformer.from_checkpoint(checkpoint, device=self.device)<br>   │        │    │     │<br>   │        │    │     ▼<br>   │        │    │     TransformerBlock<br>   │        │    │         ├── self.attn = AttentionBlock       # attention block<br>   │        │    │         │     ├── RMSNorm                    # RMS normalization<br>   │        │    │         │     ├── self.qkv = torch.nn.Linear  # query, key, value linear layers<br>   │        │    │         │     ├── Linear                     # linear layers<br>   │        │    │         │     └── RotaryEmbedding            # rotary embedding      <br>   │        │    │         └── self.mlp = MLPBlock              # MLP block<br>   │        │    │<br>   │        │    └─── generate(*model_config_args)              # generate logits<br>   │        │         └─ logits = self.model(prompt_tokens)     # calls model<br>   │        │<br>   │        └─── weights.py     # torch model weights<br>   │<br>   ├──── triton         # Triton framework<br>   │<br>   └──── vllm           # VLLM framework<br>   │<br>   ▼<br>  load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)    # OpenAI Harmony encoding<br>   |<br>   ▼<br>  OpenAI Role configuration     # OpenAI role configuration<br>   │<br>   ▼<br>  output = generator.generate(prompt_tokens)    # calls generate method for next token prediction<br>   │<br>   │<br>   ▼<br>Harmony parser:                                 # Harmony parser for deciding channels<br>  • openai-harmony reads channels from tokens<br>  • Splits analysis, commentary, final segments<br>  • Enables tool invocation logic if the &quot;commentary&quot; channel appears</pre><h3><strong>Running gpt-oss-20b on a GPU with RunPod</strong></h3><p><strong>Step 1</strong>: Spin up a machine on RunPod. We have selected an A40 GPU with 50 GB of memory and a 30 GB container disk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/0*dPdpC6pvAEYNGnq7" /></figure><p><strong>Step 2</strong>: From the RunPod console, click on JupyterLab to open the interactive environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/0*rYOqlNcO3pSnvZCQ" /></figure><p><strong>Step 3</strong>: You’ll be working inside the old school Jupyter notebook-style interface with terminal access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cpUntW6vcNX9-RjT" /></figure><p><strong>Step 4</strong>: Clone the GPT-OSS repository</p><pre>git clone https://github.com/openai/gpt-oss<br>cd gpt-oss</pre><p><strong>Step 5</strong>: Verify prerequisites. Ensure your machine has the required system dependencies (Python, pip, CUDA drivers if using GPU).</p><p><strong>Step 6</strong>:<strong> </strong>RunPod machines may not come with Python 3.12 by default. Install it and upgrade pip.</p><p><strong>Step 7</strong>: Install GPT-OSS dependencies.<strong> </strong>For PyTorch-based inference, install GPT-OSS with:</p><pre>pip install gpt-oss[torch]</pre><p><strong>Step 8: </strong>Download the model (20B). For example, to download the 20B checkpoint. (Install huggingface_hub if needed: pip install huggingface_hub)</p><pre># gpt-oss-20b<br>hf download openai/gpt-oss-20b --include &quot;original/*&quot; --local-dir gpt-oss-20b/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y5pn8OKMwmSCFV0T1HCy1Q.png" /></figure><p><strong>Step 9: </strong>Verify setup.<strong> </strong>At this stage, GPT-OSS and all requirements should be in place.</p><p><strong>Step 10: </strong>Run inference. Try a simple prompt:</p><pre>torchrun --nproc-per-node=1 -m gpt_oss.generate gpt-oss-20b/original/ --prompt &quot;number of days in a week?&quot;</pre><p><strong>Step 11</strong>: Handle errors<strong>. </strong>This may throw an error depending on your environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EHzEXxtKHMCoaG7Dzq7FnQ.png" /></figure><p><strong>Step 12: </strong>Apply fix (if needed). If you see the error, go to ./gpt-oss/gpt_oss/torch/<a href="http://model.py/">model.py</a> and add these 2 lines after line 458.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*Pf1-pzBpKyZ1H7_i_-lEuw.png" /></figure><p>Step 13: Now retry. The following is the output you would get from running generate.py, a parallel function to chat.py that generates the next k tokens and ends the session (a building block of chat.py; multiple generates-&gt; chat.py). Below shows the prediction results for the question:</p><blockquote>“Number of days in a week?”</blockquote><p>The output shows the log probabilities of each token and we see the message:</p><blockquote>“7 is the answer”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WVC7i9FRqXQlPhjkf7k3VQ.png" /></figure><p>References:</p><p>[1] <a href="https://newsletter.languagemodels.co/p/the-illustrated-gpt-oss">https://newsletter.languagemodels.co/p/the-illustrated-gpt-oss</a></p><p>[2] <a href="https://openai.com/index/introducing-gpt-oss/">https://openai.com/index/introducing-gpt-oss/</a></p><p>[3] OpenAI Harmony cookbook: <a href="https://cookbook.openai.com/articles/openai-harmony">https://cookbook.openai.com/articles/openai-harmony</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a71d06d8bf71" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choose, Drop, Route: MOE vs MOD vs MOR]]></title>
            <link>https://medium.com/@ketaki.kolhatkar99/choose-drop-route-moe-vs-mod-vs-mor-b553d38171c1?source=rss-53ab7f6ac8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b553d38171c1</guid>
            <category><![CDATA[mixture-of-experts]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Ketaki]]></dc:creator>
            <pubDate>Thu, 14 Aug 2025 12:58:29 GMT</pubDate>
            <atom:updated>2025-08-14T14:27:18.058Z</atom:updated>
            <content:encoded><![CDATA[<p>Scaling Transformers isn’t just about adding more compute, it is about adopting strategies that make them smarter and more efficient (a common theme from previous blogs! 😃). This blog walks you through the evolution of conditional computation strategies in large language models with Mixture of Experts, Mixture of Depths and Mixture of Recursion architectures. Conditional computation is a way for a model to only activate parts of its network for each input, saving compute while keeping performance high. Each addresses the limitations of the previous for model efficiency.</p><h3><strong>Mixture of Experts (MOE)</strong></h3><p>Paper Links: <a href="https://arxiv.org/pdf/1701.06538">https://arxiv.org/pdf/1701.06538</a></p><p>Back in January 2017, before the advent of the revolutionary Transformers, language modelling still heavily relied on LSTMs, RNNs and neural networks. Researchers focused on increasing model performance using computational efficiency algorithms. Sparsely Gated Mixture of Experts, introduced by Google Brain, explores the <a href="https://intellabs.github.io/distiller/conditional_computation.html">conditional computation</a> concept, where parts of the network are active on a per-example basis that dramatically increases model capacity without a proportional increase in computation.</p><p>Mixture of Experts consists of n expert networks “E1.. En”, and a gating network G. Each expert is a simple feed-forward neural network with its own set of parameters that characterises a unique aspect of each token. The gating network is a trainable model that determines the combination of experts to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZBgkqCMUNIuBOZxsVLbaA.png" /><figcaption>Mixture of Experts (MoE) layer embedded within a recurrent language model</figcaption></figure><p>The output of an MOE module can be written as follows, where G(x) and Ei(x) outputs from the Gating network and ith expert network. Whenever G(x) = 0, E(x) does not need to be calculated. In simpler words, when a token isn’t required to be sent to an expert, G(x) is zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*uSoIQCda0Jbf7Si_z_sesA.png" /><figcaption>Output of MoE module</figcaption></figure><p>The following represents a softmax gating network that applies softmax over the product of the input with a trainable weight matrix Wg. Noisy Top-K gating network is another variant that the authors propose in the paper, adding noise and sparsity to the regular gating function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*ONOCXOqib3uqAIPHZcp_hA.png" /><figcaption>Softmax Gating</figcaption></figure><p>Since each token goes to only a few experts, each expert sees only a fraction of the tokens, resulting in a drop in the batch size. Small batches mean less-efficient matrix multiplications and poor GPU utilisation. The authors propose data parallelism and model parallelism techniques that duplicate and split the model across multiple devices. With data parallelism, data is passed in batches in parallel, and after each step, gradients are synced across devices. The model is split across devices in model parallelism since it is too big to fit into one. Devices pass activations to each other during forward/backwards passes.</p><h3><strong>Mixture of Depths (MOD)</strong></h3><p>Paper Links: <a href="https://arxiv.org/pdf/2404.02258">https://arxiv.org/pdf/2404.02258</a></p><p>Fast-forward to April 2024, when Google DeepMind introduced a Mixture of Depths for dynamically allocating compute to the Transformers architecture. MOD focuses on having a compute budget by capping the number of tokens participating Transformer layers, making it a static computation graph unlike the conditional computation techniques discussed earlier. The compute capacity is lowered and tokens not requiring as much processing as others are chosen are identified through learning while still preserving the performance.</p><p>Unlike the MOE approach with the dynamic token-level routing approach, MOD decides whether a token should pass through the regular Transformer computation or go through a residual connection (remaining unchanged and saving compute). The FLOPs per forward pass will shrink as compared to a vanilla Transformer depending on the capacity we set it at.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QU4Q5cPwMkGy9d4evzRwHQ.png" /><figcaption>Mixture-of-Depths Transformer, where the left figures show the residual connection and the regular Transformer pass. The MOD approach can be thought of as an early-exit strategy at a token level, and the figure on the right is a visual representation of what the sequence would look like.</figcaption></figure><p>MOD considers the token-choice routing and the expert-choice routing techniques. As the name suggests, token-choice routing chooses the expert that is best suited to them and similarly the expert chooses the top-k tokens based on the tokens’ preferences. One of the reasons the authors go with expert-choice routing due to the obvious load balancing advantage. The router can hence decide which are the top-k most critical tokens to process. This also prevents the token over- or under-processing problem, where tokens only prefer one path over the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LgXLaZITMBohffOCkopulw.png" /><figcaption>Routing schemes</figcaption></figure><p>The top-k tokens are not causal which means to choose a token in the top-k list we need to have access to all the tokens, even the ones that come after it. The model does not have access to the autoregressive sampling. The researchers add a simple auxiliary loss that affects the model’s objective allowing for autoregressively sampling. Another method they explored was introducing an auxiliary MLP predictor with the same input and predicting whether the token should be among the top-𝑘 or not in the sequence.</p><h3><strong>Mixture of Recursions (MOR)</strong></h3><p>Paper Links: <a href="https://arxiv.org/pdf/2507.10524">https://arxiv.org/pdf/2507.10524</a></p><p>Code: <a href="https://github.com/raymin0223/mixture_of_recursions">https://github.com/raymin0223/mixture_of_recursions</a></p><p>In July 2025, Google DeepMind dropped another paper, Mixture of Recursions, which combines the strengths of the two approaches. MOR is introduced as the new Pareto frontier for being a single framework for efficient language modelling by converging efficiency paradigms. It significantly lowers validation perplexity across a wide range of model sizes, without incurring the large-model cost.</p><p>The authors introduce the concepts of recursive transformers, parameter-sharing strategies and enhanced training, which form the foundations of the MOR approach. Recursive Transformers reduce the parameter count by reusing layers across depth. The model is partitioned into recursion blocks, where each block shares the pool of parameters. Parameter sharing reduces the unique trainable parameters by a factor of the recursion number, effectively reducing the memory footprint of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fc8POMBxHDx9PLlZ1oM7cQ.png" /><figcaption>Mixture-of-Recursions</figcaption></figure><p>The two components of the MOR architecture are:</p><ol><li><strong>Routing mechanisms</strong> that assign token-specific steps to focus on the more challenging tokens.</li><li><strong>KV caching strategy</strong> that defines how KV pairs are stored and selectively utilised for attention at each recursive step.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6PXFDhhV240dHI54G1naag.png" /><figcaption>Architectural components of Mixture-of-Recursions</figcaption></figure><p><strong>Routing mechanisms</strong><em>-</em></p><p>Expert choice routing: Inspired by the top-k gating in MoD, each recursion depth becomes an expert and selects its preferred top-𝑘 tokens. Top-𝑘 tokens are selected to pass through the recursion block, where for every recursion step r, H is the hidden state, g represents the scalar score, and Φ ′ is the parameter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*1X3EWLBddoYk2f8TkuFHuw.png" /><figcaption>Expert choice routing</figcaption></figure><p>Token-choice routing: Token-choice routing commits each token to a full sequence of recursion blocks from the start. The token is assigned to expert i which corresponds to sequentially applying the recursion 𝑖 times. Tokens choose the expert best suited to them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*WOh8dgMHvySqFApFqC_akA.png" /><figcaption>Token-choice routing</figcaption></figure><p>Each routing technique has its own advantages and challenges where expert choice routing offers a perfect load balancing with static top-𝑘 selection, it suffers from information leakage. On the other hand, token-choice routing is free from such leakage, but typically requires a balancing loss or loss-free algorithms.</p><p><strong>KV Caching Strategies</strong><em>-</em></p><p>Recursion-wise KV caching: KV pairs are cached selectively for only the tokens routed to a recursion step at that level. Attention is then restricted to those locally cached tokens improving memory efficiency and reducing IO demands.</p><p>Recursive KV sharing: All tokens pass at least through the first recursion block according to the MOR principle. The KV pairs cached at the first step are reused across all subsequent recursions. Hence, context is maintained without recomputation in spite of the query length getting shorter at each recursion depth.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b553d38171c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Power of a Few: LoRa]]></title>
            <link>https://medium.com/@ketaki.kolhatkar99/the-power-of-a-few-lora-fe4dd7b3bde4?source=rss-53ab7f6ac8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe4dd7b3bde4</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ketaki]]></dc:creator>
            <pubDate>Sun, 03 Aug 2025 13:33:21 GMT</pubDate>
            <atom:updated>2025-08-03T13:33:21.450Z</atom:updated>
            <content:encoded><![CDATA[<p>Paper Link: <a href="https://arxiv.org/pdf/2106.09685">https://arxiv.org/pdf/2106.09685</a></p><p>Low Rank Adaptation research is built on the strong foundation of the massive pre-trained models with trillions of parameters. Back in 2021, even after the advent of the revolutionary ChatGPT, Microsoft was still questioning whether these models could actually become commercially viable. As Edward Hu, the research lead behind LoRa at Microsoft, explains in his <a href="https://www.youtube.com/watch?v=DhRoTONcyZE">talk</a>, few-shot learning was not enough to adapt to different use cases, and gradient updates required immense storage and hence the invention of LoRa was necessary.</p><p>What did this research mean for businesses?</p><p>Suppose company A wanted to use a language model for four different tasks, like writing emails, detecting spam, handling sales and managing marketing campaigns. Earlier, that would mean fine-tuning 4 separate models on a pre-trained model with say 175B parameters. That is 4x the cost, compute and storage. This makes it unfeasible to scale the large models. Using a pre-trained model does not work for diverse use cases, therefore fine-tuning is necessary.</p><p>LoRa cuts this mess. It recognises the fact that the weight updates during fine-tuning lie in a low-rank subspace. It introduces low rank matrices (say 18M parameters) while keeping the original model weights frozen. This method gave the same performance, with much less storage and scalability across diverse tasks. LoRa comes under the PEFT- Parameter-Efficient Fine-Tuning techniques that aim at fine-tuning language models by updating only a small subset of the parameters instead of the entire model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*Nx2_VYKPv7HeroRHA3k0Cw.png" /><figcaption>LoRa Architecture</figcaption></figure><p>As mentioned earlier, LoRa is based on the hypothesis that a change in weights during model adaptation has a low “intrinsic rank”. Instead of fine-tuning the full model, LoRA trains a pair of low-rank matrices (A and B) inserted into specific layers of the neural network. We can use the same base model for different tasks by keeping it frozen and replacing the matrices A and B, reducing storage requirements and task switching overhead. Training is more efficient and requires fewer computing resources, given that we are only optimising the low-rank matrices. Best of all, LoRa adds no additional inference latency in comparison to the fully trained model, making it a simple yet practical solution for scalable adaptation.</p><p>For reference, during traditional fine-tuning, model weights Φ0 are updated to Φ0 + ∆Φ by computing gradients. This update is done to minimise the cross-entropy loss for the conditional language modelling objective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ygKjn-JMui6V1L34miPAg.png" /><figcaption>Cross Entropy loss on the fine tuned model</figcaption></figure><p>In this method, the fine-tuned model learns a different set of parameters for each downstream task, ∆Φ. LoRa adopts a parameter-efficient approach where it learns task-specific parameters that are encoded by a smaller set of parameters Θ with the dimension of |Θ|&lt;&lt;|Φ0|.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*QU93XizHIovmY7xT3cDRRQ.png" /><figcaption>Cross Entropy loss using LoRa</figcaption></figure><p>How do the Low Rank Updates work?</p><ol><li>Given a parameter budget, the authors focus on only the <strong>self-attention</strong> weight matrices (Wq, Wk, Wv, Wo).</li><li>Surprisingly, LoRa performs comparatively with very small ranks, suggesting the “intrinsic rank” of the update matrix to be quite low. The authors perform subspace similarity between ranks 8 and 64. Using singular value decomposition and the Grassmann distance, they found significant overlap, suggesting meaningful information can be captured by lower-dimensional updates. Refer to Figure 5 in the paper to understand the concept better.</li><li>∆W has a higher correlation with W. ∆W not only amplifies the important features of W for the downtream task, but also emphasises those features that were not focused on in W.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*4nlgkdXa9FB2QKoKCJOzLA.png" /><figcaption>LoRa Equation: W0- base model; ∆W- BA low rank matrices</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PuR8ECsV4Zbtbni4OD3K6Q.png" /><figcaption>Validation accuracy on WikiSQL and MultiNLI with different rank r. To our surprise, a rank as small as one suffices for adapting both Wq and Wv on these datasets while training Wq alone needs a larger r.</figcaption></figure><p>LoRa proves to perform better on validation accuracy with lower trainable parameters as compared to other methods like <a href="https://medium.com/@razavipour6/understanding-prefix-tuning-a-novel-approach-to-fine-tuning-language-models-dc7dafeb32e4">prefix embedding</a>, prefix layering and adapters when tested on GPT3 175B. We also observe a 25% speedup during training on GPT-3 175B compared to full fine-tuning. For a large Transformer trained with Adam, we reduce that VRAM usage by up to 2/3 if r&lt;&lt;dmodel as we do not need to store the optimizer states for the frozen parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ugzJFv7uKBuNGMw-EnL-CQ.png" /><figcaption>GPT-3 175B validation accuracy vs. number of trainable parameters of several adaptation methods on WikiSQL and MNLI-matched. LoRA exhibits better scalability and task performance.</figcaption></figure><p>With the LoRa research a few future work directions include</p><ul><li>Combination of LoRa with efficient adaptation methods, potentially providing orthogonal improvement</li><li>Rank-deficiency of ∆W suggests that W could be rank-deficient as well, suggesting training itself could be done with fewer parameters</li></ul><p>References:</p><p>[1] <a href="https://www.youtube.com/watch?v=DhRoTONcyZE">https://www.youtube.com/watch?v=DhRoTONcyZE</a></p><p>[2] <a href="https://www.youtube.com/watch?v=PXWYUTMt-AU&amp;t=861s">https://www.youtube.com/watch?v=PXWYUTMt-AU&amp;t=861s</a></p><p>[3] <a href="https://www.youtube.com/watch?v=dA-NhCtrrVE&amp;t=2s">https://www.youtube.com/watch?v=dA-NhCtrrVE&amp;t=2s</a></p><p>[4] <a href="https://en.wikipedia.org/wiki/Singular_value_decomposition">https://en.wikipedia.org/wiki/Singular_value_decomposition</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe4dd7b3bde4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Language Model is Secretly a Reward Model]]></title>
            <link>https://medium.com/@ketaki.kolhatkar99/your-language-model-is-secretly-a-reward-model-9559a80ce9c2?source=rss-53ab7f6ac8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9559a80ce9c2</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[rlhf]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Ketaki]]></dc:creator>
            <pubDate>Fri, 25 Jul 2025 14:24:01 GMT</pubDate>
            <atom:updated>2025-07-25T20:25:15.618Z</atom:updated>
            <content:encoded><![CDATA[<p>Paper Link: <a href="https://arxiv.org/pdf/2305.18290">https://arxiv.org/pdf/2305.18290</a></p><p>Your language model isn’t just a stack of Transformers spitting out tokens; it is a well-engineered feedback loop that gives positive reinforcement on responses that align better with human preferences. We want our models to be respectful, not racist or rude. The data we train these models on is a mixed bag, maybe even less of the good stuff we want. Respectful responses are often lost in the noisy, cluttered internet data.</p><p>The <a href="https://en.wikipedia.org/wiki/Reinforcement_learning">Reinforcement Learning</a> from Human Feedback (<a href="https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback">RLHF</a>) technique used in ChatGPT for introducing a bias to the model toward preferred output is one of the core algorithms in this space. This layer of reinforcement bridged the gap between a raw model response and a more human-aligned behaviour, proving to be a pivotal moment in the language model breakthroughs. Models like Mistral, Meta Llama 3.1 Instruct and Zephyer have been fine-tuned using DPO a simpler and more efficient successor to RLHF.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0jzJ0GYICUMkYaQLnYm5uQ.png" /><figcaption>The core of Reinforcement Learning: model trying to maximize the reward</figcaption></figure><p>So how does RLHF really fit into the whole LLM story? It has three phases:</p><ol><li>Supervised Fine Tuning (SFT)- we start with <a href="https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)">fine-tuning</a> a <a href="https://en.wikipedia.org/wiki/Generative_pre-trained_transformer">pre-trained</a> language model on high-quality data of our objective</li><li>Reward Modelling- the SFT model is then prompted with prompts to produce pairs of answers y1, y2. Here is where we have the feedback loop, where humans are asked to mark the answer for their preference. The equation below uses the Bradley-Terry (BT) model to represent the human preference distribution of the preferred and dispreferred solution (y1 being the preferred, and y2 the dispreferred one).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*COD6vj6d4S2k0mGHapwaag.png" /><figcaption>BT model to represent reward modeling</figcaption></figure><p>Framing the problem as a binary classification, we would have the negative log likelihood loss as follows. To summarise, in this step, we get a reward function using probabilistic models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*UWzJ3yC5l_ssCtr8skiA3A.png" /><figcaption>Loss function for the reward model</figcaption></figure><p>3. RL Fine Tuning- here, the learned reward function from step 2 is used to provide feedback to the language model. πθ is the optimised model using reinforcement learning, while πref is the base model. The first part of the equation is the reward function, and the second is the constraint added to prevent the optimised model πθ from deviating too far from the reference model. <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence">KL divergence</a> is used to measure and minimise this divergence. Given the discrete nature of the language generation, the equation is not differentiable and hence optimised using reinforcement learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*4NrnvkLqF6NlKxgXZlQF7w.png" /><figcaption>RLHF RL fine tuning</figcaption></figure><p>While RLHF seems like a neat approach, taking into consideration the human feedback, it does have some challenges. Human intervention comes with uncertainty and subjectivity, hence responses can be inconsistent. The process is resource-intensive and compounds errors at every step (labelling, reward learning, RL).</p><p>This is where Direct Preference Optimisation (DPO) steps in, where we derive a simple approach for policy optimisation using preferences directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yBfxOLlsCLDIoXdf-Tke4g.png" /><figcaption>RLHF vs DPO</figcaption></figure><p>DPO utilises mathematical tricks to get rid of the reward function from the equation hence learning preferences directly from human feedback!</p><p>Refer to Appendix A.1 of the paper for a detailed derivation breakdown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_0rRd26BhYXQy3JUoeFNKw.png" /><figcaption>DPO loss function</figcaption></figure><p>Since this equation is not discrete (given no reward function), it is now differentiable! Hence, we can take the gradient of the loss function and apply gradient descent to obtain the optimal outcome. More intuitively, the loss function increases the likelihood of the preferred completions yw (preferred answer) and decreases the likelihood of dispreferred completions yl (dispreferred answer).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gudTiaXpfG8LiDp7akfvZQ.png" /><figcaption>DPO loss function gradient</figcaption></figure><p>The authors go on to prove that although DPO does not have the reward function in the equation, it implicitly models one. The shift in the log probabilities of the fine-tuned and the reference model tells how much the response aligns with human preferences, like a reward score. The language model is secretly a reward model!</p><p>We see in a summarisation task, DPO-generated summaries were preferred by humans 58% of the time vs just 17% for PPO. Even GPT-4 agreed more with human preferences when evaluating DPO, showing DPO aligns better <em>and</em> simpler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*y07Dkd95yYXrdpmJTU6RKg.png" /><figcaption>Comparing human and GPT-4 win rates and per-judgment agreement on summarisation samples. Humans agree with GPT-4 about as much as they agree with each other. Each experiment compares a summary from the stated method with a summary from PPO with temperature 0.</figcaption></figure><p>While DPO comes out as an outstanding alternative to RLHF in terms of the cost of training, accuracy and efficiency, there still lie some questions that need to be answered. How does the DPO policy generalise out of distribution, compared with learning from an explicit reward function? Additionally, while we evaluate models up to 6B parameters, exploration of scaling DPO to state-of-the-art models orders of magnitude larger is an exciting direction for future work.</p><p>References:</p><p>[1] <a href="https://www.youtube.com/watch?v=k2pD3k1485A">https://www.youtube.com/watch?v=k2pD3k1485A</a></p><p>[2] <a href="https://www.youtube.com/watch?v=hvGa5Mba4c8">https://www.youtube.com/watch?v=hvGa5Mba4c8</a></p><p>[3] <a href="https://www.superannotate.com/blog/direct-preference-optimization-dpo">https://www.superannotate.com/blog/direct-preference-optimization-dpo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9559a80ce9c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flash Attention: Beginning the IO-Aware Model Era]]></title>
            <link>https://medium.com/@ketaki.kolhatkar99/flash-attention-beginning-the-io-aware-model-era-96ff25a09d50?source=rss-53ab7f6ac8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/96ff25a09d50</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[flash-attention]]></category>
            <dc:creator><![CDATA[Ketaki]]></dc:creator>
            <pubDate>Fri, 18 Jul 2025 15:08:44 GMT</pubDate>
            <atom:updated>2025-07-18T15:08:44.664Z</atom:updated>
            <content:encoded><![CDATA[<p>Paper Link: <a href="https://arxiv.org/pdf/2205.14135">https://arxiv.org/pdf/2205.14135</a></p><p>Continuing the theme on optimising Transformers, we move from <a href="https://medium.com/@ketaki.kolhatkar99/think-faster-with-speculative-decoding-05ff1ea7c470">speculative decoding</a> to flash attention, where our bottleneck moves from inference latency to memory! Flash Attention isn’t just a clever trick, but has proven to be a core optimising technique at OpenAI, HuggingFace, MosaicML and more. The technique allows longer context windows, the same attention output quality and better GPU utilisation.</p><p>Transformers are slow and memory-hungry on long sequences. Despite having a compute-rich environment, transformers are limited by how fast they can move data; we could say the transformers are memory-bound.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*qtUd1Z5x6KOl1uzi1t_vyQ.jpeg" /><figcaption>FLASH ATTENTION!</figcaption></figure><p>The Flash Attention method focuses on the self-attention piece of the Transformer model. This method aims at computing the exact attention with far fewer memory accesses. Self-attention is the crucial part of the Transformer that allows it to have longer context windows, something that its predecessors lacked. The following blog by Jay Alamaar (<em>The Best Transformers blog out there!</em>) breaks down the architecture and goes deeper into each layer of the model.</p><p><a href="https://jalammar.github.io/illustrated-transformer/">The Illustrated Transformer</a></p><p>Self-Attention revolves around the 3 key vectors: Q, K, and V. This process involves a bunch of matrix manipulations with these vectors and the input token window. As shown below, the final step of self-attention is taking the <a href="https://en.wikipedia.org/wiki/Softmax_function">softmax</a> on the matrix multiplication of Q with K transpose and then multiplying the result with the V vector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5A4_sbuTbah0S4T5qdQAqg.png" /></figure><p>This might seem like a simple operation, and could be optimised by throwing more compute at it. But the real bottleneck here is the memory overhead, not the compute. As the algorithm below shows, we need to read and write from memory multiple times for the final output. This is what Flash Attention tries to solve: the latency of this memory read and write.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uAzDt5_cOI2bo4L8UkjDVQ.png" /><figcaption>Standard Attention Algorithm</figcaption></figure><p>Hence, we focus on the GPU memory hierarchy. A GPU consists of multiple memory components of different sizes and speeds, with the size of the memory being inversely proportional to the speed. SRAM is an order of magnitude faster than HBM (High Bandwidth Memory) and orders of magnitude smaller in size. The researchers exploit this memory hierarchy by utilising the fast SRAM and minimising the data written back to the GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*47cOxjBJWrqtvvvHFpqc8w.png" /></figure><p>Coming back to the Flash Attention algorithm, the paper talks about two methods to achieve its objectives of minimum memory access and exact attention output: 1. computing the softmax operation without accessing the whole memory. 2. not storing the large intermediate attention matrix for the backwards pass. The researchers apply the following techniques to achieve these:</p><p>I. T<a href="https://en.wikipedia.org/wiki/Tiled_rendering">iling</a>, a method where we restructure the attention into blocks and compute softmax incrementally over these blocks</p><p>II. Store the softmax normalisation factor from the forward pass to have a quick recompute attention on-chip in the backwards pass faster than the traditional approach of reading and writing the intermediate attention matrix to and from the HBM</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*e6aH3hRY-VMSy24OqsWwUg.png" /><figcaption>Flash Attention with Tiling left and performance on GPT-2 on right</figcaption></figure><p>With these two techniques of tiling and having the softmax normalisation factor stored on chip, Flash Attention achieves fine-grained control over memory access and fuses all the attention operations into one GPU kernel, also known as kernel fusion. The algorithm utilises online softmax. The algorithm below showcases tiling, the online softmax methods mentioned in the paper, and the use of SRAM instead of HBM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YBD3n0wH6nCWXWn7DXcaVQ.png" /><figcaption>Flash Attention Algorithm</figcaption></figure><p>Flash Attention has proven to be an all-star. The method has shown faster training speeds, higher output quality, and allows us to increase the input context length.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ChzQgqyrJb3C98gLpYgyjw.png" /><figcaption>GPT-2 small and medium using FlashAttention achieve up to 3× speed up compared to Huggingface implementation and up to 1.7× compared to Megatron-LM. Training time reported on 8×A100s GPUs.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bSDR-XqhFCUYaf6yJbH3LQ.png" /><figcaption>GPT-2 small with FlashAttention, with 4× larger context length compared to Megatron-LM, is still 30% faster while achieving 0.7 better perplexity. Training time on 8×A100 GPUs is reported.</figcaption></figure><p>A few limitations of this approach are configuring the IO-aware implementation of attention to a new kernel that requires writing in a low-level language and a significant engineering effort. The current approach focuses on a single GPU implementation, and parallelisation across multiple GPUs would require additional IO analysis.</p><p>Future work could involve working on these parallelisations and making the systems IO-aware. Self-attention is just one key component of the Transformer, and there’s still plenty of room to explore and optimise other components that heavily access HBM.</p><p>References</p><p>[1] A complete overview with code: <a href="https://www.youtube.com/watch?v=zy8ChVd_oTM">https://www.youtube.com/watch?v=zy8ChVd_oTM</a></p><p>[2] <a href="https://jalammar.github.io/illustrated-transformer/">https://jalammar.github.io/illustrated-transformer/</a></p><p>[3] <a href="https://www.youtube.com/watch?v=LKwyHWYEIMQ">https://www.youtube.com/watch?v=LKwyHWYEIMQ</a></p><p>[4] <a href="https://huggingface.co/docs/text-generation-inference/en/conceptual/flash_attention">https://huggingface.co/docs/text-generation-inference/en/conceptual/flash_attention</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96ff25a09d50" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>