<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bahadır AKDEMİR on Medium]]></title>
        <description><![CDATA[Stories by Bahadır AKDEMİR on Medium]]></description>
        <link>https://medium.com/@akdemir_bahadir?source=rss-711524cb67b0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*h5DmlGcFtgk-z3veiR9HMg@2x.jpeg</url>
            <title>Stories by Bahadır AKDEMİR on Medium</title>
            <link>https://medium.com/@akdemir_bahadir?source=rss-711524cb67b0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 05:02:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@akdemir_bahadir/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[TwinFlow: An Interesting Approach for 1-Step Image Generation]]></title>
            <link>https://medium.com/@akdemir_bahadir/twinflow-an-interesting-approach-to-1-step-image-generation-c98e7589dd30?source=rss-711524cb67b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c98e7589dd30</guid>
            <category><![CDATA[image-generation]]></category>
            <category><![CDATA[ai-image-generator]]></category>
            <category><![CDATA[twinflow]]></category>
            <category><![CDATA[flowmatching]]></category>
            <dc:creator><![CDATA[Bahadır AKDEMİR]]></dc:creator>
            <pubDate>Sat, 10 Jan 2026 21:43:56 GMT</pubDate>
            <atom:updated>2026-01-11T08:14:26.646Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kpRUIBL_K0h7R2ULM4qa5Q.png" /></figure><p>Current image generation models deliver impressive visual correctness, but they do so at a remarkable computational cost. Training demands extremely high memory usage, large-scale GPU usages, and specialized hardwares. Even at inference time, generating a single high-quality image typically requires 20 - 100 sequential computation steps (generally), resulting in high latency and significant energy consumption. These requirements remain a major barrier to practical deployment, especially for real-world and resource-constrained settings.</p><blockquote><em>Note: In my last two posts, I focused on </em><strong><em>Z-Image</em></strong><em>, an approach that also targets efficient image generation by reducing the number of inference steps (nearly 4–8 steps). If you are interested in related ideas and techniques, you can read more about </em><strong><em>Z-Image </em></strong><em>in</em><strong><em> </em></strong><a href="https://medium.com/@akdemir_bahadir/what-makes-z-image-so-efficient-part-1-introduction-data-infrastructure-12bb86b61a9c"><em>here</em></a><em> and </em><a href="https://medium.com/@akdemir_bahadir/what-makes-z-image-so-efficient-part-2-architecture-training-9bae9d7d947e"><em>here</em></a><em>.</em></blockquote><p>Against this backdrop, a research team has introduced TwinFlow, a method designed to significantly improve generation efficiency. Instead of scaling models or hardware further, TwinFlow focuses on rethinking the generation process itself. This shift enables reductions in memory while training for reducing generation steps and computational overhead while preserving the visual quality expected from modern image generation systems.</p><h3>Understanding the Inference Bottleneck</h3><p>Modern image generation models (like Qwen Image, Stable Diffusion etc.) achieve impressive results, but they share a fundamental limitation: computational inefficiency.</p><p>These models require multiple sequential steps to generate a single image. Each step involves a function evaluation that gradually transforms noise into the final output. This process resembles iterative refinement, where the model makes incremental improvements at each stage.</p><p>For a 20-billion parameter model like Qwen-Image-20B, generating one 1024×1024 image takes approximately 40 seconds on a single A100 GPU (<em>According to the paper</em>).</p><p>This matters because inference costs compound over time. Unlike training, which is largely a one-time expense, inference incurs a recurring cost with every generation request. As usage scales, these repeated computations quickly dominate total system expenditure. For organizations deploying image generation models in production environments, inference efficiency is therefore not a secondary concern but the primary cost driver that determines economic viability.</p><h3>Why Previous Acceleration Methods Face Limitations?</h3><p>Researchers have explored several approaches to reduce inference steps. Each has encountered specific challenges.</p><h4>Distillation Methods</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Za0Jm48nXUEx1h9pf21tug.png" /><figcaption>Progressive Distillation Example. I am shocked that I did generate this by nano banana pro. It drew nearly perfect explanation!</figcaption></figure><p>Knowledge distillation trains a smaller (or equal size) “student” model to replicate a “teacher” model’s behavior. The student learns to generate similar outputs in fewer steps .</p><p><strong>The limitation:</strong> The teacher model must remain loaded in memory during training. For models already approaching 20 billion parameters, maintaining both student and teacher in memory creates significant hardware constraints. Additionally, output quality typically degrades when reducing to fewer than 4 steps.</p><h4>GAN-Based Approaches</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q3BE4wMCbW0hs4I7UHnkWQ.png" /><figcaption>A typical GAN approach</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_PczNUuy0_7g4lM2-OyKQ.png" /><figcaption>A figure of DMD2 approach from <a href="https://arxiv.org/abs/2405.14867">its original paper</a>. Simply, it is a Distribution Matching approach with the help of GAN-like discriminator in order to reduce iterations of a diffusion model.</figcaption></figure><p>GAN-based approaches aim to achieve one-step (or few step) image generation by introducing adversarial training. A discriminator is added to judge whether generated images appear realistic, enabling these methods to produce higher-quality outputs even with a single generation step. In theory, the discriminator provides a strong learning signal that pushes the generator to converge rapidly toward visually plausible results.</p><p>This efficiency, however, comes with notable trade-offs. Adversarial training requires optimizing multiple components simultaneously, specifically the generator and the discriminator. This setup significantly increases GPU memory consumption and introduces fragile training dynamics. Models may exploit loopholes in the discriminator’s objective, leading to unstable behavior or visually absurd outputs. As model size increases, maintaining stable optimization becomes increasingly difficult. At scale, these issues become prohibitive. For instance, when researchers attempted to apply DMD2, a GAN-based method, to Qwen-Image-20B, the training process exceeded available GPU memory and could not be completed.</p><h3>The TwinFlow Method</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kpRUIBL_K0h7R2ULM4qa5Q.png" /></figure><p>TwinFlow addresses these limitations through a self-contained training framework. The key innovation lies in how it structures the learning process.</p><h4>Twin Trajectories</h4><p>Traditional flow models operate on a time interval from t=0 to t=1. At t=1, the model starts with fully random noise. At t=0, it produces the final image. The model learns the transformation pathway between these states.</p><p>TwinFlow extends this interval to range from t=-1 to t=1:</p><ul><li><strong>Positive time (t=0 to t=1)</strong>: The standard trajectory from noise to real data</li><li><strong>Negative time (t=-1 to t=0)</strong>: A parallel trajectory from noise to the model’s current (fake) outputs (which may be imperfect)</li></ul><p>This creates two symmetric trajectories originating from the same noise distribution. One trajectory represents the path to real images, while the other represents the path to the model’s generated outputs.</p><h4>…But why?</h4><p>The twin-trajectory approach in TwinFlow effectively replaces the role of a discriminator of a GAN-like approach without introducing an additional model. Instead of learning realism through a separate network, the generator itself is repurposed to perform discrimination.</p><p>The process works as follows.</p><p>1- First, a “fake” output is produced using the standard denoising procedure to generate an image.</p><p>2- This generated image is then noised again and the model is asked to predict the fake* image, but this time along a negative trajectory.</p><p>3-By contrasting the negative prediction with the positive one, the model implicitly learns the difference between its own imperfect output and the target distribution. In this sense, the generator acts as both producer and critic.</p><p>4- The negative part will be more similar when model learns its mistakes.</p><p>This design avoids the heavy memory overhead of approaches like DMD2, since no additional discriminator or teacher model is required. GPU memory usage remains close to that of standard diffusion training.</p><p>You might wonder whether this comparison could be performed within the positive trajectory alone, without introducing a separate negative path. In principle, this seems plausible. However, the generated image represents an incorrect or “fake” sample from the model’s perspective. Mixing such samples into the positive trajectory would contaminate the generator’s core learning signal. By isolating the fake–real contrast in a separate negative trajectory, TwinFlow preserves the integrity of the positive path, which remains responsible for standard image generation.</p><blockquote><em>TL;DR: We want to add a discriminator without additional model and negative time part helps to achieve this.</em></blockquote><h3>Self-Adversarial Learning</h3><p>Instead of requiring an external discriminator, TwinFlow minimizes the difference between the velocity fields of these two trajectories.</p><p>Consider the implication: if the model’s generated trajectory matches the real data trajectory, the model can produce realistic images in a single step (Like in GANs). The model learns by comparing its own velocity fields rather than relying on external components.</p><blockquote><strong><em>Note:</em></strong><em> In summary, the core idea is to make the real and fake trajectories so similar that denoising a generated image becomes indistinguishable from denoising a real one. By enforcing this alignment, the model is encouraged to correct its own mistakes and produce stronger predictions at each step. As a result, the first generation step becomes significantly more powerful than in standard approaches. While additional steps can still be used to further improve quality, the paper shows that a single step is already sufficient to achieve strong results.</em></blockquote><h3>Loss Function</h3><p>The overall training objective is composed of three components: <strong>Base Loss</strong>, <strong>Self-Adversarial Loss</strong>, and <strong>Rectification Loss</strong>.</p><h4><strong>Base Loss</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uut-8A68FhkjOw55O9nugA.png" /><figcaption>Yeah, I also do not easily understand such formulas when they are presented in this form. The important part is to understand the aim and the conceptual use of the loss. Here, the formula states that we first predict the trajectory, and then measure how different the predicted trajectory is from the ground-truth trajectory.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*piqrzkBSlP2Pr8kydSD0iw.png" /></figure><p>This term corresponds to the standard flow-matching (or any-step) objective. It trains the model to predict the velocity field that transports samples from the noise distribution to the real data distribution along the positive-time trajectory. This loss ensures accurate multi-step generation behavior and serves as the foundational training signal for learning the underlying probability flow.</p><h4><strong>Self-Adversarial Loss</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rp4HwxypPX_Z2P1EF6JoxQ.png" /><figcaption>It is the same as the base loss, except this time it is applied at the negative state, and the target data is the fake data generated earlier by the model.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*emkL5QpzOHmEsXcKYHUjDA.png" /></figure><p>The self-adversarial loss is defined on the negative-time branch of the extended trajectory. Instead of using an external discriminator, the model learns to map noise to its <em>own generated (fake) samples</em>. By training the model to predict velocity fields for these fake trajectories under negative time conditioning, the model implicitly plays both generator and discriminator roles within a single network. This self-contained adversarial signal improves the realism and robustness of one-step generation without introducing auxiliary adversarial networks</p><h4><strong>Rectify Loss</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zxKerBPkGvaoAsS0ymoJTA.png" /><figcaption>This loss compares the trajectories obtained from real data and fake data, and penalizes their difference, encouraging the model to align the fake trajectory with the real one.</figcaption></figure><p>The rectification loss enforces consistency between the real and fake trajectories by directly minimizing the discrepancy between their corresponding velocity fields. Concretely, it aligns the velocity predicted for the fake trajectory (negative time) with that of the real trajectory (positive time). This process effectively “straightens” the generative path from noise to data, enabling the model to approximate the full integration process with much larger step sizes. As a result, the model learns stronger and more direct transitions, substantially reducing the number of inference steps required for high-quality generation</p><blockquote><strong><em>TL;DR: </em></strong><em>The base loss compares the predicted trajectory with the real-data trajectory, the adversarial loss compares the negative time state predicted trajectory with the fake-data trajectory, and the rectification loss compares the fake-data trajectory with the real-data trajectory.</em></blockquote><h3>Results</h3><p>I do not go into the results too much here. You cannot find, in most cases, papers where a novel method has worse results 😄. If you are curious about the numbers and comparisons, the paper already does a much better job showing them.</p><h4>Memory Efficiency by Applying Self-Adversarial Method</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*aNrWfDosDKhsnm7RimB2NA.png" /><figcaption>The figure is from <a href="https://arxiv.org/abs/2512.05150">the original paper.</a></figcaption></figure><h4>Inference Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y-xhME2rsjGgOU-edfFp5A.png" /><figcaption>The figure is from <a href="https://arxiv.org/abs/2512.05150">the original paper.</a></figcaption></figure><h4>Evaluation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hecrGa4_RQXuFxA_Vc8hmg.png" /><figcaption>The table is from <a href="https://arxiv.org/abs/2512.05150">the original paper.</a></figcaption></figure><blockquote><strong><em>Note:</em></strong><em> I could not try the original InclusionAI/TwinFlow Qwen-Image model myself. Instead, I tried to use their Z-Image model. Unfortunately, I could not run it again (due to some issues with my computer). So I looked for other Hugging Face accounts claiming to share the TwinFlow Z-Image model, which I was able to run.</em></blockquote><blockquote><em>Based on my own one-step tests between TwinFlow Z-Image and the original Z-Image model, I did not see particularly impressive results at one step. However, since I could not test the original model directly, I do not want to share these results here. You can also check this </em><a href="https://github.com/leejet/stable-diffusion.cpp/issues/1153"><em>issue</em></a><em> where other people try and see similar results with me. If I can run the original model in the future (and hopefully do not forget), I plan to share my results. If you have tried it yourself, I would be happy to read your comments below.</em></blockquote><h3>Summary</h3><p>TwinFlow’s key contributions can be summarized as follows.</p><p><strong>Architectural simplicity.</strong> By removing auxiliary networks and frozen teacher models, TwinFlow significantly reduces training complexity and memory overhead compared to GAN-style approaches.</p><p><strong>Performance efficiency.</strong> Single-step inference achieves quality comparable to traditional N-step generation, delivering an effective Nx reduction in computation.</p><blockquote><strong><em>Note</em></strong><em>: While the paper primarily evaluates TwinFlow using Qwen-Image, the method is also applied to Z-Image. The resulting models are publicly available through </em><a href="https://huggingface.co/inclusionAI"><em>the authors’ Hugging Face repository.</em></a></blockquote><h3>References / Sources</h3><p><a href="https://arxiv.org/abs/2512.05150">TwinFlow: Realizing One-step Generation on Large Models with Self-adversarial Flows</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c98e7589dd30" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Makes Z-Image So Efficient? (Part 2): Architecture & Training]]></title>
            <link>https://medium.com/@akdemir_bahadir/what-makes-z-image-so-efficient-part-2-architecture-training-9bae9d7d947e?source=rss-711524cb67b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bae9d7d947e</guid>
            <category><![CDATA[z-image-turbo]]></category>
            <category><![CDATA[image-generation]]></category>
            <category><![CDATA[z-image]]></category>
            <dc:creator><![CDATA[Bahadır AKDEMİR]]></dc:creator>
            <pubDate>Tue, 06 Jan 2026 14:00:43 GMT</pubDate>
            <atom:updated>2026-01-06T14:00:43.537Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z6vntAyeQbyzP9xHV4_7ig.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@pawel_czerwinski?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Pawel Czerwinski</a> on <a href="https://unsplash.com/photos/body-of-water-g0eRErPBoTA?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In the <a href="https://medium.com/@akdemir_bahadir/what-makes-z-image-so-efficient-part-1-introduction-data-infrastructure-12bb86b61a9c">first part</a>, we examined the data-centric foundations of Z-Image, focusing on the profiling engines, knowledge graphs, and active curation loops that underpin the system.</p><p>In this second part, we shift to the model side, detailing the technical architecture of the <strong><em>Scalable Single-Stream Diffusion Transformer (S3-DiT)</em></strong> and the multi-stage training pipeline that enables its performance and scalability.</p><blockquote><strong><em>Note:</em></strong><em> While writing this post, I noticed that explaining every small detail makes it much longer than necessary. Splitting it into multiple parts would also make it harder to follow. So, I decided to explain only the parts that are specific to the Z-Image architecture and skip well-known or previously explained concepts. If anything is unclear, feel free to ask in the comments.</em></blockquote><h3>S3-DiT Architecture</h3><p>The architectural design of Z-Image is guided by two primary objectives: <em>efficiency</em> and <em>stability</em>. Many diffusion models employ complex dual-stream designs that process textual and visual modalities through separate representation pathways before fusing them. Z-Image instead adopts a Single-Stream Multi-Modal Diffusion Transformer (MM-DiT) paradigm.</p><p>In this approach, all modalities are embedded into a unified token space and processed by a single transformer backbone. This enables tighter cross-modal interactions, reduced architectural overhead, and more stable optimization dynamics.</p><h4>The Unified Input Stream</h4><p>The defining characteristic of S3-DiT is that it treats text and images as a single, unified sequence. Instead of keeping modalities separate, the model concatenates:</p><ul><li><strong>Text tokens</strong> from a lightweight <strong>Qwen3–4B</strong> encoder.</li><li><strong>Visual semantic tokens</strong> (used for editing tasks) derived from <strong>SigLIP</strong>.</li><li><strong>VAE image tokens</strong> extracted from the <strong>Flux VAE</strong>.</li></ul><blockquote><strong><em>Note:</em></strong><em> In the paper, the authors state that they use Qwen3–4B as the text encoder, even though it is originally a decoder-style model. To make this work, they remove the language modeling head and use the final hidden representations as the encoded input for the generation stage.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tfk7jZcBd_GFBd1YFrQBjw.png" /><figcaption><a href="https://arxiv.org/abs/2511.22699">The figure is from original Z-Image paper</a></figcaption></figure><p>By merging these into a single stream, the model achieves a modality-agnostic architecture that maximizes <strong>cross-modal parameter reuse</strong>. Every parameter in the model is available to process both visual and textual information simultaneously, ensuring that no compute is wasted on redundant, modality-specific layers.</p><blockquote><strong><em>Note:</em></strong><em> As far as I understand, the visual semantic token components are disabled in the base (Turbo) model rather than completely removed. As a result, both the Edit and Turbo models have the same number of parameters. If you have read the paper and think this interpretation is incorrect, please correct me.</em></blockquote><h4>Managing the Sequence: 3D Unified RoPE</h4><p>Processing mixed sequences of different modalities requires a sophisticated way to handle positional information. Z-Image uses <strong>3D Unified RoPE</strong> (Rotary Positional Embeddings).</p><ul><li><strong>Image tokens</strong> expand across the spatial dimensions.</li><li><strong>Text tokens</strong> increment along the temporal dimension.</li><li><strong>For Editing Tasks</strong>: Reference image tokens and target image tokens share the same spatial coordinates but they are separated by an <strong>offset</strong> in the temporal dimension. This allows the model to “align” the source and target images in space while understanding they represent different points in time or state.</li></ul><blockquote><strong><em>Note:</em></strong><em> In simple terms, this means that the target image should be closer to the </em><strong><em>text</em></strong><em> representation than the reference image in the RoPE space. I tried to illustrate this idea with a small Python example, shown below.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F76zggtxL5val-yxYVNHEQ.png" /><figcaption><strong>A simple illustration to understand 3D RoPE.</strong> The numbers in the figure are arbitrary. In practice, the representations are separated by unit offsets, and the distance in RoPE space also changes with the time step (from too noisy to lessnoisy). This figure does not show the exact RoPE distances. It is only meant to give a visual intuition about how the distances relate to each other.</figcaption></figure><h3>Training Pipeline</h3><p>Scaling to high-resolution generation often leads to a “memory wall.” The Z-Image team try to pass this through a multi-faceted efficiency strategy.</p><h4>Solving the Padding Problem</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xScZ02z6aFaKY48OmaTMkg.png" /><figcaption><strong>A simple illustration of smart padding.</strong> While real-world behavior may not look exactly like this, the idea captures the intuition and clearly shows how smart padding can significantly reduce computation.</figcaption></figure><p>In mixed-resolution training, batching images of different sizes usually results in excessive padding, which wastes GPU cycles. Z-Image solves this with <strong>sequence length-aware batch construction</strong>:</p><ol><li><strong>Estimation</strong>: The sampler estimates the sequence length of each sample based on its metadata resolution (height/width) before training.</li><li><strong>Grouping</strong>: Samples with similar sequence lengths are grouped into the same batch, minimizing computational waste from padding.</li><li><strong>Dynamic Batch Sizing</strong>: The system assigns smaller batch sizes to long-sequence batches (preventing Out-Of-Memory errors) and larger batch sizes to short-sequence batches (ensuring high hardware utilization).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3YEnaXx164UGqO9cYEelJA.png" /><figcaption><strong>A simple illustration of dynamic batching.</strong> If we process randomly sized images in a fixed batch, we may encounter out-of-memory (OOM) errors, or we may need to reduce the batch size to be safe — both of which lead to inefficient memory usage. With a basic intuition about memory requirements and dynamic batching, we can better utilize available memory and increase the batch size as much as possible without exceeding limits.</figcaption></figure><h3>Stage 1: Pre-training and Flow Matching</h3><p>Z-Image departs from standard diffusion objectives by using <strong>Flow Matching</strong>. Instead of predicting noise, the model learns the <strong>velocity</strong> v_t of a vector field that defines the path between Gaussian noise x_0 and the original image x_1.</p><blockquote><strong><em>Note:</em></strong><em> Flow matching is a separate topic and deserves its own explanation. For now, if you are not familiar with it, you can think of it this way: instead of denoising an image step by step without knowing how much magnitute of steps are needed, the model predicts the velocity of change for each pixel at the start, and then applies those changes over time using the predicted velocities.</em></blockquote><p><strong>The Two-Phase Pre-training</strong></p><ol><li><strong>Low-Resolution Pre-training</strong>: Conducted at 256² resolution, this stage focuses on efficient cross-modal alignment and “knowledge injection”. It accounts for over half of the total pre-training compute, as this is where the model learns foundational visual knowledge.</li><li><strong>Omni-Pre-training</strong>: Here, the model learns to handle <strong>arbitrary resolutions</strong> and aspect ratios. It also integrates <strong>Image-to-Image</strong> tasks to provide a strong initialization for downstream editing. Crucially, they use <strong>Z-Captioner</strong> to provide multi-level bilingual captions (short, medium, long, and user-simulated prompts) to ensure the model follows diverse instructions.</li></ol><blockquote><strong><em>Note:</em></strong><em> I just noticed that I did not mention Z-Captioner in the previous post. I am too lazy to add it now, but you can check the original paper :)</em></blockquote><h3>Stage 2: Supervised Fine-Tuning (SFT)</h3><p>Once the model has a broad understanding of the world, the goal of supervised fine-tuning is to <strong>narrow</strong> its behavior. Pre-training data is noisy, and SFT helps move the model toward a cleaner, higher-quality sub-manifold with more consistent aesthetics and better alignment.</p><p><strong>Concept Balancing and Tagged Resampling</strong></p><p>One major risk during SFT is catastrophic forgetting. The model may focus too heavily on common, high-quality images and lose the ability to generate rare concepts. To address this, the team uses tagged resampling.</p><p>They maintain a target distribution over concepts using a knowledge graph. Rare or under-represented concepts, such as specific entities or artistic styles, are sampled more often during training. Common concepts are sampled less frequently. This keeps the model balanced without hurting overall quality.</p><blockquote><strong><em>Note</em></strong><em>: Remember from the previous post. Tthey use a smart scheduling graph to select the next batch, prioritizing tasks that have been learned less. This way, each batch contains as diverse a set of tasks as possible, helping the model learn more effectively.</em></blockquote><h4>Model Merging for Robustness</h4><p>Rather than choosing between strong instruction following and photorealism, the team fine-tunes several SFT variants with different priorities and then merges them by linearly interpolating their weights.</p><p>This simple merging strategy smooths training artifacts and results in a final model that behaves more consistently across a wide range of prompts.</p><blockquote><strong><em>Note:</em></strong><em> this is similar to learning from different cases and averaging them to achieve better generalization.</em></blockquote><h3>Stage 3: Few-Step Distillation</h3><p>The standard SFT model requires roughly 100 steps, measured as the number of function evaluations, to generate high-quality samples. To achieve real-time performance, the team applied a few-step distillation strategy that collapses the stochastic sampling process into a deterministic 8-step inference path.</p><blockquote><strong><em>Note:</em></strong><em> as an inference tip, I tested this myself and found that even 5 steps can produce valid results when using 512² or smaller outputs and when text is not required in the image.</em></blockquote><h4>Decoupled DMD and DMDR</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x03osAJIkQ48MTOW9c3AUQ.png" /><figcaption><a href="https://arxiv.org/abs/2511.22699">The figure is from original Z-Image paper</a></figcaption></figure><p>They discovered that traditional Distribution Matching Distillation (DMD) often leads to detail loss and color shifts. Their solution involved two innovations:</p><ul><li><strong>Decoupled DMD (D-DMD)</strong>: The distillation process is split into two separate components. CFG augmentation serves as the main driver for few-step generation, while distribution matching acts as a regularizer that helps suppress artifacts.</li><li><strong>DMDR</strong>: This method combines distillation with reinforcement learning. Reinforcement learning helps align the model with human preferences, while the distribution matching term constrains optimization and prevents reward hacking, where the model produces images that score well but look visually incorrect.</li></ul><p>As a result, <strong>Z-Image-Turbo</strong> achieves 8-step inference that is often indistinguishable in quality from the original 100-step teacher model.</p><h3>Stage 4 Aligning with Human Feedback (RLHF)</h3><p>Even after distillation, a model may still fall short in capturing subtle human preferences. To address this, Z-Image applies a two-stage reinforcement learning from human feedback (RLHF) framework.</p><h4>Reward Annotation</h4><p>The reward model evaluates generated images across three main cases:</p><ul><li><strong>Instruction following:</strong> Human annotators break down prompts into entities, attributes, actions, and constraints, then assess how well the image satisfies them.</li><li><strong>AI-content detection:</strong> Evaluates whether an image appears overly synthetic or “too AI-generated.”</li><li><strong>Aesthetic quality:</strong> Measures overall visual appeal and coherence.</li></ul><h4>DPO and GRPO</h4><p>The RLHF process is split into two complementary stages.</p><ul><li><strong>Stage 1: Offline Alignment (DPO)</strong>: This stage focuses on objective and verifiable criteria, such as accurate text rendering and correct object counts. Vision-language models are used to automatically generate pairs of preferred (“chosen”) and non-preferred (“rejected”) images, which are then used to train the model via Direct Preference Optimization.</li><li><strong>Stage 2: Online Refinement (GRPO)</strong>: The second stage targets more subjective qualities, including photorealism and aesthetics. GRPO optimizes the model using a composite advantage function that balances multiple, often competing, quality signals at the same time.</li></ul><h3>Prompt Enhancer (PE)</h3><p>A 6B-parameter model has inherent limits in world knowledge when compared to much larger models in the 80B range. To reduce this gap, the Z-Image system includes a Prompt Enhancer (PE) driven by a reasoning chain.</p><p>The PE functions as a lightweight reasoning layer that processes the user prompt before it is passed to the image generation model. This allows the system to resolve implicit meaning, add missing context, and translate abstract inputs into concrete visual instructions.</p><blockquote><strong><em>Note:</em></strong><em> Prompt Enhancer is used in Fine-Tune part.</em></blockquote><ul><li><strong>Without Reasoning</strong>: If you give the model coordinates like 30° 09&#39; 36&quot; N, 120° 07&#39; 12&quot; E, the model might just render the text of the numbers.</li><li><strong>With Reasoning</strong>: The PE infers the location (West Lake, Hangzhou) and instructs the model to generate the correct scenic context.</li><li><strong>Content Filling</strong>: For vague prompts like “Five key habits to improve grades,” the PE generates specific content points based on common sense logic (e.g., “Consistent Study Schedule,” “Sleep Routine”) to ensure the final image has concrete details.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZwNRqwlOefozD2p78E0LEw.png" /><figcaption><a href="https://arxiv.org/abs/2511.22699">The figure is from original Z-Image paper</a></figcaption></figure><h3>Discussion</h3><p>We can clearly see that the training pipeline involves many steps and is fairly complex( at least me). As model size decreases, so does effective capacity, which forces us to improve training quality in order to build small yet competitive models. This limited capacity pushes us toward more detailed supervision, heavier engineering, and multi-stage optimization. In practice, we are trading simpler and faster inference for increasingly complex training pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ruhTCKABRfrzepv2VTTqXw.png" /><figcaption><a href="https://arxiv.org/abs/2511.22699">The figure is from original Z-Image paper</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2fI134luxAS7VKQktuJH4Q.png" /><figcaption><a href="https://arxiv.org/abs/2511.22699">The figure is from original Z-Image paper</a></figcaption></figure><h3>Conclusion</h3><p>Z-Image represents a clear shift in how model scale is approached. Instead of relying on raw parameter count, it prioritizes architectural efficiency, a unified single-stream design, intelligent batching strategies, and a carefully constructed multi-stage training curriculum that spans flow matching, distillation, and reasoning-enhanced RLHF. Through this approach, the team at <strong>Alibaba</strong> achieved results that are disproportionate to the model’s size.</p><p>As of late 2025, <strong>Z-Image-Turbo</strong> ranks among the top performers on the Text-to-Image Elo leaderboard published by the <strong>Artificial Analysis AI Arena</strong>. Notably, it remains competitive with large proprietary systems such as <strong>FLUX.2</strong> and <strong>Gemini</strong>, despite using only a fraction of their parameters and operating at a very low, single-digit dollar cost per thousand images. This makes Z-Image a compelling example of how thoughtful system design and training strategy can rival scale-driven approaches.</p><h3>Reference / Sources</h3><p><a href="https://arxiv.org/abs/2511.22699">Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bae9d7d947e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Makes Z-Image So Efficient? (Part 1): Introduction & Data Infrastructure]]></title>
            <link>https://medium.com/@akdemir_bahadir/what-makes-z-image-so-efficient-part-1-introduction-data-infrastructure-12bb86b61a9c?source=rss-711524cb67b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/12bb86b61a9c</guid>
            <category><![CDATA[ai-image-generator]]></category>
            <category><![CDATA[image-generation]]></category>
            <category><![CDATA[z-image-turbo]]></category>
            <category><![CDATA[z-image]]></category>
            <dc:creator><![CDATA[Bahadır AKDEMİR]]></dc:creator>
            <pubDate>Sat, 27 Dec 2025 20:45:15 GMT</pubDate>
            <atom:updated>2025-12-28T09:29:04.385Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How a 6B parameter model challenges industry giants through efficient archiecture and cost-effective training lifecycle</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pTdqvzYIyX27sWPHtVRYNA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@johnygoerend?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Johny Goerend</a> on <a href="https://unsplash.com/photos/galaxy-wallpaper-Oz2ZQ2j8We8?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a>. I know that the image is not related anything with the paper but I could not find a proper cover.</figcaption></figure><h3>Introduction</h3><p>Over the past few years, image generation has advanced at a remarkable improvements. Systems that once produced blurry, dreamlike outputs are now capable of generating photorealistic scenes that are nearly indistinguishable from real photographs.</p><p>Today, the image generation landscape is largely dominated by major technology companies. Flagship models set the benchmark for what is currently achievable. While open-source alternatives such as FLUX and Qwen-Image exist and perform reasonably well, a fundamental challenge remains: these models are simply too large to run efficiently on consumer hardware. With parameter counts ranging from 20 to 80 billion, they often require costly enterprise-grade GPUs to generate even a single image within a practical timeframe.</p><p>The computational cost is equally staggering. Training models of this scale can require investments of millions of dollars, and even inference demands resources far beyond the reach of most developers and researchers. As a result, the field has become increasingly polarized, with access to massive compute infrastructure determining who can meaningfully participate.</p><p>Last month, the research team at Tongyi-MAI/Alibaba introduced a notably different approach with the release of <strong><em>Z-Image</em></strong>. At a time when increasing model size is often treated as the default path to progress, they presented a 6-billion-parameter model that directly challenges the prevailing scale-at-all-costs mindset. The release includes two variants: Z-Image-Turbo, designed for text-to-image generation, and Z-Image-Edit, focused on image-to-image editing tasks.</p><blockquote><strong><em>Note: </em></strong><em>You can run a quantized version of this model with just 4GB of VRAM and still get approximately 2 seconds per step at 512x512 resolution. Since the model produces usable results in just 5–7 steps, you can generate an image in roughly 10–15 seconds on consumer-grade hardware. That’s remarkable (at least for me).</em></blockquote><blockquote><strong><em>Note:</em></strong><em> Z-Image-Edit has not been released yet, though I am eagerly waiting to try it.</em></blockquote><p>The researchers introduced innovations across the entire model lifecycle, spanning data processing, training strategies, and inference optimization. To keep this post readable, I will break their approach into several parts. We begin with how data is handled.</p><h3>Data Handling in Z-Image</h3><p>The importance of data in AI projects is already well understood. Beyond its mere availability, data quality plays a decisive role in model performance. More data does not automatically lead to better models; poorly balanced distributions, ambiguous samples, or noisy contexts can degrade performance and introduce unnecessary complexity during training.</p><p>To address these challenges, the Z-Image team designed a four-phase data management system. Below, we break down each phase in detail.</p><h4>1. Data Profiling Engine</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0uuronKPTsqZxumylF14Ag.gif" /><figcaption>Data Profiling Engine Diagram. Based on my current understanding, it should look like this. If there are any inaccuracies or missing elements, please let me know so I can correct them.</figcaption></figure><p><strong>What it is: </strong>The Data Profiling Engine acts as a comprehensive quality control layer that evaluates every image–text pair before it enters the training pipeline. Rather than treating data as a homogeneous input, the system builds a detailed profile for each sample, allowing the model to train only on data that meaningfully contributes to learning.</p><p><strong>Why it is needed: </strong>Not all data contributes equally to model performance. Images may suffer from blurriness, heavy compression, watermarks, or low informational content. Text descriptions may be poorly aligned with their corresponding images. Training on such data not only wastes valuable compute resources but can actively harm convergence and final output quality.</p><p><strong>How it works: </strong>The engine evaluates each image–text pair across multiple dimensions:</p><ul><li><strong>Basic metadata</strong><br>Fundamental properties such as resolution, file size, and perceptual hashes are extracted. Perceptual hashing provides a compact visual fingerprint, enabling efficient detection of duplicate or near-duplicate images at scale.</li></ul><blockquote><em>Note: Image resolution metadata also be helpful at the pre-training part. I will mention it on the next post.</em></blockquote><ul><li><strong>Technical quality assessment</strong><br>The system identifies over-compressed images by comparing the expected uncompressed file size to the actual file size. Specialized quality assessment models are used to detect blurriness, color distortions, visible watermarks, and excessive noise.</li><li><strong>Content complexity</strong><br>Images with minimal informational value, such as large uniform backgrounds or overly simplistic visuals, are filtered out. This is achieved by measuring pixel variance and estimating information entropy, ensuring that training focuses on visually rich samples.</li><li><strong>Semantic and aesthetic properties</strong><br>An aesthetics scoring model evaluates visual appeal, while a dedicated classifier filters out AI-generated images. Apparently, prior work has shown that synthetic imagery can degrade realism and physical consistency when overrepresented in training data.</li><li><strong>Text–image alignment</strong><br>To ensure strong cross-modal supervision, the system measures semantic alignment between images and their associated captions. Pairs with weak correspondence are discarded, preventing noisy supervision signals.</li></ul><p><strong>Outcome: </strong>By filtering low-quality, redundant, and misaligned samples at an early stage, the Data Profiling Engine significantly reduces wasted computation. As a result, training is both more efficient and more effective, allowing the model to learn exclusively from high-quality, semantically consistent data.</p><h4>2. Cross-modal Vector Engine</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NI5IgKjnjMSemnEQ79O3WQ.gif" /><figcaption>Cross-modal Vector Engine Diagram. This is how I understand it based on my current knowledge. If you spot any inaccuracies or missing elements, please let me know so I can update it. I know the diagram looks quite long, but do not let that intimidate you. I just wanted to walk through it in detail.</figcaption></figure><p><strong>What it is:</strong> A sophisticated system that converts images and text into mathematical representations (vectors) and uses them to find duplicates and related content at massive scale.</p><p><strong>Why it is needed:</strong> Even after basic deduplication, datasets contain many semantically similar images which do not contribute new knowledge. Additionally, when a model fails at specific tasks, you need a way to find and fix the problematic data quickly.</p><p><strong>How it works: </strong>The system serves two critical functions:</p><ul><li><strong>Semantic deduplication</strong>: Groups similar images into clusters and removes redundant examples, ensuring the dataset teaches diverse concepts.</li><li><strong>Targeted retrieval</strong>: When the model struggles with specific concepts (like generating a particular object), they can query the system to find relevant training examples or identify problematic data clusters to remove.</li></ul><p><strong>Achievement:</strong> This enabled them to build a non-redundant dataset while maintaining the ability to quickly diagnose and fix model weaknesses through targeted data curation.</p><h4>3. World Knowledge Topological Graph</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EhNAfFuuw1hAiJh5MOCSJg.gif" /><figcaption>World Knowledge Topological Graph Generation Diagram. Based on my current understanding, it should look like this. If there are any inaccuracies or missing elements, please let me know so I can correct them. (First Box would be “World Knowledge Topological Graph”)</figcaption></figure><p><strong>What it is:</strong> A structured knowledge graph that organizes concepts hierarchically, like a massive, interconnected encyclopedia.</p><p><strong>Why it is needed:</strong> Random sampling from your dataset might leave gaps. Entire categories of knowledge that the model never sees. You need a structured way to ensure balanced coverage across different concepts.</p><p><strong>How it works:</strong></p><p>They built this in three stages:</p><ol><li><strong>Foundation</strong>: Started with Wikipedia entities and their hyperlink structure</li><li><strong>Refinement</strong>: Pruned irrelevant nodes using PageRank scores and filtered out abstract concepts that can’t be visualized</li><li><strong>Expansion</strong>: Used their internal image dataset to extract tags and organize them into a hierarchical tree, then manually curated high-frequency concepts from actual user prompts</li></ol><p><strong>Achievement:</strong> Instead of hoping for balanced coverage, they <em>engineered</em> it. The model learns concepts systematically, filling knowledge gaps rather than reinforcing what it already knows again and again.</p><h4>4. Active Curation Engine</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zuAgs63nwRzOZrp-TfoaGQ.png" /><figcaption><a href="https://arxiv.org/abs/2511.22699">Figure is from Z-Image paper.</a></figcaption></figure><p><strong>What it is:</strong> A feedback loop that continuously improves data quality by having the model itself identify its weaknesses.</p><p><strong>Why it’s needed:</strong> Static datasets become outdated, and initial curation may not catch everything. A system that evolves with your model, addressing emerging issues and long-tail concepts may improve the performance (Apparently).</p><p><strong>How it works:</strong> The system works in a cycle:</p><ol><li><strong>Diagnosis</strong>: Generate images from various prompts and identify where the model fails or produces poor results</li><li><strong>Retrieval</strong>: Use the Cross-modal Vector Engine to find relevant data that could address these failures</li><li><strong>Curation</strong>: Apply filtering and deduplication to this targeted data</li><li><strong>Human-in-the-loop</strong>: Proposed captions and quality scores go through both AI and human verification. Failed cases trigger manual correction by experts.</li><li><strong>Retraining</strong>: Use this refined data to improve both the captioning model and quality assessment models</li></ol><p><strong>Achievement:</strong> Rather than a one-time data collection effort, they created a self-improving system that continuously refines data quality and addresses model weaknesses throughout development.</p><h3>Discussion</h3><p>While the core ideas in the paper are easy to understand conceptually, mapping them into a clear end to end diagram is much harder. The data pipeline includes many interdependent components, feedback loops, and decision points, which makes it difficult to reason about how everything connects in practice. Although the individual components are not especially hard to implement on their own, understanding their real world effects and deciding how to act on the signals they produce is the truly challenging part.</p><p>In my opinion, one notable limitation is the lack of isolated evaluations for the proposed data infrastructure ideas. The paper does not provide comparisons like how much Active Curation improves scores or whether similar performance could be achieved with fewer training iterations compared to a more traditional pipeline (etc etc). This is likely because such ablations are costly and difficult at scale. Still, without them, it is hard to assess the marginal impact of each new concept or to extract actionable insights beyond the full system as a whole.</p><blockquote><strong><em>Note:</em></strong><em> I did not mention any evaluation results here since we are at the beginning. But you can check it from the paper.</em></blockquote><h3>Conclusion</h3><p>The data infrastructure presented in this work highlights that efficiency in AI development is not solely a function of model architecture or parameter count. It begins with how data is curated, structured, and continuously refined. Through systematic profiling, intelligent deduplication, structured knowledge organization, and active feedback loops, the authors build a foundation that extracts maximal learning value from each GPU hour.</p><p>This approach directly challenges the assumption that better performance necessarily requires more data and larger models. By prioritizing data quality, diversity, and strategic sampling, the system achieves competitive results with a dramatically lower computational budget. The complete training pipeline reportedly costs around $628K, a small fraction of what is typically required for large-scale generative models. While the precise contribution of each data component remains difficult to quantify in isolation, the end-to-end results make a strong proof for treating data infrastructure as a beneficial design.</p><p><strong>In the next part</strong>, we will explore the architecture of Z-Image and training mechanisms, including their Single-Stream Diffusion Transformer (S3-DiT) design and the progressive training curriculum that leverages this carefully curated data.</p><h3>References/Sources</h3><p><a href="https://arxiv.org/abs/2511.22699">Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12bb86b61a9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BRAID: How Structured Reasoning Can Make Your LLM Smarter and Cheaper]]></title>
            <link>https://medium.com/@akdemir_bahadir/braid-how-structured-reasoning-can-make-your-llm-smarter-and-cheaper-c90045537b91?source=rss-711524cb67b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c90045537b91</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[reasoning]]></category>
            <category><![CDATA[braids]]></category>
            <category><![CDATA[bounded-reasoning]]></category>
            <category><![CDATA[mermaid]]></category>
            <dc:creator><![CDATA[Bahadır AKDEMİR]]></dc:creator>
            <pubDate>Sun, 21 Dec 2025 14:48:11 GMT</pubDate>
            <atom:updated>2025-12-21T14:48:11.932Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jz3cg4hVc40LrPCTsj_dvA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@dmjdenise?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Denise Jans</a> on <a href="https://unsplash.com/photos/yellow-rope-on-white-table-tD8qQ1d1u8s?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a>. Beyond the wordplay, a rope mirrors BRAID’s replacement of linear monologues with an interwoven structure of bounded, symbolic graphs to eliminate reasoning drift.</figcaption></figure><blockquote><em>I wanted to create content about the BRAID paper by </em><a href="https://www.youtube.com/@Arma%C4%9FanAmcalar"><em>Armağan Amcalar, whose YouTube content I enjoy following</em></a><em>. </em><a href="https://arxiv.org/abs/2512.15959"><em>The full research and performance logs are available here</em></a><em>.</em></blockquote><h3>Introduction: The Cost of Intelligence</h3><p>Large Language Models (LLMs) have transformed what is possible in natural language processing. From translation to question answering, from code generation to creative writing, these models have become the backbone of modern AI applications. Newer model iterations show that scaling model size is sufficient to unlock impressive zero-shot and few-shot reasoning performance without explicit task-specific optimization.</p><p>But <strong>cost efficiency remains a significant bottleneck</strong>. Running these massive models is not cheap, and for companies deploying AI at scale, the token costs quickly add up. Moreover, despite their impressive capabilities, LLMs still struggle with complex logical problems and can produce errors in multi-step reasoning tasks.</p><p>To reduce these errors, several mitigation strategies are typically used.</p><blockquote><strong><em>Note:</em></strong><em> I will address the cost efficiency issue and the BRAID approach later in the post. But before, let’s focus on the current solutions for complex problem reasoning.</em></blockquote><h3>Chain-of-Thought Prompting and Unstructured Reasoning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CRnp-9wU6kmeDDwx-U9n3A.png" /></figure><p>One of the most influential techniques to emerge for improving LLM reasoning is <strong>Chain-of-Thought (CoT) prompting</strong>. Instead of asking a model to jump straight to an answer, CoT encourages the model to show its work by generating explanatory steps before the final response:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nO3s64brXZv2q5jrgdmjWg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gWvzVJMP9Cn-KflFx3VcBQ.png" /></figure><p>CoT prompting dramatically improved accuracy on reasoning tasks. Researchers even discovered that simply adding “Let’s think step by step” to prompts triggered better reasoning in zero-shot scenarios.</p><p>But there’s a problem: <strong>all that extra explanation costs tokens</strong>. The model might use 200 tokens to solve a problem that could be expressed in 20. Worse, these lengthy explanations can contain “reasoning drift” where the model loses coherence or contradicts earlier statements.</p><h3>Structured Prompting</h3><p>Researchers developed more sophisticated techniques to guide model reasoning. Here are some of them:</p><ul><li><strong>Least-to-Most Prompting</strong>: Break complex problems into simpler sub-problems, solving them sequentially.</li><li><strong>Plan-and-Solve Prompting</strong>: First devise a high-level plan, then execute each sub-task according to that plan.</li><li><strong>Program-of-Thought</strong>: Generate executable code instead of natural language for calculations in order to avoid calculation errors.</li><li><strong>Tree of Thoughts</strong>: Explore multiple reasoning paths simultaneously.</li><li><strong>Universal Self-Adaptive Prompting (USP)</strong>: Let the model help “prompt itself” by generating its own reasoning examples</li></ul><blockquote><strong><em>Note: </em></strong><em>Many of these topics deserve standalone discussions, but to keep this post focused on BRAID and avoid unnecessary length, I summarizeed them briefly.</em></blockquote><p>These methods improve accuracy, but often at the cost of substantially higher token usage. Therefore, this trade-off motivates the search for alternative approaches that deliver a greater gain in accuracy per token.</p><h3>BRAID: Bounded Reasoning for Autonomous Inference and Decisions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vehswgz8hFPmCzpL8zIq6g.gif" /></figure><p>BRAID takes an interesting approach: instead of letting the model “think” in natural language, it converts the reasoning process into <strong>Mermaid flowcharts</strong> which are symbolic diagrams that encode logic as a graph structure.</p><p>Think of it like the difference between:</p><ul><li><strong>Giving someone turn-by-turn directions</strong> (traditional CoT): “Go straight for 200 meters, then turn left at the bakery, walk past three buildings…”</li><li><strong>Handing them a map</strong> (BRAID): Here’s the entire route visualized with all decision points clearly marked</li></ul><blockquote><em>Note: This approach enforces a more deterministic step structure while avoiding and mitigating unnecessary token usage, as each token (word, term, etc.) serves a specific role in constructing the diagram.</em></blockquote><h3>How BRAID Works</h3><p>BRAID operates in two stages:</p><ol><li><strong>Generation Phase</strong>: A model generates a BRAID (Mermaid) diagram encoding the logical reasoning process for a given question</li><li><strong>Solving Phase</strong>: Another model (same or different model) uses the BRAID diagram as a system message and produces the final answer</li></ol><blockquote><em>Note: We can use a capable (but expensive) model to generate the reasoning structure </em><strong><em>once</em></strong><em>, then reuse that structure with a cheap, fast model.</em></blockquote><p>Here’s a simplified example for a mermaid format:</p><pre>flowchart TD<br>    A[Read constraints] --&gt; B{Check condition 1}<br>    B --&gt;|Yes| C[Apply rule A]<br>    B --&gt;|No| D[Apply rule B]<br>    C --&gt; E[Verify solution]<br>    D --&gt; E<br>    E --&gt; F[Output answer]</pre><p>This diagram is:</p><ul><li><strong>Deterministic</strong>: No ambiguity about which path to take</li><li><strong>Compact</strong>: Uses far fewer tokens than explaining the same logic in prose</li><li><strong>Reusable</strong>: Can be cached and used repeatedly</li><li><strong>Visual: </strong>We can also visualize the steps to see how the model approached for the task.</li></ul><blockquote><strong><em>Note:</em></strong><em> In order to prevent misunderstanding, I want to mention something. “Fewer tokens” here refers to achieving substantially greater reasoning effectiveness per token. While BRAID approach may require more tokens for definition or setup, the resulting gains significantly exceed those of common reasoning techniques. This situation will become clearer in the results section.</em></blockquote><h3>Three Key Advantages</h3><p><strong>1. Comparable Accuracy Gains:</strong> Smaller models using BRAID match or exceed the performance of larger models using traditional prompting. On complex reasoning tasks, a “Nano” tier model with BRAID can outperform a “Medium” tier model without it.</p><p><strong>2. Quantitative Economic Analysis:</strong> BRAID reveals a crucial insight. Reasoning quality isn’t just about model size, it is about <strong>Model Capacity and Prompt Structure</strong>. By maximizing structure, you can minimize required capacity.</p><p><strong>3. Efficiency Gains:</strong> The researchers introduce a “Performance-per-Dollar” (PPD) metric. On some benchmarks, BRAID configurations achieved 30–74x better (!!!) cost-efficiency than baseline approaches while maintaining or improving accuracy.</p><h3>The Experiment</h3><p>The researchers tested BRAID across three challenging benchmarks:</p><ul><li><strong>GSM-Hard</strong>: Mathematical reasoning problems</li><li><strong>SCALE MultiChallenge</strong>: Complex multi-turn conversations with constraints</li><li><strong>AdvancedIF</strong>: Advanced instruction-following tasks</li></ul><h3>Key Metrics</h3><p><strong>Cost Calculation:</strong> The study distinguished between:</p><ul><li><strong>C_BRAID</strong>: Cost of generating the reasoning diagram (one-time or amortized)</li><li><strong>C_inference</strong>: Cost of executing the model to get the final answer</li></ul><blockquote><strong><em>Note:</em></strong><em> For now, the BRAID approach operates by providing the model with a Mermaid-style prompt and having it solve the problem according to the structure defined in that prompt. In an inference setting, the cost of generating the prompt itself does not need to be considered, as it is not part of the inference process. Once a task is predefined, simply updating or injecting the required parameters and reusing the same Mermaid-style prompt incurs negligible additional cost. However, if prompt generation is still to be included in the cost analysis, the following equation can be used as a cost baseline.</em></blockquote><p>For deployed systems where diagrams are reused, the amortized cost per query becomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dmg7FwWg0xNh8ETKHS_Osg.png" /></figure><p>Where N is the number of times the diagram is reused.</p><p><strong>Performance-per-Dollar (PPD):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wEVlE0DY3ZNoIjq-9Uh6yA.png" /></figure><p>In the paper, Baseline Model is “GPT5-Medium”. Values above 1.0 indicate better cost-efficiency than the baseline.</p><blockquote><strong><em>Note:</em></strong><em> According to the paper version of the formula, a score greater than 1 indicates that the method is more efficient than GPT-5-Medium. However, it is also essential to ensure that the accuracy remains at an acceptable level. A higher efficiency score is not desirable if it is achieved primarily by sacrificing accuracy in favor of extremely low token cost.</em></blockquote><h3>Results</h3><h4>Accuracy Analysis</h4><p><strong>SCALE MultiChallenge:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SvFLvMcIiPUqfNgzKIxHVw.png" /><figcaption><a href="https://arxiv.org/abs/2512.15959">Accuracy scores for MultiChallenge Dataset from the original paper.</a></figcaption></figure><p>The improvements are dramatic:</p><ul><li>GPT-4o: 19.9% → 53.7% (a 170% increase!)</li><li>GPT-5-Nano-Minimal: 23.9% → 45.2%</li></ul><blockquote><strong><em>Note: </em></strong><em>Remarkably, the Nano model with BRAID outperformed the much larger Minimal model.</em></blockquote><h4>Cost Analysis</h4><blockquote><strong><em>Note:</em></strong><em> To keep this concise, I will present only a single figure. For more detailed analysis, please checkoutt he paper.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VxIQcBi1CT4_dHppL6TAqw.png" /><figcaption><a href="https://arxiv.org/abs/2512.15959">Average costs for MultiChallenge Dataset from the original paper.</a></figcaption></figure><p>Here’s where it gets economically interesting. The cost of generating BRAID diagrams varies by model, but when amortized across many queries, the solving cost is often <em>lower</em> than traditional prompting.</p><blockquote><strong><em>Note:</em></strong><em> Also in the next table, we will see PPD scores which will show the smaller models vs baseline model comparisons.</em></blockquote><h3>Performance-per-Dollar Efficiency</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pe-xQgzt9KYvdQ8QWl4J7g.png" /><figcaption><a href="https://arxiv.org/abs/2512.15959"><em>PPD scores from the original paper</em></a><em>.</em> Column-wise comparison shows that using larger models for generation consistently yields better performance. Row-wise comparison indicates that larger reasoning models generally achieve higher PPD scores. Notably, all cases with PPD &lt; 1 (highlighted in red) occur when using the GPT-5 Medium model, which already serves as the baseline; therefore, in these settings, the accuracy–cost trade off may be inefficient. In contrast, smaller models (highlighted in green) demonstrate a more favorable accuracy–cost trade off, achieving competitive performance with up to 30x efficiency gains.</figcaption></figure><h3>Discussion: Impact and Strategic Implications of BRAID</h3><h4>Differential Performance Across Benchmarks</h4><p>The impact of BRAID varies significantly based on the inherent failure modes of the target dataset.</p><ul><li><strong>Mitigating Reasoning Drift:</strong> In the <strong>SCALE MultiChallenge</strong> benchmark, accuracy improved from <strong>19.9%</strong> to <strong>53.7%</strong> for <strong>gpt-4o. </strong>This leap occurs because the benchmark’s multi-turn constraints often induce “reasoning drift” in traditional prompting, where models lose track of instructions or contradict earlier turns. BRAID’s graph structure serves as a procedural scaffold, explicitly encoding each constraint as a node to prevent the model from wandering off-path.</li><li><strong>Optimizing Saturation through Efficiency:</strong> Conversely, <strong>GSM-Hard</strong> results showed modest accuracy gains (e.g., <strong>94%</strong> to <strong>98%</strong> for <strong>gpt-5-nano-minimal</strong>) due to the high mathematical saturation of the base models. However, this dataset revealed massive <strong>Performance-per-Dollar (PPD)</strong> gains of up to <strong>74x</strong>. BRAID converts math problems into pure computational templates, allowing low-cost models to execute the logic without the linguistic overhead of verbose natural-language explanations.</li></ul><h3>Structural Advantages and Ideal Use Cases</h3><p><strong>In my opinion</strong>, BRAID’s deterministic branching logic and symbolic representation make it uniquely suited for specific architectural requirements:</p><ul><li><strong>State Machine Logic:</strong> Relationship puzzles and non-linear logic problems are naturally represented as graphs, whereas traditional LLMs often struggle to process these structures linearly.</li><li><strong>Deterministic Workflows:</strong> The framework transforms probabilistic token prediction into a directed traversal of a decision tree, which is critical for autonomous agents choosing between specific tools or workflows.</li><li><strong>Structural Scaffolding for Code:</strong> By encoding program logic (control flows, state transitions) into Mermaid diagrams before generation, BRAID mirrors the semantic structure of software engineering tasks, potentially reducing logic errors.</li></ul><blockquote><strong><em>Note:</em></strong><em> In future work, I wish to see benchmark results on coding tasks included, to better evaluate model performance in practical programming scenarios.</em></blockquote><h3>Limitations and Future Research Directions</h3><p>While BRAID may offer significant economic advantages, it likely introduces trade-offs related to potential structural brittleness. A flawed reasoning graph may lead directly to failure, as the solver model lacks a natural-language mechanism to correct itself. To address these limitations, there are several directions that future work could take. Some are suggested by the researchers, while others reflect my own opinion based on a closer look at the results.</p><ul><li><strong>Specialized Architect Models:</strong> Fine-tuning smaller, high-efficiency models solely to act as “Architects” that convert natural language into Mermaid topologies.</li><li><strong>Open-Source Validation:</strong> Evaluating the framework across open-source ecosystems like <strong>LLaMA</strong>, <strong>Mistral</strong>, or <strong>Qwen</strong> to determine if the “BRAID Parity Effect” scales across different model architectures.</li><li><strong>Dynamic Self-Correction:</strong> Developing execution loops where solver models can detect “Topology Errors” and trigger localized re-generation of the graph.</li><li><strong>Agentic Tool-Choice Benchmarking:</strong> Measuring how effectively models select and invoke tools in agentic settings when operating under graph-constrained reasoning.</li><li><strong>Coding Benchmarks:</strong> Extending evaluation to programming tasks to understand how well structured reasoning graphs support code generation and debugging.</li></ul><h3>Conclusion</h3><p>BRAID is more than a new prompting trick. It suggests a different way of thinking about how AI systems can apply reasoning in practice.</p><p><strong>A few takeaways stand out:</strong></p><ul><li><strong>Structure also matters:</strong> Smaller models with explicit reasoning structures can outperform larger models that rely on unstructured prompting.</li><li><strong>Splitting roles is effective:</strong> Using a larger model once to generate a reasoning structure, and then smaller models to execute it, turns out to be both effective and efficient.</li><li><strong>Cost-performance trade-offs improve:</strong> In many cases, this approach achieves similar (or better) results compared to larger models, at a much lower cost.</li></ul><p>For teams building autonomous agents, BRAID points toward a practical strategy: spend effort on designing good reasoning structures, then run them cheaply and repeatedly. This lowers the barrier to using more sophisticated reasoning without relying on constant access to top-tier models.</p><h3>References/Sources</h3><p><a href="https://arxiv.org/abs/2512.15959">BRAID: Bounded Reasoning for Autonomous Inference and Decisions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c90045537b91" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Encoder–Decoder (T5)Gemma: Rethinking the Quality/Efficiency Trade-off Through Architectural…]]></title>
            <link>https://medium.com/@akdemir_bahadir/encoder-decoder-t5-gemma-rethinking-the-quality-efficiency-trade-off-through-architectural-8e505925e92a?source=rss-711524cb67b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e505925e92a</guid>
            <category><![CDATA[gemma-2-adaptation]]></category>
            <category><![CDATA[asymmetric-llm]]></category>
            <category><![CDATA[encoder-decoder-gemma]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[t5gemma]]></category>
            <dc:creator><![CDATA[Bahadır AKDEMİR]]></dc:creator>
            <pubDate>Sun, 23 Nov 2025 14:13:53 GMT</pubDate>
            <atom:updated>2025-11-23T14:13:53.078Z</atom:updated>
            <content:encoded><![CDATA[<h3>Encoder–Decoder (T5)Gemma: Rethinking the Quality/Efficiency Trade-off Through Architectural Adaptation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pymfhkfTPwhflTQ0.png" /><figcaption><a href="https://developers.googleblog.com/en/t5gemma/">The image is from here</a></figcaption></figure><p>Decoder-only architectures have become the standard for large language models thanks to their simplicity and strong performance. But trending approaches do not always define the way of progress. While much attention centers on Gemini 3 today, studies like T5Gemma show that creative and alternative model designs can uncover stronger quality–efficiency outcomes. In particular, encoder–decoder architectures, when carefully adapted from pretrained decoder-only models, can achieve superior trade-offs on real-world tasks.</p><p>This blog post tries to explore the technical foundations of Encoder–Decoder Gemma, a family of models built on this idea. In here, we will examine the initialization strategies, pretraining objectives, and architectural modifications that enable these models to challenge conventional assumptions and, in many cases, outperform their decoder-only counterparts while maintaining comparable or better inference efficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qBR5PyNdxIArpux3KGcTxg.png" /></figure><h3>Introduction</h3><p><strong>Decoder-only architectures</strong> have shaped the landscape of large language models since the rise of <strong>GPT family</strong>, demonstrating impressive versatility across natural language understanding and generation tasks. Their success, however, may overshadow an importance of something: <em>A</em><strong><em>re decoder-only models inherently the best choice for every scenario, or have they simply benefited from the majority of our scaling efforts?</em></strong></p><p><strong>Encoder–decoder architectures</strong>, introduced in the original Transformer paper and later popularized by models like <strong>T5</strong>, offer a distinctly different approach. By separating the representation of input context from the generation of output sequences, they provide <strong>architectural flexibility</strong> that can be better aligned with certain task distributions. Yet the major barrier remains the cost of pretraining such models from scratch, which has limited broader exploration of this design space.</p><p><strong>Encoder–Decoder Gemma </strong>tackles this challenge with a practical adaptation strategy. Instead of starting from a random initialization, the model is <strong>initialized from pretrained decoder-only checkpoints</strong>. This warm-start approach retains the linguistic knowledge already captured by the original model while adding the benefits of a <strong>bidirectional encoder</strong>. It significantly <strong>lowers the computational cost</strong> and <strong>enables a more systematic study of the quality–efficiency trade-offs</strong> that encoder–decoder models can offer.</p><h3><strong>Rethinking the Role of Encoder–Decoder Models</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qKhA8mpHPiORQkd-fNVYkA.png" /><figcaption><a href="https://arxiv.org/abs/2504.06225">This figure is from the original paper Encoder-Decoder Gemma</a></figcaption></figure><h4>Architectural Philosophy</h4><p>The core difference between decoder-only and encoder–decoder architectures lies in how they handle input and output sequences.</p><p><strong>Decoder-only models</strong> process all tokens, both input and output, through a single autoregressive stack with causal (<strong>unidirectional</strong>) attention. Each token can only attend to previous tokens. This preserves the autoregressive property needed for generation, but restricts the ability of models to leverage full bidirectional context when interpreting inputs.</p><p><strong>Encoder–decoder models</strong> use two distinct components:</p><ul><li>An <strong>encoder</strong> with bidirectional attention, allowing every input token to attend to all others.</li><li>A <strong>decoder</strong> that generates output tokens autoregressively while attending to the encoder’s full representation through cross-attention.</li></ul><p>This separation introduces several <strong>theoretical</strong> benefits:</p><p><strong>Bidirectional Context Understanding: </strong>With full-sequence bidirectional attention, the encoder can build richer, globally informed representations. This is especially important for tasks involving complex input comprehension, such as <em>long-context question answering</em>, <em>document summarization</em>, or s<em>emantic analysis.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D6VkxM4_bEXTWT3O98Y00g.png" /></figure><p><strong>Computational Flexibility: </strong>Encoder–decoder architectures allow asymmetric scaling of the encoder and decoder. This matters when input sequences are much longer than output sequences, or when tasks demand more representational depth in understanding than in generation. Such targeted allocation is difficult to achieve in decoder-only models.</p><p><strong>Task Alignment: </strong>Many real-world applications naturally split into an understanding step followed by a generation step. Encoder–decoder models embed this structure directly into their design, providing an inductive bias that can improve generalization and sample efficiency.</p><h4>Empirical Performance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DoXyiYOpbCt9PbzEur3n2g.png" /><figcaption><a href="https://arxiv.org/abs/2504.06225">This figure is from the original paper Encoder-Decoder Gemma</a></figcaption></figure><p>These architectural advantages translate into clear performance improvements on real-world benchmarks:</p><ul><li>Adapting a <strong>2B parameter decoder-only model</strong> into a <strong>2B-2B parameter encoder–decoder model</strong> yields <strong>improvement</strong> on aggregate benchmark scores.</li><li>Encoder–decoder models show particularly strong gains on <strong>SuperGLUE</strong>, which emphasizes complex natural language understanding.</li><li>When evaluated under <strong>matched inference budgets</strong>, encoder–decoder models achieve <strong>comparable or superior performance</strong> to larger decoder-only models.</li></ul><p>Taken together, these results indicate that when total computational cost is held constant (nearly), the structural benefits of encoder–decoder architectures can outweigh the seeming simplicity and efficiency of decoder-only designs.</p><h3>Adaptation Methodology</h3><p>The core innovation of Encoder–Decoder Gemma lies in its adaptation strategy. Instead of pretraining an encoder–decoder model from scratch, <strong>the approach initializes the model from an existing decoder-only checkpoint</strong> and then continues training with encoder–decoder–specific objectives. This warm-start method preserves valuable linguistic knowledge while enabling the architectural flexibility of an encoder–decoder design.</p><h4>Architectural Modifications</h4><p>The adaptation process begins by transforming a decoder-only network into an encoder–decoder architecture while preserving as much of the learned representation space as possible.</p><p><strong>Encoder Construction</strong></p><p>The encoder is created by reusing the layers of the decoder-only model, modifying only the attention mask:</p><ul><li><strong>Weight Initialization:</strong> Encoder parameters (including embeddings, feed-forward layers, and normalization layers) are copied directly from the decoder-only checkpoint.</li><li><strong>Attention Modification:</strong> The causal mask is removed to convert autoregressive self-attention into bidirectional self-attention. No attention weights need to change; only the mask used during computation is updated.</li></ul><p>Formally, decoder-only self-attention is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0EWLrfiq-ycsNqv5w-EiOQ.png" /></figure><p>where M_causal​ is the lower-triangular mask that blocks future tokens.</p><p>In the encoder, the mask is removed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HyonLZKuCMwgJuLaj3mUKQ.png" /></figure><p>This simple change enables the encoder to build fully contextualized representations while benefiting from the pretrained attention patterns of the original model.</p><p><strong>Decoder Construction</strong></p><p>The decoder retains the original decoder-only architecture but incorporates cross-attention:</p><ul><li><strong>Self-Attention Preservation:</strong> Causal self-attention layers and their parameters are copied unchanged.</li><li><strong>Cross-Attention Introduction:</strong> New cross-attention layers are inserted between each self-attention block and its feed-forward network, enabling the decoder to condition on the encoder’s representations.</li></ul><p>Cross-attention initialization depends on architectural symmetry:</p><p><strong>Symmetric Configuration: </strong>If the encoder and decoder share the same hidden dimension and number of layers, cross-attention parameters can be initialized by copying from the self-attention parameters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZApff74s9bmf78qz1BlChA.png" /></figure><p><strong>Asymmetric Configuration: </strong>If dimensions differ (e.g., a 9B encoder paired with a 2B decoder), cross-attention parameters are randomly initialized. To avoid early-stage instability, these parameters receive a gradual K-step learning-rate warmup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2_rJ6-INeAg0Hla8LdMmIw.png" /></figure><h4>Pretraining Objectives</h4><p>Encoder–decoder models require paired input–target sequences during training. The adaptation process relies on two complementary objectives.</p><p><strong>PrefixLM Objective</strong></p><p>PrefixLM divides a raw text sequence into an input and a target portion:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Do6lH0ICeVKSUpLqorJG_A.png" /></figure><p>The encoder processes the [x_n] prefix bidirectionally, while the decoder generates the [y_m] suffix autoregressively:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nfJRGz7LNRY8NA12Vj6pVw.png" /></figure><p><strong>Advantages:</strong></p><ul><li>Requires no task-specific preprocessing; any text corpus can be split automatically</li><li>Preserves the original autoregressive modeling objectives</li><li>Facilitates implicit knowledge distillation from the decoder-only checkpoint</li></ul><h4>UL2 Objective</h4><p>UL2 (Unified Language Learning) introduces a mixture of denoising tasks with varying corruption levels:</p><ol><li><strong>Span Corruption:</strong> Predict masked spans.</li><li><strong>Prefix Denoising:</strong> Given a prefix, generate the remainder.</li><li><strong>Extreme Summarization:</strong> Compress long sequences into shorter outputs.</li></ol><p><strong>Advantages:</strong></p><ul><li>Encourages deeper contextual reasoning through diverse corruption patterns</li><li>Improves handling of tasks requiring reorganization or interpretation of complex inputs</li><li>Reinforces the encoder–decoder structural separation</li></ul><h4>Optimization Strategy</h4><p>Adaptation training typically requires only <strong>5–10% of the compute</strong> used for original decoder-only pretraining. Key considerations include:</p><ul><li><strong>Learning Rate:</strong> A reduced learning rate (0.1–0.5x the original peak) helps maintain the pretrained model’s learned structure.</li><li><strong>Warmup for New Parameters:</strong> Randomly initialized cross-attention weights receive gradual learning-rate warmup to stabilize early training.</li><li><strong>Mixed Objective Training:</strong> Jointly training with PrefixLM and UL2 provides complementary supervision, balancing generative modeling with contextual understanding.</li></ul><h3>Architectural Flexibility: Asymmetric Encoder–Decoder Models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M_RxOHCEBLEuFOb0JASSPw.png" /></figure><p>One of the most compelling advantages of the adaptation framework is its support for <strong>asymmetric encoder–decoder configurations</strong>. This flexibility allows model capacity to be distributed according to the needs of a task, rather than fixed by a single-stack architecture.</p><h4>Configuration Design</h4><p>Suppose we begin with a pretrained 9B decoder-only model and want to build an encoder–decoder system suitable for deployment under tighter inference constraints. Two strategies emerge:</p><ol><li><strong>Symmetric Adaptation:</strong><br> Convert a smaller 2B decoder-only model into a <strong>2B encoder + 2B decoder</strong> configuration (4B total).</li><li><strong>Asymmetric Adaptation:</strong><br>Adapt the 9B decoder-only model into a <strong>9B encoder + 2B decoder</strong> configuration (11B total).</li></ol><p>Despite the larger parameter count, inference remains efficient because the decoder has same(nearly) amount of parameter.</p><h4>Performance Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tsJe5PzAKHI5sptbFdHN9Q.png" /><figcaption><a href="https://arxiv.org/abs/2504.06225">This figure is from the original paper Encoder-Decoder Gemma</a></figcaption></figure><p>Empirical results show that the asymmetric configuration (9B-2B) achieves roughly <strong>+3% higher performance</strong> than the symmetric configuration (2B-2B). Several factors contribute to this improvement:</p><ul><li><strong>Richer Encoder Representations:</strong><br>A 9B encoder provides far greater capacity for modeling complex inputs, leading to significantly stronger contextual understanding.</li><li><strong>Efficient Generation:</strong><br>A 2B decoder is sufficient for most generation tasks, which typically demand less representational depth than comprehension.</li></ul><p>This asymmetric flexibility is something decoder-only architectures cannot naturally provide. In decoder-only models, the same stack must handle both understanding and generation, making it impossible to allocate capacity independently across these phases.</p><blockquote><strong><em>Note:</em></strong><em> MoE approach may be shown as alternative for quality/efficiency trade off but in there, only FFN parts are used as experts, not the attention parts. So MoE can help to increase performance but not will be more context aware than bidirectional attention (theoratically)</em></blockquote><h3>Experimental Results and Discussion</h3><h4>Performance Summary</h4><p>Experiments across standard benchmarks reveal several consistent trends:</p><ul><li><strong>Adaptation vs. Continued Training:</strong></li></ul><p>Converting a decoder-only model into an encoder–decoder model reliably outperforms simply continuing to train the decoder-only model. This indicates that the gains come from architectural advantages, not just extra compute.</p><ul><li><strong>Adaptation vs. Training from Scratch:</strong></li></ul><p>At moderate and large scales (around one billion parameters and above), adaptation from a pretrained decoder-only checkpoint outperforms training an encoder–decoder model from random initialization, even when both use the same total training budget.</p><ul><li><strong>SuperGLUE Performance:</strong></li></ul><p>Encoder–decoder models deliver especially strong results on SuperGLUE, which emphasizes deeper linguistic understanding, reasoning, and compositional generalization.</p><h4>Why Does Adaptation Work?</h4><p>Several factors explain why adaptation is more effective than training from scratch:</p><ol><li><strong>Knowledge Transfer:</strong></li></ol><p>The pretrained decoder-only model already contains rich linguistic structure (token embeddings, attention patterns, and semantic representations). Adaptation preserves this foundation while integrating the inductive bias of an encoder–decoder architecture.</p><p><strong>2. Favorable Optimization Starting Point:</strong></p><p>Beginning from a pretrained checkpoint places the model in a region of parameter space with smoother loss geometry. Starting from scratch forces the model to navigate a more difficult optimization landscape during early training.</p><p><strong>3. More Efficient Use of Compute:</strong></p><p>Because the model does not need to relearn basic language features, the available training budget can focus on encoder–decoder–specific abilities such as bidirectional encoding and cross-attention.</p><h3>Pretraining Objective Trade-offs: UL2 vs. PrefixLM</h3><p>Choosing between UL2 and PrefixLM involves meaningful trade-offs, as each objective encourages different strengths in the adapted model.</p><h4>UL2 Strengths</h4><ul><li><strong>Richer Contextual Representations:</strong><br> UL2’s mix of denoising tasks forces the model to build flexible, multi-purpose representations.</li><li><strong>Improved Reasoning:</strong><br> Tasks such as span corruption and extreme summarization encourage the model to reorganize information, making UL2 better suited for reasoning-heavy workloads.</li></ul><h4>PrefixLM Strengths</h4><p><strong>Alignment with Generative Modeling: </strong>PrefixLM resembles classic autoregressive training, helping preserve the generative strengths of the original decoder-only model.</p><p><strong>Implicit Distillation: </strong>Because the setup closely matches the decoder-only objective, much of the pretrained knowledge transfers naturally.</p><p><strong>Simplicity: </strong>PrefixLM requires no additional data corruption or preprocessing, making it straightforward to apply at scale.</p><blockquote><strong><em>Note: </em></strong><em>In practice, combining UL2 and PrefixLM during adaptation probably provides the best overall performance, offering a balance between contextual understanding and strong generative ability. However, the relative effectiveness of using only PrefixLM, only UL2, or a combination of the two will ultimately depend on the specifics of your task.</em></blockquote><h3>Practical Considerations and Deployment</h3><h4>Model Selection Guidelines</h4><p>When choosing between decoder-only and encoder–decoder architectures, several factors help guide the decision:</p><p><strong>Task Type</strong></p><ul><li>Understanding-heavy tasks (QA, classification, reasoning) benefit from encoder–decoder models.</li><li>Pure generation tasks (creative writing, open-ended dialogue) often perform well on decoder-only models.</li><li>Mixed tasks such as summarization or translation typically favor encoder–decoder designs.</li></ul><p><strong>Inference Pattern</strong></p><ul><li>Long inputs with short outputs play to encoder–decoder efficiency.</li><li>Short inputs with long outputs may tilt the advantage toward decoder-only models.</li><li>Generating multiple outputs from the same input (such as in beam search) strongly benefits from encoder–decoder amortization.</li></ul><p><strong>Deployment Constraints</strong></p><ul><li>Memory-limited environments can take advantage of asymmetric encoder–decoder setups.</li><li>Latency-sensitive applications benefit from a smaller decoder, which reduces per-token generation cost.</li></ul><h4>Adaptation Best Practices</h4><ol><li><strong>Use High-Quality Base Models:</strong><br>The performance of the adapted model largely depends on the strength of the original decoder-only checkpoint.</li><li><strong>Lower the Learning Rate:</strong><br>A reduced learning rate (roughly 10–50% of the original peak) is generally ideal for stable adaptation.</li><li><strong>Mix Objectives:</strong><br>A balanced combination of PrefixLM and UL2 provides complementary signals, often leading to the strongest downstream performance.</li><li><strong>Leverage Asymmetric Designs:</strong><br>If large pretrained models are available, building a large encoder with a smaller decoder <strong>can outperform</strong> symmetric adaptations of smaller models.</li><li><strong>Allocate Modest Training Budget:</strong><br>Effective adaptation usually requires only 5–10% of the compute used in original pretraining, making the approach considerably more accessible.</li></ol><h3>Conclusion</h3><p>Encoder–Decoder Gemma shows that progress in language modeling does not always require scaling up decoder-only architectures. Through architectural adaptation, it is possible to achieve stronger quality–efficiency trade-offs that better match real-world needs.</p><h4>Key Takeaways</h4><ol><li><strong>Efficient Adaptation:</strong><br>Transforming pretrained decoder-only models into encoder–decoder models requires only a small fraction of the original training compute while delivering meaningful performance gains.</li><li><strong>Architectural Advantages:</strong><br>Under comparable inference budgets, encoder–decoder models often outperform decoder-only models, especially on tasks that demand deeper understanding.</li><li><strong>Flexible Capacity Allocation:</strong><br>Asymmetric encoder–decoder designs allow capacity to be placed where it matters most, rather than being tied to a single shared stack.</li><li><strong>Non-Obvious Efficiency:</strong><br>Even with more parameters, encoder–decoder models can run faster because the encoder executes once and the decoder can remain smaller.</li></ol><h3>References/Sources</h3><ul><li><a href="https://deepmind.google/models/gemma/t5gemma/">T5Gemma</a></li><li><a href="https://developers.googleblog.com/en/t5gemma/">T5Gemma: A new collection of encoder-decoder Gemma models</a></li><li><a href="https://arxiv.org/abs/2504.06225">Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e505925e92a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DeepSeek-OCR: Rethinking Text Compression Through Vision]]></title>
            <link>https://medium.com/@akdemir_bahadir/deepseek-ocr-rethinking-text-compression-through-vision-86fe9aefb273?source=rss-711524cb67b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/86fe9aefb273</guid>
            <category><![CDATA[deepseek-ocr]]></category>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[deepseek]]></category>
            <category><![CDATA[text-compression]]></category>
            <dc:creator><![CDATA[Bahadır AKDEMİR]]></dc:creator>
            <pubDate>Thu, 30 Oct 2025 05:01:52 GMT</pubDate>
            <atom:updated>2025-10-30T11:02:50.599Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><h4>Inefficiency related to tokenization and transformer mechanism:</h4><p>Large Language Models are highly capable of understanding and generating human language, but this ability comes with a significant computational cost. These systems rely on tokenization, a process that breaks text into small units such as subwords. Each token must be processed individually, which means that longer texts directly translate into larger computational workloads. As the number of tokens increases, memory use and processing time grow rapidly, often at a rate that makes large-scale applications inefficient.</p><p>This limitation becomes most evident when handling long documents or extended conversations. A model that easily manages a few hundred words may struggle with thousands, not because it lacks comprehension but because the underlying computation expands quadratically with sequence length. In practical terms, analyzing a 10,000-word document requires far more resources than processing a 1,000-word one. This challenge limits how effectively these models can scale without significant infrastructure or optimization.</p><h4>How we approach texts:</h4><p>When we read a page from a book, our brain processes the entire visual scene almost instantly. We recognize words, structure, and meaning without focusing on each individual character. A language model, however, must process each token sequentially.</p><p>This contrast raises an important question: can we find a way to compress textual information more efficiently, reducing computation while preserving meaning and accuracy?</p><p>DeepSeek-OCR explores this idea by rethinking text compression through a visual perspective. Instead of treating text purely as a sequence of tokens, it represents written language as an image to make text processing and understanding more efficient.</p><h3>DeepSeek-OCR: More Than Just OCR</h3><p>Despite its name, DeepSeek-OCR is not just another optical character recognition system. It is a proof of concept designed to explore a broader idea: whether visual representations can compress text more efficiently than conventional token-based methods.</p><p>The concept is straightforward but powerful. When a page containing bunch of words is converted into an image, the same information can be represented with far fewer vision tokens than text tokens. DeepSeek-OCR demonstrates that this approach can achieve compression ratios between seven and twenty times, depending on the nature and depth of the context being processed. This technique points toward a promising way to overcome the long-context limitations that continue to challenge large language models.</p><blockquote><em>Think of it this way: vision-language models are typically used to describe images, for example, “This photo shows a large sea beneath scattered clouds.” In the case of DeepSeek-OCR, the goal is reversed. Instead of analyzing pictures of natural scenes, the model is asked to interpret an image that contains text (of course it can also contain images, graphs etc). Rather than feeding a language model the text of a book and asking for a summary, we can provide “images” of the pages and ask for the same result. This approach allows the model to perform the same task while using far fewer tokens to understand the given context.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9galTjVPkndMUzRlC4Ywhg.png" /><figcaption>A simple visualization illustrating the proof of concept that DeepSeek-OCR aims to demonstrate. The number of tokens is hypothetical.</figcaption></figure><h3>How Does This Differ From Similar Models?</h3><p>You might already be familiar with vision-language models such as PaliGemma2, which combine visual and textual understanding. DeepSeek-OCR, however, takes a fundamentally different direction.</p><p>Traditional vision-language models like PaliGemma2 focus on interpreting images that humans create and find meaningful, such as photographs, charts, or diagrams. They are designed to answer questions like “What is shown in this image?” or “Describe what you see.”</p><p>DeepSeek-OCR approaches the problem from the perspective of large language models. It asks a different question: “How can vision help language models process text more efficiently?” Instead of emphasizing human-style visual understanding, it treats images as a medium for compressing textual information.</p><blockquote><strong><em>Note:</em></strong><em> DeepSeek-OCR can also perform image understanding tasks. However, its main contribution lies elsewhere. The researchers use it as an experimental framework to explore a key question “</em><strong><em>Can large amounts of textual information be represented as an image using fewer tokens than traditional text-based methods</em></strong><em>?”</em></blockquote><p>In this framework, OCR tasks serve as a bridge between vision and language. They provide an ideal test environment for studying vision-based text compression, since they naturally create a compression–decompression relationship between visual and textual representations.</p><h3>Architecture</h3><p>DeepSeek-OCR consists of two main components:</p><h4>1. DeepEncoder</h4><p>The DeepEncoder is responsible for understanding the semantic information contained in images. It is designed to be as simple, fast, and content-aware as possible. While existing vision encoders used in popular vision-language models can perform similar tasks, they often have limitations in efficiency or semantic precision. To address these shortcomings, the authors developed a new encoder architecture specifically optimized for visual text understanding.</p><p><strong>Some examples of popular encoders:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*orB8DDyYuzefs1n5HtJDnQ.png" /><figcaption><a href="https://www.arxiv.org/abs/2510.18234">The figure is from the original paper.</a></figcaption></figure><p><strong>DeepEncoder approach</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LNzFOQoAaa_4Ui9y7rQLSQ.png" /><figcaption><a href="https://www.arxiv.org/abs/2510.18234">Deep Encoder Architecture (The figure is from the original paper.)</a></figcaption></figure><p>The heart of the system is <strong>DeepEncoder</strong>, a carefully engineered vision encoder with approximately <strong>380 million parameters</strong>. Its architecture is designed to efficiently process high-resolution images while minimizing GPU memory usage.</p><p>The DeepEncoder connects two main components in sequence, linked by a 16x convolutional compressor:</p><p><strong>SAM-base (80M): Perception Stage</strong></p><ul><li>Acts as the visual perception module, responsible for low-level feature extraction.</li><li>Uses<strong><em> window-based attention</em></strong> to efficiently handle high-resolution images (for example, a 1024x1024 image with 4,096 tokens).</li><li>Keeps activation memory low through localized attention.</li></ul><p><strong>16x Convolutional Compressor: Token Reduction</strong></p><ul><li>Reduces token count by a factor of 16 (for example, from 4,096 to 256 tokens).</li><li>Minimizes memory use and computation before global processing.</li></ul><p><strong>CLIP-large (300M): Knowledge Stage</strong></p><ul><li>Serves as the visual knowledge extraction module, leveraging CLIP’s large-scale pretraining.</li><li>Uses <strong><em>dense global attention</em></strong> to capture high-level semantic relationships.</li><li>Processes the compressed tokens efficiently to achieve deep contextual understanding.</li></ul><blockquote><strong><em>Note: </em></strong><em>CLIP is originally designed to take raw image tokens as input. In DeepSeek-OCR, the authors remove CLIP’s embedding layer so that it can directly process the compressed tokens produced by the convolutional module.</em></blockquote><p>The architecture supports multiple resolution modes that allow flexible experimentation with compression and detail. For example, the <strong><em>Tiny</em></strong> mode <strong>(512x512)</strong> uses only<strong> 64</strong> vision tokens, making it suitable for testing efficient compression. The <strong><em>Large</em></strong> mode <strong>(1280x1280)</strong> increases fidelity with <strong>400</strong> tokens, offering greater detail when needed. There is also a dynamic <strong><em>Gundam</em></strong> mode that combines multiple resolutions to handle ultra-high-resolution inputs such as newspaper scans or densely formatted documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AqbjGKRsSY1WSeaZFdWAmQ.png" /><figcaption><a href="https://www.arxiv.org/abs/2510.18234">The figure is from the original paper.</a></figcaption></figure><blockquote><strong><em>Note:</em></strong><em> “Gundam” name is coming from an anime about giant humanoid robots known as Mobile Suits, with the most iconic models referred to as </em><strong><em>Gundams</em></strong><em>. In the context of DeepSeek-OCR, the term </em><strong><em>Gundam</em></strong><em> describes a dynamic-resolution vision model configuration. The name is likely inspired by (I hope so) the way the model combines multiple image segments, much like how a Gundam mecha is assembled from interconnected components.</em></blockquote><h4>2. DeepSeek-3B-MoE Decoder</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*YXmdxHpga7iFfqbG_ffYPg.png" /><figcaption><a href="https://www.arxiv.org/abs/2510.18234">The figure is from the original paper.</a></figcaption></figure><p>The decoder uses a Mixture-of-Experts architecture with 3B total parameters but only 570M activated during inference. This makes it suitable for domain-centric research, obtaining the expressive capability of a 3B model while enjoying the inference efficiency of a 500M small model.</p><p>The decoder’s job is to take those compressed vision tokens and reconstruct the original text. It learns a non-linear mapping that decompresses the visual representation back into readable text.</p><h3>Benchmarks: Proving the Concept</h3><blockquote><strong><em>Note:</em></strong><em> I will go through this part quickly, since it is generally understood that a published research paper already demonstrates some improvement :). While it may not achieve the best results across all text, formula, table, and OCR benchmarks, it performs competitively with </em><strong><em>dots.ocr</em></strong><em>. However, this blog post mainly focuses on the </em><strong><em>text compression aspect</em></strong><em>, so I won’t dive into the OCR results. If you’re interested in those, you can check them out in the paper.</em></blockquote><p>The experimental results validate DeepSeek-OCR’s compression capabilities impressively:</p><h4>Compression Ratio Tests</h4><p>When testing on diverse document layouts, the model achieves 96%+ OCR decoding precision at 9–10x text compression, approximately 90% at 10–12x compression, and roughly 60% at 20x compression.</p><p>To put this in perspective: imagine a document page containing 1,000 text tokens. DeepSeek-OCR can represent this using just 100 vision tokens while maintaining over 96% accuracy. Even at extreme compression (20x, or 50 vision tokens for 1,000 text tokens), accuracy remains at 60%.</p><h4>Practical Performance</h4><p>On OmniDocBench, DeepSeek-OCR surpasses GOT-OCR2.0 (which uses 256 tokens per page) using only 100 vision tokens, and outperforms MinerU2.0 (which requires 6000+ tokens per page on average) while utilizing fewer than 800 vision tokens.</p><h3>Real-World Capabilities</h3><p>Beyond compression, DeepSeek-OCR demonstrates practical utility:</p><ul><li><strong>Deep Parsing</strong>: The model can perform secondary analysis on documents, parsing charts into structured HTML tables, converting chemical formulas to SMILES format, and even understanding geometric figures.</li><li><strong>Multilingual Support</strong>: It handles nearly 100 languages, from English and Chinese to Arabic and Sinhala.</li></ul><blockquote><strong><em>Note</em></strong><em>: The huge amount of data is Chinese and English. About ~16% of training data is other languages.</em></blockquote><ul><li><strong>Flexible Output</strong>: Through different prompts, users can get either raw text extraction or formatted output with layout information preserved.</li></ul><h3>Discussion</h3><p>The most exciting aspect of DeepSeek-OCR isn’t just what it achieves today, but what it <em>suggests for tomorrow</em>. Innovation often comes from asking unconventional questions, and this project does exactly that by reframing a fundamental assumption about how AI should process text.</p><h4>Probable Use Cases and Future Directions</h4><p><strong>1. Historical Document Compression</strong></p><p>Consider a conversational AI system that maintains context across many turns of dialogue. Currently, storing all previous conversation history as text tokens becomes computationally expensive. With optical compression, older conversation rounds could be rendered as images and compressed by 10x, maintaining most information while dramatically reducing token count.</p><p>The researchers propose an intriguing parallel: just as human memories fade over time and distant objects appear blurrier, older context could be progressively downsampled to smaller resolutions. Recent conversation stays crystal clear (high resolution), while older exchanges become increasingly compressed (lower resolution). This mimics biological memory decay while managing computational resources efficiently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_eeklQlZp2b8A5bkutb13w.png" /><figcaption><a href="https://www.arxiv.org/abs/2510.18234">The figure is from the original paper.</a></figcaption></figure><p><strong>2. Memory Mechanisms in LLMs</strong></p><p>Current language models treat all tokens equally, whether they represent something from the first sentence or the last. The optical compression approach can simulate memory forgetting mechanisms by rendering previous rounds of historical text onto images for initial compression, then progressively resizing older images to achieve multi-level compression, where token counts gradually decrease and text becomes increasingly blurred.</p><p>This approach enables <strong>theoretically unlimited context architectures</strong> that effectively balance <strong>information retention</strong> and <strong>computational efficiency</strong>. Recent information is preserved with high fidelity, while distant memories gradually compress, fading in detail as their relevance diminishes.</p><p><strong>3. Enhanced Document Parsing</strong></p><p>Traditional OCR systems extract text but lose valuable formatting information. DeepSeek-OCR’s visual approach inherently preserves formatting cues:</p><ul><li><strong>Font emphasis</strong>: Bold, italic, and colored text remain distinguishable in the visual representation</li><li><strong>Size variations</strong>: Headings versus body text retain their relative importance</li><li><strong>Spatial relationships</strong>: Tables, columns, and layout structure are naturally preserved</li><li><strong>Visual elements</strong>: Charts, formulas, and diagrams can be parsed alongside text</li></ul><p>This holistic understanding could enhance how AI systems interpret documents where formatting conveys meaning, such as academic papers, legal texts, technical manuals, and other structured materials.</p><p><strong>4. Efficient Data Generation</strong></p><p>The practical throughput of 200k+ pages per day makes DeepSeek-OCR valuable for creating training data. As language models grow larger and require more diverse training data, efficient document processing becomes crucial. The ability to parse and structure documents at scale, while using fewer computational resources than traditional approaches, has immediate practical value.</p><h3>Thinking Differently About Modalities</h3><p>DeepSeek-OCR encourages us to rethink how different modalities such as vision and language can work together. Instead of viewing vision as something separate that language models must learn to interpret, it asks whether vision itself could be a more efficient way to represent certain types of linguistic information.</p><h3>Final Words</h3><p>DeepSeek-OCR represents more than a technical achievement; it exemplifies the value of open research and development in AI. The team has made their code and model weights publicly accessible, allowing researchers worldwide to build upon this work, validate the findings, and explore new applications.</p><p><strong>Why does this matter?</strong></p><p>We cannot know whether large enterprises have already developed similar technologies behind closed doors. Major AI labs have significant resources and may have explored comparable approaches internally. However, without open publications and shared code, that knowledge remains isolated. The wider research community, including academics, independent researchers, and developers, cannot learn from it, build upon it, or apply it to address real-world problems.</p><p>The research also demonstrates something equally important: innovation doesn’t always require the biggest models or the most resources. Sometimes it requires asking different questions and challenging assumptions. By reframing OCR as a compression problem and treating vision as a computational tool rather than just a perceptual task, DeepSeek-OCR opens new research directions that might not have emerged from conventional approaches.</p><p>As we continue to push the boundaries of what AI systems can achieve, projects like this remind us that progress comes not only from scaling up but also from thinking differently.</p><h3>References / Sources</h3><ul><li><a href="https://www.arxiv.org/abs/2510.18234">DeepSeek-OCR: Contexts Optical Compression</a></li><li><a href="https://deepseek.ai/blog/deepseek-ocr-context-compression">DeepSeek AI | Leading AI Language Models &amp; Solutions</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FuWrBH4iN5y4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DuWrBH4iN5y4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FuWrBH4iN5y4%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d87ed51f177c43782ff6345fd17a1ad/href">https://medium.com/media/5d87ed51f177c43782ff6345fd17a1ad/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86fe9aefb273" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside nanochat Part 6: Understanding Evaluation]]></title>
            <link>https://medium.com/@akdemir_bahadir/inside-nanochat-part-6-understanding-evaluation-f675ced9e736?source=rss-711524cb67b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f675ced9e736</guid>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[nanochat]]></category>
            <dc:creator><![CDATA[Bahadır AKDEMİR]]></dc:creator>
            <pubDate>Mon, 27 Oct 2025 09:57:44 GMT</pubDate>
            <atom:updated>2025-10-27T09:57:44.503Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iOFF2VhcpERf0YWjGqQlZw.png" /></figure><p>Finally, we’ve reached the final part of this series. In this last part, we explore the evaluation framework that measures model performance across a broad range of tasks and metrics. Evaluation serves as the key feedback mechanism within the training pipeline, guiding optimization and validating the model’s capabilities. The <strong>nanochat</strong> evaluation framework consists of two main components: <strong>task-specific evaluations</strong>, which measure <strong>reasoning</strong> and <strong>generation</strong> skills, and <strong>training metrics</strong>, which monitor model efficiency and language modeling performance.</p><blockquote><strong><em>Note:</em></strong><em> For me, this has been the most educational (and the hardest) part. I was already familiar with LLM training pipelines, but since I had never trained a full model before, I often overlooked the evaluation stage. This time, I finally understood (I hope so) which datasets are available, what each one measures, and how they reflect different model abilities.</em></blockquote><h3>Overview of the Evaluation Framework</h3><p>The evaluation system is designed around several key principles:</p><ol><li><strong>Diverse Task Coverage</strong>: Evaluations span multiple choice questions, mathematical reasoning, code generation, and conversational ability</li><li><strong>Benchmark Standardization</strong>: All tasks follow established benchmarks with public datasets and baselines</li><li><strong>Efficiency-Aware Metrics</strong>: Beyond accuracy, we track computational efficiency through metrics like Model FLOPs Utilization (MFU)</li><li><strong>Tokenization Independence</strong>: Core metrics like Bits Per Byte (BPB) enable fair comparisons across different tokenizers</li><li><strong>Distributed Evaluation</strong>: All evaluation tasks support multi-GPU parallelization for faster iteration (<strong>I didn’t used and did not mentioned here</strong>).</li></ol><p>The framework distinguishes between <strong>base model evaluation</strong> (focused on language understanding and reasoning) and <strong>chat model evaluation</strong> (focused on instruction following and task completion).</p><h3>Evaluation Tasks</h3><p>Evaluation tasks are organized into three categories based on their evaluation methodology: <strong>multiple choice tasks</strong>, <strong>generative tasks</strong>, and <strong>language modeling tasks</strong>. Each category requires different prompt rendering and answer extraction strategies.</p><h3>CORE Metric</h3><p>The <strong>CORE</strong> metric is a composite score used to evaluate language models, calculated as the “CORE centered accuracy” over a specific subset of 22 tasks. It was designed to provide a reliable, low-variance signal of model quality, which is particularly useful for getting a stable assessment even when using smaller-scale models. The accuracy for each task is rescaled so that 0 represents random guessing and 1 represents perfect performance</p><blockquote><strong><em>Note: </em></strong><em>Apparently, The functions used for evaluating the CORE metric are implemented as described in the </em><a href="https://arxiv.org/abs/2406.11794"><em>DCLM paper</em></a><em>.</em></blockquote><blockquote><strong><em>Note: </em></strong><em>As mentioned in the nanochat repository (in </em><em>run1000.sh), </em><a href="https://karpathy-public.s3.us-west-2.amazonaws.com/eval_bundle.zip"><em>you can find the evaluation bundle here.</em></a></blockquote><h4>Component Tasks</h4><p>The CORE metric aggregates performance across a diverse set of 22 tasks designed to assess various capabilities. These include:</p><ul><li><strong>Commonsense and Reasoning:</strong> Tasks like <strong>HellaSwag</strong> , <strong>CommonsenseQA</strong> , <strong>COPA</strong> (causal reasoning) , <strong>PIQA</strong> (physical commonsense) , and the <strong>Winograd Schema Challenge</strong> evaluate a model’s understanding of everyday scenarios and implicit knowledge.</li><li><strong>Question Answering and Comprehension:</strong> This includes several datasets such as <strong>BoolQ</strong> (binary QA) , <strong>SQUAD</strong> (extractive QA from a passage) , <strong>CoQA</strong> (conversational QA) , and <strong>OpenBookQA</strong> (multi-step reasoning with commonsense).</li><li><strong>Scientific and Factual Knowledge:</strong> The <strong>ARC easy and challenge</strong> datasets use questions from science exams , while the <strong>Jeopardy</strong> dataset covers a wide variety of topics. <strong>QA Wikidata</strong> from Big-Bench requires completing factual statements.</li><li><strong>Symbolic and Algorithmic Reasoning:</strong> Several tasks from the <strong>Big-Bench</strong> suite are included to test formal reasoning, such as <strong>CS Algorithms</strong> (executing recursion and dynamic programming), <strong>Dyck languages</strong> (completing balanced expressions), and <strong>Operators</strong> (computing with newly defined operators).</li><li><strong>Broad Language Understanding:</strong> Tasks like <strong>LAMBADA</strong> test a model’s ability to use broader narrative context to predict a final word. The suite also includes specialized tasks like <strong>Language Identification</strong> and analytical reasoning from the legal domain via <strong>AGI Eval LSAT-AR</strong>.</li></ul><blockquote><strong><em>Note: </em></strong><em>It would be too exhausting to explain all datasets. You can check the datasets and their aim in eval_meta_data.csv file in zip mentioned above.</em></blockquote><h4>Metric Calculation</h4><p>The CORE metric is computed as the mean accuracy across all tasks, centered by subtracting the random baseline:</p><pre># Evaluate each task<br>results = {}<br>centered_results = {}<br>for task in tasks:<br>    # Run evaluation for this task<br>    accuracy = evaluate_task(model, tokenizer, data, device, task_meta)<br>    <br>    results[label] = accuracy<br>    row = eval_metadata[eval_metadata[&quot;Eval Task&quot;] == label]<br>    random_baseline = row[&quot;Random baseline&quot;].values[0]<br>    centered_result = (accuracy - 0.01 * random_baseline) / (1.0 - 0.01 * random_baseline)<br>    centered_results[label] = centered_result<br>core_metric = sum(centered_results.values()) / len(centered_results)</pre><p><strong>Centering the metric by the random baseline</strong> ensures that:</p><ul><li>A score of 0.0 corresponds to random guessing</li><li>A score of 1.0 corresponds to perfect accuracy</li><li>The metric captures genuine model capability above chance</li></ul><p>This normalization is crucial because <strong>different tasks have different numbers of answer choices and thus different random baselines</strong> (e.g., 25% for 4-choice questions, 50% for 2-choice questions). The comprehensive nature of the CORE metric (22 tasks) provides a robust measure of general reasoning capability across diverse domains.</p><h4>Implementation Details</h4><p>The CORE evaluation uses a sophisticated prompt rendering system that handles few-shot examples and task-specific formatting:</p><pre>def evaluate_example(idx, model, tokenizer, data, device, task_meta):<br>    &quot;&quot;&quot;Evaluate a single example, return True if correct, False otherwise&quot;&quot;&quot;<br>    item = data[idx]<br>    task_type = task_meta[&#39;task_type&#39;]<br>    num_fewshot = task_meta[&#39;num_fewshot&#39;]<br>    continuation_delimiter = task_meta[&#39;continuation_delimiter&#39;]<br><br>    # Sample few-shot examples (excluding current item)<br>    fewshot_examples = []<br>    if num_fewshot &gt; 0:<br>        rng = random.Random(1234 + idx)<br>        available_indices = [i for i in range(len(data)) if i != idx]<br>        fewshot_indices = rng.sample(available_indices, num_fewshot)<br>        fewshot_examples = [data[i] for i in fewshot_indices]<br>    # Render prompts and batch sequences based on task type<br>    if task_type == &#39;multiple_choice&#39;:<br>        prompts = render_prompts_mc(item, continuation_delimiter, fewshot_examples)<br>        tokens, start_idxs, end_idxs = batch_sequences_mc(tokenizer, prompts)<br>    #(...)</pre><blockquote><strong><em>Note: </em></strong><em>These classes/functions are longer than mentioned part. I am trying to be simple as much as possible.</em></blockquote><h3>Multiple Choice Tasks</h3><p>Multiple choice tasks form the backbone of standardized model evaluation. They provide reliable, objective measurements of model knowledge and reasoning across diverse domains.</p><h4><a href="https://huggingface.co/datasets/allenai/ai2_arc">ARC (AI2 Reasoning Challenge)</a></h4><p>The ARC dataset contains grade-school science questions with two difficulty levels:</p><ul><li><strong>ARC-Easy</strong></li><li><strong>ARC-Challenge</strong></li></ul><p>Each question has 4 answer choices, making the random baseline 25%. Questions cover topics from elementary and middle school science curricula.</p><p>The implementation loads <a href="https://huggingface.co/datasets/allenai/ai2_arc">questions from HuggingFace datasets</a> and formats them for evaluation:</p><pre>class ARC(Task):<br>    #(...)<br>    def __init__(self, subset, split, **kwargs):<br>        super().__init__(**kwargs)<br>        assert subset in [&quot;ARC-Easy&quot;, &quot;ARC-Challenge&quot;], &quot;ARC subset must be ARC-Easy or ARC-Challenge&quot;<br>        assert split in [&quot;train&quot;, &quot;validation&quot;, &quot;test&quot;], &quot;ARC split must be train|validation|test&quot;<br>        self.ds = load_dataset(&quot;allenai/ai2_arc&quot;, subset, split=split).shuffle(seed=42)<br>    def get_example(self, index):<br>        row = self.ds[index]<br>        question = row[&quot;question&quot;] # the question text<br>        choices = row[&quot;choices&quot;][&quot;text&quot;] # the text of each choice<br>        answer_string = row[&quot;answerKey&quot;] # e.g. &quot;A&quot;, &quot;B&quot;, &quot;C&quot;, &quot;D&quot;<br>        letters = row[&quot;choices&quot;][&quot;label&quot;] # e.g. [&quot;A&quot;, &quot;B&quot;, &quot;C&quot;, &quot;D&quot;]<br>        <br>        # create and return the Conversation object<br>        user_message = render_mc(question, letters, choices)<br>        messages = [<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_message},<br>            {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: answer_string}<br>        ]<br>        conversation = {<br>            &quot;messages&quot;: messages,<br>            &quot;letters&quot;: letters, # useful during evaluation<br>        }<br>        return conversation<br>    #(...)</pre><h4><a href="https://huggingface.co/datasets/cais/mmlu">MMLU (Massive Multitask Language Understanding)</a></h4><p>MMLU is a comprehensive benchmark covering 57 subjects across STEM, humanities, and social sciences. Each question has 4 answer choices.</p><p>The dataset structure allows for few-shot evaluation using auxiliary training examples:</p><pre>class MMLU(Task):<br>    letters = (&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;)<br>    groups = (&#39;abstract_algebra&#39;, &#39;anatomy&#39;, &#39;astronomy&#39;, &#39;business_ethics&#39;, <br>              &#39;clinical_knowledge&#39;, &#39;college_biology&#39;, ...) # 57 total subjects. Note used. Just for visualization.<br>    #(...)<br>    def get_example(self, index):<br>        row = self.ds[index]<br>        question = row[&quot;question&quot;] # the question text<br>        choices = row[&quot;choices&quot;] # the text of each choice<br>        answer = row[&quot;answer&quot;] # index of the answer, e.g. 0,1,2,3 (for A,B,C,D)<br>        subject = row[&quot;subject&quot;] # e.g. &quot;college_biology&quot;, &quot;college_chemistry&quot;, etc.<br>        <br>        user_message = render_mc(question, self.letters, choices)<br>        assistant_message = self.letters[answer]<br>        messages = [<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_message},<br>            {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: assistant_message}<br>        ]<br>        conversation = {<br>            &quot;messages&quot;: messages,<br>            &quot;subject&quot;: subject,<br>            &quot;letters&quot;: self.letters,<br>        }<br>        return conversation<br>    #(...)</pre><h4>Evaluation Method</h4><p>Multiple choice evaluation follows a consistent methodology:</p><ol><li><strong>Prompt Rendering</strong>: Format the question with letter-labeled choices</li><li><strong>Model Forward Pass</strong>: Compute losses for each answer choice continuation</li><li><strong>Answer Selection</strong>: Choose the option with lowest average loss</li><li><strong>Accuracy Computation</strong>: Compare selected answer to ground truth</li></ol><p>The key insight is that for multiple choice questions, we can evaluate which continuation the model assigns highest probability:</p><pre>def render_prompts_mc(item, continuation_delimiter, fewshot_examples=None):<br>    &quot;&quot;&quot;Render complete prompts for a multiple choice question&quot;&quot;&quot;<br>    template_str = &quot;&quot;&quot;<br>{%- for example in fewshot_examples -%}<br>{{ example.query }}{{ continuation_delimiter }}{{ example.choices[example.gold] }}{% endfor -%}<br>{{ item.query }}{{ continuation_delimiter }}{{ choice }}&quot;&quot;&quot;.strip()<br>    template = Template(template_str)<br>    fewshot_examples = fewshot_examples or []<br>    context = {<br>        &#39;fewshot_examples&#39;: fewshot_examples,<br>        &#39;continuation_delimiter&#39;: continuation_delimiter,<br>        &#39;item&#39;: item<br>    }<br>    prompts = [template.render(choice=choice, **context) for choice in item[&#39;choices&#39;]]<br>    return prompts<br></pre><p>The evaluation computes the loss for each possible answer continuation:</p><pre># For MC/schema: find the option with lowest average loss<br>if task_type in [&#39;multiple_choice&#39;, &#39;schema&#39;]:<br>    mean_losses = [losses[i, si-1:ei-1].mean().item()<br>                    for i, (si, ei) in enumerate(zip(start_idxs, end_idxs))]<br>    pred_idx = mean_losses.index(min(mean_losses))<br>    is_correct = pred_idx == item[&#39;gold&#39;]</pre><p>This loss-based approach works well for base models that haven’t been fine-tuned for instruction following, as it directly measures the model’s probability assignment over continuations.</p><h4>Prompt Format Design</h4><p>The multiple choice rendering uses a carefully designed format optimized for smaller models:</p><pre>def render_mc(question, letters, choices):<br>    &quot;&quot;&quot;<br>    The common multiple choice rendering format we will use.Note two important design decisions:<br>    1) Bigger models don&#39;t care as much, but smaller models prefer to have<br>       the letter *after* the choice, which results in better binding.<br>    2) There is no whitespace between the delimiter (=) and the letter.<br>       This is critical because the tokenizer has different token ids<br>       for &quot; A&quot; vs. &quot;A&quot;. The assistant responses will be just the letter itself,<br>       i.e. &quot;A&quot;, so it is important that here in the prompt it is the exact same<br>       token.<br>    &quot;&quot;&quot;<br>    query = f&quot;Multiple Choice question: {question}\n&quot;<br>    query += &quot;&quot;.join([f&quot;- {choice}={letter}\n&quot; for letter, choice in zip(letters, choices)])<br>    query += &quot;\nRespond only with the letter of the correct answer.&quot;<br>    return query</pre><p>This format places the letter after the choice text (choice=A rather than A. choice) because smaller models better associate the letter with the preceding context. Additionally, ensuring no whitespace before the letter ensures tokenization consistency with the expected answer format.</p><h3>Generative Tasks</h3><p>Generative tasks require the model to produce free-form text that is then evaluated against specific criteria. These tasks better reflect real-world usage where models must generate complete solutions rather than select from predefined options.</p><h4><a href="https://huggingface.co/datasets/openai/gsm8k">GSM8K (Grade School Math)</a></h4><p><a href="https://huggingface.co/datasets/openai/gsm8k">GSM8K</a> is a dataset including grade-school level math word problems. Each problem requires multi-step reasoning and produces a numerical answer.</p><p><strong>Example Problem:</strong></p><pre>Question:<br>Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes <br>of babysitting. How much did she earn?<br><br>Answer:<br>Weng earns 12/60 = $&lt;&lt;12/60=0.2&gt;&gt;0.2 per minute.<br>Working 50 minutes, she earned 0.2 x 50 = $&lt;&lt;0.2*50=10&gt;&gt;10.<br>#### 10</pre><p>The dataset uses a special format with calculator tool calls inside &lt;&lt; &gt;&gt; tags. The final answer appears after the #### marker.</p><p><strong>Implementation:</strong></p><pre>GSM_RE = re.compile(r&quot;#### (\-?[0-9\.\,]+)&quot;)<br><br>def extract_answer(completion):<br>    &quot;&quot;&quot;<br>    Extract the numerical answer after #### marker.<br>    &quot;&quot;&quot;<br>    match = GSM_RE.search(completion)<br>    if match:<br>        match_str = match.group(1).strip()<br>        match_str = match_str.replace(&quot;,&quot;, &quot;&quot;)<br>        return match_str<br>    return None<br><br>class GSM8K(Task):<br>    def evaluate(self, conversation, assistant_response):<br>        &quot;&quot;&quot;<br>        Compare extracted numerical answers.<br>        &quot;&quot;&quot;<br>        assert isinstance(assistant_response, str), &quot;Assuming simple string response for now&quot;<br>        # Extract ground truth answer<br>        assistant_message = conversation[&#39;messages&#39;][-1]<br>        last_text_part = assistant_message[&#39;content&#39;][-1][&#39;text&#39;]<br>        # Extract both the ground truth answer and the predicted answer<br>        ref_num = extract_answer(last_text_part)<br>        pred_num = extract_answer(assistant_response)<br>        # Compare and return the success as int<br>        is_correct = int(pred_num == ref_num)<br>        return is_correct</pre><p>The evaluation extracts the final numerical answer from both the reference solution and the model’s generated response, comparing them for exact match (after normalizing comma separators).</p><p><strong>Conversation Format:</strong></p><p>The GSM8K dataset includes tool calls, which are represented as structured message parts:</p><pre>def get_example(self, index):<br>    row = self.ds[index]<br>    question = row[&#39;question&#39;]<br>    answer = row[&#39;answer&#39;]<br>    <br>    # Parse tool calls from answer<br>    assistant_message_parts = []<br>    parts = re.split(r&#39;(&lt;&lt;[^&gt;]+&gt;&gt;)&#39;, answer)<br>    for part in parts:<br>        if part.startswith(&#39;&lt;&lt;&#39;) and part.endswith(&#39;&gt;&gt;&#39;):<br>            inner = part[2:-2]<br>            if &#39;=&#39; in inner:<br>                expr, result = inner.rsplit(&#39;=&#39;, 1)<br>            else:<br>                expr, result = inner, &quot;&quot;<br>            assistant_message_parts.append({&quot;type&quot;: &quot;python&quot;, &quot;text&quot;: expr})<br>            assistant_message_parts.append({&quot;type&quot;: &quot;python_output&quot;, &quot;text&quot;: result})<br>        else:<br>            assistant_message_parts.append({&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: part})<br>    <br>    messages = [<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: question},<br>        {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: assistant_message_parts},<br>    ]<br>    return {&quot;messages&quot;: messages}</pre><p>This structured representation allows the model to learn when and how to use computational tools during mathematical reasoning.</p><blockquote><strong><em>Note:</em></strong><em> This task can be completed without using calculation tools. Although the Reinforcement Learning component could be improved, performing mathematical calculations directly with a language model cannot solve the calculation errors.</em></blockquote><h4><a href="https://huggingface.co/datasets/openai/openai_humaneval">HumanEval</a></h4><p><a href="https://huggingface.co/datasets/openai/openai_humaneval">HumanEval</a> is a code generation benchmark containing Python programming problems. Each problem provides a function signature and docstring, and the model must complete the function implementation.</p><p><strong>Dataset Structure:</strong></p><pre>class HumanEval(Task):<br><br>    def __init__(self, **kwargs):<br>        super().__init__(**kwargs)<br>        self.ds = load_dataset(&quot;openai/openai_humaneval&quot;, split=&quot;test&quot;).shuffle(seed=42)<br>    def get_example(self, index):<br>        row = self.ds[index]<br>        prompt = row[&#39;prompt&#39;] # function signature and docstring<br>        solution = row[&#39;canonical_solution&#39;] # reference implementation<br>        entry_point = row[&#39;entry_point&#39;] # function name to test<br>        test = row[&#39;test&#39;] # test cases<br>        <br>        complete_solution = f&quot;{prompt}\n{solution}&quot;<br>        messages = [<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>            {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: complete_solution},<br>        ]<br>        conversation = {<br>            &quot;messages&quot;: messages,<br>            &quot;entry_point&quot;: entry_point,<br>            &quot;test&quot;: test,<br>        }<br>        return conversation</pre><p><strong>Evaluation Method:</strong></p><p>HumanEval evaluation requires executing generated code against test cases. This is implemented using a sandboxed execution environment:</p><pre>def evaluate(self, conversation, completion):<br>    &quot;&quot;&quot;Given (conversation, completion), return boolean success of the completion.&quot;&quot;&quot;<br>    # Extract imports from the prompt<br>    imports = extract_imports(conversation[&#39;messages&#39;][0][&#39;content&#39;])<br>    # Extract code from completion (handles markdown blocks)<br>    completion_code = extract_program(completion)<br>    <br>    # Construct complete program<br>    program = (<br>        imports<br>        + &quot;\n\n&quot;<br>        + completion_code<br>        + &quot;\n\n&quot;<br>        + conversation[&#39;test&#39;]<br>        + &quot;\n&quot;<br>        + f&quot;check({conversation[&#39;entry_point&#39;]})&quot;<br>    )<br>    <br>    # Execute in sandbox<br>    result = execute_code(program)<br>    success = result.success<br>    return success</pre><p>The extract_program function handles various output formats, including code wrapped in markdown blocks:</p><pre>def extract_program(completion):<br>    &quot;&quot;&quot;<br>    Extract Python code from LLM completion.<br>    Handles various output formats:<br>    - Code wrapped in ```python ... ``` or ``` ... ``` blocks<br>    - Plain code without markdown blocks<br>    &quot;&quot;&quot;<br>    pattern = r&#39;```(?:python)?\s*\n(.*?)\n```&#39;<br>    matches = re.findall(pattern, completion, re.DOTALL)<br>    <br>    if matches:<br>        return matches[0].strip()<br>    <br>    # No code blocks found, return the whole completion<br>    return completion.strip()</pre><p><strong>Sandboxed Execution:</strong></p><p>Code execution uses a multi-process sandbox with several safety features:</p><pre>def execute_code(<br>    code: str,<br>    timeout: float = 5.0, # 5 seconds default<br>    maximum_memory_bytes: Optional[int] = 256 * 1024 * 1024, # 256MB default<br>) -&gt; ExecutionResult:<br>    &quot;&quot;&quot;<br>    Execute Python code in a sandboxed environment.<br>    <br>    Safety features:<br>    - Each execution runs in its own process<br>    - Execution limited by timeout (default 5s)<br>    - Memory limits enforced (default 256MB)<br>    - Dangerous functions disabled (os.system, subprocess.Popen, etc.)<br>    - Runs in temporary directory<br>    &quot;&quot;&quot;<br>    manager = multiprocessing.Manager()<br>    result_dict = manager.dict()<br>    <br>    p = multiprocessing.Process(<br>        target=_unsafe_execute,<br>        args=(code, timeout, maximum_memory_bytes, result_dict)<br>    )<br>    p.start()<br>    p.join(timeout=timeout + 1)<br>    <br>    if p.is_alive():<br>        p.kill()<br>        return ExecutionResult(success=False, error=&quot;Execution timed out&quot;, timeout=True)<br>    <br>    return ExecutionResult(<br>        success=result_dict[&quot;success&quot;],<br>        stdout=result_dict[&quot;stdout&quot;],<br>        stderr=result_dict[&quot;stderr&quot;],<br>        error=result_dict[&quot;error&quot;],<br>        timeout=result_dict[&quot;timeout&quot;],<br>        memory_exceeded=result_dict[&quot;memory_exceeded&quot;],<br>    )</pre><p>This sandbox protects against:</p><ul><li>Infinite loops (timeout enforcement)</li><li>Memory exhaustion (resource limits)</li><li>Destructive filesystem operations (disabled dangerous functions)</li><li>Process interference (separate process per execution)</li></ul><p>However, it’s important to note that this is not a security sandbox. It protects against accidental issues but not malicious code.</p><blockquote><strong><em>Note: </em></strong><em>We already know that nanochat is not enterprise pipeline. Its for educational purpose.</em></blockquote><p><strong>Metric:</strong></p><p>HumanEval uses <strong>Pass@1 accuracy</strong>: the percentage of problems where the first generated solution passes all test cases. This is a strict metric that requires both syntactic correctness and semantic correctness.</p><h4>ChatCORE: Conversational Evaluation</h4><p>ChatCORE is a custom chat evaluation metric in nanochat that aggregates performance across the main chat evaluation tasks. It provides a single number summarizing chat model quality.</p><p><strong>Metric Calculation:</strong></p><pre>all_tasks = [&#39;ARC-Easy&#39;, &#39;ARC-Challenge&#39;, &#39;MMLU&#39;, &#39;GSM8K&#39;, &#39;HumanEval&#39;]<br>baseline_accuracies = {<br>    &#39;ARC-Easy&#39;: 0.25, # multiple choice 1 of 4 =&gt; 25%<br>    &#39;ARC-Challenge&#39;: 0.25,<br>    &#39;MMLU&#39;: 0.25,<br>    &#39;GSM8K&#39;: 0.0, # open-ended =&gt; 0%<br>    &#39;HumanEval&#39;: 0.0, # open-ended =&gt; 0%<br>}<br><br># Calculate ChatCORE metric (mean centered accuracy)<br>if all_tasks_were_evaluated:<br>    centered_mean = 0<br>    for task_name, acc in results.items():<br>        baseline_acc = baseline_accuracies.get(task_name, 0.0)<br>        centered_acc = (acc - baseline_acc) / (1.0 - baseline_acc)<br>        centered_mean += centered_acc<br>    chatcore_metric = centered_mean / len(results)</pre><p>Like the CORE metric, ChatCORE centers each task’s accuracy by its random baseline before averaging. This ensures that:</p><ul><li>The metric ranges from 0 (random baseline) to 1 (perfect performance)</li><li>Multiple choice and generative tasks contribute equally</li><li>The metric captures improvement above chance</li></ul><h3>Task Abstraction</h3><p>All evaluation tasks inherit from a common Task base class that provides a consistent interface:</p><pre>class Task:<br>    &quot;&quot;&quot;<br>    Base class of a Task. Allows for lightweight slicing of the underlying dataset.<br>    &quot;&quot;&quot;<br>    def __init__(self, start=0, stop=None, step=1):<br>        # allows a lightweight logical view over a dataset<br>        self.start = start<br>        self.stop = stop # could be None here<br>        self.step = step<br>    @property<br>    def eval_type(self):<br>        # one of &#39;generative&#39; | &#39;categorical&#39;<br>        raise NotImplementedError<br>    def num_examples(self):<br>        raise NotImplementedError<br>    def get_example(self, index):<br>        raise NotImplementedError<br>    def __len__(self):<br>        start = self.start<br>        stop = self.num_examples() if self.stop is None else self.stop<br>        step = self.step<br>        span = stop - start<br>        num = (span + step - 1) // step # ceil_div(span, step)<br>        return num<br>    def __getitem__(self, index: int):<br>        physical_index = self.start + index * self.step<br>        conversation = self.get_example(physical_index)<br>        return conversation<br>    def evaluate(self, problem, completion):<br>        raise NotImplementedError</pre><p>This abstraction enables:</p><ul><li><strong>Dataset Slicing</strong>: Efficiently evaluate on subsets without copying data</li><li><strong>Consistent Interface</strong>: All tasks expose the same methods</li><li><strong>Evaluation Type</strong>: Tasks declare whether they are ‘categorical’ or ‘generative’</li><li><strong>Flexible Evaluation</strong>: Each task implements custom evaluation logic</li></ul><h3>Evaluation Metrics</h3><p>Beyond task-specific accuracy measurements, nanochat tracks several key metrics that provide insight into model quality and training efficiency.</p><h4>Bits Per Byte (BPB)</h4><p>Bits Per Byte is a tokenization-independent metric for measuring language modeling performance. Unlike raw cross-entropy loss, which depends on vocabulary size, BPB normalizes by the actual information content of the target text.</p><h4>Definition</h4><p>BPB is computed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*tS1TZ3EsngpIYKvfe4zI0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lyFVoNcoxeDEKWalZoxKXw.png" /></figure><p>However, the proper implementation requires careful handling of tokenization effects. The key insight is that different tokens represent different amounts of information (measured in bytes). A token representing “ing” contains 3 bytes, while a token representing “a” contains 1 byte.</p><h4>Implementation</h4><p>The implementation normalizes loss by the byte content of target tokens:</p><pre>@torch.no_grad()<br>def evaluate_bpb(model, batches, steps, token_bytes):<br>    &quot;&quot;&quot;<br>    Calculate bits per byte (bpb), a tokenization vocab size-independent metric.<br>    <br>    The way this works is that instead of just calculating the average loss as usual,<br>    you calculate the sum loss, and independently also the sum bytes (of all the <br>    target tokens), and divide. This normalizes the loss by the number of bytes <br>    that the target tokens represent.<br>    <br>    Args:<br>        model: The model to evaluate<br>        batches: Iterator of (input, target) batches<br>        steps: Number of batches to evaluate<br>        token_bytes: 1D tensor of shape (vocab_size,) indicating the number of bytes <br>                     for each token id, or 0 if the token should not be counted<br>    &quot;&quot;&quot;<br>    # Record the losses<br>    total_nats = torch.tensor(0.0, dtype=torch.float32, device=model.get_device())<br>    total_bytes = torch.tensor(0, dtype=torch.int64, device=model.get_device())<br>    <br>    batch_iter = iter(batches)<br>    for _ in range(steps):<br>        x, y = next(batch_iter)<br>        loss2d = model(x, y, loss_reduction=&#39;none&#39;) # (B, T)<br>        loss2d = loss2d.view(-1) # flatten<br>        y = y.view(-1) # flatten<br>        <br>        if (y &lt; 0).any():<br>            # Handle ignore_index (e.g. -1) for masked tokens<br>            valid = y &gt;= 0<br>            y_safe = torch.where(valid, y, torch.zeros_like(y))<br>            num_bytes2d = torch.where(<br>                valid,<br>                token_bytes[y_safe],<br>                torch.zeros_like(y, dtype=token_bytes.dtype)<br>            )<br>            total_nats += (loss2d * (num_bytes2d &gt; 0)).sum()<br>            total_bytes += num_bytes2d.sum()<br>        else:<br>            # Fast path: no ignored targets<br>            num_bytes2d = token_bytes[y]<br>            total_nats += (loss2d * (num_bytes2d &gt; 0)).sum()<br>            total_bytes += num_bytes2d.sum()<br>    <br>    # Sum reduce across all ranks<br>    world_size = dist.get_world_size() if dist.is_initialized() else 1<br>    if world_size &gt; 1:<br>        dist.all_reduce(total_nats, op=dist.ReduceOp.SUM)<br>        dist.all_reduce(total_bytes, op=dist.ReduceOp.SUM)<br>    <br>    # Calculate bpb and return<br>    total_nats = total_nats.item()<br>    total_bytes = total_bytes.item()<br>    bpb = total_nats / (math.log(2) * total_bytes)<br>    return bpb</pre><h4>Token Byte Mapping</h4><p>The token_bytes tensor is precomputed during tokenizer training:</p><pre># Cache a mapping from token id to number of bytes<br>vocab_size = tokenizer.get_vocab_size()<br>special_set = set(tokenizer.get_special_tokens())<br>token_bytes = []<br>for token_id in range(vocab_size):<br>    token_str = tokenizer.decode([token_id])<br>    if token_str in special_set:<br>        token_bytes.append(0)  # special tokens are not counted<br>    else:<br>        id_bytes = len(token_str.encode(&quot;utf-8&quot;))<br>        token_bytes.append(id_bytes)</pre><p>This mapping ensures:</p><ol><li>Normal tokens are weighted by their byte length</li><li>Special tokens (like &lt;|bos|&gt;) are excluded from the metric (weight 0)</li><li>Masked tokens (with ignore_index=-1) are excluded from the metric</li></ol><h4>Why BPB Matters</h4><p>BPB enables fair comparisons across:</p><ul><li><strong>Different tokenizers</strong>: A model with vocab size 32k vs 64k can be compared</li><li><strong>Different languages</strong>: UTF-8 encoding ensures consistent byte measurement</li><li><strong>Training progress</strong>: BPB improvements directly reflect better compression</li></ul><p>A lower BPB indicates better language modeling, as the model needs fewer bits to represent each byte of text.</p><h4>Measurement Context</h4><p>BPB is measured on a validation subset of FineWeb during base pretraining:</p><pre>if last_step or step % eval_every == 0:<br>    model.eval()<br>    val_loader = build_val_loader()<br>    eval_steps = eval_tokens // (device_batch_size * max_seq_len * ddp_world_size)<br>    with autocast_ctx:<br>        val_bpb = evaluate_bpb(model, val_loader, eval_steps, token_bytes)<br>    print0(f&quot;Step {step:05d} | Validation bpb: {val_bpb:.4f}&quot;)</pre><h3>Model FLOPs Utilization (MFU)</h3><p>Model FLOPs Utilization measures training efficiency by comparing the actual computational throughput to the theoretical hardware peak.</p><h4>Definition</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*laUPBWtAYj_DG0Tu8OMnow.png" /></figure><p>For NVIDIA H100 SXM with bfloat16:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*R8qpq_B8DtHzxErza1gckw.png" /></figure><p>Full expression:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*SqCfFjqaDt01-IEXgdBjsA.png" /></figure><blockquote><strong><em>Note:</em></strong><em> Here, 100% means mulitplying with 100 to find the percentage (0.1 * 100 = 10 -&gt; 10%)</em></blockquote><h4>Interpretation</h4><p>Achieving high MFU requires:</p><ol><li><strong>Large batch sizes</strong>: Maximize parallelism and amortize memory latency</li><li><strong>Optimized kernels</strong>: Use Flash Attention and fused operations</li><li><strong>Efficient data loading</strong>: Prevent GPU starvation</li><li><strong>Mixed precision</strong>: Leverage tensor cores with bfloat16/float16</li></ol><h4>Calculation</h4><p>The implementation tracks tokens processed per second and converts to FLOPs:</p><pre># Training step<br>t0 = time.time()<br>optimizers_zero_grad()<br>for micro_step in range(grad_accum_steps):<br>    with autocast_ctx:<br>        loss = model(x, y)<br>        loss = loss / grad_accum_steps<br>    loss.backward()<br>torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=grad_clip_norm)<br>optimizers_step()<br>torch.cuda.synchronize()<br>t1 = time.time()<br><br># Compute metrics<br>dt = t1 - t0<br>total_training_time += dt<br>flops_per_sec = num_flops_per_token * total_batch_size / dt<br>promised_flops_per_sec_h100 = 989e12 * ddp_world_size # bfloat16 H100 SXM<br>mfu = 100 * flops_per_sec / promised_flops_per_sec_h100</pre><h3>FLOPs per Token Estimate</h3><p>Understanding the computational cost of training requires estimating FLOPs per token. This metric helps predict training time and cost for different model configurations.</p><h4>Formula</h4><p>For a transformer model, the forward pass FLOPs per token are approximately:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*ZYu6ZEl8QcVpKlvwxdxsAw.png" /></figure><p>where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*P4yFzJJPz0olzGKPIDsx4A.png" /></figure><h4>Derivation</h4><p>The formula comes from two main components:</p><p><strong>1. Matrix Multiplications (dominant term):</strong></p><p>Each transformer layer contains:</p><ul><li>Query, Key, Value projections: 3*2*d*d FLOPs</li><li>Attention output projection: 2*d*d FLOPs</li><li>Two MLP layers: 2*2*d*4*d FLOPs</li></ul><blockquote><strong><em>Note:</em></strong><em> Where does extra “2” come from? A projection process includes d multiplications and (d-1) additions. therefore it can be counted as 2d operations</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*dL5a-5J0VxIVs4E1IHITsA.png" /></figure><p><strong>2. Attention Computation:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*WXPsrThZiu43uZQuUIZRCQ.png" /></figure><blockquote><strong><em>Note: </em></strong><em>If there is miscalculation, please inform me at comments. So that I can fix it. My calculations are from </em><a href="https://arxiv.org/abs/2204.02311"><em>PaLM paper (nanochat repo refers here in flop calculation</em></a><em>).</em></blockquote><h4>Implementation</h4><pre># Model configuration<br>@dataclass<br>class GPTConfig:<br>    sequence_len: int = 1024<br>    vocab_size: int = 50304<br>    n_layer: int = 12<br>    n_head: int = 6 # number of query heads<br>    n_kv_head: int = 6 # number of key/value heads (MQA)<br>    n_embd: int = 768<br><br># FLOPs calculation<br>def estimate_flops(self):<br>    &quot;&quot;&quot;Return the estimated FLOPs per token for the model.&quot;&quot;&quot;<br>    nparams = sum(p.numel() for p in self.parameters())<br>    nparams_embedding = self.transformer.wte.weight.numel()<br>    l, h, q, t = self.config.n_layer, self.config.n_head, self.config.n_embd // self.config.n_head, self.config.sequence_len<br>    num_flops_per_token = 6 * (nparams - nparams_embedding) + 12 * l * h * q * t<br>    return num_flops_per_token</pre><h4>Backward Pass Consideration</h4><p>The full training cost is approximately <strong>3x the forward pass FLOPs</strong>:</p><ul><li>1x forward pass</li><li>2x backward pass (computing gradients w.r.t. inputs and weights)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*m0ODj2C_NJAY6cuiqhPJcQ.png" /></figure><h4>Practical Usage</h4><p>This estimate helps with:</p><ol><li><strong>Training time prediction</strong>: Given target token count and hardware, estimate wall-clock time</li><li><strong>Cost estimation</strong>: Convert FLOPs to GPU-hours to estimate cloud costs</li><li><strong>Hyperparameter selection</strong>: Compare compute budgets for different model sizes</li><li><strong>Efficiency debugging</strong>: If measured FLOPs differ significantly, investigate bottlenecks</li></ol><h3>Evaluation During Training</h3><p>During base pretraining, evaluation is integrated into the training loop:</p><pre># CORE metric evaluation every 2000 steps<br>if last_step or (step &gt; 0 and step % core_metric_every == 0):<br>    model.eval()<br>    with autocast_ctx:<br>        results = evaluate_model(orig_model, tokenizer, device, <br>                                max_per_task=core_metric_max_per_task)<br>    print0(f&quot;Step {step:05d} | CORE metric: {results[&#39;core_metric&#39;]:.4f}&quot;)<br>    wandb_run.log({<br>        &quot;step&quot;: step,<br>        &quot;total_training_flops&quot;: flops_so_far,<br>        &quot;core_metric&quot;: results[&quot;core_metric&quot;],<br>        &quot;centered_results&quot;: results[&quot;centered_results&quot;],<br>    })<br>    model.train()</pre><p>Note the use of orig_model rather than the compiled model, since evaluation inputs vary in sequence length.</p><h3>Putting It All Together</h3><p>The evaluation framework serves multiple purposes throughout the nanochat training pipeline:</p><h4>During Base Pretraining</h4><ul><li><strong>BPB</strong> (every 250 steps): Fast feedback on language modeling quality</li><li><strong>CORE metric</strong> (every 2000 steps): Measure reasoning capability development across 22 diverse tasks</li><li><strong>MFU</strong>: Monitor training efficiency and hardware utilization</li><li><strong>Generation samples</strong>: Qualitatively assess model behavior</li></ul><h4>During Chat Fine-Tuning</h4><ul><li><strong>Task-specific accuracy</strong>: Measure performance on ARC, MMLU, GSM8K, HumanEval</li><li><strong>ChatCORE metric</strong>: Single number summarizing chat model quality</li><li><strong>Instruction following</strong>: Evaluate response format and structure</li></ul><h3>Conclusion</h3><blockquote><strong><em>Note:</em></strong><em> Finally… the end! This part was especially challenging to understand and explain as clearly as possible. I believe that knowing something and being able to explain it to someone unfamiliar with the concept are two very different skills. I hope I managed to present the ideas clearly and accurately so that they’re helpful to you.</em></blockquote><p>The nanochat evaluation framework demonstrates several key principles for measuring language model performance:</p><ol><li><strong>Diverse Tasks</strong>: Evaluation spans reasoning, math, code, and conversation</li><li><strong>Standardized Baselines</strong>: All metrics are centered by random performance</li><li><strong>Efficiency Metrics</strong>: Track computational efficiency alongside accuracy</li><li><strong>Tokenization Independence</strong>: BPB enables fair comparisons across vocabularies</li><li><strong>Distributed Execution</strong>: Multi-GPU support for fast evaluation</li></ol><p>This comprehensive approach provides the feedback signals necessary to train capable language models while maintaining efficiency and reproducibility. The framework balances quantitative benchmarks (CORE, ChatCORE, BPB) with qualitative assessment (generation samples) and efficiency monitoring (MFU), giving developers a complete picture of model development progress.</p><blockquote><strong><em>Note:</em></strong><em> Please also keep in mind that this repository is still new and may change over time. I probably won’t update these posts when changes occur (unless I need to fix an error I made). That’s why I’ve tried to include as many code snippets as possible.</em></blockquote><h3>References / Sources</h3><p><a href="https://github.com/karpathy/nanochat">GitHub - karpathy/nanochat: The best ChatGPT that $100 can buy.</a></p><p><strong>Note:</strong> If you’d like to learn more about LLMs or the latest developments in Deep Learning, I’d recommend following the channels and accounts below (some of them might be obvious, but just in case):</p><p>1- Me (Of course! 😄)</p><p><a href="https://medium.com/@akdemir_bahadir">Bahadır AKDEMİR - Medium</a></p><p>2- <strong>Andrej Karpathy’s YouTube Channel (</strong>He doesn’t post videos often, but whenever he does, they’re very insightful and valuable.</p><p><a href="https://www.youtube.com/@AndrejKarpathy">Andrej Karpathy</a></p><p>3-<strong>Julia Turc’s YouTube Channel</strong> (Simple yet meaningful explanations and effective visualizations of AI concepts.)</p><p><a href="https://www.youtube.com/@juliaturc1">Julia Turc</a></p><p>4- <strong>Yannic Kilcher’s YouTube Channel (</strong>Before reading a popular paper, I usually check if Yannic Kilcher has already explained it.)</p><p><a href="https://www.youtube.com/@YannicKilcher">Yannic Kilcher</a></p><p>5- <strong>bycloud (</strong>You don’t need to speed up his videos because he’s already fast enough 😄 Funny, meme-style explanations make learning both entertaining and effective.<strong>)</strong></p><p><a href="https://www.youtube.com/@bycloudAI">bycloud</a></p><p>6- <strong>AI Coffee Break</strong> (Great visualizations with a soft, calm voice. It’s the perfect channel to grab a coffee and relax while learning the concepts you’re interested in.)</p><p><a href="https://www.youtube.com/@AICoffeeBreak">AI Coffee Break with Letitia</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f675ced9e736" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside nanochat Part 5: Understanding Training]]></title>
            <link>https://medium.com/@akdemir_bahadir/inside-nanochat-part-5-understanding-training-5e729522ac2a?source=rss-711524cb67b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e729522ac2a</guid>
            <category><![CDATA[llm-training]]></category>
            <category><![CDATA[nanochat]]></category>
            <category><![CDATA[lrs]]></category>
            <category><![CDATA[pre-training]]></category>
            <category><![CDATA[sft]]></category>
            <dc:creator><![CDATA[Bahadır AKDEMİR]]></dc:creator>
            <pubDate>Tue, 21 Oct 2025 12:23:13 GMT</pubDate>
            <atom:updated>2025-10-21T12:23:13.684Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w23oTnxIQ3ZsxzOyEpLsig.png" /></figure><h3>Introduction</h3><p>In this part, we explore the multi-phase training pipeline that turns raw text data into a functional conversational model. The training process unfolds across five distinct stages, each addressing a specific aspect of the model’s development. This structured progression reflects contemporary language model training practices, where models acquire skills step by step through a well-defined curriculum.</p><h3>Overview of the Training Pipeline</h3><p>The <strong>nanochat</strong> training pipeline progresses through a series of well-defined stages, each building on the previous one:</p><ol><li><strong>Tokenizer Training</strong> : Construct a vocabulary tailored to the training corpus.</li><li><strong>Base Model Pretraining: </strong>Learn general language patterns from large-scale web text.</li><li><strong>Midtraining: </strong>Introduce conversational formats and tool usage to shape interactive behavior.</li><li><strong>Supervised Fine-Tuning</strong>: Adapt the model to specific domains and structured tasks.</li><li><strong>Reinforcement Learning (Optional):</strong> Refine performance by optimizing for task-specific feedback signals.</li></ol><p>Together, these phases guide the model from an untrained neural network to a system capable of coherent dialogue and problem-solving.</p><h3>Tokenizer Training</h3><p>The first stage in training a language model is to build a <strong>tokenizer, </strong>a system that converts raw text into sequences of integers the model can understand. In <strong>nanochat</strong>, a <strong>Byte Pair Encoding (BPE)</strong> tokenizer is trained with a vocabulary size of 2¹⁶ (65,536) tokens (can be customized).</p><blockquote><strong><em>Note:</em></strong><em> I already explained the training tokenizer in detail </em><a href="https://medium.com/@akdemir_bahadir/inside-nanochat-part-1-understanding-tokenization-00f48f429693"><em>in part 1.</em></a></blockquote><h4>Training Process</h4><p>Tokenizer training consists of several key steps:</p><ol><li><strong>Data Collection:</strong> Load the first eight data shards, containing roughly two billion characters of web text.</li><li><strong>Vocabulary Construction: </strong>Apply the BPE algorithm to identify the most frequent character sequences and merge them into tokens.</li><li><strong>Compression Evaluation</strong>: Assess how efficiently the tokenizer represents text while minimizing sequence length. (Will be covered in next post. Not here)</li><li><strong>Artifact Persistence:</strong> Save the trained tokenizer for consistent use across all subsequent training stages.</li></ol><p>A <strong>Rust-based BPE implementation (</strong>rustbpe<strong>)</strong> is used for performance efficiency, as tokenizer training involves extensive computation over large text datasets.</p><pre>def text_iterator():<br>    &quot;&quot;&quot;<br>    1) Flatten the batches into a single iterator<br>    2) Crop every document to args.doc_cap characters<br>    3) Break when we&#39;ve seen args.max_chars characters<br>    &quot;&quot;&quot;<br>    nchars = 0<br>    for batch in parquets_iter_batched(split=&quot;train&quot;):<br>        for doc in batch:<br>            doc_text = doc<br>            if len(doc_text) &gt; args.doc_cap:<br>                doc_text = doc_text[:args.doc_cap]<br>            nchars += len(doc_text)<br>            yield doc_text<br>            if nchars &gt; args.max_chars:<br>                return<br><br>text_iter = text_iterator()<br><br># Train the tokenizer<br>t0 = time.time()<br>tokenizer = RustBPETokenizer.train_from_iterator(text_iter, args.vocab_size)<br>t1 = time.time()</pre><p>The BPE algorithm works by iteratively merging the most frequent pairs of byte sequences. This creates a vocabulary that efficiently represents common words and subwords while handling rare words through byte-level fallback.</p><h3>Phase 2: Base Model Pretraining</h3><p>Base pretraining is where the model learns the fundamental patterns of language. We train an autoregressive language model on web text, teaching it to predict the next token given all previous tokens. This phase consumes the majority of computational resources and determines the model’s core capabilities.</p><blockquote><em>Note: You can find more information about pretraining by these two videos (</em><a href="https://youtu.be/Zar2TJv-sE0"><em>[here]</em></a><em>, </em><a href="https://youtu.be/7xTGNNLPyMI"><em>[here]</em></a><em>)</em></blockquote><h4>Optimizer Configuration</h4><p>The training employs a hybrid optimization strategy, using different optimizers for different parameter groups:</p><pre># Initialize the Optimizer (Muon for Linear layers, AdamW for embedding and lm_head)<br>optimizers = model.setup_optimizers(<br>    unembedding_lr=unembedding_lr,<br>    embedding_lr=embedding_lr,<br>    matrix_lr=matrix_lr,<br>    weight_decay=weight_decay<br>)<br>adamw_optimizer, muon_optimizer = optimizers</pre><p>The Muon optimizer, designed specifically for transformer weight matrices, provides better convergence properties than standard optimizers. Meanwhile, AdamW handles the embedding and unembedding layers, which benefit from adaptive learning rates.</p><blockquote><strong><em>Note:</em></strong><em> </em><a href="https://medium.com/@akdemir_bahadir/inside-nanochat-part-3-understanding-optimizers-7f989d407153"><em>All explained in my blog post here (part 3).</em></a></blockquote><h4>Data Loading Pipeline</h4><p>The data loading pipeline streams tokenized text from disk, ensuring efficient GPU utilization:</p><pre>def tokenizing_distributed_data_loader(B, T, split, tokenizer_threads=4, tokenizer_batch_size=128):<br>    &quot;&quot;&quot;Stream pretraining text from parquet files, tokenize, yield training batches.&quot;&quot;&quot;<br>    needed_tokens = B * T + 1  # +1 for target at last position<br>    tokenizer = get_tokenizer()<br>    bos_token = tokenizer.get_bos_token_id()<br>    token_buffer = deque()<br>    <br>    while True:<br>        # Accumulate enough tokens for one iteration<br>        while len(token_buffer) &lt; needed_tokens:<br>            doc_batch = next(batches)<br>            token_lists = tokenizer.encode(doc_batch, prepend=bos_token, num_threads=tokenizer_threads)<br>            for tokens in token_lists:<br>                token_buffer.extend(tokens)<br>        <br>        # Create inputs/targets and yield<br>        for i in range(needed_tokens):<br>            scratch[i] = token_buffer.popleft()<br>        inputs = scratch[:-1].view(B, T).to(device=&quot;cuda&quot;, dtype=torch.int32)<br>        targets = scratch[1:].view(B, T).to(device=&quot;cuda&quot;, dtype=torch.int64)<br>        yield inputs, targets</pre><p>This streaming approach allows training on datasets far larger than available memory.</p><blockquote><strong><em>Note: </em></strong><em>currently nanochat is designed for cuda devices. There are some parts like this. May be they will be more generalized later.</em></blockquote><h4>Training Loop</h4><p>The core training loop implements gradient accumulation for large effective batch sizes:</p><pre>for step in range(num_iterations + 1):<br>    # Single training step<br>    for micro_step in range(grad_accum_steps):<br>        with autocast_ctx:<br>            loss = model(x, y)<br>        train_loss = loss.detach()<br>        loss = loss / grad_accum_steps<br>        loss.backward()<br>        x, y = next(train_loader)  # prefetch next batch<br>    <br>    # Gradient clipping<br>    if grad_clip &gt; 0.0:<br>        torch.nn.utils.clip_grad_norm_(orig_model.parameters(), grad_clip)<br>    <br>    # Update learning rates<br>    lrm = get_lr_multiplier(step)<br>    for opt in optimizers:<br>        for group in opt.param_groups:<br>            group[&quot;lr&quot;] = group[&quot;initial_lr&quot;] * lrm<br>    <br>    # Update momentum for Muon<br>    muon_momentum = get_muon_momentum(step)<br>    for group in muon_optimizer.param_groups:<br>        group[&quot;momentum&quot;] = muon_momentum<br>    <br>    # Step optimizers<br>    for opt in optimizers:<br>        opt.step()<br>    model.zero_grad(set_to_none=True)</pre><h4>Learning Rate Schedule</h4><p>The learning rate follows a warmup-stable-warmdown schedule (<a href="https://medium.com/@akdemir_bahadir/inside-nanochat-part-3-understanding-optimizers-7f989d407153">explained in part 3</a>):</p><pre>def get_lr_multiplier(it):<br>    warmup_iters = round(warmup_ratio * num_iterations)<br>    warmdown_iters = round(warmdown_ratio * num_iterations)<br>    if it &lt; warmup_iters:<br>        return (it + 1) / warmup_iters<br>    elif it &lt;= num_iterations - warmdown_iters:<br>        return 1.0<br>    else:<br>        progress = (num_iterations - it) / warmdown_iters<br>        return progress * 1.0 + (1 - progress) * final_lr_frac</pre><p>With default settings (no warmup, 20% warmdown), the learning rate remains constant for 80% of training, then linearly decays to zero. This schedule prevents overfitting while ensuring thorough exploration early in training.</p><h3>Phase 3: Midtraining</h3><p>The concept of <strong>midtraining</strong> is introduced in this repository and is not commonly seen in typical language model pipelines (As far as I know). Standard LLM training usually follows the sequence of <strong>Pretraining → Supervised Fine-Tuning (SFT) → Reinforcement Learning (RL)</strong>. Midtraining serves as a bridge between general language modeling and conversational capability.</p><p>In this phase, the model learns to recognize conversation boundaries, interpret special tokens, and use external tools (in this case, a <strong>calculator)</strong>. These skills are essential for transforming a pretrained language model into one suited for interactive dialogue.</p><blockquote><strong><em>Note:</em></strong><em> In most frameworks, midtraining is integrated into the SFT stage. However, </em><strong><em>nanochat</em></strong><em> defines it as a distinct and separate phase.</em></blockquote><h4>Data Mixture</h4><p>The midtraining dataset combines three sources:</p><pre>train_dataset = TaskMixture([<br>    SmolTalk(split=&quot;train&quot;),                          # 460K conversations<br>    MMLU(subset=&quot;auxiliary_train&quot;, split=&quot;train&quot;),   # 100K multiple choice<br>    GSM8K(subset=&quot;main&quot;, split=&quot;train&quot;),             # 8K math + calculator<br>])  # Total: 568K examples</pre><ul><li><strong>SmolTalk</strong>: General conversations teaching dialogue patterns</li><li><strong>MMLU Auxiliary</strong>: Multiple-choice questions for knowledge grounding</li><li><strong>GSM8K</strong>: Math word problems requiring calculator tool use</li></ul><h4>Conversation Rendering</h4><p>Unlike pretraining, midtraining uses structured conversations. The tokenizer converts these into sequences while preserving turn boundaries:</p><pre>def render_conversation(self, conversation, max_tokens=2048):<br>    &quot;&quot;&quot;<br>    Tokenize a single conversation.<br>    Returns:<br>    - ids: list[int] - token ids<br>    - mask: list[int] - 1 for supervised tokens, 0 otherwise<br>    &quot;&quot;&quot;<br>    ids, mask = [], []<br>    <br>    # Add BOS token<br>    add_tokens(bos, 0)<br>    <br>    for i, message in enumerate(messages):<br>        if message[&quot;role&quot;] == &quot;user&quot;:<br>            add_tokens(user_start, 0)<br>            add_tokens(self.encode(content), 0)<br>            add_tokens(user_end, 0)<br>        elif message[&quot;role&quot;] == &quot;assistant&quot;:<br>            add_tokens(assistant_start, 0)<br>            if isinstance(content, str):<br>                add_tokens(self.encode(content), 1)  # Supervised<br>            elif isinstance(content, list):<br>                for part in content:<br>                    if part[&quot;type&quot;] == &quot;text&quot;:<br>                        add_tokens(self.encode(part[&quot;text&quot;]), 1)<br>                    elif part[&quot;type&quot;] == &quot;python&quot;:<br>                        add_tokens(python_start, 1)<br>                        add_tokens(self.encode(part[&quot;text&quot;]), 1)<br>                        add_tokens(python_end, 1)<br>            add_tokens(assistant_end, 1)<br>    <br>    return ids[:max_tokens], mask[:max_tokens]</pre><p>The mask array indicates which tokens should contribute to the loss. User messages and system-generated content (like <strong><em>tool outputs</em></strong>) are masked out, ensuring the model only learns to generate assistant responses.</p><h4>Variable-Length Sequences</h4><p>Unlike pretraining’s fixed-length sequences, conversations have natural boundaries:</p><pre>def mid_data_generator(split):<br>    dataset = train_dataset if split == &quot;train&quot; else val_dataset<br>    needed_tokens = device_batch_size * max_seq_len + 1<br>    token_buffer = deque()<br>    cursor = ddp_rank<br>    <br>    while True:<br>        # Accumulate tokens from multiple conversations<br>        while len(token_buffer) &lt; needed_tokens:<br>            conversation = dataset[cursor]<br>            ids, _ = tokenizer.render_conversation(conversation)<br>            token_buffer.extend(ids)<br>            cursor += ddp_world_size<br>            if cursor &gt;= dataset_size:<br>                cursor -= dataset_size  # wrap around<br>                if split == &quot;train&quot;:<br>                    last_step = True<br>        <br>        # Build batch and yield<br>        for i in range(needed_tokens):<br>            scratch[i] = token_buffer.popleft()<br>        inputs = scratch[:-1].view(device_batch_size, max_seq_len)<br>        targets = scratch[1:].view(device_batch_size, max_seq_len)<br>        yield inputs, targets</pre><p>This approach packs multiple conversations into each batch, maximizing GPU utilization while respecting conversation boundaries through special tokens.</p><h4>Key Differences from Base Training</h4><ol><li><strong>Shorter Duration</strong>: One epoch over the conversation dataset (~1000 steps vs. 20,000+ for pretraining)</li><li><strong>Same Optimizer Settings</strong>: Reuse the tuned hyperparameters from pretraining</li><li><strong>Structured Data</strong>: Conversations with explicit turn boundaries</li><li><strong>Tool Examples</strong></li></ol><h4>Learning Rate Schedule</h4><pre>def get_lr_multiplier(progress):<br>    # First 80% of training: constant, then linear decay<br>    return 1 if progress &lt; 0.8 else 1 - (progress - 0.8) / 0.2</pre><p>The schedule maintains full learning rate for most of training, then gently reduces it toward the end.</p><h4>Expected Outcomes</h4><p>After midtraining, the model exhibits:</p><ul><li><strong>Conversation awareness</strong>: Responds appropriately to user messages</li><li><strong>Special token understanding</strong>: Uses &lt;|assistant_start|&gt; and &lt;|assistant_end|&gt; correctly</li><li><strong>Calculator tool usage</strong>: Invokes Python expressions wrapped in &lt;|python_start|&gt;...&lt;|python_end|&gt;</li><li><strong>Improved chat metrics</strong>: Better performance on conversational benchmarks</li></ul><h3>Phase 4: Supervised Fine-Tuning</h3><p>Supervised Fine-Tuning (SFT) adapts the midtrained model to specific task domains. While midtraining provides general conversation abilities, SFT focuses on particular capabilities like science question answering or mathematical reasoning. This phase demonstrates rapid adaptation to new tasks with minimal additional training.</p><h4>Dataset Mixture</h4><p>The SFT phase uses a curated mixture emphasizing the target capabilities:</p><pre>train_ds = TaskMixture([<br>    ARC(subset=&quot;ARC-Easy&quot;, split=&quot;train&quot;),        # 2.3K science questions<br>    ARC(subset=&quot;ARC-Challenge&quot;, split=&quot;train&quot;),   # 1.1K harder science<br>    GSM8K(subset=&quot;main&quot;, split=&quot;train&quot;),          # 8K math problems<br>    SmolTalk(split=&quot;train&quot;, stop=10_000),         # 10K conversations<br>])  # Total: 21.4K examples</pre><p>This mixture balances task-specific data (ARC, GSM8K) with general conversation ability (SmolTalk) to prevent catastrophic forgetting.</p><h4>Data Collation</h4><p>SFT requires careful handling of variable-length sequences:</p><pre>def sft_data_generator(dataset, batch_size):<br>    pad_token_id = tokenizer.encode_special(&quot;&lt;|assistant_end|&gt;&quot;)<br>    <br>    def collate_and_yield(batch):<br>        nrows = len(batch)<br>        ncols = max(len(ids) for ids, mask in batch) - 1<br>        inputs = torch.full((nrows, ncols), pad_token_id, dtype=torch.long)<br>        targets = torch.full((nrows, ncols), -1, dtype=torch.long)<br>        <br>        for i, (ids, mask) in enumerate(batch):<br>            n = len(ids)<br>            ids_tensor = torch.tensor(ids, dtype=torch.long)<br>            inputs[i, :n-1] = ids_tensor[:-1]<br>            <br>            # Apply mask: set target to -1 where mask is 0<br>            row_targets = ids_tensor[1:]<br>            mask_tensor = torch.tensor(mask[1:], dtype=torch.long)<br>            row_targets[mask_tensor == 0] = -1<br>            targets[i, :n-1] = row_targets<br>        <br>        return inputs.to(device), targets.to(device)<br>    <br>    # Iterate over dataset, tokenize, and yield batches<br>    batch = []<br>    while True:<br>        for i in range(ddp_rank, len(dataset), ddp_world_size):<br>            doc = dataset[i]<br>            ids, mask = tokenizer.render_conversation(doc)<br>            batch.append((ids, mask))<br>            if len(batch) == batch_size:<br>                yield collate_and_yield(batch)<br>                batch = []</pre><p>The padding token is set to &lt;|assistant_end|&gt;, and crucially, padded positions have targets set to -1 (PyTorch&#39;s ignore index), ensuring they don&#39;t contribute to the loss.</p><h4>Training Process</h4><p>The SFT training loop is simpler than pretraining due to the smaller dataset:</p><pre>num_iterations = (len(train_ds) // target_examples_per_step) * num_epochs<br>train_loader = sft_data_generator(train_ds, batch_size=device_batch_size)<br><br>for step in range(num_iterations):<br>    # Evaluate gradient<br>    for micro_step in range(grad_accum_steps):<br>        train_inputs, train_targets = next(train_iter)<br>        with autocast_ctx:<br>            loss = model(train_inputs, train_targets)<br>        train_loss = loss.detach()<br>        loss = loss / grad_accum_steps<br>        loss.backward()<br>    <br>    # Update learning rate<br>    lrm = get_lr_multiplier(step)<br>    for opt in optimizers:<br>        for group in opt.param_groups:<br>            group[&quot;lr&quot;] = group[&quot;initial_lr&quot;] * lrm<br>    <br>    # Step optimizers<br>    for opt in optimizers:<br>        opt.step()<br>    model.zero_grad(set_to_none=True)</pre><blockquote><strong><em>Note: </em></strong><em>I’ve always felt that LLM architectures look simpler than vision models like YOLO. But when I examine the full training pipeline, language models seem far more complex. I’m not sure if it’s just me, but the gap feels surprisingly wide. The code snippet above may look straightforward, yet once you dive into the underlying functions and classes, the complexity quickly becomes apparent.</em></blockquote><h4>Masked Loss Computation</h4><p>A critical detail: the model computes loss only on supervised tokens (where targets &gt;= 0):</p><pre># In the model&#39;s forward pass<br>def forward(self, inputs, targets):<br>    logits = self.transformer(inputs)  # (B, T, vocab_size)<br>    <br>    # Cross-entropy loss, automatically ignoring targets == -1<br>    loss = F.cross_entropy(<br>        logits.view(-1, logits.size(-1)),<br>        targets.view(-1),<br>        ignore_index=-1<br>    )<br>    return loss</pre><blockquote><strong><em>Note:</em></strong><em> This ensures the model learns only from assistant responses, not user inputs or system-generated content.</em></blockquote><h4>Learning Rate Schedule</h4><pre>def get_lr_multiplier(it):<br>    lrm = 1.0 - it / num_iterations<br>    return lrm</pre><p>The linear decay from full learning rate to zero over the course of training prevents overfitting while allowing the model to fine-tune its representations for the target tasks.</p><h4>Conservative Initialization</h4><pre># Set initial LR as fraction of base LR<br>for opt in optimizers:<br>    for group in opt.param_groups:<br>        group[&quot;lr&quot;] = group[&quot;lr&quot;] * init_lr_frac  # 0.02<br>        group[&quot;initial_lr&quot;] = group[&quot;lr&quot;]</pre><p>Starting with 2% of the pretraining learning rate ensures we adapt the model without destroying the knowledge acquired during earlier phases.</p><h3>Phase 5: Reinforcement Learning (Optional)</h3><blockquote><strong><em>Note: </em></strong><em>From my own experience, I found PPO and GRPO quite hard to understand. Maybe it is just me, but I really recommend watching videos to get a deeper grasp of the concepts. Without that, you might feel like you understand them at first, but when you look at the code again or try to edit it, you realize the logic is still unclear. I will probably forget some parts again once I step away from the RL section. So it is worth taking the time to understand these topics well. Here are a few helpful resources on GRPO: </em><a href="https://youtu.be/xT4jxQUl0X8?si=TBYMLodU0d2RNnXS"><em>[here1]</em></a><em>, </em><a href="https://youtu.be/bAWV_yrqx4w?si=xqLUmXws7i3jWIjv"><em>[here2]</em></a><em>, </em><a href="https://youtu.be/Yi1UCrAsf4o?si=0SGozm3Y7YyfyLi_"><em>[here3]</em></a><em>.</em></blockquote><blockquote><strong><em>Note:</em></strong><em> If you notice any part that I may have explained incorrectly, please let me know in the comments so I can correct it.</em></blockquote><p>Reinforcement Learning (RL) optimizes the model for task-specific rewards beyond simple imitation learning. While SFT teaches the model to mimic human demonstrations, RL allows the model to discover strategies that maximize success metrics. In nanochat, it applies RL to improve performance on GSM8K, a mathematical reasoning benchmark.</p><h4>Algorithm: Simplified Policy Gradient</h4><p>The implementation uses a simplified variant of GRPO (Group Relative Policy Optimization), reducing to essentially REINFORCE with advantages:</p><pre>&quot;&quot;&quot;<br>I put GRPO in quotes because we actually end up with something a lot<br>simpler and more similar to just REINFORCE:<br><br>1) Delete trust region, so there is no KL regularization to a reference model<br>2) We are on policy, so there&#39;s no need for PPO ratio+clip<br>3) We use GAPO style normalization that is token-level, not sequence-level<br>4) Instead of z-score normalization (r - mu)/sigma, only use (r - mu) as advantage<br>&quot;&quot;&quot;</pre><p>This stripped-down approach works surprisingly well for mathematical reasoning, where the reward signal (correct/incorrect) is unambiguous.</p><h3>Training Process</h3><p>The RL training loop alternates between sampling and learning:</p><p><strong>1. Rollout Generation</strong>: Sample multiple completions per problem</p><pre>@torch.no_grad()<br>def get_batch():<br>    for example_idx in itertools.cycle(rank_indices):<br>        conversation = train_task[example_idx]<br>        <br>        # Tokenize, keeping only the prompt (delete assistant&#39;s answer)<br>        tokens = tokenizer.render_for_completion(conversation)<br>        prefix_length = len(tokens)<br>        <br>        # Generate num_samples completions<br>        generated_token_sequences = []<br>        masks = []<br>        for sampling_step in range(num_samples // device_batch_size):<br>            seed = hash((step, example_idx, sampling_step)) &amp; 0x7FFFFFFF<br>            with autocast_ctx:<br>                seqs, masks_batch = engine.generate_batch(<br>                    tokens,<br>                    num_samples=device_batch_size,<br>                    max_tokens=max_new_tokens,<br>                    temperature=temperature,<br>                    top_k=top_k,<br>                    seed=seed,<br>                )<br>            generated_token_sequences.extend(seqs)<br>            masks.extend(masks_batch)</pre><p><strong>2. Reward Computation</strong>: Evaluate each completion</p><pre># Calculate rewards for each sample<br>        rewards = []<br>        for sample_tokens in generated_token_sequences:<br>            generated_tokens = sample_tokens[prefix_length:]<br>            generated_text = tokenizer.decode(generated_tokens)<br>            reward = train_task.reward(conversation, generated_text)<br>            rewards.append(reward)</pre><p>For GSM8K, the reward is binary: 1.0 if the answer is correct, 0.0 otherwise. The evaluation parses the model’s output, extracts the final numerical answer, and compares it to the ground truth.</p><p><strong>3. Advantage Calculation</strong>: Compute deviation from mean</p><pre>rewards = torch.tensor(rewards, dtype=torch.float, device=device)<br>        mu = rewards.mean()<br>        advantages = rewards - mu</pre><p>This simple advantage estimation encourages completions that perform better than average on this problem.</p><p><strong>4. Policy Update</strong>: Backpropagate weighted by advantages</p><pre># Prepare training data<br>        ids = torch.tensor(padded_sequences, dtype=torch.long, device=device)<br>        mask_ids = torch.tensor(padded_masks, dtype=torch.long, device=device)<br>        inputs = ids[:, :-1]<br>        targets = ids[:, 1:].clone()<br>        targets[mask_ids[:, 1:] == 0] = -1  # ignore prompt and tool use<br>        <br>        # Forward pass to get log probabilities<br>        with autocast_ctx:<br>            logp = -model(inputs, targets, loss_reduction=&#39;none&#39;).view_as(inputs)<br>        <br>        # Policy gradient objective<br>        pg_obj = (logp * advantages.unsqueeze(-1)).sum()<br>        num_valid = (targets &gt;= 0).sum().clamp(min=1)<br>        pg_obj = pg_obj / num_valid<br>        <br>        # Minimize negative objective<br>        loss = -pg_obj<br>        loss.backward()</pre><p>The key insight: we weight the log-probability of each token by the advantage of the entire completion. Tokens in high-reward completions receive positive gradients (encouraged), while those in low-reward completions receive negative gradients (discouraged).</p><h4>Masking Details</h4><pre>targets[mask_ids[:, 1:] == 0] = -1<br># NOTE: the Engine returns mask=0 for BOTH the prompt tokens AND the tool use tokens.<br># So we will (correctly) end up not training on the prompt tokens, or the tool use forced tokens.</pre><p>The mask ensures gradients flow only through the model’s sampled tokens, not the prompt or calculator outputs (which are deterministically inserted by the tool use system).</p><h4>On-Policy Sampling</h4><p>Critically, the model generates fresh samples at each step using the current policy:</p><pre># Generate samples<br>model.eval()<br>generated_token_sequences, masks = engine.generate_batch(...)<br><br># Train on those samples<br>model.train()<br>loss.backward()<br>optimizer.step()</pre><p>This on-policy approach ensures the samples reflect the current policy’s distribution, avoiding the distribution shift problems of off-policy methods.</p><h4>When to Use RL</h4><p>Reinforcement learning is most valuable when:</p><ul><li><strong>Clear reward signals exist:</strong> The task has measurable success criteria, such as correctness or user satisfaction (like math).</li><li><strong>Exploration is beneficial:</strong> The model can discover better strategies by trying multiple responses or reasoning paths.</li><li><strong>Supervised data is limited:</strong> RL can improve behavior beyond what labeled examples provide.</li><li><strong>Fine-grained control is needed:</strong> You want to guide the model’s tone, safety, or style based on specific feedback signals.</li><li><strong>Long-term outcomes matter:</strong> The model’s output quality depends on multi-step reasoning or delayed rewards.</li></ul><h3>Conclusion</h3><p>The <strong>nanochat</strong> training pipeline shows how a language model evolves through clear, structured phases. The tokenizer defines the input space, pretraining builds language understanding, midtraining adds conversation and tool use, fine-tuning adapts to tasks, and reinforcement learning refines performance.</p><p>This step-by-step approach makes the process transparent and practical for building conversational models on any scale.</p><h3>References / Sources</h3><p><a href="https://github.com/karpathy/nanochat">GitHub - karpathy/nanochat: The best ChatGPT that $100 can buy.</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZar2TJv-sE0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZar2TJv-sE0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZar2TJv-sE0%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c622fd371fae9474c00405248879ea52/href">https://medium.com/media/c622fd371fae9474c00405248879ea52/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F7xTGNNLPyMI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D7xTGNNLPyMI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F7xTGNNLPyMI%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a07e71939d7814a6375872c5408129d9/href">https://medium.com/media/a07e71939d7814a6375872c5408129d9/href</a></iframe><p><a href="https://medium.com/@akdemir_bahadir/inside-nanochat-part-3-understanding-optimizers-7f989d407153">Inside nanochat Part 3: Understanding Optimizers</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FxT4jxQUl0X8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DxT4jxQUl0X8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FxT4jxQUl0X8%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/aad75f0f522ffae66db6671b35df5405/href">https://medium.com/media/aad75f0f522ffae66db6671b35df5405/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FbAWV_yrqx4w%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbAWV_yrqx4w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FbAWV_yrqx4w%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/872b379fd57ba2f7c527911823b78276/href">https://medium.com/media/872b379fd57ba2f7c527911823b78276/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FYi1UCrAsf4o%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DYi1UCrAsf4o&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FYi1UCrAsf4o%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/694fc6b8f0d984f6ea34b86b539ec9a7/href">https://medium.com/media/694fc6b8f0d984f6ea34b86b539ec9a7/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e729522ac2a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside nanochat Part 4: Understanding Inference]]></title>
            <link>https://medium.com/@akdemir_bahadir/inside-nanochat-part-4-understanding-inference-56a2e00ca45a?source=rss-711524cb67b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/56a2e00ca45a</guid>
            <category><![CDATA[top-k]]></category>
            <category><![CDATA[temperature]]></category>
            <category><![CDATA[kv-cache]]></category>
            <category><![CDATA[nanochat]]></category>
            <category><![CDATA[inference]]></category>
            <dc:creator><![CDATA[Bahadır AKDEMİR]]></dc:creator>
            <pubDate>Mon, 20 Oct 2025 12:55:32 GMT</pubDate>
            <atom:updated>2025-10-20T12:56:55.181Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z0pSbb7jgsCxj1HsYkjpeA.png" /></figure><h3>Introduction</h3><p>This is the fourth article in the <em>Inside nanochat</em> series. In the previous parts, we explored <a href="https://medium.com/@akdemir_bahadir/inside-nanochat-part-1-understanding-tokenization-00f48f429693">tokenization</a>, <a href="https://medium.com/@akdemir_bahadir/inside-nanochat-part-2-understanding-architecture-6290bb81374d">architecture</a>, and <a href="https://medium.com/@akdemir_bahadir/inside-nanochat-part-3-understanding-optimizers-7f989d407153">optimiziers</a>. This part focuses on inference, the stage where trained language models generate text one token at a time.</p><p>Training enables the model to predict the next token, but inference is where this ability is applied to turn prompts into coherent responses. Efficient inference introduces major computational challenges. Each generated token requires a complete forward pass through the transformer, and straightforward implementations can be computationally expensive.</p><p>This article explains how <strong>nanochat</strong> achieves efficient inference through three core techniques:</p><ul><li><strong>KV Cache:</strong> Removing redundant computation during autoregressive generation</li><li><strong>Tool Integration:</strong> Allowing models to use external functions such as calculator</li><li><strong>Sampling Strategies:</strong> Managing randomness and maintaining the quality of generated text</li></ul><h3>The Inference Challenge</h3><p>Autoregressive language models generate text sequentially, with each token dependent on all previously generated tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hocjZWABGYpEuN8Z.gif" /><figcaption>LLM inference example: Of course, outputs aren’t full words, they’re generally tokens. This is just a simple visualization.</figcaption></figure><p>A straightforward implementation performs a full forward pass through the transformer at every generation step, recalculating attention scores for all previous tokens. For a sequence of length <em>T</em>, this leads to <em>O(T²)</em> total computation, since the same attention keys and values are repeatedly recomputed.</p><p>The key to improving efficiency is <strong>KV caching</strong>, which stores these previously computed keys and values and reuses them in later steps.</p><h3>KV Cache: Optimizing Inference Efficiency</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wUSm0yFgc42qxx-323RFXQ.png" /><figcaption>A simple representation of what we need only to calcualte for the new token attention</figcaption></figure><h3>Understanding KV Cache</h3><p>In transformer attention mechanisms, each token produces three vectors: queries (Q), keys (K), and values (V). During generation:</p><ul><li><strong>Queries</strong> originate from newly generated tokens</li><li><strong>Keys and Values</strong> are required for all tokens in the sequence (both prompt and generated)</li></ul><p>The critical insight: once K and V are computed for a token, they remain constant. These values can be cached and reused for all subsequent generation steps.</p><h3>Cache Architecture</h3><p>In nanochat, the KV cache maintains a 6-dimensional tensor:</p><pre>cache_shape = (L, 2, B, H, T, D)</pre><p>Where:</p><ul><li>L = number of transformer layers</li><li>2 = separate storage for keys and values</li><li>B = batch size (number of parallel sequences)</li><li>H = number of attention heads</li><li>T = maximum sequence length</li><li>D = head dimension (embedding size / number of heads)</li></ul><blockquote><strong><em>Wait… the first attention layer seems to stay mostly the same except for the new key and value pairs of the latest token. But how can we be sure that all attention layers keep the same values? After all, the outputs pass through MLP layers too???.</em></strong></blockquote><blockquote><em>At first, this can be confusing. It feels like each layer should change its outputs as new tokens are added. But when we look closely at how the MLP layer works, we see that it is </em><strong><em>token independent</em></strong><em>. It applies the same operations to every token in the batch and does not mix information between tokens. It simply processes each token embedding based on its dimensions.</em></blockquote><blockquote><em>Therefore, if the attention outputs for previous tokens remain the same, their MLP outputs also remain the same. This means that, except for the new token, the values from earlier tokens stay identical across all layers during inference.</em></blockquote><p>The complete KVCache implementation from nanochat/engine.py:</p><pre>class KVCache:<br>    &quot;&quot;&quot;<br>    Works hand-in-hand with the GPT model to maintain the KV cache.<br>    Note that the .pos advances automatically after the last layer of the Transformer inserts.<br>    &quot;&quot;&quot;<br>    def __init__(self, batch_size, num_heads, seq_len, head_dim, num_layers):<br>        # Each of K/V is of shape (B, H, T, D) and we have one per layer of the Transformer.<br>        self.kv_shape = (num_layers, 2, batch_size, num_heads, seq_len, head_dim)<br>        self.kv_cache = None<br>        self.pos = 0 # current position in time in the cache<br>    def reset(self):<br>        self.pos = 0<br>    def get_pos(self):<br>        return self.pos</pre><h4>Lazy Initialization Strategy</h4><p>An optimization approach: the cache is not allocated until the first forward pass. This design allows the cache to automatically inherit the appropriate dtype (bfloat16, float32, etc.) and device (CPU/GPU) from the model:</p><pre>def insert_kv(self, layer_idx, k, v):<br>    # Lazy initialize the cache here because we need to know the dtype/device<br>    if self.kv_cache is None:<br>        self.kv_cache = torch.empty(self.kv_shape, dtype=k.dtype, device=k.device)<br>    <br>    # Insert new keys/values to the cache and return the full cache so far<br>    B, H, T_add, D = k.size()<br>    t0, t1 = self.pos, self.pos + T_add</pre><h4>Dynamic Memory Allocation</h4><p>When generation exceeds the initial sequence length estimate, the cache dynamically expands in 1024-token increments:</p><pre># Dynamically grow the cache if needed<br>    if t1 &gt; self.kv_cache.size(4):<br>        t_needed = t1 + 1024 # as much as we need plus buffer of 1024<br>        t_needed = (t_needed + 1023) &amp; ~1023 # round up to nearest multiple of 1024<br>        current_shape = list(self.kv_cache.shape)<br>        current_shape[4] = t_needed<br>        self.kv_cache.resize_(current_shape)<br>    <br>    # Insert k, v into the cache<br>    self.kv_cache[layer_idx, 0, :, :, t0:t1] = k<br>    self.kv_cache[layer_idx, 1, :, :, t0:t1] = v<br>    <br>    # Return the full cached keys/values up to current position (as a view)<br>    key_view = self.kv_cache[layer_idx, 0, :, :, :t1]<br>    value_view = self.kv_cache[layer_idx, 1, :, :, :t1]<br>    <br>    # Increment pos after the last layer of the Transformer processes<br>    if layer_idx == self.kv_cache.size(0) - 1:<br>        self.pos = t1<br>    <br>    return key_view, value_view</pre><p>The bitwise operation (t_needed + 1023) &amp; ~1023 efficiently rounds up to the nearest multiple of 1024, ensuring optimal memory allocation.</p><h4>Attention Mechanism Integration</h4><p>The cache integrates directly into the attention mechanism in nanochat/gpt.py:</p><pre>def forward(self, x, cos_sin, kv_cache):<br>    B, T, C = x.size()<br>    # Project the input to get queries, keys, and values<br>    q = self.c_q(x).view(B, T, self.n_head, self.head_dim)<br>    k = self.c_k(x).view(B, T, self.n_kv_head, self.head_dim)<br>    v = self.c_v(x).view(B, T, self.n_kv_head, self.head_dim)<br>    # Apply Rotary Embeddings to queries and keys<br>    cos, sin = cos_sin<br>    q, k = apply_rotary_emb(q, cos, sin), apply_rotary_emb(k, cos, sin)<br>    q, k = norm(q), norm(k) # QK norm<br>    q, k, v = q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2)<br>    # Apply KV cache: insert current k,v into cache, get the full view so far<br>    if kv_cache is not None:<br>        k, v = kv_cache.insert_kv(self.layer_idx, k, v)<br>    <br>    Tq = q.size(2) # number of queries in this forward pass<br>    Tk = k.size(2) # number of keys/values in total (cached + current)</pre><h4>Batch Prefilling via Cache Replication</h4><p>nanochat implements efficient multi-sample generation from a single prompt:</p><pre>def prefill(self, other):<br>    &quot;&quot;&quot;<br>    Prefill given another KV cache. Optionally expand along batch dim.<br>    This is used when we do batch 1 prefill and then want to generate<br>    multiple samples in parallel from there.<br>    &quot;&quot;&quot;<br>    # 1) validate the shapes<br>    assert self.kv_cache is None, &quot;Cannot prefill a non-empty KV cache&quot;<br>    assert other.kv_cache is not None, &quot;Cannot prefill with a None KV cache&quot;<br>    <br>    # 2) initialize the cache<br>    dtype, device = other.kv_cache.dtype, other.kv_cache.device<br>    self.kv_cache = torch.empty(self.kv_shape, dtype=dtype, device=device)<br>    <br>    # 3) copy the data over<br>    self.kv_cache[:, :, :, :, :other.pos, :] = other.kv_cache<br>    <br>    # 4) update the pos<br>    self.pos = other.pos</pre><p>This capability enables:</p><ol><li>Single prompt processing with batch size 1</li><li>Cache cloning N times</li><li>Parallel generation of N different completions</li></ol><p>This approach is essential for techniques such as best-of-N sampling and reinforcement learning from human feedback (RLHF).</p><h3>Tool Integration: Extending Model Capabilities</h3><p>Language models demonstrate well-documented limitations in arithmetic operations. A model may struggle with “What is 847 × 923?” despite the deterministic nature of the answer. The solution: enable tool usage.</p><h4>The Calculator Implementation</h4><p>nanochat implements a secure calculator for evaluating arithmetic expressions:</p><pre>def use_calculator(expr):<br>    &quot;&quot;&quot;Evaluate a math expression safely.&quot;&quot;&quot;<br>    expr = expr.replace(&quot;,&quot;, &quot;&quot;)<br>    if any([x not in &quot;0123456789*+-/.() &quot; for x in expr]):<br>        return None  # disallow non-numeric chars<br>    if &quot;**&quot; in expr:<br>        return None  # disallow power operator (could be expensive)<br>    return eval_with_timeout(expr)<br><br>def eval_with_timeout(formula, max_time=3):<br>    try:<br>        with timeout(max_time, formula):<br>            with warnings.catch_warnings():<br>                warnings.simplefilter(&quot;ignore&quot;, SyntaxWarning)<br>                return eval(formula)<br>    except Exception as e:<br>        signal.alarm(0)<br>        return None</pre><p>The 3-second timeout protection prevents malicious or computationally expensive operations from blocking the system.</p><h4>Tool Use Protocol</h4><p>The model learns calculator usage through designated special tokens:</p><pre>Model generates: Let&#39;s calculate: &lt;|python_start|&gt;847*923&lt;|python_end|&gt;<br>Engine detects python block completion<br>Engine evaluates: 847*923 = 781681<br>Engine injects: &lt;|output_start|&gt;781681&lt;|output_end|&gt;<br>Model continues: So the answer is 781681.</pre><p>This protocol establishes:</p><ol><li>When to invoke tools (expressions delimited by &lt;|python_start|&gt; and &lt;|python_end|&gt;)</li><li>How to interpret tool outputs (delimited by &lt;|output_start|&gt; and &lt;|output_end|&gt;)</li></ol><h4>State Machine Architecture</h4><p>Tool integration employs a per-sample state machine to track tool usage:</p><pre>class RowState:<br>    # Per-row state tracking during generation<br>    def __init__(self, current_tokens=None):<br>        self.current_tokens = current_tokens or [] # Current token sequence<br>        self.forced_tokens = deque() # Queue of tokens to force inject<br>        self.in_python_block = False # Whether we are inside a python block<br>        self.python_expr_tokens = [] # Tokens of the current python expression<br>        self.completed = False # Whether this row has completed generation</pre><p>State transitions occur during token generation:</p><pre># Handle tool logic<br>if next_token == python_start:<br>    state.in_python_block = True<br>    state.python_expr_tokens = []<br>elif next_token == python_end and state.in_python_block:<br>    state.in_python_block = False<br>    if state.python_expr_tokens:<br>        # Decode the accumulated tokens to get the expression<br>        expr = self.tokenizer.decode(state.python_expr_tokens)<br>        result = use_calculator(expr)<br>        <br>        if result is not None:<br>            # Tokenize the result and prepare to inject it<br>            result_tokens = self.tokenizer.encode(str(result))<br>            state.forced_tokens.append(output_start)<br>            state.forced_tokens.extend(result_tokens)<br>            state.forced_tokens.append(output_end)<br>    state.python_expr_tokens = []<br>elif state.in_python_block:<br>    # Accumulate tokens inside the python block<br>    state.python_expr_tokens.append(next_token)</pre><h4>Token Forcing Mechanism</h4><p>When tool results become available, they are injected into the generation stream, overriding the model’s sampling:</p><pre># Select the next token in this row<br>is_forced = len(state.forced_tokens) &gt; 0<br>token_masks.append(0 if is_forced else 1) # 0 = forced, 1 = sampled<br>next_token = state.forced_tokens.popleft() if is_forced else sampled_tokens[i]<br>token_column.append(next_token)</pre><p>The token_masks array tracks whether tokens were sampled or forced, which is critical for training (forced tokens should not contribute to the loss function).</p><h4>Multi-Tool Support</h4><p>The state machine architecture naturally accommodates multiple tool invocations within a single response:</p><pre>Calculate the area of a rectangle: &lt;|python_start|&gt;5*3&lt;|python_end|&gt;<br>&lt;|output_start|&gt;15&lt;|output_end|&gt;<br>Now the perimeter: &lt;|python_start|&gt;2*(5+3)&lt;|python_end|&gt;<br>&lt;|output_start|&gt;16&lt;|output_end|&gt;<br>So area is 15 and perimeter is 16.</pre><p>Each &lt;|python_end|&gt; token triggers tool evaluation, with results queued in forced_tokens for subsequent injection.</p><h3>Sampling Strategy: Controlling Generation Quality</h3><p>After obtaining logits from the model, selecting the next token requires careful consideration. The sampling strategy significantly impacts generation quality, creativity, and determinism.</p><h4>The Sampling Function</h4><p>nanochat implements a classical sampling function with temperature and top-k filtering:</p><pre>@torch.inference_mode()<br>def sample_next_token(logits, rng, temperature=1.0, top_k=None):<br>    &quot;&quot;&quot;Sample a single next token from given logits of shape (B, vocab_size). <br>    Returns (B, 1).&quot;&quot;&quot;<br>    assert temperature &gt;= 0.0, &quot;temperature must be non-negative&quot;<br>    <br>    if temperature == 0.0:<br>        return torch.argmax(logits, dim=-1, keepdim=True)<br>    <br>    if top_k is not None:<br>        k = min(top_k, logits.size(-1))<br>        vals, idx = torch.topk(logits, k, dim=-1)<br>        vals = vals / temperature<br>        probs = F.softmax(vals, dim=-1)<br>        choice = torch.multinomial(probs, num_samples=1, generator=rng)<br>        return idx.gather(1, choice)<br>    else:<br>        logits = logits / temperature<br>        probs = F.softmax(logits, dim=-1)<br>        return torch.multinomial(probs, num_samples=1, generator=rng)</pre><h4>Greedy Decoding (temperature = 0.0)</h4><pre>if temperature == 0.0:<br>    return torch.argmax(logits, dim=-1, keepdim=True)</pre><p><strong>Greedy decoding</strong> consistently selects the highest-probability token. Characteristics include:</p><ul><li>Deterministic behavior (identical input yields identical output)</li><li>Computational efficiency (no probability computation required)</li><li>Potential for repetitive outputs</li><li>Suitable for precision-critical tasks (code generation, mathematics)</li></ul><p><strong>Example:</strong> Given logits for “The cat sat on the ___”:</p><pre>[(&quot;mat&quot;, 0.45), (&quot;floor&quot;, 0.30), (&quot;table&quot;, 0.15), (&quot;car&quot;, 0.10)]<br>→ Consistently selects &quot;mat&quot;</pre><h4>Temperature Sampling</h4><p><strong>Temperature (τ)</strong> controls the level of randomness in token selection by scaling the logits before applying the softmax function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*sNpkYShAG9SPWZQrJ4Aecg.png" /></figure><pre>logits = logits / temperature<br>probs = F.softmax(logits, dim=-1)<br>return torch.multinomial(probs, num_samples=1, generator=rng)</pre><p><strong>Temperature (τ)</strong> directly influences how confidently the model selects the next token.</p><ul><li><strong>τ &lt; 1 (e.g., 0.7):</strong> Sharpens the probability distribution and increases determinism.<br> <strong>Before:</strong> [(“mat”, 0.45), (“floor”, 0.30), (“table”, 0.15), (“car”, 0.10)]<br> <strong>After:</strong> [(“mat”, 0.60), (“floor”, 0.25), (“table”, 0.10), (“car”, 0.05)]</li><li><strong>τ = 1:</strong> Leaves the distribution unchanged. The model samples directly from its original probabilities.</li><li><strong>τ &gt; 1 (e.g., 1.5):</strong> Flattens the probability distribution and increases randomness.<br> <strong>Before:</strong> [(“mat”, 0.45), (“floor”, 0.30), (“table”, 0.15), (“car”, 0.10)]<br> <strong>After:</strong> [(“mat”, 0.35), (“floor”, 0.30), (“table”, 0.20), (“car”, 0.15)]</li></ul><p><strong>Application contexts:</strong></p><ul><li>Low temperature (0.5–0.7): Factual writing, code generation, translation</li><li>Medium temperature (0.8–1.0): Conversational agents, question answering, general text</li><li>High temperature (1.2–1.5): Creative writing, brainstorming</li></ul><h4>Top-K Filtering</h4><p>Top-k filtering restricts sampling to the k most probable tokens:</p><pre>if top_k is not None:<br>    k = min(top_k, logits.size(-1))<br>    vals, idx = torch.topk(logits, k, dim=-1)<br>    vals = vals / temperature<br>    probs = F.softmax(vals, dim=-1)<br>    choice = torch.multinomial(probs, num_samples=1, generator=rng)<br>    return idx.gather(1, choice)</pre><p><strong>Process:</strong></p><ol><li>Sort tokens by logit value, retain top k</li><li>Apply temperature scaling to these k tokens exclusively</li><li>Renormalize probabilities (sum to 1.0)</li><li>Sample from this restricted distribution</li></ol><p><strong>Example with k=2:</strong></p><pre>Original: [(&quot;mat&quot;, 0.45), (&quot;floor&quot;, 0.30), (&quot;table&quot;, 0.15), (&quot;car&quot;, 0.10)]<br>Top-2:    [(&quot;mat&quot;, 0.60), (&quot;floor&quot;, 0.40)]  # renormalized<br>→ &quot;table&quot; and &quot;car&quot; have zero probability of selection</pre><p><strong>Advantages:</strong></p><ul><li>Prevents sampling of highly improbable tokens (reduces incoherent outputs)</li><li>Maintains diversity among probable tokens</li><li>Typical values: k=20 for focused generation, k=50 for creative generation</li></ul><h4>Combined Temperature and Top-K</h4><p>In practice, both techniques are often combined:</p><pre># Example parameters<br>temperature = 0.8  # Moderately focused<br>top_k = 40        # Consider 40 most likely tokens<br># Process:<br># 1. Filter to top 40 tokens<br># 2. Apply temperature scaling (slightly sharpen distribution)<br># 3. Sample from resulting distribution</pre><p>This approach provides:</p><ul><li><strong>Safety:</strong> from top-k (eliminates improbable tokens)</li><li><strong>Control:</strong> from temperature (adjusts randomness level)</li><li><strong>Quality.</strong> from both (balances coherence and creativity)</li></ul><h3>Multi-Sample Generation: Parallel Completions</h3><p>nanochat implements efficient parallel generation of multiple completions from a single prompt.</p><h4>Two-Phase Generation Approach</h4><pre>@torch.inference_mode()<br>def generate(self, tokens, num_samples=1, max_tokens=None, <br>             temperature=1.0, top_k=None, seed=42):<br>    &quot;&quot;&quot;Generate num_samples completions with single prefill and parallel decode.&quot;&quot;&quot;<br>    <br>    # 1) Run a batch 1 prefill of the prompt tokens<br>    m = self.model.config<br>    kv_model_kwargs = {<br>        &quot;num_heads&quot;: m.n_kv_head, <br>        &quot;head_dim&quot;: m.n_embd // m.n_head, <br>        &quot;num_layers&quot;: m.n_layer<br>    }<br>    kv_cache_prefill = KVCache(<br>        batch_size=1,<br>        seq_len=len(tokens),<br>        **kv_model_kwargs,<br>    )<br>    ids = torch.tensor([tokens], dtype=torch.long, device=device)<br>    logits = self.model.forward(ids, kv_cache=kv_cache_prefill)<br>    logits = logits[:, -1, :]<br>    next_ids = sample_next_token(logits, rng, temperature, top_k)<br>    sampled_tokens = next_ids[:, 0].tolist()</pre><p><strong>Phase 1: Single Prefill</strong></p><ul><li>Process entire prompt once (batch size = 1)</li><li>Computationally expensive (O(T²) for sequence length T)</li><li>Executed once regardless of num_samples</li></ul><pre># 2) Replicate the KV cache for each sample/row<br>    kv_length_hint = (len(tokens) + max_tokens) if max_tokens \<br>                     else self.model.config.sequence_len<br>    kv_cache_decode = KVCache(<br>        batch_size=num_samples,<br>        seq_len=kv_length_hint,<br>        **kv_model_kwargs,<br>    )<br>    kv_cache_decode.prefill(kv_cache_prefill)<br>    del kv_cache_prefill # free memory<br>    <br>    # 3) Initialize states for each sample<br>    row_states = [RowState(tokens.copy()) for _ in range(num_samples)]</pre><p><strong>Phase 2: Parallel Decode</strong></p><ul><li>Clone prefilled cache N times (efficient memory copy operation)</li><li>Batch size now equals num_samples</li><li>Each decoding step processes all samples simultaneously</li></ul><h4>Per-Sample State Management</h4><p>Each sample maintains independent state:</p><pre># 4) Main generation loop<br>num_generated = 0<br>first_iteration = True<br>while True:<br>    # Stop condition: we&#39;ve reached max tokens<br>    if max_tokens is not None and num_generated &gt;= max_tokens:<br>        break<br>    # Stop condition: all rows are completed<br>    if all(state.completed for state in row_states):<br>        break<br>    <br>    # Get sampled tokens - either from prefill or from forward pass<br>    if first_iteration:<br>        sampled_tokens = [sampled_tokens[0]] * num_samples<br>        first_iteration = False<br>    else:<br>        logits = self.model.forward(ids, kv_cache=kv_cache_decode)<br>        logits = logits[:, -1, :]<br>        next_ids = sample_next_token(logits, rng, temperature, top_k)<br>        sampled_tokens = next_ids[:, 0].tolist()</pre><p>Each forward pass returns a “token column” containing one token per sample:</p><pre>Sample 0: [prompt...] → &quot;Hello&quot;<br>Sample 1: [prompt...] → &quot;Hi&quot;<br>Sample 2: [prompt...] → &quot;Hey&quot;<br>Sample 3: [prompt...] → &quot;Greetings&quot;</pre><h3>Early Stopping Implementation</h3><p>Individual samples can complete at different times:</p><pre># Process each row: choose the next token, update state<br>    token_column = []<br>    token_masks = []<br>    for i, state in enumerate(row_states):<br>        # ... token selection and tool logic ...<br>        <br>        # On &lt;|assistant_end|&gt; or &lt;|bos|&gt;, mark the row as completed<br>        if next_token == assistant_end or next_token == bos:<br>            state.completed = True</pre><p>When a sample generates an end token:</p><ol><li>Mark as completed</li><li>Continue generating for remaining samples</li><li>Terminate loop when all samples complete</li></ol><p>This approach is more efficient than waiting for the slowest sample to reach max_tokens.</p><h3>Efficiency Analysis</h3><p>Consider generating 8 completions for a 100-token prompt:</p><p><strong>Naive approach (separate generation):</strong></p><ul><li>8 × (prefill + decode) operations</li><li>8 × O(T²) prefill cost = O(8T²)</li></ul><p><strong>Batched approach (nanochat):</strong></p><ul><li>1 × prefill operation = O(T²)</li><li>Parallel decode = O(T) per step, with 8× memory bandwidth</li></ul><p>For typical workloads, this achieves approximately <strong>6–7× speedup</strong> compared to naive separate generation.</p><h3>Complete Generation Flow</h3><p>A comprehensive generation example:</p><pre>from nanochat.engine import Engine<br>from nanochat.checkpoint_manager import load_model<br><br># Load model and tokenizer<br>model, tokenizer, meta = load_model(&quot;base&quot;, device=&quot;cuda&quot;)<br># Create engine<br>engine = Engine(model, tokenizer)<br># Prepare prompt<br>prompt = &quot;What is 15 * 23?&quot;<br>bos = tokenizer.get_bos_token_id()<br>tokens = tokenizer.encode(prompt, prepend=bos)<br># Generate 4 completions<br>for token_column, token_masks in engine.generate(<br>    tokens, <br>    num_samples=4,<br>    max_tokens=100,<br>    temperature=0.7,<br>    top_k=50,<br>    seed=42<br>):<br>    # token_column contains 4 tokens (one per sample)<br>    for i, (token, mask) in enumerate(zip(token_column, token_masks)):<br>        text = tokenizer.decode([token])<br>        forced = &quot;[FORCED]&quot; if mask == 0 else &quot;&quot;<br>        print(f&quot;Sample {i}: {text}{forced}&quot;, end=&quot;&quot;)<br>    print()</pre><p><strong>Internal process:</strong></p><p><strong>1 . Initialization</strong>:</p><ul><li>Create RNG with seed for reproducibility</li><li>Retrieve special token IDs for tool use</li></ul><p><strong>2 . Single Prefill</strong>:</p><ul><li>Process prompt through model with batch size 1</li><li>Construct initial KV cache with shape (L, 2, 1, H, T, D)</li><li>Sample first token</li></ul><p><strong>3 . Cache Replication</strong>:</p><ul><li>Create larger cache with batch size 4</li><li>Clone prefilled cache 4 times</li><li>Initialize 4 independent RowState objects</li></ul><p><strong>4 . Decoding Loop</strong>:</p><ul><li><strong>Forward pass</strong>: Generate logits for 4 samples simultaneously</li><li><strong>Sampling</strong>: Sample 4 tokens (one per sample)</li><li><strong>State machine</strong>: For each sample:</li><li>Check for forced tokens (tool results)</li><li>Update tool use state</li><li>Check for completion</li><li><strong>Tool execution</strong>: If &lt;|python_end|&gt; detected:</li><li>Decode accumulated tokens</li><li>Evaluate expression</li><li>Queue result tokens for forcing</li><li><strong>Yield</strong>: Return token column and masks</li></ul><p><strong>5 . Early Stopping</strong>:</p><ul><li>Continue until all samples reach end token or max_tokens</li></ul><p><strong>Example output:</strong></p><pre>Sample 0: Let&#39;s calculate: &lt;|python_start|&gt;15*23&lt;|python_end|&gt;<br>Sample 1: To find 15 times 23, &lt;|python_start|&gt;15*23&lt;|python_end|&gt;<br>Sample 2: 15 * 23 = &lt;|python_start|&gt;15*23&lt;|python_end|&gt;<br>Sample 3: The answer is &lt;|python_start|&gt;15*23&lt;|python_end|&gt;<br>[Tool evaluation occurs]<br>Sample 0: &lt;|output_start|&gt;[FORCED]345[FORCED]&lt;|output_end|&gt;[FORCED], so 345.<br>Sample 1: &lt;|output_start|&gt;[FORCED]345[FORCED]&lt;|output_end|&gt;[FORCED] which equals 345.<br>Sample 2: &lt;|output_start|&gt;[FORCED]345[FORCED]&lt;|output_end|&gt;[FORCED]<br>Sample 3: &lt;|output_start|&gt;[FORCED]345[FORCED]&lt;|output_end|&gt;[FORCED].<br></pre><p>The [FORCED] indicators denote tokens originating from the calculator rather than model sampling.</p><h3>Performance Considerations</h3><h4>Memory Usage</h4><p>The KV cache represents the primary memory consumer during inference:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*4DMabRwkDwMzcsFmhMaBuQ.png" /></figure><p>For nanochat’s base configuration:</p><ul><li>L = 6 layers</li><li>B = 4 samples</li><li>H = 6 heads</li><li>T = 2048 max sequence</li><li>D = 64 head dimension</li><li>dtype = bfloat16 (2 bytes)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*Et6TS0f5Zde4JclH2IaopA.png" /></figure><p>Memory usage scales linearly with num_samples, establishing practical limits for batched generation.</p><h4>Computational Efficiency</h4><p>With KV cache, each decoding step requires:</p><ul><li><strong>Without cache</strong>: O(T² · d) attention over all token pairs</li><li><strong>With cache</strong>: O(T · d) only new queries attend to cached keys</li></ul><p>For a 1000-token sequence, this represents a <strong>1000× speedup</strong> in attention computation.</p><h4>Batching Benefits</h4><p>Processing N samples in parallel:</p><ul><li><strong>Memory</strong>: N× increase (linear scaling)</li><li><strong>Compute</strong>: approximately 1.2N× increase (some memory bandwidth overhead)</li><li><strong>Wall time</strong>: approximately 1.5N× compared to serial processing (significant improvement)</li></ul><p>The optimal configuration typically ranges from num_samples=4-8</p><h3>Advanced Considerations</h3><h4><a href="https://medium.com/@akdemir_bahadir/speculative-decoding-how-to-make-large-language-models-think-faster-without-sacrificing-quality-bcdc24711c38">Speculative Decoding</a></h4><p>An extension compatible with this architecture: employ a smaller “draft” model to generate multiple candidate tokens, then verify them with the full model in parallel. This approach can further reduce latency.</p><h4><a href="https://medium.com/@akdemir_bahadir/continuous-batching-in-llm-inference-d24182b21bdf">Continuous Batching</a></h4><p>In production systems, samples can be dynamically added to or removed from the batch as they complete, maximizing GPU utilization. nanochat’s per-sample state tracking facilitates this implementation.</p><h4>Multi-Step Tool Use</h4><p>The tool integration extends naturally to more complex workflows:</p><pre>User: Plot y = x^2 from -5 to 5<br>Model: &lt;|python_start|&gt;import matplotlib...&lt;|python_end|&gt;<br>        &lt;|output_start|&gt;[image data]&lt;|output_end|&gt;</pre><p>By incorporating additional tools and expanding the state machine, support for code execution, web search, and other capabilities becomes feasible.</p><h3>Conclusion</h3><p>Efficient inference results from careful engineering across multiple layers:</p><ol><li><strong>KV Cache</strong> eliminates redundant computation through intelligent caching and dynamic memory management</li><li><strong>Tool Integration</strong> extends model capabilities through a state machine and token forcing mechanism</li><li><strong>Sampling Strategies</strong> balance creativity and coherence through temperature and top-k filtering</li><li><strong>Multi-Sample Generation</strong> maximizes hardware utilization through prefill-then-parallelize architecture</li></ol><p>These techniques transform a theoretically correct but computationally expensive generation process into a practical, real-time system capable of powering interactive applications.</p><h3>References / Sources</h3><ul><li><a href="https://github.com/karpathy/nanochat">GitHub - karpathy/nanochat: The best ChatGPT that $100 can buy.</a></li><li><a href="https://medium.com/@akdemir_bahadir/speculative-decoding-how-to-make-large-language-models-think-faster-without-sacrificing-quality-bcdc24711c38">Speculative Decoding: How to Make Large Language Models Think Faster Without Sacrificing Quality</a></li><li><a href="https://medium.com/@akdemir_bahadir/continuous-batching-in-llm-inference-d24182b21bdf">Continuous Batching in LLM Inference</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56a2e00ca45a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside nanochat Part 3: Understanding Optimizers]]></title>
            <link>https://medium.com/@akdemir_bahadir/inside-nanochat-part-3-understanding-optimizers-7f989d407153?source=rss-711524cb67b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f989d407153</guid>
            <category><![CDATA[muon-optimizer]]></category>
            <category><![CDATA[optimizer]]></category>
            <category><![CDATA[nanochat]]></category>
            <dc:creator><![CDATA[Bahadır AKDEMİR]]></dc:creator>
            <pubDate>Sun, 19 Oct 2025 14:47:51 GMT</pubDate>
            <atom:updated>2025-10-19T14:47:51.319Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OmPNNvtBxLWa-fJ1DL6FLw.png" /></figure><h3>Introduction</h3><p>Training large language models efficiently depends on the careful selection and tuning of optimization algorithms. This article examines how NanoChat combines two complementary optimizers, Muon and AdamW, together with thoughtfully designed learning rate and momentum schedules to achieve fast and stable convergence during training.</p><blockquote><em>Note: This section includes more mathematical concepts than the previous parts.</em></blockquote><h3>The Core Strategy</h3><p>The central idea behind nanochat’s optimization approach is that different types of parameters respond better to different optimization methods. Embeddings and the language model head are trained effectively with <em>adaptive learning rates provided by AdamW</em>. In contrast, the two-dimensional weight matrices in attention and MLP layers <em>benefit from the orthogonalization properties of Muon</em>, which help maintain stability and efficient gradient flow during training.</p><h3>Understanding Muon</h3><p>Muon, short for <em>MomentUm Orthogonalized by Newton–Schulz</em>, is an optimizer specifically designed for two-dimensional weight matrices. It combines the stability of <em>stochastic gradient descent</em> (sgd) with <em>momentum</em> and the geometric advantages of maintaining approximate orthogonality in weight updates.</p><h4>How Muon Works</h4><p>Muon operates in three distinct steps.</p><h4><strong>Step 1: Standard SGD with Momentum</strong></h4><p>The first equation accumulates gradients over time. The parameter <strong>β (beta)</strong> determines how much information from previous gradients is retained. For example, when β is 0.95, the optimizer keeps 95 percent of the previous momentum and adds 5 percent of the new gradient. This produces a smoother estimate of the gradient direction and reduces noise from individual batches.</p><blockquote><strong><em>Note:</em></strong><em> This concept can be intuitively understood through an analogy with momentum in physics. Imagine a car moving quickly toward a curve. Even if the driver turns the steering wheel sharply, the car cannot change direction immediately because its forward momentum resists abrupt turns. In a similar way, when the optimizer has accumulated strong momentum in a particular direction, it does not instantly follow a new gradient direction from the current batch. Instead, it adjusts gradually, verifying whether the new direction truly represents a better path before committing to it.</em></blockquote><p>The optional <strong>Nesterov</strong> term introduces a small lookahead by combining the current gradient with the momentum buffer. This anticipatory adjustment allows the optimizer to estimate the future position of the parameters, leading to more informed and stable updates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OR5VMb4QH-nl_Hi3uevn7A.png" /><figcaption>momentum formula (generic naming)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sqgYzkpHwHGDMvYTphJmGg.png" /><figcaption>momentum formula (naming with nanochat code variables)</figcaption></figure><blockquote><strong><em>Note:</em></strong><em> In this part, I encountered an interesting PyTorch function called </em><em>lerp. It stands for linear interpolation and is used to mix two tensors based on a given weight. In PyTorch syntax, it can be expressed as</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZUTK9yEVVA2aX4oGwfTHZg.png" /></figure><blockquote><em>This operation is performed </em><strong><em>in-place</em></strong><em>, which means the result overwrites the original tensor </em><strong><em>a</em></strong><em>. In that case, PyTorch uses the form:</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*CYFyOLRyt2eoH-h4VrlUHg.png" /></figure><blockquote><em>When the weight </em><strong><em>w</em></strong><em> is zero, the result is the same as </em><strong><em>a</em></strong><em>. When </em><strong><em>w</em></strong><em> is one, it becomes </em><strong><em>b</em></strong><em>. For values of </em><strong><em>w</em></strong><em> between zero and one, the output lies proportionally between </em><strong><em>a</em></strong><em> and </em><strong><em>b</em></strong><em>. In optimization, this function is often used to blend the previous momentum with the current gradient, leading to smoother and more stable updates during training.</em></blockquote><h4><strong>Step 2: Orthogonalization via Newton-Schulz Iteration</strong></h4><p>The main idea behind <strong>Muon</strong> is to make the update direction <strong>orthogonal</strong>, which helps keep training stable and efficient.</p><blockquote><strong><em>Note: Orthogonal</em></strong><em> means that vectors (or the rows of a matrix) are at right angles to each other. In this state, they do not share redundant information. For optimization, this is valuable because it keeps weight updates independent, preventing the model from repeatedly moving in the same direction in parameter space.</em></blockquote><p>This orthogonalization is achieved through a short iterative process called the <strong>Newton–Schulz iteration</strong>, usually repeated five times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*cTCrgmhX9uf1cv5p3DyIPQ.png" /></figure><blockquote><strong><em>Note</em></strong><em>: “||u_t||_​F”​ is the </em><strong><em>Frobenius norm</em></strong><em>, which measures the overall size or energy of the matrix “u_t”. Dividing by this value (plus a small ε) scales the matrix to a stable range and prevents division by zero.</em></blockquote><blockquote><em>Each iteration then moves the matrix closer to being orthogonal:</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*tkw6D6FvfHLJV66dABHxRA.png" /></figure><blockquote><em>where</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*lHZ_3t8PR3CJpk1EHasURg.png" /></figure><blockquote><strong><em>Note:</em></strong><em> “A_i”​ is the </em><strong><em>Gram matrix</em></strong><em>, which shows how similar or aligned the rows of “X_i” are. If “A_i”​ becomes close to the identity matrix, it means the rows of “X_i”​ are nearly orthogonal. In other words, they point in different, independent directions.</em></blockquote><blockquote><strong><em>Note:</em></strong><em> The constants (a,b,c)=(3.4445, −4.7750, 2.0315) are chosen to make the method converge quickly while staying numerically stable. Because of this, Muon can reach a good level of orthogonality in about five iterations, rather than the dozens usually required.</em></blockquote><blockquote><strong><em>Note</em></strong><em>: This process ensures that the update direction remains balanced. It prevents weights from growing too large or collapsing, leading to smoother and more reliable learning during training.</em></blockquote><blockquote><strong><em>Note:</em></strong><em> I might not have been able to explain Step 2 very clearly here. You can get a better understanding of the Newton–Schulz iteration and orthogonalization by watching a few video explanations </em><a href="https://youtu.be/bO5nvE289ec?si=fvgjCzaC6fOye2qg"><em>[here1]</em></a><em> </em><a href="https://youtu.be/yNwdwZPBYtM?si=3BQspOfzXzoo9uij"><em>[here2]</em></a><em> </em><a href="https://youtu.be/m1PUl-aUk8c?si=V1xnEBFQAm8Z3wY1"><em>[here3]</em></a><em> . Seeing the process visually often makes it easier to understand how the matrix gradually becomes orthogonal through repeated updates.</em></blockquote><p>Here is the related part in the nanochat code:</p><pre>@torch.compile<br>def zeropower_via_newtonschulz5(G: Tensor, steps: int) -&gt; Tensor:<br>    &quot;&quot;&quot;<br>    Newton-Schulz iteration to compute the zeroth power / orthogonalization of G. We opt to use a<br>    quintic iteration whose coefficients are selected to maximize the slope at zero. For the purpose<br>    of minimizing steps, it turns out to be empirically effective to keep increasing the slope at<br>    zero even beyond the point where the iteration no longer converges all the way to one everywhere<br>    on the interval. This iteration therefore does not produce UV^T but rather something like US&#39;V^T<br>    where S&#39; is diagonal with S_{ii}&#39; ~ Uniform(0.5, 1.5), which turns out not to hurt model<br>    performance at all relative to UV^T, where USV^T = G is the SVD.<br>    &quot;&quot;&quot;<br>    assert G.ndim &gt;= 2 # batched Muon implementation by @scottjmaddox, and put into practice in the record by @YouJiacheng<br>    a, b, c = (3.4445, -4.7750,  2.0315)<br>    X = G.bfloat16()<br>    if G.size(-2) &gt; G.size(-1):<br>        X = X.mT<br><br>    # Ensure spectral norm is at most 1<br>    X = X / (X.norm(dim=(-2, -1), keepdim=True) + 1e-7)<br>    # Perform the NS iterations<br>    for _ in range(steps):<br>        A = X @ X.mT<br>        B = b * A + c * A @ A # quintic computation strategy adapted from suggestion by @jxbz, @leloykun, and @YouJiacheng<br>        X = a * X + B @ X<br><br>    if G.size(-2) &gt; G.size(-1):<br>        X = X.mT<br>    return X</pre><p><strong>Step 3: Aspect-Ratio Scaled Update</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Lodis5GlXdG7WcWHA5cIw.png" /></figure><p>This final step applies the orthogonalized update to the weights. The scaling factor √(d_out/d_in) accounts for the aspect ratio of the weight matrix. When a matrix is tall (d_out &gt; d_in), this scaling increases the effective learning rate to compensate for the geometric properties of non-square matrices. The max(1, …) ensures we never scale down, only up when needed.</p><p>Here is the complete part of muon optimizer in the code:</p><pre>class Muon(torch.optim.Optimizer):<br> # ... some details here<br>    @torch.no_grad()<br>    def step(self):<br>        for group in self.param_groups:<br>            params: list[Tensor] = group[&quot;params&quot;]<br>            for p in params:<br>                g = p.grad<br>                assert g is not None<br>                state = self.state[p]<br>                if &quot;momentum_buffer&quot; not in state:<br>                    state[&quot;momentum_buffer&quot;] = torch.zeros_like(g)<br>                buf: Tensor = state[&quot;momentum_buffer&quot;]<br>                buf.lerp_(g, 1 - group[&quot;momentum&quot;])<br>                g = g.lerp_(buf, group[&quot;momentum&quot;]) if group[&quot;nesterov&quot;] else buf<br>                g = zeropower_via_newtonschulz5(g, steps=group[&quot;ns_steps&quot;])<br>                p.add_(g, alpha=-group[&quot;lr&quot;] * max(1, p.size(-2) / p.size(-1))**0.5)</pre><h4>Also: Distributed Muon</h4><p>For multi-GPU training, nanochat implements DistMuon, which efficiently synchronizes gradients and weights across ranks:</p><p>The distributed version uses a block-cyclic parameter assignment strategy. Each GPU owns a subset of parameters and is responsible for computing their updates. This reduces memory overhead while maintaining training efficiency.</p><p>The process works in three stages. First, reduce-scatter ensures each GPU receives the averaged gradient for its owned parameters. Second, only the owner computes the Muon update for those parameters. Third, all-gather replicates the updated weights back to all GPUs.</p><h3>Understanding AdamW</h3><p>While Muon excels at optimizing 2D matrices, embeddings and the language model head benefit from AdamW’s adaptive learning rates. AdamW (Adam with decoupled weight decay) maintains separate learning rates for each parameter based on gradient statistics.</p><blockquote><strong><em>Note: </em></strong><em>The </em><strong><em>Adam optimizer</em></strong><em> is one of the most widely used optimization algorithms today. There are many excellent educational videos and tutorials that explain its mechanics in detail. Because it is already well-documented and familiar to most readers, I will cover it here only briefly.</em></blockquote><h4>Configuration</h4><p>From nanochat/gpt.py, here&#39;s how AdamW is configured:</p><pre>def setup_optimizers(self, unembedding_lr=0.004, embedding_lr=0.2, <br>                     matrix_lr=0.02, weight_decay=0.0):<br>    model_dim = self.config.n_embd<br>    <br>    # Separate parameters into groups<br>    matrix_params = list(self.transformer.h.parameters())<br>    embedding_params = list(self.transformer.wte.parameters())<br>    lm_head_params = list(self.lm_head.parameters())<br>    <br>    # Scale LR by ∝1/√d_model (tuned for 768 dim model)<br>    dmodel_lr_scale = (model_dim / 768) ** -0.5<br>    <br>    adam_groups = [<br>        dict(params=lm_head_params, lr=unembedding_lr * dmodel_lr_scale),<br>        dict(params=embedding_params, lr=embedding_lr * dmodel_lr_scale),<br>    ]<br>    <br>    # AdamW configuration<br>    adamw_kwargs = dict(<br>        betas=(0.8, 0.95),      # Lower β₁ than standard Adam<br>        eps=1e-10,              # Numerical stability<br>        weight_decay=weight_decay<br>    )<br>    <br>    AdamWFactory = DistAdamW if ddp else partial(torch.optim.AdamW, fused=True)<br>    adamw_optimizer = AdamWFactory(adam_groups, **adamw_kwargs)</pre><h4>Also: Distributed AdamW</h4><p>Like Muon, nanochat implements a distributed version following ZeRO-2 principles:</p><p>This shards optimizer states across GPUs, reducing memory consumption by a factor equal to the number of GPUs.</p><h3>Learning Rate Schedule</h3><p><strong>nanochat</strong> uses a <strong>three-phase learning rate schedule</strong> consisting of <em>warmup</em>, <em>constant</em>, and <em>warmdown</em> stages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jtzRdlbfo1kt7y1Af0bOmg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZO5Dz91S4pg7zUW5uJtdpg.png" /></figure><p>The <strong>warmup phase</strong> gradually increases the learning rate from zero to its full value. This helps prevent instability at the start of training, especially when parameters are randomly initialized. However, since <strong>Muon</strong> already provides strong training stability, <strong>nanochat</strong> disables warmup by default (warmup_ratio = 0.0).</p><p>The <strong>constant phase</strong> keeps the learning rate fixed at its maximum value. This allows the model to explore the loss landscape freely and make large, confident updates. Typically, this phase covers about <strong>80%</strong> of total training time.</p><p>The <strong>warmdown phase</strong> linearly reduces the learning rate as training nears completion. Smaller step sizes in this stage help the optimizer settle into sharper, more stable minima, which often improves the final model performance. By default, this phase covers the last <strong>20%</strong> of training.</p><h4>Implementation</h4><p>From scripts/base_train.py:</p><pre># Default schedule configuration<br>warmup_ratio = 0.0      # ratio of iterations for LR warmup<br>warmdown_ratio = 0.2    # ratio of iterations for LR warmdown<br>final_lr_frac = 0.0     # final LR is this fraction of the initial LR<br><br>def get_lr_multiplier(it):<br>    warmup_iters = round(warmup_ratio * num_iterations)<br>    warmdown_iters = round(warmdown_ratio * num_iterations)<br>    <br>    if it &lt; warmup_iters:<br>        # Linear warmup<br>        return (it + 1) / warmup_iters<br>    elif it &lt;= num_iterations - warmdown_iters:<br>        # Constant phase<br>        return 1.0<br>    else:<br>        # Linear warmdown<br>        progress = (num_iterations - it) / warmdown_iters<br>        return progress * 1.0 + (1 - progress) * final_lr_frac</pre><p>The learning rate is applied during the training loop:</p><pre># Apply learning rate schedule<br>lrm = get_lr_multiplier(step)<br>for opt in optimizers:<br>    for group in opt.param_groups:<br>        group[&quot;lr&quot;] = group[&quot;initial_lr&quot;] * lrm</pre><h3>Momentum Schedule for Muon</h3><p>While the learning rate remains constant for most of training, <strong>Muon</strong> increases its momentum coefficient during the initial steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*USHR-QQ-FwFSBCD3zhDuEg.png" /></figure><p>This gradually increases momentum from 0.85 to 0.95 over the first 300 iterations.</p><p>Lower momentum (0.85) at the start allows the optimizer to respond quickly to the initial gradient signals. The model parameters begin at random values, and the gradients during early training often point in dramatically different directions. Lower momentum means we don’t accumulate too much history from these early, potentially misleading gradients.</p><p>Higher momentum (0.95) later provides better gradient smoothing and helps traverse flat regions. Once training stabilizes, we want to smooth out noise and maintain consistent direction even when individual batch gradients fluctuate.</p><h4>Implementation</h4><pre>def get_muon_momentum(it):<br>    frac = min(it / 300, 1)<br>    momentum = (1 - frac) * 0.85 + frac * 0.95<br>    return momentum</pre><p>Applied during training:</p><pre># Update Muon momentum<br>muon_momentum = get_muon_momentum(step)<br>for group in muon_optimizer.param_groups:<br>    group[&quot;momentum&quot;] = muon_momentum</pre><p>The 300-step ramp is quick enough not to waste iterations but long enough to provide benefits. This mirrors techniques used in other high-performance optimizers.</p><h3>Complete Training Loop</h3><p>Here’s how all the components work together:</p><pre># Initialize optimizers<br>optimizers = model.setup_optimizers(<br>    unembedding_lr=0.004,<br>    embedding_lr=0.2,<br>    matrix_lr=0.02,<br>    weight_decay=0.0<br>)<br>adamw_optimizer, muon_optimizer = optimizers<br># Training loop<br>for step in range(num_iterations):<br>    # Forward pass<br>    logits = model(x)<br>    loss = F.cross_entropy(logits.view(-1, logits.size(-1)), y.view(-1))<br>    <br>    # Backward pass<br>    loss.backward()<br>    <br>    # Apply learning rate schedule<br>    lrm = get_lr_multiplier(step)<br>    for opt in optimizers:<br>        for group in opt.param_groups:<br>            group[&quot;lr&quot;] = group[&quot;initial_lr&quot;] * lrm<br>    <br>    # Apply momentum schedule (Muon only)<br>    muon_momentum = get_muon_momentum(step)<br>    for group in muon_optimizer.param_groups:<br>        group[&quot;momentum&quot;] = muon_momentum<br>    <br>    # Optimizer step<br>    for opt in optimizers:<br>        opt.step()<br>    <br>    # Zero gradients<br>    for opt in optimizers:<br>        opt.zero_grad()</pre><h3>Performance Characteristics</h3><p>The combination of <strong>Muon</strong> and <strong>AdamW</strong> provides several key advantages.</p><p>Muon converges faster than Adam when optimizing transformer weight matrices. Its orthogonalization property ensures that updates move in geometrically meaningful directions, avoiding the zigzagging behavior that can occur with standard optimizers.</p><p>Orthogonalization also helps prevent gradient explosion. By keeping updates orthogonal, Muon naturally limits their magnitude and avoids uncontrolled growth.</p><p>In terms of memory efficiency, Muon is more lightweight than Adam. For a parameter tensor of size (d_out,d_in), AdamW stores two buffers (the first and second moments), requiring 2 × d_out × d_in memory. Muon stores only one buffer (momentum), using 1×d_out×d_in memory. This results in a <strong>50% reduction</strong> in memory usage for transformer parameters.</p><p>Each optimizer is applied where it performs best. This targeted use allows better overall performance compared to relying on a single optimizer for all parameters.</p><p>Finally, both optimizers support efficient distributed implementations. With <strong>ZeRO-2 style sharding</strong> across GPUs, memory requirements scale efficiently with the number of devices, enabling large models to train smoothly across multiple systems.</p><h3>Conclusion</h3><p>nanochat’s optimization strategy demonstrates that algorithm selection matters. By using Muon for 2D weight matrices (fast, stable, memory efficient) and AdamW for embeddings and LM head (adaptive, proven), combined with smart scheduling for learning rate and momentum, the training process achieves both speed and stability.</p><p>The careful attention to learning rates, momentum schedules, and distributed training details ensures that nanochat can efficiently train high-quality language models. Understanding these components and their interactions provides insight into modern training techniques and shows how thoughtful engineering can significantly improve training efficiency.</p><p>In the next part of this series, we’ll explore the data pipeline and how nanochat efficiently streams and tokenizes training data.</p><h3>References / Sources</h3><p><a href="https://github.com/karpathy/nanochat?source=post_page-----6290bb81374d---------------------------------------">GitHub - karpathy/nanochat: The best ChatGPT that $100 can buy.</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FbO5nvE289ec%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbO5nvE289ec&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FbO5nvE289ec%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4bd32ba7ca29d73ace9711dbeb8e7a3/href">https://medium.com/media/d4bd32ba7ca29d73ace9711dbeb8e7a3/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fm1PUl-aUk8c%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dm1PUl-aUk8c&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fm1PUl-aUk8c%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/104ebc7614b8554a905ce58bfd682dda/href">https://medium.com/media/104ebc7614b8554a905ce58bfd682dda/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FyNwdwZPBYtM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DyNwdwZPBYtM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FyNwdwZPBYtM%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b40208cb030cfe7da0311ab82a47a69/href">https://medium.com/media/3b40208cb030cfe7da0311ab82a47a69/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f989d407153" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>