<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mukul Ranjan on Medium]]></title>
        <description><![CDATA[Stories by Mukul Ranjan on Medium]]></description>
        <link>https://medium.com/@mukulranjan?source=rss-abdbc773a16e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TWEY7oW_o_JNjS_28nLkkw.jpeg</url>
            <title>Stories by Mukul Ranjan on Medium</title>
            <link>https://medium.com/@mukulranjan?source=rss-abdbc773a16e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 14:24:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mukulranjan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Serving Thousands of Concurrent LoRA Adapters]]></title>
            <link>https://medium.com/@mukulranjan/serving-thousands-of-concurrent-lora-adapters-6b407e8df516?source=rss-abdbc773a16e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b407e8df516</guid>
            <dc:creator><![CDATA[Mukul Ranjan]]></dc:creator>
            <pubDate>Fri, 16 Jan 2026 00:15:25 GMT</pubDate>
            <atom:updated>2026-01-17T14:44:52.704Z</atom:updated>
            <content:encoded><![CDATA[<h4>[Paper Review] S-LoRA: Serving Thousands of Concurrent LoRA Adapters</h4><p>Low Rank Adaptation (LoRA) is very often used to efficiently finetune a base model for deployment across different tasks. This results in a huge collection of LoRA adapters from a single base model. One major use case for this can be seen in recommender systems. For example, say you train a foundation model for user modeling and you want to use this for different segments of users. The most straightforward solution is to train an adapter for each cohort of users. This will result in thousands of adapters and efficient serving becomes critical in that case.</p><p>S-LoRA paper solves the problem of efficient serving when you have thousands of adapters from the same base model.</p><h4>TL;DR and Contributions</h4><p>This paper presents a system for the scalable serving of many task-specific fine-tuned LoRA adapters. It stores all the adapters in the main memory and fetches the adapters used by the currently running queries to the GPU memory. It proposes, 1.) <strong>Unified Paging </strong>to efficiently use GPU memory and reduce fragmentation and 2.) <strong><em>a new tensor parallelism strategy (S-LoRA TP)</em></strong> and <strong><em>optimised custom CUDA kernel</em></strong> for heterogeneous batching of LoRA computation. Contributions can be summarised as below:</p><ol><li><strong>Heterogeneous Batching and Scheduling</strong>: Introduces optimised CUDA kernels to <strong>operate directly on non-contiguous memory,</strong> in order to batch <em>different adapters of varying rank.</em></li><li><strong>Unified Paging</strong>: Introduces <strong>unified memory pool</strong> to reduce <strong>memory fragmentation and increase batch size</strong>. The pool manages dynamic adaptor weights and KV cache tensors by <strong><em>unified paging mechanism</em></strong>.</li><li><strong>S-LoRA TP</strong>: Introduces a new <strong><em>tensor parallel</em></strong> strategy <em>across multiple GPUs</em>, to ensure that it incurs minimal communication cost for added LoRA communication compared to base model.</li></ol><h4>Preliminary</h4><p>Here, I would assume that you are aware of LoRA so I won’t go in too much detail. LoRA is a parameter-efficient fine-tuning method designed to adapt pre-trained large language models to new task. The motivation behind LoRA comes from the <strong><em>low intrinsic dimensionality of model update during adaptation (</em></strong><a href="https://aclanthology.org/2021.acl-long.568.pdf">another interesting paper</a><strong><em>).</em></strong> In the training phase LoRA freezes the weight of pre-trained base model and adds <em>trainable low-rank matrices to each layer</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*cpoY9ktTsZRDUZRsUpuBpA.png" /><figcaption>Figure 1: Reparametrization of the LoRA. Only A and B are trained and the weight of pre-trained base model W is kept frozen.(<a href="https://arxiv.org/pdf/2106.09685">source</a>)</figcaption></figure><p>Formally, for a pre-trained weight matrix <strong>W₀</strong> ∈ <strong>ℝ</strong>ʰ ˣ ᵈ, LoRA introduces a low-rank update <strong>∆W</strong> = <strong>AB</strong>, where <strong>A</strong> ∈ <strong>ℝ</strong>ʰ ˣ ʳ, <strong>B</strong> ∈ <strong>ℝ</strong>ʳ ˣ ᵈ, and the rank <em>r</em> ≪ min(<em>h</em>, <em>d</em>) and hence the update become <strong>W = W₀ + ∆W = W₀ + AB. </strong>If the forward pass of a base model is defined by <em>h</em> = <em>x</em><strong>W₀</strong>, then after applying LoRA, the forward pass becomes:</p><p>h = <em>x</em>(W₀ + ∆W) = <em>x</em>(W₀ + AB) (1)<br> = <em>x</em>W₀ + <em>x</em>AB (2)</p><p>Typically, this adjustment is only applied to the query, key, value, and output projection matrices in the self-attention module, and excludes the feed-forward module.</p><h3><strong>Heterogeneous Batching and Scheduling</strong></h3><h4>The Batching Challenge</h4><p>The paper’s batching strategy enables online high-throughput serving of multiple LoRA adapters simultaneously. The original LoRA paper suggests <em>to merge the weight into the base model (equation 1)</em>. This leads to no additional overhead during the inference since the <em>resulting model has the same parameters count as the base models</em>.</p><p><strong>However, when serving multiple adapters </strong>this creates problems of multiple weight copy of the same model, consuming eccessive memory. While the LoRA paper propose adding and subtracting adapter weights on-the-fly to avoid memory overhead, <em>this approach doesn’t support concurrent inference on separate LoRA adapters and therefore limiting batching opportunities</em>.</p><h4>S-LoRA’s Solution</h4><p>S-LoRA proposes LoRA computation <strong><em>x</em>AB</strong> on-the-fly (equation 2) rather than merging weight. This eliminates duplication and enables batching of the more costly <strong>xW₀</strong> operation. While this adds an additional overhead of <strong>xAB</strong> computation, this computation is substantially lower than <strong>xW₀</strong> and the savings from batching the base model computation more than compensate for it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*18j1TjsSDORKX-KMMn5PUw.png" /><figcaption>Figure 2: Separated batched computation for the base model and LoRA computation. The batched computation of the base model is implemented by GEMM. The batched computation for LoRA adapters is implemented by custom CUDA kernels which support batching various sequence lengths and adapter ranks. (<a href="https://arxiv.org/pdf/2311.03285">source</a>)</figcaption></figure><h4><strong>Custom CUDA Kernels</strong></h4><p>Directly implementing the factored computation of the base model and individual LoRA adapters using the batch GEMM kernel from the existing BLAS libraries would require significant padding due to <em>the heterogeneity of sequence lengths and adapter ranks </em>resulting in poor hardware utilization.</p><p>Instead, S-LoRA batches the computation of the base model <strong>xW₀ </strong>and then uses custom CUDA kernels to efficiently compute <strong><em>x</em>AB</strong> for each adapter separately without padding (see <strong><em>Figure 2</em></strong>). This custom implementation achieves much better performance than naive batched approaches.</p><h4><strong>Memory Management Strategy</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LamQPYQFuX9-1iMxId22Xw.png" /><figcaption>Figure 3: Overview of memory allocation in S-LoRA. S-LoRA stores all adapters in the main memory and fetches the active adapters for the current batch to the GPU memory. The GPU memory is used to store the KV cache, adapter weights, base model weights, and other temporary tensors (<a href="https://arxiv.org/pdf/2311.03285">source</a>).</figcaption></figure><p>While the total number of LoRA adapters can be large, only a subset is needed for any given batch, and <em>the batch size itself is constrained by GPU memory.</em> <strong><em>S-LoRA exploits this by storing all adapters in main memory (RAM)</em></strong> and dynamically loading only the adapters required for the current batch onto the GPU. <em>This means the system’s adapter capacity is bounded by main memory rather than GPU memory, enabling it to serve thousands of adapters.</em></p><h4><strong>Token-Level Scheduling</strong></h4><p>To maximize throughput, S-LoRA adopts iteration-level scheduling from <a href="https://www.usenix.org/system/files/osdi22-yu.pdf">Orca</a>, where <em>requests are scheduled at the token level rather than the sequence level.</em> New requests are immediately added to the running batch whenever space becomes available, and requests exit the batch once they complete generation or meet stopping criteria. This approach significantly reduces GPU memory usage compared to traditional batching, but introduces new memory management challenges that we will discuss later.</p><h4><strong>Adapter Clustering</strong></h4><p>To further improve batching efficiency, <strong>S-LoRA can reduce the number of active adapters in a running batch</strong>. With fewer adapters consuming memory, more space becomes available for the KV cache, enabling larger batch sizes. <em>Since GPUs are often underutilized during the decoding phase, increasing batch size directly translates to higher throughput</em>. A straightforward way to reduce active adapters is through <strong><em>“adapter clustering”</em></strong>, <strong>prioritizing requests that use the same adapter when forming batches.</strong> However, this comes with trade-offs: it can increase average latency and create fairness issues, as some adapters may be prioritized over others.</p><h4><strong>Admission Control</strong></h4><p>When traffic exceeds system capacity, S-LoRA employs admission control to maintain service quality. Serving systems typically operate under a <em>service level objective (SLO)</em> that specifies target latency. Without admission control, an overloaded system will eventually violate its SLO for all requests as the queue grows unbounded. S-LoRA implements an <strong><em>“early abort” strategy: it estimates which recent requests can be served within the SLO and processes them in arrival order, dropping requests that cannot meet the latency requirement.</em></strong> This ensures the system maintains its SLO for accepted requests even under high load.</p><h3>Unified Paging and Memory Management</h3><p>Serving multiple LoRA adapters introduces unique memory challenges. <strong><em>S-LoRA stores adapters in main memory (RAM) and dynamically loads them to GPU RAM as needed.</em></strong> This creates two key problems:</p><ol><li><strong>Memory fragmentation</strong> from loading and unloading adapters of varying sizes</li><li><strong>I/O latency overhead</strong> when transferring adapters between main memory and GPU</li></ol><p>S-LoRA addresses these through <strong><em>Unified Paging and intelligent prefetching</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*Mh3ErlnhF4LBNRpI67Cjkw.png" /><figcaption>Figure 4: Unified memory pool. We use a unified memory pool to store both KV caches and adapter weights in a non-contiguous way to reduce memory fragmentation. The page size is H elements.</figcaption></figure><h4>Unified Paging</h4><p>The key observation: adapter weights behave similarly to KV caches.</p><p><strong>Similarities between adapter weights and KV caches:</strong></p><ul><li><strong>Variable sizes</strong>: KV cache size varies with sequence length; adapter sizes vary with rank. Both are allocated on request arrival and deallocated on completion, leading to fragmentation if mismanaged.</li><li><strong>Shared dimensionality</strong>: KV cache tensors have shape (S, H) where S is sequence length and H is hidden dimension. LoRA weights have shape (R, H) where R is rank. Both share the H dimension, which we can exploit.</li></ul><p>Inspired by <a href="https://arxiv.org/abs/2309.06180">PagedAttention</a>, S-LoRA introduces <strong>Unified Paging, <em>a single memory pool managing both KV caches and adapter weights</em></strong>.</p><p><strong>Algorithm</strong></p><ol><li>Allocate a large static buffer using all available space (excluding base model weights and temporary activations)</li><li>Store both KV caches and adapter weights in pages of size H</li><li>A KV cache with sequence length S uses S pages; a LoRA weight with rank R uses R pages</li><li>KV caches and adapter weights are stored interleaved and non-contiguously (Figure 4)</li></ol><p>This reduces fragmentation, allowing adapters of various ranks to coexist with dynamic KV caches efficiently.</p><h4>Prefetching and Overlapping</h4><p>While Unified Paging solves fragmentation, I/O overhead remains, especially with many or large adapters. S-LoRA addresses this through <strong>dynamic prediction.</strong></p><ol><li>While processing the current batch, predict which adapters are needed next based on the waiting queue.</li><li>Prefetch and load predicted adapters into available memory.</li><li>By the time the next batch runs, most required adapters are already loaded, minimizing I/O latency</li></ol><h4>Custom CUDA Kernels</h4><p>The non-contiguous memory layout requires specialized computation kernels. <strong><em>S-LoRA implements custom CUDA kernels that efficiently handle heterogeneous LoRA batching across varying ranks and sequence lengths</em></strong>:</p><p><strong>Prefill stage</strong> (processing sequences): At this stage kernel handles a sequence of tokens and gathers adapter weight with different rank from the memory pool. This kernel is called <strong>MBGMM</strong> (Multi-size Batched Gather Matrix-Matrix Multiplication) kernel.</p><p><strong>Decode stage</strong> (processing single tokens): At this stage the kernel handles a single token and gather adapter weights with different rank from the memory pool. It is called <strong>MBGMV</strong> (Multi-size Batched Gather Matrix-Vector Multiplication) kernel. The paper tested two implementations: Triton and modified Punica kernels. Modified Punica version was faster, supporting non-contiguous memory, multiple ranks per batch, and fine-grained memory gathering</p><h3><strong>S-LoRA </strong>Tensor Parallelism</h3><p>To support multi-GPU inference of large transformer models with batched LoRA adapters, S-LoRA introduces novel tensor parallelism strategies. Tensor parallelism is the most widely used parallelism method due to its <strong><em>single-program multiple-data (SPMD)</em></strong> pattern, which simplifies implementation and integration. It reduces per-GPU memory usage and latency when serving large models. However, LoRA adapters introduce new weight matrices and matrix multiplications that require careful partitioning strategies.</p><h4>Partition Strategy</h4><p>The base model uses the Megatron-LM tensor parallelism strategy, so S-LoRA’s approach aligns the partition strategies of the added LoRA computation with those of the base model. This alignment minimizes communication costs by avoiding unnecessary data transfers and enabling communication fusion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K4DlyL9K4WxCNYLR7S8TrQ.png" /><figcaption>Figure 5: Tensor parallelism partition strategy for batched LoRA computation. This is a computational graph where nodes represent tensors/operators and the edges represent dependency. We use different colors to represent different partition strategies, which include column partition, row partition, partial sum, and replication. The per-GPU shape of each tensor is also annotated in gray. Note that B is the number of tokens, h is the input dimension, N is the number of devices, d is the hidden size, and r is the adapter rank (<a href="https://arxiv.org/pdf/2311.03285">source</a>).</figcaption></figure><p><strong>Base Model Partitioning (Megatron-LM):</strong></p><p>Let’s examine the feed-forward module (2-layer MLP) shown in Figure 5:</p><ul><li>First weight matrix (W₁): column-partitioned</li><li>Second weight matrix (W₂): row-partitioned</li><li>Final step: all-reduce operation to accumulate partial sums</li></ul><p><strong>LoRA Adapter Partitioning:</strong></p><p>For adapters on W₁ (matrices A₁ and B₁):</p><ul><li>Both A₁ and B₁: column-partitioned</li><li>Intermediate results: collected via all-gather operation</li></ul><p>For adapters on W₂ (matrices A₂ and B₂):</p><ul><li>A₂: row-partitioned</li><li>B₂: column-partitioned</li><li>Intermediate results: summed via all-reduce operation</li></ul><p><strong>Optimization:</strong> The all-reduce for LoRA computation is fused with the base model’s final all-reduce. Instead of two separate all-reduce calls, S-LoRA performs just one, <strong><em>essentially fusing an all-gather operation for the final LoRA matrix multiplication with the base model’s reduction</em></strong>.</p><p><strong>Extension to self-attention layers:</strong> The strategy adapts naturally by partitioning along the head dimension (following Megatron-LM):</p><ul><li>Query-key-value projection weight matrix → treated as W₁</li><li>Output projection weight matrix → treated as W₂</li></ul><h4>Communication and Memory Analysis</h4><p>Given N devices, B tokens, hidden size h, and adapter rank r:</p><ul><li><strong>Base model:</strong> One all-reduce = 2(N−1)Bh/N</li><li><strong>LoRA computation:</strong> Three all-gather (Q, K, V) + one all-reduce (output) = 5(N−1)Br/N</li></ul><p><strong>Why this is efficient:</strong> Since r ≪ h (adapter rank is much smaller than hidden size), the additional communication cost from LoRA is negligible. This is achieved by:</p><ul><li>Carefully scheduling communications on small intermediate LoRA tensors</li><li>Fusing LoRA operations with base model communications</li></ul><p><strong>Memory usage:</strong> This strategy is optimal, all weight matrices are partitioned across devices with zero replication, maximizing memory efficiency.</p><h3>Reference</h3><ol><li><a href="https://arxiv.org/abs/2311.03285">S-LoRA: Serving Thousands of Concurrent LoRA Adapters</a></li><li><a href="https://arxiv.org/abs/2106.09685">LoRA: Low-Rank Adaptation of Large Language Models</a></li><li><a href="https://aclanthology.org/2021.acl-long.568.pdf">Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning</a></li><li><a href="https://www.usenix.org/system/files/osdi22-yu.pdf">ORCA: A Distributed Serving System for Transformer-Based Generative Models</a></li><li><a href="https://arxiv.org/pdf/2309.06180">Efficient Memory Management for Large Language Model Serving with PagedAttention</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b407e8df516" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Register Buffer and Register Parameter in PyTorch]]></title>
            <link>https://medium.com/@mukulranjan/register-buffer-and-register-parameter-in-pytorch-3e86d310de6c?source=rss-abdbc773a16e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e86d310de6c</guid>
            <dc:creator><![CDATA[Mukul Ranjan]]></dc:creator>
            <pubDate>Wed, 14 Jan 2026 01:54:38 GMT</pubDate>
            <atom:updated>2026-01-14T01:54:38.458Z</atom:updated>
            <content:encoded><![CDATA[<p>When building neural networks in PyTorch, we often need to store tensors that aren’t meant to be trained. The most common examples being std and mean for the batch norm (check the pytorch codebase <a href="https://github.com/pytorch/pytorch/blob/v2.9.1/torch/nn/modules/batchnorm.py#L380">here</a> and look for register_buffer). When I first saw this being used frequently in multiple codebases, I found it a bit confusing. After some small google search, I came across really good discussion in the pytorch forum(see references).</p><h3>Basics</h3><p>In PyTorch, there are two main ways to store tensors in our model:</p><p><strong>Parameters</strong> are learnable tensors that get updated during training. They require gradients and are returned by model.parameters(), which means the optimizer will update them.</p><p><strong>Buffers</strong> are fixed tensors that don’t require gradients. They’re not returned by model.parameters(), so the optimizer ignores them. Think of them as constants or non-learnable state that the model needs to remember.</p><h3>Why Not Just Use self.my_tensor?</h3><p>The first confusion I had was if I don’t want something trained, can I just assign it directly like self.my_tensor = torch.randn(1) inside my nn.Module? Technically yes, but you&#39;ll run into problems.</p><p>Let’s see what happens:</p><pre>class MyModel(nn.Module):<br>    def __init__(self):<br>        super(MyModel, self).__init__()<br>        self.my_tensor = torch.randn(1)<br>        self.register_buffer(&#39;my_buffer&#39;, torch.randn(1))<br>        self.my_param = nn.Parameter(torch.randn(1))<br>        <br>    def forward(self, x):<br>        return x<br><br>model = MyModel()<br>print(model.my_tensor)<br># tensor([0.9329])<br>print(model.state_dict())<br># OrderedDict([(&#39;my_param&#39;, tensor([-0.2471])), <br>#              (&#39;my_buffer&#39;, tensor([1.2112]))])</pre><p>Notice that my_tensor doesn&#39;t appear in the state_dict(). This means if we save and load the model, my_tensor won&#39;t be restored. we will lose that state. Now see what happens when we move the model to GPU:</p><pre>model.cuda()<br>print(model.my_tensor)<br># tensor([0.9329])  # Still on CPU!<br>print(model.state_dict())<br># OrderedDict([(&#39;my_param&#39;, tensor([-0.2471], device=&#39;cuda:0&#39;)), <br>#              (&#39;my_buffer&#39;, tensor([1.2112], device=&#39;cuda:0&#39;))])</pre><p>The buffer and parameter moved to CUDA, but my_tensor stayed on CPU. This will cause errors when you try to use it in forward passes.</p><p><strong>major benefits of </strong><strong>register_buffer:</strong></p><ol><li>Automatically included in state_dict() for saving/loading</li><li>Moved to the correct device when you call model.cuda() or model.to(device)</li><li>Makes your code’s intent clear to other developers</li></ol><h3>Why Not Use nn.Parameter with requires_grad=False?</h3><p>Another confusion that I had was: why not just use nn.Parameter(tensor, requires_grad=False) for buffers?</p><p>This technically works for training, but it’s confusing and inefficient:</p><ol><li><strong>Misleading code</strong>: Other developers expect parameters to be learnable. Seeing non-learnable “parameters” is confusing.</li><li><strong>Optimizer overhead</strong>: If you pass model.parameters() to your optimizer, it includes these fake parameters. The optimizer has to check and skip them on every step, which wastes computation.</li><li><strong>Code clarity</strong>: Separating buffers and parameters makes your intent obvious at a glance.</li></ol><h3>Examples</h3><p>Here’s a simple example of batch normalization where we track running statistics:</p><pre>class SimpleBatchNorm(nn.Module):<br>    def __init__(self, num_features):<br>        super(SimpleBatchNorm, self).__init__()<br>        self.weight = nn.Parameter(torch.ones(num_features))<br>        self.bias = nn.Parameter(torch.zeros(num_features))<br>        <br>        # These are not learned, but need to be saved and moved to device<br>        self.register_buffer(&#39;running_mean&#39;, torch.zeros(num_features))<br>        self.register_buffer(&#39;running_var&#39;, torch.ones(num_features))<br>        self.register_buffer(&#39;num_batches_tracked&#39;, torch.tensor(0))<br>    <br>    def forward(self, x):<br>        # Use running_mean and running_var for normalization<br>        # Update them during training<br>        return normalized_output</pre><p>The weights and biases are parameters (learned). The running statistics are buffers (tracked but not learned).</p><h3>When to Use register_parameter</h3><p>We also see register_parameter() in PyTorch code. It works similarly to directly assigning an nn.Parameter, but we pass the name as a string:</p><pre>self.my_param = nn.Parameter(torch.randn(10))<br># vs<br>self.register_parameter(&#39;my_param&#39;, nn.Parameter(torch.randn(10)))</pre><p>Both do the same thing. The register_parameter approach is handy when we create parameters in a loop or need dynamic naming:</p><pre>for i in range(5):<br>    self.register_parameter(f&#39;weight_{i}&#39;, nn.Parameter(torch.randn(10)))</pre><p>Otherwise, it’s just a style choice.</p><h3>Summary</h3><ul><li>Use <strong>nn.Parameter</strong> for learnable tensors that require gradients</li><li>Use <strong>register_buffer</strong> for non-learnable tensors that need to be saved and moved with the model</li><li>Don’t use plain attributes like self.my_tensor for anything you want preserved in the model state</li><li>Don’t use nn.Parameter(requires_grad=False) as a hack for buffers</li></ul><h3>References</h3><ol><li><a href="https://discuss.pytorch.org/t/what-is-the-difference-between-register-buffer-and-register-parameter-of-nn-module/32723">PyTorch Discussion on register_buffer vs register_parameter</a></li><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_buffer">PyTorch Documentation — nn.Module.register_buffer</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e86d310de6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Complete Sony Camera Guide: Every Series Explained]]></title>
            <link>https://medium.com/@mukulranjan/complete-sony-camera-guide-every-series-explained-faf8eaece081?source=rss-abdbc773a16e------2</link>
            <guid isPermaLink="false">https://medium.com/p/faf8eaece081</guid>
            <dc:creator><![CDATA[Mukul Ranjan]]></dc:creator>
            <pubDate>Sun, 04 Jan 2026 13:06:57 GMT</pubDate>
            <atom:updated>2026-01-04T13:06:57.246Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Acknowledgement: Most content of this blog is based on this youtube </strong><a href="https://www.youtube.com/watch?v=Y5V6feRnLkk"><strong>video</strong></a><strong>[1] by Jason Vong.</strong></p><p>Recently I wanted to buy a new camera and got stuck with different camera lingo. I watched several youtube videos but I think Jason Vong’s video was really great. I used most of the content from his videos as it is in this blog. If you want you can first watch his video and then think of this blog as the short note of the video with some additional details.</p><p>Sony offers an extensive range of cameras that can be overwhelming for newcomers and even experienced photographers. This comprehensive guide breaks down Sony’s camera lineup into distinct series, helping you understand which camera might be right for your needs and budget.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RLDJS6x5aTALDv7qULla8g.png" /></figure><h3>Understanding Camera Sensor Sizes</h3><p>Before diving into specific camera series, it’s important to understand sensor sizes. Sony cameras range from small 1/2.55&quot; sensors in compact cameras to full-frame sensors measuring 35.9 x 24.0mm. Larger sensors generally provide better image quality, especially in low light, but result in larger, more expensive cameras.</p><p>The sensor size progression from smallest to largest:</p><ul><li><strong>1/2.55&quot;</strong> (6.17 x 4.55mm) — Smartphone-sized sensors</li><li><strong>1&quot;</strong> (12.80 x 9.60mm) — Premium compact cameras</li><li><strong>Micro 4/3</strong> (17.00 x 13.00mm) — Mirrorless standard</li><li><strong>APS-C</strong> (23.6 x 15.60mm) — Crop sensor DSLRs and mirrorless</li><li><strong>Full-Frame</strong> (35.90 x 24.00mm) — Professional standard</li><li><strong>Medium Format</strong> (53.0 x 40.20mm) — High-end professional</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*STVkKs58VE2Jw1EJJY1lRQ.png" /><figcaption>Different camera sensor sizes (<a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">source</a>)</figcaption></figure><h3>1. RX Series: Premium Compact Cameras</h3><p>The RX series represents Sony’s premium compact camera lineup, designed for users who want high image quality in a portable package.</p><h4>Entry-Level Compacts</h4><ul><li><strong>RX0 II</strong> ($700, 2019): Ultra-compact with 1&quot; sensor and 24mm f/4 lens. Perfect for travel and situations where discretion is key.</li><li><strong>RX1R II</strong> ($3200, 2015): Full-frame sensor with fixed 35mm f/2 lens, delivering exceptional image quality in a compact body.</li></ul><h4>Advanced Compacts</h4><ul><li><strong>RX100 VII</strong> ($1300, 2019): Features 24–200mm lens, making it ideal for versatile shooting situations. Includes advanced autofocus and 20 FPS burst shooting.</li><li><strong>RX10 IV</strong> ($1700, 2017): Sports a massive 24–600mm lens with constant aperture, perfect for wildlife and sports photography where you can’t get close to the action.</li></ul><p><strong>Who should consider RX cameras?</strong> High-end enthusiasts and professional photographers who need a smaller body with the same capabilities as their bigger setup. These cameras excel when portability is crucial without sacrificing image quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N8OoP-D2CsUsb5C3VKuSQQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IeOWGfjb276L2miom9OQxg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*zKclgV5HjruqQfvvtjAZ9Q.png" /><figcaption>Different Images for RX Series Specifications (Above two images are from <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">this video</a> by Jason Vong)</figcaption></figure><h3>2. ZV Series: Content Creator Focused</h3><p>Sony’s ZV series specifically targets vloggers, content creators, and social media influencers.</p><ul><li><strong>ZV-1</strong> ($750, 2020): Features 24–70mm lens with older UI but longer reach for versatile content creation.</li><li><strong>ZV-1 II</strong> ($950, 2023): Updated with 18–50mm lens, new UI, AI autofocus, and enhanced beginner-friendly features plus cinematic vlog mode.</li></ul><p>These cameras prioritize video features like flip-out screens, superior microphones, and simplified controls for content creation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W_Cr1aTNNr9qk-wQVi2B1Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*Y73vQYzCIQLjm7IhG8Ygew.png" /><figcaption>Different Images for ZV Series Specifications (Left images is from <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">this video</a> by Jason Vong)</figcaption></figure><h3>3. Alpha 6x00 Series: APS-C Mirrorless</h3><p>The Alpha 6x00 series offers a significant step up in quality over smartphones while remaining more affordable than full-frame systems.</p><ul><li><strong>α6400</strong> ($900, 2019): Entry-level with 24MP APS-C sensor, 4K video up to 30fps, and 11 frames per second burst shooting.</li><li><strong>α6600</strong> ($1200, 2019): Adds in-body image stabilization and improved battery life over the α6400.</li><li><strong>α6700</strong> ($1400, 2023): Latest model with 26MP sensor, improved video capabilities, and enhanced autofocus system.</li></ul><p><strong>Key advantage:</strong> These cameras provide a huge step up in quality over smartphones yet cost less than a full-frame system, making them ideal for enthusiasts ready to move beyond phone photography.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bki7ZDVjjZPf9_2EY7UuQw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FfP3mPzeurNEJSeR_cgnrw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*V2PWfItimOqyZMnhA9fBmA.png" /><figcaption>Different Images for a6x00 Series Specifications (Above two images are from <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">this video</a> by Jason Vong)</figcaption></figure><h3>4. FX Series: Cinema-Focused Professional Cameras</h3><p>The FX series represents Sony’s cinema-focused lineup, dedicated to providing the most efficient workflow for filmmakers and video professionals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*osGe0cTN8EtC3zDk4r1u8w.png" /><figcaption>Some Specifications of FX series Camera</figcaption></figure><h4>Cinema Cameras</h4><ul><li><strong>FX9</strong> and <strong>FX6</strong>: Professional camcorder-style cameras for broadcast and cinema production</li><li><strong>FX3</strong> ($3900, 2021): Mirrorless cinema camera in a compact body</li><li><strong>FX30</strong> ($1800, 2022): APS-C cinema camera offering true 24p recording with anamorphic de-squeezing and electronic shutter only</li></ul><h4>Comparison: FX30 vs α6700</h4><p>While both use APS-C sensors, the <strong>FX30</strong> focuses entirely on video with features like:</p><ul><li>True 24p recording</li><li>Anamorphic de-squeezing</li><li>Cine EI workflow</li><li>Electronic shutter only</li></ul><p>The <strong>α6700</strong> offers:</p><ul><li>Mechanical and electronic shutter</li><li>Better burst photography (11 FPS)</li><li>More balanced photo/video hybrid approach</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WoV4qFHQoicyB_af2WeFHA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*875OyUKBzgS3Xr-ywioYpA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dgDrIP72Nc4xH9g4O8PJcw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*II26l7RU-i4cb5A7Qgnl8Q.png" /><figcaption>Different images from the <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">Json Vong Video</a></figcaption></figure><h3>5. a7 Series: Full-Frame Versatility</h3><p>Sony’s a7 series represents their full-frame mirrorless lineup, with different models optimized for specific use cases.</p><h4>a7 Base Model</h4><p>The standard a7 cameras offer balanced performance for general photography and are the most affordable entry into full-frame photography.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*QJbp0RlM2HrQ1kwiVuj0UA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSQSxhzC818qTM30RdaVIA.png" /><figcaption>Different images from the <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">Json Vong Video</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*5ge_eR6qKgZ0U2V24coxLg.jpeg" /><figcaption>Different images from the <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">Json Vong Video</a></figcaption></figure><h4>a7R Series: Resolution Priority</h4><ul><li><strong>a7R III</strong> ($1798): 42MP sensor</li><li><strong>a7R IV</strong> ($2498): 61MP sensor</li><li><strong>a7R V</strong> ($3900, 2022): Latest 61MP with 8K video capabilities</li></ul><p>The “R” stands for resolution — these cameras prioritize maximum image detail and are ideal for landscape, architecture, and studio photography where fine detail matters most.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WqdQjXfST8otHukLcDNtow.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlNz9ZCSOMAjybGpODYUiA.png" /><figcaption>Different images from the <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">Json Vong Video</a></figcaption></figure><h4>a7S Series: Sensitivity Priority</h4><ul><li><strong>a7S III</strong> ($3500, 2020): Optimized for low-light performance with excellent high ISO capabilities</li></ul><p>The “S” indicates sensitivity — these cameras excel in challenging lighting conditions and are favored by videographers and photographers who frequently shoot in low light.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNRt6vjXDDExOGlXDPWJ5g.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IJhtUzZpubmQHEotbgna-Q.png" /><figcaption>Different images from the <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">Json Vong Video</a></figcaption></figure><h4>a7C Series: Compact Full-Frame</h4><ul><li><strong>a7C Series</strong>: Compact full-frame cameras designed for casual users who want full-frame quality in a smaller, more approachable package with fewer custom buttons and simplified controls.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eeerOH6UuwWPizwmoDP-xQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tec1ANSYDDy0wtQDPWW5PQ.png" /><figcaption>Different images from the <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">Json Vong Video</a></figcaption></figure><h3>6. a9 Series: Speed and Performance</h3><p>The a9 series focuses on ultimate performance for action photography.</p><p><strong>Frame Rate Comparison:</strong></p><ul><li><strong>α6X00 Series</strong>: 11 FPS</li><li><strong>A7 IV</strong>: 10 FPS</li><li><strong>A9 II</strong>: 20 FPS</li><li><strong>A9 III</strong>: 120 FPS (revolutionary speed)</li></ul><p>The a9 series cameras are the go-to choice for sports photographers, wildlife shooters, and anyone who needs to capture fast-moving subjects with precision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*fRx770QYFOzMFKRmWRGMYQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2vx15CY-4GzatViJ1ASCyg.png" /><figcaption>Different images from the <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">Json Vong Video</a></figcaption></figure><h3>7. a1 Series: The Ultimate Sony Camera</h3><p>The <strong>Sony a1</strong> represents the pinnacle of Sony’s camera technology — “The camera that DOES IT ALL!”</p><p><strong>Key Specifications:</strong></p><ul><li><strong>51MP</strong> full-frame sensor</li><li><strong>30 FPS</strong> continuous shooting</li><li><strong>8K 30FPS</strong> video recording</li></ul><p>The a1 combines the resolution of the a7R series, the speed of the a9 series, and the video capabilities of the FX series into one flagship camera. It’s designed for professionals who need the absolute best performance regardless of price.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*NuivnUOOAJFK8YvqTiXhrQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qCw1qw-IFQZ5RmbbbkJGwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zmd98zOqXjB_lUQhXrjFZw.png" /><figcaption>Different images from the <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">Json Vong Video</a></figcaption></figure><h3>Choosing the Right Sony Camera</h3><p><strong>For Content Creators</strong>: ZV series offers the best features for vlogging and social media content. The ZV-E10 II at $999 is particularly recommended as “outstanding value for the dollar” with professional reviewers noting it corrects all major issues from its predecessor.</p><p><strong>For Budget-Conscious Enthusiasts</strong>: Alpha 6x00 series provides excellent image quality without full-frame costs. The α6700 at $1400 represents the current sweet spot for hybrid photo/video performance.</p><p><strong>For Filmmakers</strong>: FX series delivers professional video features and workflows. Consider the FX30 for APS-C cinema needs or wait for the anticipated FX3 II.</p><p><strong>For General Photography</strong>: a7 base models offer balanced full-frame performance. The A7 IV remains highly rated by professionals as “one of the best cameras tested, period” and “the mirrorless hybrid to beat.”</p><p><strong>For Specialized Needs</strong>:</p><ul><li><strong>a7R V</strong> for maximum resolution (61MP) and landscape photography</li><li><strong>a7S III</strong> for low-light performance and video excellence</li><li><strong>a9 III</strong> for action and sports with revolutionary global shutter technology</li><li><strong>a1 II</strong> for no-compromise professional work with the latest AI autofocus</li></ul><p><strong>For Portability</strong>: RX series or a7C II provide premium features in compact bodies.</p><h3>Professional Insights and Market Position</h3><p><strong>Why Professionals Choose Sony in 2025:</strong></p><ul><li><strong>Lens Ecosystem</strong>: Over 70 native E-mount lenses plus excellent third-party support</li><li><strong>Autofocus Leadership</strong>: Industry-leading AI-powered subject detection</li><li><strong>Video Excellence</strong>: Only manufacturer offering both professional photography and cinema-grade video in the same bodies</li><li><strong>Innovation Pace</strong>: Consistent updates with meaningful improvements (ZV-E10 II, A1 II)</li><li><strong>Professional Support</strong>: Extensive adapter system even for legacy lenses</li></ul><p><strong>Value Considerations</strong>: While Sony cameras can seem expensive initially, the investment pays off through the extensive lens ecosystem, consistent firmware updates, and strong resale values. Professional photographers particularly appreciate that Sony doesn’t abandon older models — they continue supporting them with firmware updates for years.</p><h3>Current Best Buys for 2025</h3><p><strong>Best Overall Value</strong>: <strong>Sony A7 IV</strong> — Still the gold standard for hybrid photography/video despite being 4 years old</p><p><strong>Best Creator Camera</strong>: <strong>Sony ZV-E10 II</strong> — Fixes all predecessor issues and provides incredible value at $999</p><p><strong>Best Professional Investment</strong>: <strong>Sony A1 II</strong> — Latest flagship with cutting-edge AI autofocus and maximum versatility</p><p><strong>Best Beginner Full-Frame</strong>: <strong>Sony A7C II</strong> — Compact full-frame with excellent performance in a smaller body</p><p><strong>Best APS-C Hybrid</strong>: <strong>Sony A6700</strong> — Professional features in affordable APS-C package</p><h3>Conclusion</h3><p>Sony’s camera lineup covers every conceivable photography and videography need, from compact travel cameras to professional cinema equipment. The key is identifying your primary use case, budget, and whether you prioritize portability, image quality, video features, or specialized performance characteristics.</p><p>Remember that the camera body is just one part of the equation — lenses, accessories, and your skill development will ultimately determine the quality of your final images and videos.</p><p><em>This guide provides an overview of Sony’s current camera lineup. Prices and specifications may vary by region and change over time. Always check current specifications and pricing before making a purchase decision.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bUzdkAhJBWJwHtR8Sy_Y4A.png" /><figcaption>Different images from the <a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">Json Vong Video</a></figcaption></figure><h3>References</h3><ol><li><a href="https://www.youtube.com/watch?v=Y5V6feRnLkk">https://www.youtube.com/watch?v=Y5V6feRnLkk</a></li><li><a href="https://www.cameraegg.org/tag/sony-a1-ii/">https://www.cameraegg.org/tag/sony-a1-ii/</a></li><li><a href="https://alphauniverse.com/stories/sony-electronics-announces-the-nextgeneration-zve10-ii-and-e-pz-1650mm-f3-55-6-oss-ii-lens-kit/">https://alphauniverse.com/stories/sony-electronics-announces-the-nextgeneration-zve10-ii-and-e-pz-1650mm-f3-55-6-oss-ii-lens-kit/</a></li><li><a href="https://www.dpreview.com/reviews/sony-zv-e10-ii-vlogging-camera-review">https://www.dpreview.com/reviews/sony-zv-e10-ii-vlogging-camera-review</a></li><li><a href="https://www.rtings.com/camera/reviews/sony">https://www.rtings.com/camera/reviews/sony</a></li><li><a href="https://alphauniverse.com/stories/sony-electronics-announces-the-nextgeneration-zve10-ii-and-e-pz-1650mm-f3-55-6-oss-ii-lens-kit/">https://alphauniverse.com/stories/sony-electronics-announces-the-nextgeneration-zve10-ii-and-e-pz-1650mm-f3-55-6-oss-ii-lens-kit/</a></li><li><a href="https://www.sony.co.in/electronics/interchangeable-lens-cameras/zv-e10m2">https://www.sony.co.in/electronics/interchangeable-lens-cameras/zv-e10m2</a></li><li><a href="https://www.cameraegg.org/tag/sony-a1-ii/">https://www.cameraegg.org/tag/sony-a1-ii/</a></li><li><a href="https://www.digitalcameraworld.com/news/camera-rumors/4">https://www.digitalcameraworld.com/news/camera-rumors/4</a></li><li><a href="https://www.sonyalpharumors.com/sony-2025-camera-roadmap-a7v-fx3ii-and-100-megapixel-sony-a7rvi/">https://www.sonyalpharumors.com/sony-2025-camera-roadmap-a7v-fx3ii-and-100-megapixel-sony-a7rvi/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=faf8eaece081" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Cars Work: Complete Beginner’s Guide to Car Fundamentals]]></title>
            <link>https://medium.com/@mukulranjan/how-cars-work-complete-beginners-guide-to-car-fundamentals-6a1d5cd27a18?source=rss-abdbc773a16e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a1d5cd27a18</guid>
            <dc:creator><![CDATA[Mukul Ranjan]]></dc:creator>
            <pubDate>Sun, 04 Jan 2026 12:57:17 GMT</pubDate>
            <atom:updated>2026-01-07T16:31:57.488Z</atom:updated>
            <content:encoded><![CDATA[<h4>Cars 101: Learning how to talk car</h4><h3>Acknowledgement and Disclaimer</h3><p>Most of the content and images in this blog are from different youtube videos/blogs listed in the reference section. I am a no car expert but a beginner enthusiast like most of you, who likes learning about different stuffs related to cars. Some content of the blogs have been modified for grammatical correctness and reading flow with the help of Claude Sonnet 4.5.</p><h3>How Does a Car Work?</h3><p>There are three major components of a car: Engine, Transmission, and Drivetrain.</p><p><strong>The engine</strong> is the most important part of the car. It converts fuel into mechanical energy through controlled combustions. This then flows to the <strong>transmission</strong>, which acts as <strong>a mediator that adjusts the engine’s output to match driving conditions</strong>, whether you need gentle acceleration for city driving or maximum power for highways superspeed. Finally, the <strong>drivetrain</strong> takes this managed power and delivers it to the wheels that actually move the car forward or backward.</p><p>Think of it like a relay race where:</p><ul><li>the engine creates the energy, the transmission optimizes it for the situation, and the drivetrain carries it to where it’s needed.</li><li>Each component must work in harmony for smooth, efficient operation.</li></ul><p>This power flow, from combustion to wheels, forms the basis of how every car operates, regardless of whether it’s a compact economy car or a high-performance sports car.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jpU_ChDTJn8nOBAePCG4IA.png" /></figure><h3>Engines</h3><h4>Engine Volume (Displacement)</h4><p>Engine displacement refers to the total volume of all cylinders in an engine, typically measured in liters or cubic centimeters.</p><p><strong>Example:</strong> 3.0L V6 = 3 liter V-layout 6 cylinder, so each cylinder has 3L ÷ 6 = 0.5L or 500cc volume.</p><p>The larger the displacement, the more air and fuel the engine can burn per cycle, generally resulting in more power. However, larger displacement also typically means higher fuel consumption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBSndwzT-owVhZAeebVrYQ.png" /><figcaption>Source [1]</figcaption></figure><h4>Engine Cycle Types</h4><p><strong>4-Stroke Engines:</strong> It is the most common in cars today. The four strokes are:</p><ol><li>Intake (piston draws in air/fuel),</li><li>Compression (piston compresses mixture),</li><li>Power (combustion pushes piston down), and</li><li>Exhaust (piston pushes out burnt gases).</li></ol><p><strong>2-Stroke Engines:</strong> Complete the cycle in just two strokes, found mainly in <strong>motorcycles, chainsaws, and some small engines</strong>. They’re simpler but less fuel-efficient and produce more emissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*9ayBrxLsnJya-HYhRnw1UQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*q0NwdNe9XYB7q8NFwsy5KA.jpeg" /><figcaption>4-stroke and 2-stroke engines (Sources [5, 6])</figcaption></figure><h4>Fuel Delivery Systems</h4><p><strong>Carburetors:</strong> Older mechanical system that mixes air and fuel using vacuum and jets. Simple but less precise than modern systems.</p><p><strong>Fuel Injection:</strong> Modern electronic system that precisely controls fuel delivery through computer-controlled injectors. Offers better fuel economy, emissions, and performance than carburetors.</p><h4>Induction Types of Engines</h4><p><strong>Naturally Aspirated (NA):</strong> The movement of pistons in cylinders draws fresh air through atmospheric pressure alone, providing oxygen for combustion. This is the most common and simplest form of induction.</p><p><strong>Better ways to draw more power:</strong></p><ol><li><strong>Turbocharger</strong> Uses exhaust gases to spin a turbine which compresses incoming air and forces it into the cylinders. This allows more air (and therefore more fuel) to be burned, significantly increasing power output. Turbochargers can provide substantial power gains but may have some lag at low RPMs.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xp3iBUvUXjZE6cUDaEmWNg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqGebTxgON7Isbe5mHR0jQ.png" /><figcaption>Turbocharger (Sources: [1, 3])</figcaption></figure><p><strong>2. Supercharger</strong> Mechanically driven by the engine’s crankshaft to compress air before it enters the cylinders. Unlike turbochargers, superchargers provide immediate power response since they’re directly connected to the engine. You can use a smaller displacement engine to produce the same power as a larger naturally aspirated engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ipoPAC8DxRFzh3oy7h45lA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EYR9tr2BhI--7D-r-3MaDw.png" /><figcaption>Supercharger (Sources: [1, 4])</figcaption></figure><p><strong>Types of Superchargers:</strong></p><ul><li><strong>Roots Supercharger:</strong> Uses two intermeshing rotors to move air, provides immediate boost but less efficient at high RPMs</li><li><strong>Twin-Screw Supercharger:</strong> Uses helical rotors that compress air internally, more efficient than Roots type</li><li><strong>Centrifugal Supercharger:</strong> Uses an impeller like a turbocharger but mechanically driven, most efficient at higher RPMs</li><li><strong>Electric Supercharger: </strong>An electric supercharger uses an electric motor to compress the intake air for an internal combustion engine, increasing its power. Unlike a mechanical supercharger, which is driven by the <strong>engine’s crankshaft, an electric supercharger operates independently, providing instant boost on demand</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*x3I7_5uiPKFoBA7T.jpg" /><figcaption>Types of Supercharger (<a href="https://www.youtube.com/watch?v=iq8KWnFGNf8">source</a>)</figcaption></figure><h4>Engine Layouts</h4><p><strong>1. Inline Layout (I4, I6)</strong> Cylinders arranged in a straight line. Most common configuration due to simplicity and cost-effectiveness. Inline-4 engines are standard in most economy cars, while inline-6 engines offer smooth operation and good power.</p><p><strong>2. V-Layout (V6, V8, V10, V12)</strong> Cylinders arranged in a “V” shape with two banks of cylinders. This design is more compact than inline engines with the same number of cylinders. Popular in performance cars and trucks. BMW, Ford, Chevrolet, and many other manufacturers use this layout.</p><p><strong>3. Flat/Boxer Layout (H4, H6)</strong> Cylinders arranged horizontally in opposing banks, creating a flat profile. Subaru and Porsche are famous for using this layout. Benefits include lower center of gravity and better balance, but they’re wider and more complex to service.</p><p><strong>4. W-Layout (W8, W12, W16)</strong> Multiple banks of cylinders arranged in a “W” configuration, essentially two V-engines joined together. Bentley, Bugatti, and some Volkswagen Group vehicles use this complex but compact layout for high-cylinder-count engines.</p><p><strong>5. VR Layout</strong> A narrow-angle V-engine where cylinders are so close together they can share a single cylinder head. Volkswagen developed this layout to fit more cylinders in tight engine bays while keeping costs manageable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VBuR3HvldIMZfsidDkXELg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*peOXSJ5VH6j4cshqydi_Mg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*31nUt6fgSIFHAOGNSiU9XQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OQWSgvves0NNYhu8FaWpZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0KzHoomaiXjPmhsHkhe3VA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fUGh6HvNqnUbwE5CgMmU5Q.png" /><figcaption>Different Engine Layouts (<a href="https://www.youtube.com/watch?v=ZkMqRx8VP6o">Source</a>)</figcaption></figure><h4>Engine Cooling Systems</h4><p><strong>Air-Cooled:</strong> Uses airflow over fins to dissipate heat. Simple but less effective, mainly found in older vehicles and motorcycles.</p><p><strong>Liquid-Cooled:</strong> Uses coolant circulated through the engine block and radiator. More effective temperature control and allows for more consistent engine temperatures.</p><h4>Engine Management</h4><p><strong>ECU (Engine Control Unit):</strong> The computer brain that controls fuel injection, ignition timing, emissions systems, and other engine functions. Modern engines rely heavily on electronic management for optimal performance and efficiency.</p><p><strong>Sensors:</strong> Various sensors (oxygen, temperature, pressure, etc.) provide real-time data to the ECU to optimize engine operation under different conditions.</p><h3>Transmission (Clutch)</h3><p><strong>1. Manual Transmission</strong> Uses a clutch pedal to manually disconnect the engine from the transmission, allowing the driver to shift gears by hand. The clutch is a friction disc that engages and disengages the engine’s power to the transmission. Manual transmissions offer more driver control, better fuel economy, and are generally less expensive to maintain, but require more skill to operate smoothly.</p><p><strong>2. Automatic Transmission</strong> Uses a torque converter filled with fluid instead of a mechanical clutch to transfer power from the engine to the transmission. The transmission automatically selects the appropriate gear based on speed, throttle position, and other factors. While more convenient and easier to drive, automatic transmissions are typically more complex, expensive, and slightly less fuel-efficient than manual transmissions.</p><p><strong>3. CVT (Continuously Variable Transmission)</strong> Uses belts and pulleys to provide an infinite number of gear ratios within its range. Offers smooth acceleration and optimal fuel economy but lacks the engaging feel many drivers prefer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/0*j-dZvUcQ0EGJH8QL" /><figcaption>Different Types of Transmission (<a href="https://go4trans.com/technical-transmission-general-articles/possible-consequences-of-using-atf-in-cvt-transmission/">source</a>)</figcaption></figure><h3>Drivetrain</h3><p><strong>AWD (All Wheel Drive)</strong> Sends power to all four wheels simultaneously, providing maximum traction in various conditions. More expensive due to additional components and complexity, but offers superior handling in snow, rain, and off-road conditions. Popular in SUVs and performance vehicles.</p><p><strong>RWD (Rear Wheel Drive)</strong> Sends power to the rear wheels only. Common in performance cars, trucks, and luxury vehicles because it provides better weight distribution, handling balance, and allows for more aggressive acceleration without front-wheel torque steer.</p><p><strong>FWD (Front Wheel Drive)</strong> Sends power to the front wheels only. The cheapest and most space-efficient layout, commonly found in economy cars like most Honda Civics. Offers good fuel economy and traction in light snow, but can suffer from torque steer and understeer during aggressive driving.</p><p><strong>4WD (Four Wheel Drive)</strong> Similar to AWD but typically part-time, meaning it can be engaged or disengaged by the driver. More common in trucks and off-road vehicles, designed for challenging terrain rather than on-road performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QGSnoDKYfeLxOpu0sNxGug.png" /><figcaption>Drivetrain Configuration (<a href="https://www.youtube.com/watch?v=ZkMqRx8VP6o">Source</a>)</figcaption></figure><h3>Differentials</h3><p><strong>Open Differential</strong> Power takes the path of least resistance, meaning if one wheel loses traction (like on ice), all power goes to that spinning wheel while the other wheel gets no power. This is the most basic and cheapest differential type.</p><p><strong>Limited Slip Differential (LSD)</strong> Ensures both tires receive more equal power distribution by limiting the speed difference between wheels. When one wheel starts to slip, the LSD transfers power to the wheel with better traction, improving acceleration and handling.</p><p><strong>Locking Differential</strong> Can completely lock both wheels together to rotate at the same speed, providing maximum traction for off-road or extreme conditions. Common in serious off-road vehicles and some performance cars.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Ne-h6lhsaEcbZvp34yEl8Q.jpeg" /><figcaption>Different Types of Differential (<a href="https://www.differential-lsd.com/open-differential-vs-limited-slip-vs-locking-differential-whats-the-difference/">source</a>)</figcaption></figure><h3>Brakes</h3><p><strong>Disc Brakes:</strong> Use brake pads that squeeze against a rotor (disc) attached to the wheel. More effective at heat dissipation and stopping power, commonly found on front wheels and performance vehicles.</p><p><strong>Drum Brakes:</strong> Use brake shoes that press outward against a drum. Less expensive but not as effective as disc brakes, typically found on rear wheels of economy cars.</p><p><strong>ABS (Anti-lock Braking System):</strong> Prevents wheels from locking up during hard braking by rapidly pulsing the brakes, allowing the driver to maintain steering control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C5zQbxddO1LTOf7i.jpg" /><figcaption>Types of Brakes (<a href="https://www.theengineeringchoice.com/what-is-brake/">source</a>)</figcaption></figure><h3>Control Systems</h3><p><strong>Traction Control</strong> Prevents wheel spin during acceleration by reducing engine power or applying brakes to spinning wheels. Helps maintain grip on slippery surfaces like wet pavement, snow, or loose gravel.</p><p><strong>Stability Control (ESC: Electronic Stability Control)</strong> Detects when the car is beginning to slide or lose control and automatically applies individual brakes and reduces engine power to help the driver maintain control. Since 2012, it has been required on all new cars sold in the United States due to its effectiveness in preventing crashes.</p><p><strong>ABS (Anti-lock Braking System)</strong> Prevents wheels from locking during hard braking, allowing the driver to maintain steering control while stopping. Works by rapidly pulsing the brakes when wheel lockup is detected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QtkFE3b-VzlobT98u06ybA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*QzsARpDu1vYTAqmtnxAXPA.avif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*llyP57h9KRHPAiXiJROVPw.jpeg" /><figcaption>Different Types of Control System in a car (Sources, [7, 8, 9])</figcaption></figure><h3>Car Types</h3><p><strong>Traditional Body Styles:</strong></p><ul><li><strong>Sedan:</strong> Four-door passenger car with separate trunk compartment</li><li><strong>Hatchback:</strong> Compact car with rear door that opens upward to access cargo area</li><li><strong>SUV:</strong> Sport Utility Vehicle, larger and higher-riding with truck-based or unibody construction</li><li><strong>Coupe:</strong> Two-door car typically focused on style and performance</li><li><strong>Convertible:</strong> Car with removable or retractable roof</li><li><strong>Wagon:</strong> Extended sedan with larger cargo area but lower profile than SUVs</li><li><strong>Pickup Truck:</strong> Open cargo bed with separate cab for passengers</li></ul><p><strong>Performance Categories (Common Builds):</strong></p><p><strong>1. Econobox</strong> Basic economy cars designed for maximum fuel efficiency and affordability. Typically feature small engines (under 2.0L), front-wheel drive, and minimal luxury features. Examples include base model Honda Civic, Toyota Corolla, or Nissan Versa. Built for daily commuting with low running costs.</p><p><strong>2. Sport</strong> Performance-oriented vehicles that balance everyday usability with enhanced driving dynamics. Feature more powerful engines, sport-tuned suspension, and aggressive styling. Examples include Honda Civic Si, Subaru WRX, or BMW 3 Series. Designed for enthusiasts who want excitement without extreme compromises.</p><p><strong>3. Super</strong> High-performance supercars with exotic styling, powerful engines (typically 500+ horsepower), and premium materials. Examples include Ferrari 488, Lamborghini Huracán, or McLaren 570S. Built for maximum performance and exclusivity with price tags typically starting around $200,000+.</p><p><strong>4. Hyper</strong> The ultimate expression of automotive engineering, featuring cutting-edge technology, hybrid powertrains, and extreme performance figures (1000+ horsepower). Examples include Bugatti Chiron, Koenigsegg Jesko, or McLaren P1. Limited production vehicles often costing $1 million or more.</p><p><strong>5. Muscle</strong> American-style performance cars emphasizing large displacement engines and straight-line acceleration. Feature powerful V8 engines, rear-wheel drive, and aggressive styling. Examples include Dodge Challenger Hellcat, Chevrolet Camaro ZL1, or Ford Mustang GT. Built for maximum horsepower and classic American performance character.</p><p><strong>6. Luxury</strong> Premium vehicles focusing on comfort, refinement, and high-end materials rather than pure performance. Feature advanced technology, superior interior quality, and smooth ride characteristics. Examples include Mercedes S-Class, BMW 7 Series, or Lexus LS. Built for those who prioritize comfort and prestige over sportiness.</p><h3>Exhaust System</h3><p><strong>X-Pipe:</strong> Connects left and right exhaust banks with an X-shaped crossover, providing better high-RPM power and a sharper exhaust note.</p><p><strong>H-Pipe:</strong> Uses an H-shaped connection between exhaust banks, offering better low-end torque and a deeper exhaust sound.</p><p><strong>Y-Pipe:</strong> Merges two exhaust pipes into one, commonly used where space is limited or in single-exit exhaust systems.</p><p><strong>Straight-Through Design:</strong> No crossover, maintains separate exhaust paths for each bank of cylinders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HMbyDmg7-z-vysYduURUug.png" /><figcaption>Different Types of Exhaust Pipes</figcaption></figure><h3>CAT (Catalytic Converter)</h3><p>A crucial emissions control device that uses precious metals (platinum, palladium, rhodium) to convert harmful exhaust gases into less toxic substances. It transforms carbon monoxide into carbon dioxide, nitrogen oxides into nitrogen and oxygen, and unburned hydrocarbons into carbon dioxide and water. Catalytic converters are legally required on all street vehicles and removing them results in failed emissions tests and significant fines. They operate most efficiently when heated to proper operating temperature, which is why cars pollute more during cold starts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0CvFq1wcy-tARhDIczwntw.png" /></figure><p>This ends our discussion about Cars 101.</p><p>The following appendix is based on different youtube videos/blogs listed in the reference section, and google images.</p><h3>Appendix</h3><h4>Different Car Terms</h4><p><strong>WOT (Wide Open Throttle)</strong>: This occurs when the accelerator pedal is pressed completely to the floor, allowing maximum airflow into the engine. It’s the condition for achieving peak power output from the engine.</p><p><strong>HEMI</strong>: A hemispherical combustion chamber design that allows for larger valves and better airflow compared to traditional flat-head designs. This configuration typically produces more power and efficiency, though it’s more expensive to manufacture.</p><p><strong>BOV (Blow Off Valve)</strong>: A pressure relief valve that releases excess boost pressure from turbocharged engines when the throttle closes suddenly. It prevents compressor surge and makes the characteristic “pssh” sound associated with turbocharged cars.</p><p><strong>Double Clutching</strong>: A manual transmission technique where you press the clutch twice during a gear change, once to take it out of gear, rev-match the engine, then clutch again to engage the new gear. This method helps synchronize gear speeds for smoother shifting, especially in older transmissions without synchros.</p><p><strong>Muffler</strong>: An exhaust component designed to reduce noise from the engine by using chambers and baffles to cancel out sound waves. It’s typically located at the rear of the exhaust system and can significantly affect both sound and back-pressure.</p><p><strong>Resonator</strong>: An exhaust component that targets specific frequency ranges to eliminate particular sounds or drone. Unlike mufflers that reduce overall volume, resonators fine-tune the exhaust note by canceling unwanted frequencies.</p><p><strong>Downpipe</strong>: The exhaust pipe section that connects directly to the turbocharger or exhaust manifold. Upgrading to a larger diameter or less restrictive downpipe can improve turbo response and power output.</p><p><strong>Unibody</strong>: A vehicle construction method where the body and frame are integrated into a single welded structure. This design is lighter and more rigid than traditional body-on-frame construction, commonly used in cars and crossovers.</p><p><strong>Ladder Frame</strong>: A truck chassis design that resembles a ladder with two parallel rails connected by cross-members. This robust construction method provides excellent strength for towing and off-road use but is heavier than unibody designs.</p><p><strong>Carburetors and Fuel Injection</strong>: Two different fuel delivery systems, carburetors use vacuum and mechanical components to mix air and fuel, while fuel injection uses electronic controls and pressurized injectors. Fuel injection is more precise and efficient but carburetors are simpler and easier to tune.</p><p><strong>CEL (Check Engine Light)</strong>: A dashboard warning light that illuminates when the engine control unit detects a problem with emissions or engine performance. The underlying issue can range from a loose gas cap to serious engine problems requiring immediate attention.</p><p><strong>FWD, RWD, 4WD, AWD</strong>: Different drivetrain configurations, Front-Wheel Drive sends power to front wheels only, Rear-Wheel Drive to rear wheels only, 4-Wheel Drive can be engaged for all four wheels, and All-Wheel Drive automatically distributes power to all wheels. Each system has distinct advantages for different driving conditions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QGSnoDKYfeLxOpu0sNxGug.png" /><figcaption>Drivetrain Configuration (<a href="https://www.youtube.com/watch?v=ZkMqRx8VP6o">Source</a>)</figcaption></figure><p><strong>Torque</strong>: The rotational force produced by an engine, measured in pound-feet or Newton-meters. High torque provides strong acceleration and pulling power, especially important for towing or off-road driving.</p><p><strong>I4, I6, V8</strong>: Engine configurations describing cylinder arrangement, Inline-4 has four cylinders in a row, Inline-6 has six in a row, and V8 has eight cylinders arranged in a V-shape. Each configuration offers different characteristics in terms of power, smoothness, and packaging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*31nUt6fgSIFHAOGNSiU9XQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*peOXSJ5VH6j4cshqydi_Mg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VBuR3HvldIMZfsidDkXELg.png" /><figcaption>Types of Engine (<a href="https://www.youtube.com/watch?v=ZkMqRx8VP6o">Source</a>)</figcaption></figure><p><strong>F-R/R-R/F-A (Front-RWD/Front-AWD/Rear-RWD)</strong>: Vehicle layout designations where F-F means front-mounted engine driving front wheels, and F-R means front-mounted engine driving rear wheels. These layouts affect weight distribution, handling characteristics, and interior space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3syEbUTDLlnlRGQixgdMpA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xeeK-GFvizi39mSM72TOIg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7FQfjJYFp_auKEHL5d8wTQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=ZkMqRx8VP6o">Source</a></figcaption></figure><p><strong>Bondo</strong>: A brand name for auto body filler, a putty-like substance used to repair dents and rust holes in car bodies. While useful for repairs, excessive Bondo use can indicate poor bodywork or hidden damage.</p><p><strong>N/A (Naturally Aspirated)</strong>: An engine that draws air into the cylinders using only atmospheric pressure and vacuum created by the pistons. Unlike turbocharged or supercharged engines, N/A engines don’t use forced induction to increase power output.</p><p><strong>Turbo/Supercharger</strong>: Both are forced induction systems that compress air entering the engine to increase power output. Turbochargers use exhaust gases to spin a turbine, while superchargers are mechanically driven by the engine itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqGebTxgON7Isbe5mHR0jQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EYR9tr2BhI--7D-r-3MaDw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xp3iBUvUXjZE6cUDaEmWNg.png" /><figcaption>Turbo and Supercharger Engine (Sources: 1, 3)</figcaption></figure><p><strong>Spinny Boi</strong>: Internet slang for a turbocharger, referring to the high-speed rotation of the turbine and compressor wheels. This playful term has become popular in car enthusiast communities and memes.</p><p><strong>Roots SC (Roots Supercharger)</strong>: A positive displacement supercharger type that uses two intermeshing rotors to move air. It provides immediate boost response but becomes less efficient at higher RPMs compared to other supercharger designs.</p><p><strong>Twin Screw SC</strong>: A supercharger design using two helical rotors that compress air internally as it moves through the housing. This design is more efficient than Roots-type superchargers and produces less heat.</p><p><strong>Centrifugal SC</strong>: A supercharger that uses an impeller to accelerate air outward, similar to a turbocharger compressor but mechanically driven. It’s most efficient at higher RPMs and provides a boost curve similar to turbochargers.</p><p><strong>X-Pipe or H-Pipe</strong>: Exhaust system crossover designs that connect the left and right exhaust banks to balance pressure and improve flow. X-pipes typically provide better high-RPM power while H-pipes offer more low-end torque and a different exhaust note.</p><p><strong>CAT (Catalytic Converter)</strong>: An emissions control device that uses precious metals to convert harmful exhaust gases into less toxic substances. It’s legally required on street vehicles and removing it can result in significant fines and failed emissions tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0CvFq1wcy-tARhDIczwntw.png" /><figcaption>Source [3]</figcaption></figure><p><strong>Muffler or CAT Delete</strong>: The removal of mufflers or catalytic converters from the exhaust system to reduce restriction and increase sound. While this may improve performance, it often makes vehicles too loud for street use and illegal in many areas.</p><p><strong>Sleeper</strong>: A vehicle that appears stock or unassuming but has significant performance modifications hidden underneath. These cars are designed to surprise unsuspecting opponents in street racing scenarios.</p><p><strong>Solid Axle or Live Axle</strong>: A rear suspension design where both wheels are connected by a solid beam that also contains the differential. While simpler and stronger than independent suspension, it provides less ride comfort and handling precision.</p><p><strong>Torsion Beam</strong>: A semi-independent rear suspension design using a beam that twists to allow some independent wheel movement. It’s commonly found in compact cars as a compromise between cost, space efficiency, and ride quality.</p><p><strong>MacPherson Strut</strong>: A popular front suspension design that combines the shock absorber and spring into a single unit mounted to the wheel hub. It’s space-efficient and cost-effective but has some geometric limitations compared to more complex designs.</p><p><strong>Double Wishbone</strong>: A suspension design using two A-shaped control arms (wishbones) to locate the wheel, allowing precise control of wheel geometry. This design offers superior handling characteristics but requires more space and complexity than MacPherson struts.</p><p><strong>Multi-Link</strong>: An advanced suspension design using multiple links and arms to control wheel movement with great precision. It offers the best combination of ride quality and handling but is more complex and expensive than simpler designs.</p><p><strong>Camber</strong>: The angle of the wheel when viewed from the front or rear of the vehicle, negative camber tilts the top of the wheel inward. Proper camber settings optimize tire contact patch for cornering performance, while excessive camber can cause uneven tire wear.</p><p><strong>AFR (Air-Fuel Ratio)</strong>: The ratio of air to fuel in the combustion mixture, typically expressed as a number like 14.7:1 for gasoline engines. Maintaining proper AFR is crucial for engine performance, fuel economy, and emissions control.</p><p><strong>BHP (Brake Horsepower)</strong>: Engine power measured at the flywheel before any drivetrain losses occur. This represents the raw power output of the engine itself, typically higher than wheel horsepower due to transmission and differential losses.</p><p><strong>WHP (Wheel Horsepower)</strong>: Power measured at the wheels after accounting for drivetrain losses through the transmission, differential, and other components. This represents the actual power available to move the vehicle and is typically 10–20% lower than BHP.</p><p><strong>Limp Mode</strong>: A protective engine management strategy that severely limits power and RPM when the ECU detects a potentially damaging condition. The vehicle remains drivable but at reduced performance to prevent further engine damage until repairs can be made.</p><p><strong>Heel-Toe</strong>: An advanced driving technique used during braking and downshifting where the driver uses different parts of their right foot to operate both brake and throttle pedals simultaneously. This allows for smooth rev-matching while maintaining braking pressure through corners.</p><p><strong>Rev Hang</strong>: A characteristic of some modern engines where RPM takes longer to drop after lifting off the throttle, often implemented for emissions reasons. While it can help reduce emissions during gear changes, many drivers find it interferes with smooth shifting.</p><p><strong>Money Shift</strong>: An accidental gear change that results in severe engine damage, typically from shifting into a lower gear at high RPM. The term refers to the expensive repair costs that result from this mistake, often requiring complete engine rebuilds.</p><p><strong>Final Drive</strong>: The last gear reduction in the drivetrain, typically located in the differential, that determines the overall gear ratio. Changing the final drive ratio affects acceleration, top speed, and fuel economy characteristics of the vehicle.</p><p><strong>LSD (Limited Slip Differential)</strong>: A differential that can limit the speed difference between two wheels, helping to transfer power to the wheel with better traction. This improves acceleration and handling compared to open differentials, especially in performance and off-road applications.</p><p><strong>CVT (Continuously Variable Transmission)</strong>: A transmission that can seamlessly change through an infinite number of gear ratios within its range, typically using belts and pulleys. While CVTs offer smooth operation and good fuel economy, they often lack the engaging feel preferred by driving enthusiasts.</p><p><strong>Blinker Fluid</strong>: A humorous non-existent automotive fluid used as a joke to trick inexperienced car owners or new mechanics. Like “muffler bearings,” it’s part of automotive culture’s tradition of harmless pranks on newcomers.</p><p><strong>Transaxle</strong>: A transmission and differential combined into a single unit, commonly found in front-wheel-drive and some rear-engine vehicles. This design saves space and weight compared to separate transmission and differential units.</p><p><strong>Racing from a Dig</strong>: Street racing terminology for starting a race from a complete stop, as opposed to rolling starts. This type of racing emphasizes launch technique and low-end power rather than top-end speed.</p><p><strong>Roll Racing</strong>: Street racing that begins while both vehicles are already moving at a predetermined speed, typically around 40–60 mph. This format reduces the importance of launch technique and focuses more on aerodynamics and high-RPM power.</p><p><strong>Ground Effect</strong>: Aerodynamic phenomenon where air flowing under a vehicle creates downforce, improving handling and stability at high speeds. Modern race cars and some supercars use sophisticated underbody designs to maximize this effect.</p><p><strong>Firing Order</strong>: The sequence in which engine cylinders fire during the combustion cycle, designed to minimize vibration and optimize power delivery. Different firing orders can affect engine smoothness, sound characteristics, and mechanical stress patterns.</p><p><strong>Rod Knock</strong>: A serious engine problem caused by excessive clearance between connecting rod bearings and the crankshaft, creating a distinctive knocking sound. This condition typically requires immediate engine shutdown and complete rebuild, as continued operation will cause catastrophic damage.</p><h3>References</h3><ol><li><a href="https://www.youtube.com/watch?v=9ngnCrBKWZA">https://www.youtube.com/watch?v=9ngnCrBKWZA</a></li><li><a href="https://www.youtube.com/watch?v=Uzk-po3mAHA">https://www.youtube.com/watch?v=Uzk-po3mAHA</a></li><li><a href="https://www.youtube.com/watch?v=ZkMqRx8VP6o">https://www.youtube.com/watch?v=ZkMqRx8VP6o</a></li><li><a href="https://www.thedrive.com/cars-101/40371/how-does-a-supercharger-work">https://www.thedrive.com/cars-101/40371/how-does-a-supercharger-work</a></li><li><a href="https://www.britannica.com/technology/gasoline-engine/Two-stroke-cycle">https://www.britannica.com/technology/gasoline-engine/Two-stroke-cycle</a></li><li><a href="https://www.britannica.com/technology/four-stroke-cycle">https://www.britannica.com/technology/four-stroke-cycle</a></li><li><a href="https://www.kbb.com/car-advice/electronic-stability-control/">https://www.kbb.com/car-advice/electronic-stability-control/</a></li><li><a href="https://www.sellanycar.com/guides/understanding-traction-control-system-capabilities/">https://www.sellanycar.com/guides/understanding-traction-control-system-capabilities/</a></li><li><a href="https://www.motor-works.com/blog/inside-the-tech--anti-lock-braking-system--abs-">https://www.motor-works.com/blog/inside-the-tech--anti-lock-braking-system--abs-</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a1d5cd27a18" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Perplexity]]></title>
            <link>https://medium.com/@mukulranjan/understanding-perplexity-9df50dfa8827?source=rss-abdbc773a16e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9df50dfa8827</guid>
            <dc:creator><![CDATA[Mukul Ranjan]]></dc:creator>
            <pubDate>Sat, 09 Aug 2025 23:24:05 GMT</pubDate>
            <atom:updated>2025-11-26T18:57:24.769Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog is extracted from my previous blog (<a href="https://medium.com/@mukulranjan/combining-pruning-and-knowledge-distillation-1b5b7407c8c8">Combining Pruning and Knowledge Distillation</a>).</p><blockquote><strong>Perplexity</strong> measures how <strong><em>perplexed</em></strong><em> 😕 </em>a language model is by a sequence of token.</blockquote><p>So, if you know how does generation works in a language model then you must be aware of the fact that, the model assign the highest (in most of the cases, top-k and top-p sampling is the topic of some other discussion) probability to the actual next token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*IOJv2Feaa-YZg-00tSgu4A.png" /><figcaption>Equation for the perplexity.</figcaption></figure><p>Intuitively, since the model assigns high probability to the actual next token, the negative log probability part of the above equation will be low, leading to low perplexity. Hence <em>the better the model is the lower is it’s perplexity</em>. So if a model is worse, it will predict some random tokens, and hence negative log probability part of the above equation will be high, leading to <strong>high perplexity</strong> and hence model is <strong>more</strong> <strong>surprised</strong> by the sequence of tokens.</p><p>Now I will try to explain it more formally with the help of the examples from <a href="https://medium.com/nlplanet/two-minutes-nlp-perplexity-explained-with-simple-probabilities-6cdc46884584">this</a>[1] short blog.</p><p>Perplexity quantifies how well the model predicts a given sequence of words or tokens. Lower perplexity indicates better prediction. A perplexity of k means the model is as confused as if it had to <em>choose uniformly and independently from k possibilities for each word</em>. You can have a look at <a href="https://medium.com/nlplanet/two-minutes-nlp-perplexity-explained-with-simple-probabilities-6cdc46884584">this</a>[1] short blog for understanding it in detail. I have used that blog to write this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jFGW-HuyqW77w2o8ZwyaTw.png" /><figcaption>An example of a toy language model taken from [1] for understanding perplexity in detail.</figcaption></figure><p>Consider a very small language model with probability distribution shown above and with a vocabulary of only six words: “a”,<em> “the”</em>,<em> “red”</em>, <em>“fox”</em>,<em> “dog”</em>,<em> </em>and <em>“.” . W</em>e want to compute the probability of the sentence W, <em>i.e</em> “<em>a red fox.</em>” using that language model.</p><p>p(W) = p(w1, w2, w3, w4)</p><pre>p(“a red fox.”) = p(“a”, “red”, “fox”, “.”) = p(“a”) * p(“red” | “a”) * p(“fox” | “a red”) * p(“.” | “a red fox”)</pre><p>From the above image, we can see the distribution predicted by our language model. p(“a”)=0.4; p(“red” | “a”)=0.27; p(“fox” | “a red”)=0.55; p(“.” | “a red fox”) = 0.79.</p><p>Thus, p(“a red fox.”) = 0.4 * 0.27 * 0.55 * 0.79 = 0.0469. Since p(W) is the product of probabilities it’s value will become very less for larger sentence. In order to solve this we can normalize them using geometric mean. So normalized probability of p(W) = p(“a red fox.”) can be written as pₙₒᵣₘ = 𝑃(𝑊)^{1/N}, where N is the number of words in sentence. For our example we can pₙₒᵣₘ = (0.0469)^{1/4} = 0.465. This can be used to compare probabilities of sentence. The higher is this number the better is the language model</p><blockquote>Perplexity is defined as the reciprocal of this number. So, perplexity can be defined as the reciprocal of the geometric mean of product of the probabilities of all words in sentence.</blockquote><p>More formally,</p><p>Perplexity(W) = PPL(W) = 1/pₙₒᵣₘ = p(𝑊)^{-1/N}</p><p>Hence for our example, perplexity is 1/0.465 = 2.15. Perplexity value of 1 indicate and ideal language model. Now let’s try to prove this statement stated above:</p><blockquote>A perplexity of k means the model is as confused as if it had to choose uniformly and independently from k possibilities for each word.</blockquote><p>Since our example language model can predict only 6 words, hence the probability of each word is 1/6. Hence, p(W) = p(“a red fox.”) = (1/6) ^ 4 = 0.00077. PPL(W) = p(W)^(-1/N) = ((1/6)⁴)^(-1/4) = 6.</p><p>In the more general form, from a set of k words if we are predicting the probability of W, which contains N words. Then, PPL(W) = ((1/k)^N)^(-1/N) = k</p><h4>Entropy and Perplexity</h4><p>From this equation we can figure out the relationship between the Shannon’s entropy and perplexity which is more widely accepted definition of perplexity. Note that <a href="https://en.wikipedia.org/wiki/Entropy_(information_theory)">Shannon entropy</a> of a language model for sentence W is given by the H(W) = 𝔼[-log p(W)]. We can prove that,</p><blockquote>Perplexity = 2ᴴ⁽ᵂ⁾</blockquote><p>H(W) = -1/N*∑(log₂ p(xᵢ)) where xᵢ denotes the probability of the prediction of 𝑖ᵗʰ token. For example in the previous example let’s consider:</p><p>p(“a”) = p( 𝑥₁)</p><p>p(“red” | “a”) = p(𝑥₂)</p><p>p(“fox” | “a red”) = p(𝑥₃)</p><p>p(“.” | “a red fox”) = p(𝑥₄)</p><p>Then, H(W) can be written as ∑log₂ {p(xᵢ)^(-1/N)} = ∏(p(xᵢ) )^(-1/N) = P(W)^{-1/N} = Perplexity</p><p>Note that ∏(p(xᵢ) )^(-1/N) is the inverse of the geometric mean predicted probabilities of all words in W, which is perplexity.</p><h3>Calculating Perplexity</h3><p>Here is a simple code to calculate perplexity on a sample of openwebtext dataset.</p><pre>import torch<br>from transformers import AutoModelForCausalLM, AutoTokenizer<br>from datasets import load_dataset<br>from tqdm import tqdm<br>import numpy as np<br><br># Configuration<br>MODEL_NAME = &quot;Qwen/Qwen3-14B&quot;  <br>NUM_SAMPLES = 300<br>MAX_LENGTH = 1024 <br>DEVICE = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br><br>print(&quot;Loading model and tokenizer...&quot;)<br>tokenizer = AutoTokenizer.from_pretrained(<br>    MODEL_NAME, <br>    trust_remote_code=True,<br>    padding_side=&#39;right&#39;<br>)<br>model = AutoModelForCausalLM.from_pretrained(<br>    MODEL_NAME,<br>    torch_dtype=torch.bfloat16,  <br>    device_map=&quot;auto&quot;,<br>    trust_remote_code=True<br>)<br>model.eval()<br><br>#load dataset<br>print(&quot;Loading OpenWebText dataset...&quot;)<br>dataset = load_dataset(<br>    &quot;Skylion007/openwebtext&quot;,  <br>    split=&quot;train&quot;,<br>    streaming=True,  # More efficient for large datasets<br>    trust_remote_code=True<br>)<br><br># Calculate perplexity<br>total_loss = 0.0<br>total_tokens = 0<br>samples_processed = 0<br><br>print(f&quot;Calculating perplexity on {NUM_SAMPLES} samples...&quot;)<br><br>for sample in tqdm(dataset, total=NUM_SAMPLES):<br>    if samples_processed &gt;= NUM_SAMPLES:<br>        break<br>    <br>    text = sample[&quot;text&quot;]<br>    <br>    # Tokenize<br>    inputs = tokenizer(<br>        text,<br>        return_tensors=&quot;pt&quot;,<br>        max_length=MAX_LENGTH,<br>        truncation=True,<br>        padding=False<br>    )<br>    <br>    input_ids = inputs.input_ids.to(DEVICE)<br>    <br>    <br>    with torch.no_grad():<br>        # Forward pass - model automatically computes loss when labels are provided<br>        outputs = model(input_ids, labels=input_ids)<br>        loss = outputs.loss<br>        <br>        # The loss is already averaged over tokens in the sequence<br>        num_tokens = input_ids.shape[1]<br>        total_loss += loss.item() * num_tokens<br>        total_tokens += num_tokens<br>    <br>    samples_processed += 1<br><br># Calculate final perplexity<br>average_loss = total_loss / total_tokens<br>perplexity = np.exp(average_loss)<br><br>print(f&quot;\n{&#39;=&#39;*50}&quot;)<br>print(f&quot;Results on {samples_processed} samples:&quot;)<br>print(f&quot;{&#39;=&#39;*50}&quot;)<br>print(f&quot;Total tokens evaluated: {total_tokens:,}&quot;)<br>print(f&quot;Average loss (cross-entropy): {average_loss:.4f}&quot;)<br>print(f&quot;Perplexity: {perplexity:.2f}&quot;)<br>print(f&quot;{&#39;=&#39;*50}&quot;)</pre><h3>References</h3><ol><li><a href="https://medium.com/nlplanet/two-minutes-nlp-perplexity-explained-with-simple-probabilities-6cdc46884584">https://medium.com/nlplanet/two-minutes-nlp-perplexity-explained-with-simple-probabilities-6cdc46884584</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9df50dfa8827" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the Pruning of Large Language Models]]></title>
            <link>https://medium.com/@mukulranjan/all-about-pruning-and-knowledge-distillation-for-llms-edc705b48916?source=rss-abdbc773a16e------2</link>
            <guid isPermaLink="false">https://medium.com/p/edc705b48916</guid>
            <dc:creator><![CDATA[Mukul Ranjan]]></dc:creator>
            <pubDate>Mon, 30 Jun 2025 19:29:07 GMT</pubDate>
            <atom:updated>2026-02-17T00:01:48.172Z</atom:updated>
            <content:encoded><![CDATA[<h4>Efficient LLM Part 1: Fundamental of Pruning</h4><h4>Introduction</h4><p>Large Language Models (LLMs) like GPT-4, Claude, Gemini, and their open-source counterparts such as LLaMA, Deepseek, Qwen, GLM, and Mistral have revolutionized the domain of artificial intelligence. They have the ability to perform a wide range of tasks from writing poetry and solving complex problems to engage in conversations that feel remarkably human. However, they’re <strong>massive, expensive to run, and require enormous computational resources</strong>.</p><h4>The Efficiency Problem in AI and The Lottery Ticket Hypothesis</h4><p>Before getting into the solutions, let’s first understand the problem. Modern LLMs are computational monstrous. Let’s have a look at Figure 1 and 2 and the following trend in the SOTA model parameters.</p><p>Look at these images from the recently published Artificial Intelligence Index Report 2025 [46].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GKkQTzsDOU2jJsZtxNsI_Q.png" /><figcaption>Figure 1: The number of parameters in the notable AI models.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*48VbZkJpJ5V0NxcqwUC8ww.png" /><figcaption>Figure 2: Estimated Training cost of selected AI models</figcaption></figure><ul><li><strong>GPT-4</strong> reportedly contains 1.76 trillion parameters [37, 38]</li><li><strong>Google’s GLaM</strong> weighs in at 1.2 trillion parameters [36]</li><li>Even “smaller” models like <strong>LLaMA-70B</strong> require multiple high-end GPUs just to run</li><li>and many more …</li></ul><p>These things show that even though these models work well, they are extremely expensive and out of reach of the general population of the world. To make it accessible for everyone, we must need to do something else. Is there a way? The answer to this question is yes!!</p><p>If you are familiar with the fundamental of deep learning you might be aware of the fact that deep learning models are massively overparameterized; meaning they have much more parameters than the dataset they are trained on. The exact mechanism of how it works is still unknown and explainability is one of the major area of research, however overparameterization somehow works and this is what makes sparsity techniques one of the major lever here. Pruning in Efficient ML has been sort of evergreen technique going back to 90s when papers like OBD[5], OBS[4] and EBD[42] first got published. Recent methods maked these older techniques work for modern LLMs. Methods like SparseGPT, Wanda, LLM-Pruner and NVIDIA’s Minitron[3, 4] have shown that we can compress models by 2 to 4x while maintaining performance, making powerful AI accessible to organizations with limited computational resources.</p><p><strong>Why does pruning work so well?</strong> I don’t want to discuss in detail here but the Lottery Ticket Hypothesis [2] offers a great insight in this regard. The main intuition again comes from overparameterization hypothesis. The Lottery Ticket Hypothesis propose <strong>that a dense, randomly-initialized networks contain sparse subnetworks (“winning tickets”) that, when trained in isolation with their original initialization, can match the full network’s performance.</strong> The catch is these subnetworks only work with their specific initial weights and random reinitialization kills performance. This suggests massive networks may be necessary not for capacity, but to ensure some well-initialized subnetwork exists that can learn effectively. I might write a seperate blog for this, if you are interested I would suggest you to read the paper. It is very interesting and pretty well written.</p><h3>Part I: Pruning</h3><h4><strong>What is Pruning?</strong></h4><p>Pruning is a technique used to reduce the size of a neural network by removing <strong><em>unnecessary or less significant parameters</em></strong> (weights) from the model. The primary objective of pruning is to create a smaller, more efficient model without significantly sacrificing accuracy. Pruning can be performed at different levels of the neural network, such as neurons, layers, or individual weights. Many neural networks are heavily over-parameterized. We can often remove 50–90% of a model’s weights without significantly impacting performance. For most of the cases, a significance values (or <strong>importance score</strong> or <strong>saliency</strong> or <strong>sensitivity</strong>) is used to identify the least important weight/blocks/layers in order to remove them.</p><h4><strong>Different Types of Pruning</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*shPYfq6XQ0HxWwoFdQ0oXg.png" /><figcaption>Figure 3: A general Framework for Structured, Semi-structured and unstructured pruning (source: <a href="https://arxiv.org/abs/2312.17244">https://arxiv.org/abs/2312.17244</a>)</figcaption></figure><h4><strong>Unstructured Pruning</strong></h4><p>In unstructured pruning (non-structured pruning), <strong>a fraction of total weights within the neural network are removed based on their significance score</strong>. It is the finest-grained case of the pruning which operates at the individual weight level. It is called unstructured because <strong>it prunes x% of the total weight without any structure/pattern</strong>. Typically, weights with less significance values (or <strong>importance score</strong> or <strong>saliency</strong> or <strong>sensitivity</strong>) are considered less important and are pruned. If you are interested in the more formal definition of the unstructured pruning, you can have a look at the section 3.1 of this survey paper [3].</p><p>We will discuss different pruning criteria for finding the significance score for pruning in the next section. Methods like WANDA (Weights and Activations)[15] prune weights based on the importance score calculated by multiplying wight’s magnitudes by corresponding input activations, requiring no retraining. GBLM Pruner [39] uses gradient, weight and activation to calculate the importance score and then prune the weight with the least score. Weight update based methods like SparseGPT [16] and OBS[4] uses second order gradient(Hessian) to first identify the pruning mask (weights to be pruned) and then to update the remaining weight to compensate for the pruned weight. Identification of the <strong>pruning mask works by identifying the least important weight based on the importance score</strong> which depends on the Hessian.</p><p><strong>Pros &amp; Cons:</strong></p><ul><li>Maximum flexibility in choosing what to remove</li><li>Fine-grained pruning leads to a significant reduction in model size. Can achieve very high sparsity levels (80–95%)</li><li>Can be applied to any part of the network.</li><li>The resulting sparsity can lead to inefficiencies in hardware acceleration, as many deep learning frameworks are optimized for dense matrix operations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4JGEUUos33PSkcdInk99cA.png" /><figcaption>Figure 4: Unstructured Pruning: It prunes x% of the weight without any structure/pattern based on the saliency score: Dense weight (left image), Pruned weight (right image)</figcaption></figure><h4><strong>Structured Pruning</strong></h4><p>Structured pruning takes a different approach, instead of removing individual weights, it removes entire components like neurons, attention heads, or even complete layers.</p><h4>Pros</h4><ul><li><strong>Immediate speedup</strong> on standard hardware</li><li><strong>Smaller model files</strong> for easier deployment</li><li><strong>Predictable performance gains</strong> that align with theoretical calculations</li></ul><p>The choice between structured and unstructured pruning often comes down to deployment requirements: if you need immediate inference speedup, structured pruning is typically the way to go. To understand different types of structured pruning and pruning metric used in this method, look at my blog <a href="https://medium.com/@mukulranjan/combining-pruning-and-knowledge-distillation-1b5b7407c8c8">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D2-E_6UuxY5iem3DAgXecA.png" /><figcaption>Figure 5: Comparison of pruning units. Width pruning reduces the size of projection weight matrices. Depth pruning removes Transformer blocks, or individual MHA and FFN modules.[source: Shortened LLaMA]</figcaption></figure><h4>Semi-Structured Pruning</h4><p>This is the middle ground, which combines elements of both approaches. The most popular variant is <strong>N:M sparsity</strong>, where N weights are pruned from every consecutive group of M weights. For example, in 2:4 magnitude pruning [39], 2 parameter with lowest absolute value of magnitude from the group of 4 is removed as shown below.</p><pre>2:4 Sparsity Pattern:<br>Original: [0.8, 0.2, 0.9, 0.1] → Pruned: [0.8, 0.0, 0.9, 0.0]<br>Original: [0.3, 0.7, 0.4, 0.6] → Pruned: [0.0, 0.7, 0.0, 0.6]</pre><p>This approach has gained traction because it can leverage specialized hardware features. NVIDIA’s new GPUs have built-in support for 2:4 sparsity patterns, providing automatic acceleration [40].</p><h4>Dynamic Pruning</h4><p>Dynamic pruning [10, 11, 12, 13, 14, 15, 16] adjusts the model structure during training rather than after it, allowing the pruning process to adapt to the model’s evolving understanding. Methods like “Fire Together Wire Together” use self-supervised mask prediction to identify which connections should be preserved or pruned during training.</p><h3>Part II: Pruning Criteria</h3><p>Now that we understand <em>what</em> to prune (individual weights, neurons, or layers), let’s explore <em>how</em> to decide which weights to remove. This is the central question in neural network pruning: <strong>How do we measure a weight importance? </strong>The most simple thing that we all can think for this is to just consider is the magnitude of the weight as the saliency score but it doesn’t work well for the obvious reason (A small weight can be important if removing it requires large compensatory changes elsewhere in the network or say if it corresponds to attention sink[46] in modern LLMs).</p><p>In this section, we’ll explore two fundamental approaches for the saliency measure and pruning algorithms:</p><ol><li><strong>Weight Update Based Methods (OBD, OBS, SparseGPT)</strong>: These methods use second-order information (Hessian) to both identify <em>which weights to prune</em> AND <em>how to update the remaining weights to compensate for the removal</em>.</li><li><strong>Direct Saliency Methods/Methods Without Weight Updates (WANDA, GBLM-Pruner)</strong>: These methods compute importance scores using first-order information (gradients, activations, weights) without explicitly updating other weights.</li></ol><h3>Weight Update Based Methods</h3><p>Let’s try to understand the basic mathematics of neural network pruning. I would assume that you have some basic understanding of linear algebra and neural network here. We will start with Optimal Brain damage.</p><ol><li><strong>Optimal Brain Damage (OBD)</strong></li></ol><p>OBD is considered as one of the first paper to introduce the idea of neural network pruning. The fundamental question OBD asks is: <em>If I set a weight to zero, how much will the loss function change? </em>They approximate this using a Taylor expansion of the loss function around a local minimum. When we perturb weights by δw, the loss E changes by δE, given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zyxvnuZDDDAO4Wjlm4KB3w.png" /><figcaption>Equation 1: Second-order Taylor expansion showing how loss E changes when weights are perturbed by δw. This approximation is the foundation of OBD’s saliency measure.</figcaption></figure><p>Or in the matrix notation this can be written as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*Rx5QIm_BfL2GppT7Skq3MA.png" /><figcaption>Equation 2: Matrix notation of the above equation. Where <strong>g</strong> = gradient vector (∂E/∂w_i) and <strong>H</strong> = Hessian matrix (∂²E/∂wᵢ∂wⱼ)</figcaption></figure><p>OBD makes three key approximation to simplify the above equation.</p><ol><li>At a minimum, g = 0 so the first term will become 0,</li><li>Ignore higher-order terms O(‖δw‖³) and</li><li>Assume that the hessian is diagonal which implies that H = ∂²E/∂wᵢ∂wⱼ ≈ 0 whenever i ≠ j. This means weights can be considered independently, that is changing one weight doesn’t affect the optimal values of others. Under this diagonal assumption, our equation simplifies to δE ≈ ½ Σᵢ hᵢᵢ(δwᵢ)². where hᵢᵢ = ∂²E/∂wᵢ² is the i-th diagonal element of the Hessian.</li></ol><p>When we delete a weight, <strong>we’re essentially making the perturbed weight δwᵢ + wᵢ = 0 and hence setting δwᵢ = -wᵢ<em>.</em></strong> This gives us the saliency formula to <strong>sᵢ = ½ hᵢᵢwᵢ²</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*2KM3JTbCetWkcDsrDPjvCQ.png" /><figcaption>Equation 3 (a, b, c): Final Saliency score for OBD</figcaption></figure><p>The saliency term has two components: wᵢ² captures the magnitude of the weight (larger weights generally matter more), while hᵢᵢ captures the curvature of the error surface (high curvature means the error is very sensitive to changes in this weight). Weights with low saliency can be safely deleted.</p><p><strong>2. Optimal Brain Surgeon (OBS)</strong></p><p>OBS build upon the OBD by using the <strong>full Hessian</strong> (all second-order interactions) and <strong>updating remaining weights</strong> to compensate for pruned weights <strong>without retraining</strong>. Which is basically not considering the third assumption of OBD. If we just consider the assumption 1 and 2 of OBD discussed above,<strong> </strong>equation 2 takes the form of equation 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*kSmMkKDf89vckFg09wk0vQ.png" /><figcaption>Equation 4: Change in the loss without any diagonal approximation. Note that H is not a diagonal matrix here unlike OBD.</figcaption></figure><p>Since our goal is to <strong>prune the weight wᵢ </strong>that leads to <strong>smallest increase in the error δE, </strong>we can formulate our optimization problems as shown in equation 5. Note that the condition <strong>δwᵢ + wᵢ = 0 </strong>in the equation 5 comes from the fact that we want to prune the weight <strong>wᵢ , </strong>meaning the value of the weight after perturbation must be zero as discussed in the case of OBD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*fRwB_Xnq_rNyyDZlO5k75g.png" /><figcaption>Equation 5: Constrained optimization problem for OBS. Note that since the non-diagonal elements of H can be non-zero we need to solve this constrained optimization using Lagrange multiplier and find the optimal values of saliency score and the perturbation <strong>δw</strong>.</figcaption></figure><p>We can form the a Lagrangian from above equation as shown in equation 6. Note that <strong>eᵢ</strong> is a unit vector in the weight space corresponding to (scalar) weight <strong>wᵢ</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*gRJu6P0gXlfW-GfLwq256Q.png" /><figcaption>Equation 6: Lagrangian for constrained equation 5.</figcaption></figure><p>where λ is a Lagrange multiplier. We can take derivatives w.r.t. δ<strong>w</strong> and λ and set them to zero and then employ the constraints <strong>δwᵢ + wᵢ = 0</strong>, to find that the optimal weight change <strong>δw</strong> and resulting change in error <strong>δE</strong> as shown in equation 7 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*f8tytk2Gvi4YE6f3S4A8Wg.png" /><figcaption>Equation 7: Optimal weight change <strong>δw</strong> and resulting change in error <strong>δE</strong></figcaption></figure><p>Hence, the saliency formula takes the following form in Equation 8. Note that the update formula <strong>δw </strong>adjusts all remaining weights based it’s value provided by equation 7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*ONjBVIhvrpsKoqMgbrB8mQ.png" /><figcaption>Equation 8: Saliency formula for OBS</figcaption></figure><p><strong>Comparing OBD and OBS</strong></p><p>At first glance, the saliency formulas look similar:</p><ul><li><strong>OBD</strong>: sᵢ = ½hᵢᵢwᵢ² (uses <strong>diagonal of Hessian</strong>)</li><li><strong>OBS</strong>: sᵢ = wᵢ²/2[<strong>H</strong>⁻¹]ᵢᵢ (uses <strong>diagonal of inverse Hessian</strong>)</li></ul><p>But they’re completely different, OBD uses the diagonal elements of <strong>H </strong>but<strong> </strong>OBS uses the diagonal elements of <strong>H</strong>⁻¹. For a diagonal matrix, [<strong>H</strong>⁻¹]ᵢᵢ = 1/hᵢᵢ, so OBS reduces to OBD. But when <strong>H</strong> has significant off-diagonal terms (which it <strong>always</strong> does in practice for neural networks), these give completely different results. A weight might have high hᵢᵢ (OBD says “important!”) but also high [<strong>H</strong>⁻¹]ᵢᵢ (OBS says “safe to prune because other weights can compensate”). OBD would keep the weight; OBS correctly identifies it as redundant.</p><p><strong>The Computational Challenge</strong></p><p>The difficulty with OBS is computing <strong>H</strong>⁻¹ for a network with thousands or millions of weights, the Hessian matrix is enormous. OBS shows that:</p><ol><li>The Hessian can be approximated as a sample covariance matrix: <strong>H</strong> ≈ (1/P)∑ₖ <strong>X</strong>⁽ᵏ⁾(<strong>X</strong>⁽ᵏ⁾)ᵀ where <strong>X</strong>⁽ᵏ⁾ are gradient vectors for sample k.</li><li>This allows <strong>H</strong>⁻¹ to be computed recursively during training using efficient matrix inversion formulas (the Woodbury matrix identity) shown below in Equation 9.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*KKPcaWvaBGkn5-9zf0k4jA.png" /><figcaption>Equation 9: The Woodbury matrix identity, recursive algorithm for computing inverse of Hessian. Note that, <strong>X</strong>⁽ᵏ⁾ are gradient vectors for sample k.</figcaption></figure><p><strong>3. Sparse GPT</strong></p><p>The simple OBS implementation doesn’t scale to large networks because computing and inverting <strong>H</strong> is O(n^3) where n is the number of weights. For a layer with even 4096 weights, this means ~68 billion operations. For modern LLMs with billions of weights, this is completely intractable. More recent methods like <a href="https://proceedings.neurips.cc/paper_files/paper/2021/file/b0490b85e92b64dbb5db76bf8fca6a82-Paper.pdf">Adaprune</a> [44] solves this problem converting the full model compression problems as the layerwise subproblem. For each layer with weight matrix <strong>W</strong> and input <strong>X</strong>, the goal is to find a pruning mask <strong>M</strong> and updated weights <strong>W</strong>c such that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*vSM5oqcPVDcUKeaA7P7yIA.png" /><figcaption>Equation 10: Layer wise reconstruction error</figcaption></figure><p>where ⊙ denotes element-wise multiplication. Once the mask is fixed, optimal reconstruction requires solving for each row <em>i</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*osq9Jru6jEjr6v6uiZB8-A.png" /><figcaption>Equation 11: Optimal reconstruction for the mixed mask M</figcaption></figure><p>This requires inverting <strong>Hₘᵢ = Xₘᵢ Xᵀₘᵢ</strong> separately for each row because different rows have different masks Mᵢ. For a d×d weight matrix, this costs <strong>O(d⁴)</strong>, which is completely intractable for LLMs.</p><h4>Key Observation and Column-by-Column Processing</h4><p>One key observation here is the full Hessian <strong>H = XX^T</strong> is the <strong>SAME for all rows</strong> (it only depends on inputs X, not on weights W). The problem is that masking it is different per row that is <strong>(Hₘᵢ)⁻¹ ≠ (H⁻¹)_Mi</strong> ← Can’t just mask the inverse!</p><p>To solve these issue SparseGPT makes this approximations — instead of handling all pruned weights at once, process columns left-to-right and prune one column at a time. It only updates a <em>subset</em> <strong>U</strong> of weights. This is still optimal for that subset just there is less effective compensation of the weight because there will be fewer weights available to absorb the error. Define index subsets:</p><ul><li>U₁ = {1, 2, 3, …, d} (all columns)</li><li>U₂ = {2, 3, …, d} (removed column 1)</li><li>U₃ = {3, …, d} (removed columns 1,2)</li><li>…</li><li>Uⱼ = {j, j+1, …, d} (columns j onwards)</li></ul><p>More formally this can be defined in equation 12. In words, starting with U₁ being the set of all indices, each subset Uⱼ₊₁ is created by removing the smallest index from the previous subset Uⱼ .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*iZG97-zG_2cQ9JJ-WlBoJQ.png" /><figcaption>Equation 12: a sequence of index sets. In words, starting with U₁ being the set of all indices, each subset Uⱼ₊₁ is created by removing the smallest index from the previous subset Uⱼ .</figcaption></figure><p>For each subset Uⱼ, define its Hessian: H_Uⱼ = (XX^T)_Uⱼ and it’s inverse as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*FtDLroEpEAUY7fmgw6Ay5A.png" /><figcaption>Equation 13: Inverse hessian of the subset Uⱼ</figcaption></figure><p>we can compute <strong>B = (H_</strong>Uⱼ<strong>)⁻¹</strong> recursively using Gaussian elimination in <strong>O(d²)</strong> time as shown in equation 13 and the full sequence of d inverse Hessians costs <strong>O(d³)</strong> total.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*kD5SOzT6S13t3DVJiZofCg.png" /><figcaption>Equation 14: Recursive relation for the Inverse of Hessian</figcaption></figure><p>This is just the submatrix of H corresponding to columns in Uⱼ. These Hessians impose a sequence of inverses: <strong>(H_U₁)⁻¹, (H_U₂)⁻¹, (H_U₃)⁻¹, …, (H_Uⱼ)⁻¹. </strong>More importantly when processing column j, <strong>all rows use the same H_Uⱼ</strong> because it only depends on which columns remain (j onwards), not on which specific weights in those columns are pruned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iBO1htHSyoG3vM4aUq3unw.png" /><figcaption>Figure 6: [Left] Visualization of the SparseGPT reconstruction algorithm. Given a fixed pruning mask M, we incrementally prune weights in each column of the weight matrix W, using a sequence of Hessian inverses (HUj)−1, and updating the remainder of the weights in those rows, located to the “right” of the column being processed. Specifically, the weights to the “right” of a pruned weight (dark blue) will be updated to compensate for the pruning error, whereas the unpruned weights do not generate updates (light blue). [Right] Illustration of the adaptive mask selection via iterative blocking.(Figure 4 from SparseGPT paper[16] )</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*4oCnxIbmz2Ezts376GfQ-g.png" /><figcaption>Figure 6: SparseGPT algorithm from the original paper. Annotation in the official python implementation is provided in the appendix.</figcaption></figure><p>Thus SparseGPT solves the scalability problem of pruning massive LLMs by processing the weight matrix column-by-column from left to right. For each column, it first selects which weights to prune based on their saliency scores (computed using the Hessian inverse diagonal), then zeros out the pruned weights, and finally updates all the remaining weights to the right of the current column to compensate for the introduced error. The key trick is that by processing in this left-to-right order, the algorithm can compute a sequence of Hessian inverses recursively in O(d³) time instead of O(d⁴), and more importantly, each Hessian inverse in the sequence can be shared across all rows of the weight matrix.</p><h3>Methods Without Weight Updates</h3><p>As discussed above, these methods directly prune the weight based on the importance score/saliency without updating the weight and hence offer much faster pruning compared to weight update based methods.</p><ol><li><strong>WANDA</strong></li></ol><p>It prunes weights with the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis. It is Motivated by the recent observation of emergent large magnitude features in LLMs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*O4cVa38dt-YJAsrLpbqqFQ.png" /><figcaption>Equation 15: Pruning metric or saliency for Wanda.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Md5Nui9XJrwOpPcBtYX2WQ.png" /><figcaption>Figure 7: Illustration of the proposed method Wanda (Pruning by Weights and activations), compared with the magnitude pruning approach. Given a weight matrix W and input feature activations X, it compute the weight importance as the elementwise product between the weight magnitude and the norm of input activations (|W|· ∥X∥2). Weight importance scores are compared on a per-output basis (within each row in W), rather than globally across the entire matrix. (Figure 1 from WANDA paper, <a href="https://arxiv.org/abs/2306.11695">https://arxiv.org/abs/2306.11695</a>)</figcaption></figure><p><strong>2. GBLM-Pruner</strong></p><p>GBLM-Pruner extends WANDA by adding gradient information to the pruning metric. All the methods discussed before assume that since model is trained gradient value should be 0, however that is not the case when authors empirically observe this. The pruning metric/saliency for this methods is given by equation 16.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*-cGqWSPfgQcIoN79eEcEXQ.png" /><figcaption>Equation 16: Pruning metric or saliency for GBLM-Pruner. α is an empirically determined scaling factor (α = 100 ) and ∥G[:, i, j]∥p represents the ℓp norm of gradients across samples.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sn8x_d2hkJvGE6LMAzPoAQ.png" /><figcaption>Figure 8: Illustration of method GBLM-Pruner. Given a weight matrix, W, a gradient matrix, G, and an input feature activation, X, weight importance is computed as an elementwise multiplication of weight magnitude and ℓ1 or ℓ2 norm of the gradients across multiple samples, denoted as ∥G∥p · |W|, optionally, it is promotable to add the multiplication of weight and the ℓ2 norm of input activations, denoted as |W| · ∥X∥2. (Figure 1 from GBLM-Pruner paper, <a href="https://arxiv.org/pdf/2311.04902">https://arxiv.org/pdf/2311.04902</a>)</figcaption></figure><h3>Appendix</h3><h4>Basic OBS implementation</h4><p>Here I tried to implement the naive version of OBS algorithm described in the paper. You can have a look at complete code from scratch <a href="https://github.com/mukul54/ml-basics/blob/main/pruning/obs.py">here</a>.</p><pre>import numpy as np<br><br>def compute_hessian_inverse(X, alpha=1e-5):<br>    &quot;&quot;&quot;<br>    Compute inverse Hessian H^{-1} where H = (1/P) * X * X^T<br>    <br>    Args:<br>        X: Input gradients matrix (n_samples, n_weights)<br>        alpha: Damping factor for numerical stability<br>    <br>    Returns:<br>        H_inv: Inverse Hessian (n_weights, n_weights)<br>    &quot;&quot;&quot;<br>    n_samples, n_weights = X.shape<br>    P = n_samples<br>    <br>    # Initialize with damped identity<br>    H_inv = (1.0 / alpha) * np.eye(n_weights)<br>    <br>    # Recursive update for each sample<br>    for m in range(n_samples):<br>        x = X[m:m+1, :].T  # Column vector (n_weights, 1)<br>        <br>        # Woodbury matrix identity update<br>        numerator = H_inv @ x @ x.T @ H_inv<br>        denominator = P + x.T @ H_inv @ x<br>        <br>        H_inv = H_inv - numerator / denominator<br>    <br>    return H_inv<br><br>def obs_pruning_step(weights, H_inv):<br>    &quot;&quot;&quot;<br>    Perform one step of OBS: find weight to prune and compute updates<br>    <br>    Args:<br>        weights: Current weight vector (n_weights,)<br>        H_inv: Inverse Hessian matrix (n_weights, n_weights)<br>    <br>    Returns:<br>        q: Index of weight to prune<br>        saliency: Saliency of that weight<br>        weight_updates: Updates for all weights<br>    &quot;&quot;&quot;<br>    n_weights = len(weights)<br>    <br>    # Compute saliency for each weight<br>    # L_q = w_q^2 / (2 * [H^{-1}]_{qq})<br>    H_inv_diag = np.diag(H_inv)<br>    saliencies = (weights ** 2) / (2 * H_inv_diag)<br>    <br>    # Find weight with minimum saliency (excluding already pruned weights)<br>    active_weights = weights != 0<br>    saliencies[~active_weights] = np.inf<br>    q = np.argmin(saliencies)<br>    <br>    # Compute weight updates: δw = -(w_q / [H^{-1}]_{qq}) * H^{-1}[:, q]<br>    e_q = np.zeros(n_weights)<br>    e_q[q] = 1.0<br>    <br>    weight_updates = -(weights[q] / H_inv[q, q]) * (H_inv @ e_q)<br>    <br>    return q, saliencies[q], weight_updates<br><br># Example usage<br>if __name__ == &quot;__main__&quot;:<br>    # Simulate a small example<br>    n_weights = 5<br>    n_samples = 10<br>    <br>    # Random weight gradients (in practice, computed from backprop)<br>    X = np.random.randn(n_samples, n_weights) * 0.1<br>    <br>    # Current weights (after training)<br>    weights = np.array([0.5, 0.1, 0.8, 0.2, 0.3])<br>    <br>    # Compute inverse Hessian<br>    H_inv = compute_hessian_inverse(X)<br>    <br>    print(&quot;Initial weights:&quot;, weights)<br>    print(&quot;\nPruning iterations:&quot;)<br>    <br>    # Prune 3 weights<br>    for iteration in range(3):<br>        q, saliency, updates = obs_pruning_step(weights, H_inv)<br>        <br>        print(f&quot;\nIteration {iteration + 1}:&quot;)<br>        print(f&quot;  Pruning weight {q}: {weights[q]:.4f}&quot;)<br>        print(f&quot;  Saliency: {saliency:.6f}&quot;)<br>        <br>        # Apply updates<br>        weights = weights + updates<br>        <br>        # Delete the pruned weight<br>        weights[q] = 0<br>        <br>        print(f&quot;  Updated weights: {weights}&quot;)</pre><h4>Code for SparseGPT</h4><p>You can have a look at the original code of SparseGPT <a href="https://github.com/IST-DASLab/sparsegpt/blob/master/sparsegpt.py">here</a>. It is very simple to follow and the most of the code for llm pruning is loosely based on this codebase. I have added the annotated version of the sparsegpt.py from the repo <a href="https://github.com/mukul54/ml-basics/blob/main/pruning/annotated_sparsegpt.py">here</a>.</p><h4>References</h4><ol><li><a href="https://arxiv.org/abs/2402.13116">A Survey on Knowledge Distillation of Large Language Models</a></li><li><a href="https://arxiv.org/abs/1803.03635">The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks</a></li><li><a href="https://arxiv.org/abs/2308.06767">A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations</a></li><li><a href="https://proceedings.neurips.cc/paper/1992/file/303ed4c69846ab36c2904d3ba8573050-Paper.pdf">Second order derivatives for network pruning: Optimal Brain Surgeon</a></li><li><a href="https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf">Optimal Brain Damage</a></li><li>Compact Language Models via Pruning and Knowledge Distillation: <a href="https://arxiv.org/abs/2407.14679">https://arxiv.org/abs/2407.14679</a></li><li><a href="https://developer.nvidia.com/blog/how-to-prune-and-distill-llama-3-1-8b-to-an-nvidia-llama-3-1-minitron-4b-model/">https://developer.nvidia.com/blog/how-to-prune-and-distill-llama-3-1-8b-to-an-nvidia-llama-3-1-minitron-4b-model/</a></li><li><a href="https://arxiv.org/abs/2408.11796">LLM Pruning and Distillation in Practice: The Minitron Approach</a></li><li><a href="https://openreview.net/pdf?id=0OU1ZXXxs5">Pruning vs Quantization: Which is Better?</a></li><li><a href="https://openaccess.thecvf.com/content/CVPR2022/papers/Elkerdawy_Fire_Together_Wire_Together_A_Dynamic_Pruning_Approach_With_Self-Supervised_CVPR_2022_paper.pdf">Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction</a></li><li><a href="https://arxiv.org/abs/2006.07253">Dynamic Model Pruning with Feedback</a></li><li><a href="https://medium.com/@rekalantar/pytorch-tutorial-dynamic-weight-pruning-for-more-optimized-and-faster-neural-networks-7b337e47987b">https://medium.com/@rekalantar/pytorch-tutorial-dynamic-weight-pruning-for-more-optimized-and-faster-neural-networks-7b337e47987b</a></li><li>Manifold Regularized Dynamic Network Pruning: <a href="https://openaccess.thecvf.com/content/CVPR2021/papers/Tang_Manifold_Regularized_Dynamic_Network_Pruning_CVPR_2021_paper.pdf">https://openaccess.thecvf.com/content/CVPR2021/papers/Tang_Manifold_Regularized_Dynamic_Network_Pruning_CVPR_2021_paper.pdf</a></li><li><a href="https://openaccess.thecvf.com/content/CVPR2024/papers/Huang_FedMef_Towards_Memory-efficient_Federated_Dynamic_Pruning_CVPR_2024_paper.pdf">FedMef: Towards Memory-efficient Federated Dynamic Pruning</a></li><li><a href="https://proceedings.neurips.cc/paper_files/paper/2018/file/a9a6653e48976138166de32772b1bf40-Paper.pdf">Frequency-Domain Dynamic Pruning for Convolutional Neural Networks</a></li><li><a href="https://openreview.net/forum?id=31EWrnEBsn&amp;noteId=fikquhJPEE">Accelerating Deep Learning with Dynamic Data Pruning</a></li><li><a href="https://www.datature.io/blog/a-comprehensive-guide-to-neural-network-model-pruning">https://www.datature.io/blog/a-comprehensive-guide-to-neural-network-model-pruning</a></li><li><a href="https://arxiv.org/abs/2306.11695">A Simple and Effective Pruning Approach for Large Language Models</a></li><li>Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models: <a href="https://iclr.cc/virtual/2024/poster/18549">https://iclr.cc/virtual/2024/poster/18549</a></li><li>SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot: <a href="https://arxiv.org/abs/2301.00774">https://arxiv.org/abs/2301.00774</a></li><li><a href="https://arxiv.org/abs/2302.01530">Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective</a></li><li><a href="https://openaccess.thecvf.com/content/ICCV2023W/RCV/papers/Liao_Can_Unstructured_Pruning_Reduce_the_Depth_in_Deep_Neural_Networks_ICCVW_2023_paper.pdf">Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?</a></li><li><a href="https://openreview.net/forum?id=eoSeaK4QJo">Towards Energy Efficient Spiking Neural Networks: An Unstructured Pruning Framework</a></li><li><a href="https://openreview.net/forum?id=jpm1AfJucwt">Revisiting Loss Modelling for Unstructured Pruning</a></li><li><a href="https://openreview.net/forum?id=Yg7ExbCxzt6">What Makes a Good Prune? Maximal Unstructured Pruning for Maximal Cosine Similarity</a></li><li><a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10090171">Causal Unstructured Pruning in Linear Networks Using Effective Information</a></li><li>LLM-Pruner: On the Structural Pruning of Large Language Models: <a href="https://arxiv.org/abs/2305.11627">https://arxiv.org/abs/2305.11627</a></li><li>ShortGPT: Layers in Large Language Models are More Redundant Than You Expect: <a href="https://arxiv.org/abs/2403.03853">https://arxiv.org/abs/2403.03853</a></li><li>ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression: <a href="https://arxiv.org/abs/1707.06342">https://arxiv.org/abs/1707.06342</a></li><li>Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks: <a href="https://arxiv.org/pdf/1808.06866">https://arxiv.org/abs/1808.06866</a></li><li>Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods: <a href="https://arxiv.org/abs/2402.02834">https://arxiv.org/abs/2402.02834</a></li><li>LaCo: Large Language Model Pruning via Layer Collapse: <a href="https://arxiv.org/abs/2402.11187">https://arxiv.org/abs/2402.11187</a></li><li>Structured Pruning for Deep Convolutional Neural Networks: A Survey: <a href="https://ieeexplore.ieee.org/abstract/document/10330640">https://ieeexplore.ieee.org/abstract/document/10330640</a></li><li>THE LLM SURGEON (<a href="https://arxiv.org/abs/2312.17244">https://arxiv.org/abs/2312.17244</a>), (<a href="https://github.com/Qualcomm-AI-research/llm-surgeon">https://github.com/Qualcomm-AI-research/llm-surgeon</a>)</li><li>Importance Estimation for Neural Network Pruning: (<a href="https://arxiv.org/abs/1906.10771">https://arxiv.org/abs/1906.10771</a>)</li><li><a href="https://gpt3demo.com/apps/google-glam">https://gpt3demo.com/apps/google-glam</a></li><li><a href="https://en.wikipedia.org/wiki/GPT-4">https://en.wikipedia.org/wiki/GPT-4</a></li><li><a href="https://explodingtopics.com/blog/gpt-parameters">https://explodingtopics.com/blog/gpt-parameters</a></li><li>GBLM-Pruner (<a href="https://arxiv.org/abs/2311.04902">https://arxiv.org/abs/2311.04902</a>)</li><li>Magnitude Pruning (<a href="https://proceedings.neurips.cc/paper/2015/hash/ae0eb3eed39d2bcef4622b2499a05fe6-Abstract.html">Learning both Weights and Connections for Efficient Neural Network</a>)</li><li><a href="https://developer.nvidia.com/blog/structured-sparsity-in-the-nvidia-ampere-architecture-and-applications-in-search-engines/">https://developer.nvidia.com/blog/structured-sparsity-in-the-nvidia-ampere-architecture-and-applications-in-search-engines/</a></li><li><a href="https://proceedings.neurips.cc/paper_files/paper/1996/file/2ac2406e835bd49c70469acae337d292-Paper.pdf">Early Brain Damage</a></li><li><a href="https://proceedings.neurips.cc/paper_files/paper/1993/file/b056eb1587586b71e2da9acfe4fbd19e-Paper.pdf">Optimal Brain Surgeon: Extensions and performance comparisons</a></li><li><a href="https://proceedings.neurips.cc/paper_files/paper/2021/file/b0490b85e92b64dbb5db76bf8fca6a82-Paper.pdf">Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks</a></li><li><a href="https://arxiv.org/abs/2309.17453">Efficient Streaming Language Models with Attention Sinks</a></li><li><a href="https://arxiv.org/abs/2504.07139">Artificial Intelligence Index Report 2025</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edc705b48916" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Tokenizers in LLM —  Part 1 :  Byte Pair Encoding and Beyond]]></title>
            <link>https://medium.com/@mukulranjan/understanding-tokenizers-in-llm-part-1-byte-pair-encoding-and-beyond-75e4c4e70e6f?source=rss-abdbc773a16e------2</link>
            <guid isPermaLink="false">https://medium.com/p/75e4c4e70e6f</guid>
            <dc:creator><![CDATA[Mukul Ranjan]]></dc:creator>
            <pubDate>Sat, 18 Jan 2025 21:47:01 GMT</pubDate>
            <atom:updated>2025-08-09T23:16:30.263Z</atom:updated>
            <content:encoded><![CDATA[<h3>Tokenizers in Large Language Models : Byte Pair Encoding and Beyond</h3><p>This is the first part of the blogpost series about the tokenization in Language Models.</p><p>If you have seen Karpathy Sensei’s video[1], you must be aware of the fact that tokenizer is one of the most “<em>problematically important</em>” concept in NLP. The video is probably the best explanation available on internet for understanding tokenization. Tokens are the most fundamental unit of large language models. LLMs don’t see the word like we do but they are fed these words in different way, called ‘tokens’. Different tokenization methods are used in different open and closed source large language models.</p><p>Figure below shows the rough high level block diagram of a language model depicting tokenization in it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L5LwK_jcapBpj1m2JqcCbg.png" /><figcaption>A high level rough block diagram of a language model depicting tokenization. Note that output logits will have probability score over entire vocabulary, not for input token. Note than tokenizer is a separate module and it has their own training data. LLMs only sees the tokens and never directly deals with the texts.</figcaption></figure><h3>Tokenization</h3><p>Tokenization is the process of converting texts data into token :’) . Tokens are the most fundamental unit of large language models. When we use large language models, it first converts our input texts/images/videos into some fundamental unit that we call tokens. The process of this conversion from input texts to token is called tokenization. When you write on ChatGPT, “write an ode on the end of universe.”, it first converts your input into this format:</p><pre>&lt;|im_start|&gt;system&lt;|im_sep|&gt;write an ode on the end of universe.&lt;|im_end|&gt;&lt;|im_start|&gt;assistant&lt;|im_sep|&gt;</pre><p>Which is then converted to the following numerical sequence:</p><pre>200264, 17360, 200266, 9566, 448, 58840, 402, 290, 1268, 328, 28714, 13, 200265, 200264, 173781, 200266<br></pre><p>You can use this website (<a href="https://tiktokenizer.vercel.app/">https://tiktokenizer.vercel.app/</a>) to play with different input text. In this blog we will learn how does this convert from text sequence to numerical sequence happens.</p><p>There are several methods, one of the most basic one is to use it in same way that a python code process the string, in the raw byte form. Just use “utf-8” encoding of the input and then we are done. But, we don’t!!! Why not we just use “utf-8” encoding?</p><ol><li>The vocabulary size is will be fixed (256 unique byte values). Isn’t that a good thing??</li><li>This will allow us to have smaller input embedding table and prediction at the final layer. Another good thing??</li><li>It will require modeling all possible byte combinations for meaningful language patterns and make <strong>our input sequences very very long</strong>.</li><li>Because of this reason, <strong>transformer input layer will need to process much longer input context, which is not computationally efficient for attention based models</strong>. Remember??, OpenAI charges you based on no of input tokens if we use ‘utf-8’, our api cost will become much much larger.</li><li>Also, ‘utf-8’ does not have any semantic understanding. For example it treats the word “unhappiness” as 11 completely separate characters and doesn’t recognize that this word contains meaningful subparts: “un-” (prefix), “happy” (root), “-ness” (suffix). Model will need more data to understand semantic representation if directly use utf-8 encoding.</li><li>Many predictions in language depend on previous <em>words</em> or even <em>syntactic phrases</em>, and focusing on individual characters/bytes is so low-level that it makes it more challenging to learn about those higher levels of abstraction[10].</li></ol><p>For example Japanese word for Good Morning, Ohayou gozaimasu, “おはようございます”, will be converted into 27 different tokens using ‘utf-8’ encoding: [227,129,138,227,129,175,227,130,136,227,129,134,227,129,148,227,129,150,227,129,132,227,129,190,227,129,153], however if we use current GPT-4o tokenizer it will just use 4 tokens: [8930, 5205, 72683, 59809], which provides <strong>almost 7x compression in the input token size</strong>.</p><p>Therefore, we need to find some other way to encode our data which is more efficient than simple byte encoding like ‘utf-8’. Then, why don’t we use each word as a separate token???</p><p>If we use word based tokenizations, it is <strong>practically impossible</strong> to create a dictionary of all the words in human history. Hence, word level tokenization will almost always have out of vocabulary problems and multi-linguality might become a distant dream for those types of model. Whenever we come up with new words and use that word to ask something from our model, it will throw an error.</p><p>One set of method to solve both of these problems is called <strong>subword based tokenization</strong> which is being used in all the modern language models. Subword based tokenization solves both of these problems by allowing words to be decomposed into subwords and bringing the best of these world. Byte Pair Encoding is one of the example of <strong>subword based tokenization.</strong></p><h3>Byte Pair Encoding</h3><p>BPE is one of the most common subword based tokenization. It works by replacing the <strong>highest-frequency pair of bytes</strong> with a new byte that was not there in the initial vocabulary. By it’s design (we will see with an example), It keeps the more frequent word as it is in the vocabulary and breaks down the less frequent word. Let’s see this from the example from wikipedia article:</p><p>Suppose this is the data consisting of only 4 characters {a, b, c, d}: <strong>aaabdaaabac. </strong>We can create a dictionary of each pairs of characters/byte with the count.</p><p>Pair of bytes: [aa, aa, ab, bd, da, aa, aa, ab, ba, ac]. We can see that “aa” occurs 4 times, hence we will update our vocabulary by adding extra word “aa” in it. Hence the updated dictionary will become {“aa”, “a”, “b”, “c”, “d”}. Let’s represent “aa” by “Z”, for simplicity and avoiding confusion in the next iteration. So our updated word will become: <strong>ZabdZabac.</strong></p><p>Now the updated pair of bytes will look like this: [Za, ab, bd, dZ, Za, ab, ba, ac], we can see that “ab” and Za appears similar number of times 2, but let’s add “ab” to the dictionary : {“aa”, “ab”, “a”, “b”, “c”, “d”}. Let’s represent “ab” by Y. So the updated word will become: ZYdZYac.</p><p>After this the updated pair looks like this: [ZY, Yd, dZ, ZY, Ya, ac]. Let’s add ZY= “aaab” to dictionary. Thus the dictionary is: {“aaab”, “aa”, “ab”, “a”, “b”, “c”, “d”} and represent “ZY” with X, Hence the new word is <strong>XdXac</strong>. We can no longer further split it to create more than one pair of characters. Hence we encoded the given word to create this vocabulary: {“aaab”, “aa”, “ab”, “a”, “b”, “c”, “d”}</p><p>Note that I am using byte and characters interchangingly, since they are almost the same assuming each character can be represented as a single byte (which is not the case always, but we can assume that for simplicity). However, based on this nature BPE has two different type[12]: BPE (which can be said to be character level BPE), and Byte-level BPE [14]. We can do the same thing with byte as well. We convert the entire data into sequence of bytes and use the same logic as above. Let’s look at the code for this and try to understand it.</p><pre>text = &quot;まいにち まいにち ぼくらはてっぱんの<br>うえで やかれて いやになっちゃうよ<br>あるあさ ぼくは みせのおじさんと<br>けんかして うみに にげこんだのさ<br>はじめて およいだ うみのそこ<br>とっても きもちが いいもんだ<br>おなかの あんこが おもいけど<br>うみは ひろいぜ こころがはずむ<br>ももいろ サンゴが てをふって<br>ぼくの およぎを ながめていたよ<br>まいにち まいにち たのしいことばかり<br>なんぱせんが ぼくのすみかさ<br>ときどき サメに いじめられるけど<br>そんなときゃ そうさ にげるのさ<br>いちにち およげば はらぺこさ<br>めだまも くるくる まわっちゃう<br>たまには エビでも くわなけりゃ<br>しおみず ばかりじゃ ふやけてしまう<br>いわばの かげから くいつけば<br>それは ちいさな つりばりだった<br>どんなに どんなに もがいても<br>ハリが のどから とれないよ<br>はまべで みしらぬ おじさんが<br>ぼくを つりあげ びっくりしてた<br>やっぱり ぼくは たいやきさ<br>すこし こげある たいやきさ<br>おじさん つばを のみこんで<br>ぼくを うまそうに たべたのさ&quot;<br>tokens = text.encode(&quot;utf-8&quot;) <br>tokens = list(map(int, tokens))<br>print(&#39;---&#39;)<br>print(tokens)<br>print(len(token)) # 1245<br># [10,227,129,190,227,129,132,227,129,171,227,129,161,32,227,...]</pre><p>The above code simply convert any text into “utf-8” encoding. We can now use the logic discussed above to write the code for BPE. Most of these code are from Karpathy Sensei’s video and colab notebook. We used the lyrics of Japanese song ‘<a href="https://en.wikipedia.org/wiki/Oyoge!_Taiyaki-kun">Oyoge! Taiyaki Kun</a>’ and got a total of 1245 tokens using ‘utf-8’ encoding.</p><ol><li>Get the stats of each byte pair available in the dataset. Like we did in the wikipedia example and sort it in the reverse order of token pair counts.</li></ol><pre>def get_stats(ids: list) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Calculate frequencies of adjacent token pairs in a sequence.<br><br>    Args:<br>        ids (list): A list of token IDs to analyze.<br><br>    Returns:<br>        dict: A dictionary mapping token pairs (tuples) to their frequencies.<br>    <br>    Example:<br>        &gt;&gt;&gt; get_stats([65, 66, 65, 66, 67])<br>        {(65, 66): 2, (66, 65): 1, (66, 67): 1}<br>    &quot;&quot;&quot;<br>    counts = {}<br>    # Use zip to create pairs of adjacent tokens<br>    # ids[1:] creates a shifted copy, so zip creates adjacent pairs<br>    for pair in zip(ids, ids[1:]):<br>        # Increment count for this pair, defaulting to 0 if not seen before<br>        counts[pair] = counts.get(pair, 0) + 1<br>    return counts<br><br>stats = get_stats(tokens)<br>print(sorted(((v,k) for k,v in stats.items()), reverse=True))<br># [(297, (227, 129)), (85, (227, 130)), (55, (32, 227)), (28, (132, 227)), (28, (10, 227)), ...]</pre><p>2. Now, let’s see the code for the merge operation. Merge operation will merge token pairs into a single new token like in the wikipedia example we replaced, ‘aa’ with ‘Z’; ‘ab’ with ‘Y’ and “ZY” with X. It simply takes the <strong>list of all token ids (ids)</strong>, and <strong>pair of tokens (pair)</strong> that we want to replace and the <strong>new token id (idx),</strong> that we want <strong>all the occurrences of the pair</strong> in token lists(<strong>ids</strong>) to replace with. It return the list of all the updated tokens.</p><pre>def merge(ids: list, pair: tuple, idx: int) -&gt; list:<br>    &quot;&quot;&quot;<br>    Merge all occurrences of a token pair into a single new token.<br><br>    Args:<br>        ids (list): List of token IDs to process.<br>        pair (tuple): Tuple of two token IDs to merge (t1, t2).<br>        idx (int): New token ID to assign to merged pair.<br><br>    Returns:<br>        list: New list with specified token pairs merged.<br><br>    Example:<br>        &gt;&gt;&gt; merge([65, 66, 67, 65, 66], (65, 66), 256)<br>        [256, 67, 256]<br>    &quot;&quot;&quot;<br>    newids = []<br>    i = 0<br>    while i &lt; len(ids):<br>        # Check if we&#39;ve found our target pair<br>        if (i &lt; len(ids) - 1 and          # Not at last token<br>            ids[i] == pair[0] and         # First token matches<br>            ids[i+1] == pair[1]):         # Second token matches<br>            newids.append(idx)            # Add merged token ID<br>            i += 2                        # Skip both tokens<br>        else:<br>            newids.append(ids[i])         # Keep current token<br>            i += 1<br>    return newids<br><br># Main BPE training process<br>vocab_size = 350  # Target vocabulary size<br>num_merges = vocab_size - 256  # Number of merges needed<br>                              # 256 is typically the base vocabulary (byte tokens)<br>ids = list(tokens)  # Create working copy of original tokens<br><br># Dictionary to store merge rules: (token1, token2) -&gt; new_token<br>merges = {}  <br><br># Perform merges until we reach desired vocabulary size<br>for i in range(num_merges):<br>    # Get frequency counts of all adjacent pairs<br>    stats = get_stats(ids)<br>    <br>    # Find most frequent pair<br>    pair = max(stats, key=stats.get)<br>    <br>    # Assign new token ID (starting from 256)<br>    idx = 256 + i<br>    <br>    print(f&quot;merging {pair} into a new token {idx}&quot;)<br>    <br>    # Apply the merge throughout the token list<br>    ids = merge(ids, pair, idx)<br>    <br>    # Store the merge rule for later use<br>    merges[pair] = idx<br># merging (227, 129) into a new token 256<br># merging (227, 130) into a new token 257<br># merging (32, 256) into a new token 258<br># ...<br><br># We can also calculate the compression ratio<br>print(&quot;tokens length:&quot;, len(tokens)) # 1245<br>print(&quot;ids length:&quot;, len(ids)) # 383<br>print(f&quot;compression ratio: {len(tokens) / len(ids):.2f}X&quot;) # 3.25X </pre><p>We observe that we got 3.25x compression in the tokens by using BPE as compared to ‘utf-8’. We can actually see how increasing dictionary size / vocab size improves the compression ratio for our text data. Following figure shows it for our data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*GyRWvX2u9sCy50pyeXC1CA.png" /><figcaption>Vocab size vs compression ratio. We can observe that increasing vocab size increase the compression ratio, hence less no of token will be needed if we have very large vocabulary size.</figcaption></figure><p>As we saw in the figure 1 that tokenizers is a separate module, that deals with text to token conversion. LLM never directly sees the texts and it always deals with tokens. So after getting all these secret sauces of BPE encoding we actually need to encode a string. But remember we created all the secrete sauce based on the only limited Japanese texts, so our encoder won’t be very efficient and for english words it will be same as raw ‘utf-8’ encoding, that is the number of tokens for the english word after encoding will be same as no of characters in the input texts. Therefore it is very important to train our tokenizers on diverse set of datasets from different languages.</p><pre>def encode(text: str) -&gt; list:<br>    &quot;&quot;&quot;<br>    Encode a text string into a sequence of token IDs using learned BPE merges.<br>    <br>    This function:<br>    1. Converts the input text to UTF-8 bytes<br>    2. Iteratively applies learned merge rules to create larger tokens<br>    3. Stops when no more merges are possible<br>    <br>    Args:<br>        text (str): The input text to be tokenized<br>        <br>    Returns:<br>        list: A list of integer token IDs<br>        <br>    Example:<br>        &gt;&gt;&gt; encode(&quot;hello&quot;)<br>        [256, 108, 111]  # where 256 might be the merged token for &#39;he&#39;<br>        <br>    Notes:<br>        - Uses the global &#39;merges&#39; dictionary containing learned BPE merge rules<br>        - Applies merges greedily, choosing the lowest-indexed merge rule when multiple apply<br>        - Merge rules are applied until no further merges are possible<br>    &quot;&quot;&quot;<br>    # Convert input text to list of UTF-8 bytes<br>    tokens = list(text.encode(&quot;utf-8&quot;))<br>    <br>    # Continue merging as long as we have at least 2 tokens<br>    while len(tokens) &gt;= 2:<br>        # Get statistics (frequencies) of token pairs, since we need the keys of all pairs<br>        stats = get_stats(tokens)<br>        <br>        # Find the merge rule with lowest index<br>        # merges.get(p, float(&quot;inf&quot;)) returns float(&quot;inf&quot;) if p isn&#39;t in merges<br>        # This ensures pairs without merge rules are never selected<br>        pair = min(stats, key=lambda p: merges.get(p, float(&quot;inf&quot;)))<br>        <br>        # If the best pair has no merge rule, we&#39;re done<br>        if pair not in merges:<br>            break  # nothing else can be merged<br>            <br>        # Get the new token ID for this pair<br>        idx = merges[pair]<br>        <br>        # Apply the merge throughout our token list<br>        tokens = merge(tokens, pair, idx)<br>        <br>    return tokens<br><br>encoded_tokens_eng = encode(&quot;hello world&quot;)# [104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100]<br>encoded_tokens_japanese = encode(&quot;まいにち&quot;) # [256, 290, 280]<br>print(len(&quot;hello world&quot;), len(&quot;まいにち&quot;)) # 11, 4<br>print(len(encoded_tokens_eng), len(encoded_tokens_japanese)) # 11, 3</pre><p>we can see that for english words there is no compression but for a new Japanese word “まいにち” we are seeing slight compression.</p><h4>Decoding it back</h4><p>Decoders converts back the sequence of encoded tokens to string. Since, we already discussed that LLMs always deal with tokens, it take in tokens and gives out tokens. Tokenizer module must be able to also convert a sequence of tokens back to strings for us human to understand.</p><p>Let’s understand that by following code:</p><pre>from typing import Dict, List, bytes<br><br># Initialize vocabulary with single byte tokens<br>vocab: Dict[int, bytes] = {idx: bytes([idx]) for idx in range(256)}<br><br># Extend vocabulary with merged tokens from merges dictionary<br>for (p0, p1), idx in merges.items():<br>    vocab[idx] = vocab[p0] + vocab[p1] # this addition is in bytes so it will concatenate two bytes<br><br>def decode(ids: List[int]) -&gt; str:<br>    &quot;&quot;&quot;<br>    Decodes a sequence of token IDs back into a text string using a pre-computed vocabulary.<br>    <br>    The function performs the following steps:<br>    1. Converts each token ID to its corresponding bytes using the vocabulary<br>    2. Concatenates all bytes together<br>    3. Decodes the resulting bytes into a UTF-8 string<br>    <br>    Args:<br>        ids (List[int]): A list of integer token IDs to decode.<br>        <br>    Returns:<br>        str: The decoded text string.<br>            <br>    Global Dependencies:<br>        - vocab (Dict[int, bytes]): A dictionary mapping token IDs to their byte sequences.<br>    <br>    Example:<br>        &gt;&gt;&gt; decode([104, 101, 108, 108, 111])<br>        &#39;hello&#39;<br>    <br>    Notes:<br>        - Uses &#39;replace&#39; error handling for UTF-8 decoding, which replaces invalid<br>          byte sequences with the � character rather than raising an error.<br>        - The vocabulary is expected to be initialized with single bytes (0-255)<br>          and extended with merged token pairs.<br>    &quot;&quot;&quot;<br>    # Convert token IDs to bytes and join them, concatenate all the bytes togather<br>    tokens: bytes = b&quot;&quot;.join(vocab[idx] for idx in ids)<br>    <br>    # Decode bytes to UTF-8 string, replacing invalid sequences<br>    text: str = tokens.decode(&quot;utf-8&quot;, errors=&quot;replace&quot;)<br>    <br>    return text<br><br>result = decode([104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100])<br>print(result)  # &quot;hello world&quot;<br>decode_eng = decode(encoded_tokens_eng) # &quot;hello world&quot;<br>decode_jap = decode(encoded_tokens_japanese) # &quot;まいにち&quot;</pre><h4>Let’s wrap everything up</h4><p>Now let’s create a module to do everything from training the BPE tokenizer to encoding a string and decoding it back.</p><pre>class BPETokenizer:<br>    &quot;&quot;&quot;<br>    Byte-Pair Encoding (BPE) Tokenizer Implementation.<br>    <br>    This tokenizer starts with a base vocabulary of bytes (0-255) and iteratively<br>    merges the most frequent pairs of tokens to create new tokens until the<br>    desired vocabulary size is reached.<br>    &quot;&quot;&quot;<br>    <br>    def __init__(self, vocab_size: int = 512):<br>        &quot;&quot;&quot;<br>        Initialize the BPE tokenizer.<br><br>        Args:<br>            vocab_size (int): Target size of the vocabulary. Must be &gt; 256.<br>        &quot;&quot;&quot;<br>        if vocab_size &lt;= 256:<br>            raise ValueError(&quot;Vocabulary size must be greater than 256&quot;)<br>            <br>        self.vocab_size = vocab_size<br>        self.vocab = None<br>        self.merges = None<br><br>    def get_stats(self, ids: list) -&gt; dict:<br>        &quot;&quot;&quot;<br>        Calculate frequencies of adjacent token pairs in a sequence.<br><br>        Args:<br>            ids (list): List of token IDs to analyze.<br><br>        Returns:<br>            dict: Mapping of token pairs to their frequencies.<br>        &quot;&quot;&quot;<br>        counts = {}<br>        for pair in zip(ids, ids[1:]):<br>            counts[pair] = counts.get(pair, 0) + 1<br>        return counts<br><br>    def merge(self, ids: list, pair: tuple, idx: int) -&gt; list:<br>        &quot;&quot;&quot;<br>        Merge all occurrences of a token pair into a single new token.<br><br>        Args:<br>            ids (list): List of token IDs to process.<br>            pair (tuple): Tuple of two token IDs to merge (t1, t2).<br>            idx (int): New token ID to assign to merged pair.<br><br>        Returns:<br>            list: New list with specified token pairs merged.<br>        &quot;&quot;&quot;<br>        newids = []<br>        i = 0<br>        while i &lt; len(ids):<br>            if (i &lt; len(ids) - 1 and <br>                ids[i] == pair[0] and <br>                ids[i+1] == pair[1]):<br>                newids.append(idx)<br>                i += 2<br>            else:<br>                newids.append(ids[i])<br>                i += 1<br>        return newids<br><br>    def train(self, text: str):<br>        &quot;&quot;&quot;<br>        Train the tokenizer on input text.<br><br>        Args:<br>            text (str): Input text to train on.<br>        &quot;&quot;&quot;<br>        # Initialize base vocabulary with bytes<br>        self.vocab = {idx: bytes([idx]) for idx in range(256)}<br>        self.merges = {}<br><br>        # Convert text to bytes for initial tokens<br>        tokens = list(text.encode(&quot;utf-8&quot;))<br>        ids = list(tokens)  # Working copy<br><br>        # Calculate number of merges needed<br>        num_merges = self.vocab_size - 256<br><br>        # Perform merges<br>        for i in range(num_merges):<br>            # Get pair frequencies<br>            stats = self.get_stats(ids)<br>            if not stats:<br>                break  # No more pairs to merge<br><br>            # Find most frequent pair<br>            pair = max(stats, key=stats.get)<br>            idx = 256 + i<br><br>            # Perform merge<br>            ids = self.merge(ids, pair, idx)<br>            self.merges[pair] = idx<br><br>            # Update vocabulary<br>            self.vocab[idx] = self.vocab[pair[0]] + self.vocab[pair[1]]<br><br>    def encode(self, text: str) -&gt; list:<br>        &quot;&quot;&quot;<br>        Encode text into tokens.<br><br>        Args:<br>            text (str): Text to encode.<br><br>        Returns:<br>            list: List of token IDs.<br>        &quot;&quot;&quot;<br>        if not self.vocab or not self.merges:<br>            raise RuntimeError(&quot;Tokenizer must be trained first&quot;)<br><br>        tokens = list(text.encode(&quot;utf-8&quot;))<br>        while len(tokens) &gt;= 2:<br>            stats = self.get_stats(tokens)<br>            pair = min(stats, key=lambda p: self.merges.get(p, float(&quot;inf&quot;)))<br>            if pair not in self.merges:<br>                break  # Nothing else can be merged<br>            idx = self.merges[pair]<br>            tokens = self.merge(tokens, pair, idx)<br>        return tokens<br><br>    def decode(self, ids: list) -&gt; str:<br>        &quot;&quot;&quot;<br>        Decode token IDs back into text.<br><br>        Args:<br>            ids (list): List of token IDs.<br><br>        Returns:<br>            str: Decoded text.<br>        &quot;&quot;&quot;<br>        if not self.vocab:<br>            raise RuntimeError(&quot;Tokenizer must be trained first&quot;)<br><br>        tokens = b&quot;&quot;.join(self.vocab[idx] for idx in ids)<br>        text = tokens.decode(&quot;utf-8&quot;, errors=&quot;replace&quot;)<br>        return text<br><br><br># Example usage<br>def main():<br>    # Sample text for training<br>    training_text = &quot;&quot;&quot;<br>    Hello world! This is a sample text to demonstrate BPE tokenization.<br>    We&#39;ll use this to train our tokenizer and then test encoding and decoding.<br>    &quot;&quot;&quot;<br><br>    # Initialize and train tokenizer<br>    tokenizer = BPETokenizer(vocab_size=512)<br>    tokenizer.train(training_text)<br><br>    # Test encoding and decoding<br>    test_text = &quot;Hello world!&quot;<br>    encoded = tokenizer.encode(test_text)<br>    decoded = tokenizer.decode(encoded)<br><br>    print(f&quot;Original text: {test_text}&quot;)<br>    print(f&quot;Encoded tokens: {encoded}&quot;)<br>    print(f&quot;Decoded text: {decoded}&quot;)<br>    print(f&quot;Vocabulary size: {len(tokenizer.vocab)}&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()<br><br># Original text: Hello world!<br># Encoded tokens: [72, 101, 265, 111, 32, 119, 111, 114, 108, 100, 33]<br># Decoded text: Hello world!<br># Vocabulary size: 363</pre><h3>What’s Next</h3><p>As discussed in the beginning tokenization is probably the most “problematically important” concept in NLP. A lot of problems in recent language models are because of the nature of tokenization. Here are some of the problems listed by Karpathy Sensei in his video:</p><ul><li>Why can’t LLM spell words? <strong>Tokenization</strong>.</li><li>Why can’t LLM do super simple string processing tasks like reversing a string? <strong>Tokenization</strong>.</li><li>Why is LLM worse at non-English languages (e.g. Japanese)? <strong>Tokenization</strong>.</li><li>Why is LLM bad at simple arithmetic? <strong>Tokenization</strong>.</li><li>Why is LLM not actually end-to-end language modeling? <strong>Tokenization</strong>.</li><li>What is the real root of suffering? <strong>Tokenization</strong>.</li></ul><p>Some of these problems have been solved by the newer version of LLMs however a lot of security and safety issue still exists. I would highly recommend anyone reading this blog to watch Karpathy Sensei’s video on tokenization [1].</p><p>One of the major problem caused by tokenization is mathematical ability of larger language models [5, 6, 7, 8]. There have been several recent advancement in this direction that we will discuss in the next blog (hopefully 🙃)!!</p><h4>References</h4><ol><li><a href="https://youtu.be/zduSFxRajkE?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ">https://youtu.be/zduSFxRajkE?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ</a></li><li><a href="https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation">https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation</a></li><li><a href="https://huggingface.co/learn/nlp-course/en/chapter6/5">https://huggingface.co/learn/nlp-course/en/chapter6/5</a></li><li><a href="https://en.wikipedia.org/wiki/Byte_pair_encoding">https://en.wikipedia.org/wiki/Byte_pair_encoding</a></li><li><a href="https://huggingface.co/spaces/huggingface/number-tokenization-blog">https://huggingface.co/spaces/huggingface/number-tokenization-blog</a></li><li><a href="https://arxiv.org/abs/2402.14903">https://arxiv.org/abs/2402.14903</a></li><li><a href="https://www.beren.io/2024-07-07-Right-to-Left-Integer-Tokenization/">https://www.beren.io/2024-07-07-Right-to-Left-Integer-Tokenization/</a></li><li><a href="https://arxiv.org/abs/2410.14166">https://arxiv.org/abs/2410.14166</a></li><li><a href="https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf">https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf</a></li><li><a href="https://seantrott.substack.com/p/tokenization-in-large-language-models">https://seantrott.substack.com/p/tokenization-in-large-language-models</a></li><li><a href="https://arxiv.org/pdf/2406.10229">https://arxiv.org/pdf/2406.10229</a></li><li><a href="https://datascience.stackexchange.com/questions/126715/what-are-the-differences-between-bpe-and-byte-level-bpe">https://datascience.stackexchange.com/questions/126715/what-are-the-differences-between-bpe-and-byte-level-bpe</a></li><li><a href="https://tiktokenizer.vercel.app/">https://tiktokenizer.vercel.app/</a></li><li><a href="https://arxiv.org/pdf/1909.03341">https://arxiv.org/pdf/1909.03341</a></li><li><a href="https://en.wikipedia.org/wiki/Oyoge!_Taiyaki-kun">https://en.wikipedia.org/wiki/Oyoge!_Taiyaki-kun</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75e4c4e70e6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Combining Pruning and Knowledge Distillation]]></title>
            <link>https://medium.com/@mukulranjan/combining-pruning-and-knowledge-distillation-1b5b7407c8c8?source=rss-abdbc773a16e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b5b7407c8c8</guid>
            <dc:creator><![CDATA[Mukul Ranjan]]></dc:creator>
            <pubDate>Wed, 28 Aug 2024 18:39:59 GMT</pubDate>
            <atom:updated>2025-09-10T21:14:30.628Z</atom:updated>
            <content:encoded><![CDATA[<h4>Efficient LLM Part 3: Paper Summary of the paper “Compact Language Models via Pruning and Knowledge Distillation”</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*xIZ25Sd22g4yH9QNDzOsHQ.png" /><figcaption>High-level overview of our proposed pruning and distillation approach. The total number of tokens used for each step is indicated in parentheses. Source: [4]</figcaption></figure><p>Pruning and knowledge distillation can be combined to create even more efficient models. A recent paper by Nvidia [<a href="https://arxiv.org/abs/2407.14679"><strong><em>2</em></strong></a><strong><em>, </em></strong><a href="https://developer.nvidia.com/blog/how-to-prune-and-distill-llama-3-1-8b-to-an-nvidia-llama-3-1-minitron-4b-model/"><strong><em>3</em></strong></a><strong><em>, </em></strong><a href="https://arxiv.org/pdf/2408.11796v1"><strong><em>4</em></strong></a>] combines pruning with classical knowledge distillation for resource-efficient retraining of LLMs. They efficiently applied pruning and distillation methods to Llama 3.1 8B model to obtain a Llama-3.1 4B model. It performs favorably against state-of-the-art open-source models of similar size, including Minitron 4B, Phi-2 2.7B, Gemma2 2.6B, and Qwen2–1.5B [<strong><em>3</em></strong>].</p><p>Let’s try to understand this work in detail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hxpi3MOCyQsDq0Ya9I37aA.png" /><figcaption>Figure 1: High-level overview of proposed iterative pruning and distillation approach to train a family of smaller LLMs. [<a href="https://arxiv.org/abs/2407.14679">2</a>]</figcaption></figure><p>This paper proposes a thorough empirical study of <strong>structured pruning</strong> techniques for LLMs across multiple dimensions(neurons, attention heads, embedding channels, model depth). Authors have used both width pruning and depth pruning strategies in this work.</p><p>Above diagram shows the high level overview of the iterative pruning and distillation approach to train a family of smaller LLMs. All the steps from larger LLMs to smaller LLMs can be summarized as follows:</p><ol><li>Importance Estimation: Use activation-based importance scoring for width pruning (attention heads, neurons, embedding channels) and perplexity (PPL) or Block Importance (BI) scores for depth pruning (layers).</li><li>Perform importance estimation using only forward passes on a small calibration dataset.</li><li>Ranking: Rank all components (heads, neurons, embedding channels, layers) based on their calculated importance scores.</li><li>Pruning: Remove the least important components based on the ranking and reshape the remaining weight matrices accordingly.</li><li>Architecture Search: Perform a lightweight neural architecture search to find optimal pruned configurations and evaluate multiple pruned architectures with short retraining to stabilize rankings.</li><li>Iterative Refinement: Optionally repeat the process to create progressively smaller models.</li><li>Use the largest pruned model as the starting point for further compression.</li><li>Assess the pruned and distilled model on various downstream tasks to ensure performance retention.</li></ol><h3>1. Pruning</h3><h4>Recap: Width Pruning vs Depth Pruning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*D2-E_6UuxY5iem3DAgXecA.png" /><figcaption>Figure 2: Comparison of pruning units. Width pruning reduces the size of projection weight matrices. Depth pruning removes Transformer blocks, or individual MHA and FFN modules[source: Shortened LLaMA].</figcaption></figure><h4>Width pruning</h4><p>Remove neurons in feed-forward layers, attention heads, and embedding channels by estimating the importance of different model components using only forward passes on a small calibration dataset. They used different importance estimation scores for different dimensions:</p><ol><li><strong>Attention Heads: </strong>For attention head, authors used the L2 norm of the output of the attention layer which is then aggregated along the batch and sequence dimensions as shown in the following equation.</li></ol><p>The intuition behind this method is that attention heads producing larger magnitude outputs are likely more important to the model’s function. By summing over a batch of inputs and all sequence positions, the method captures the head’s overall importance across different inputs and contexts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*VoeNMgzFE0eHXu6BCRYQaQ.png" /><figcaption>Importance score calculation for the attention head</figcaption></figure><blockquote>Since Medium doesn’t support the latex rendering we will use direct latex equations for explaining this equation.</blockquote><p><strong><em>Understanding Equation</em></strong><em>: </em><strong><em>W^{Q,i}, W^{K,i}, W^{V,i}</em></strong><em> are the weight matrices for the Query, Key, and Value projections for the i-th attention head respectively. </em><strong><em>∑_{B,S}</em></strong><em>refers to aggregation along the batch and sequence dimensions. The authors found that </em><strong><em>using L2 norm for batch aggregation and mean for sequence aggregation</em></strong><em> worked best.</em></p><p><strong>2. Neurons: </strong>For<strong> </strong>estimating the importance of neurons authors calculates the magnitude of each neuron’s output(before the non-linearity) across a batch of inputs and all positions in the sequence. Neurons that consistently produce larger outputs across various inputs are deemed more important, as they likely capture more significant features or patterns in the data.</p><p>The intuition behind this method for neuron pruning is to measure how much each neuron contributes to the overall output of the layer. Neurons that have a larger impact on the output are considered more important and are less likely to be pruned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*_N7GeYgnvEkWhhkfnsXn0g.png" /></figure><p><strong><em>Understanding Equation</em></strong><em>: X(W_1^i)^T denotes transpose of the dot product between the input X and the weights for the i-th neuron. W_1^i is the i-th row of the weight matrix W_1 (corresponding to the i-th neuron).</em></p><p>3. <strong>Embedding Channels: </strong>For estimating the importance of embedding channels authors uses the aggregation of activations produced by the LayerNorm layers across the batch and sequence dimensions. It gives an overall measure of how much each embedding dimension contributes to the model’s output after normalization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*sZTyAYQ-5u0Kx2RPbZJIig.png" /><figcaption>importance score for the i-th embedding channel.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*F5i4xN8UaRwUgTObhRMSGA.png" /><figcaption>Layer Normalization operation on input X. μ and σ² represent the mean and variance across the embedding dimensions. ϵ is a small value for numerical stability. γ and β are learnable parameters and ⊙ denotes element-wise multiplication</figcaption></figure><h4>Depth pruning</h4><p>Authors describes two main approaches for depth pruning:</p><ol><li><strong>Perplexity (PPL) based ranking</strong>: This method estimates layer importance by removing a single layer and computing its effect on the model’s perplexity. The paper uses the same equation as proposed in the <a href="https://arxiv.org/pdf/2402.02834">Shortened LLaMA</a> paper as shown below. Note that, Lower perplexity indicates better prediction.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bL53DF328vSXb4YKSu5ZlQ.png" /><figcaption>Perplexity Score for layer n, S is the number of samples in the evaluation set, L is the sequence length. Probability term inside log function is the is the probability of token 𝑥ₗ given the previous tokens, as predicted by the model with layer n removed.[<a href="https://arxiv.org/pdf/2402.02834">Shortened LLaMA</a>]</figcaption></figure><p>We have discussed about perplexity in Appendix A.1. Authors use this equation to quantify how much the model’s predictive performance degrades when a specific layer is removed. Those layers, when removed, cause a larger increase in perplexity are deemed more important and are less likely to be pruned. This approach allows for a data-driven, performance-based method of identifying which layers contribute most significantly to the model’s overall effectiveness.</p><p><strong>2. Block Importance (BI) ranking</strong>: This method uses the cosine distance between the input and output of a layer to estimate its sensitivity. The BI score for layer i is computed by the following equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*JLmpop2yERI-lW5fWqcmug.png" /><figcaption>BI score of layer. 𝑋ᵢ is the input to layer i. Xᵢ,ₜ denotes the 𝑡ᵗʰ row of 𝑋ᵢ. Eₓ,ₜ denotes the expectation over all inputs X and positions t.</figcaption></figure><p>The BI score of layer i measures how much a layer changes its input. It does this by computing the cosine similarity between the input and output of a layer, then subtracting this from 1. If a layer doesn’t change its input much, the cosine similarity will be close to 1, resulting in a low BI score, however if a layer significantly transforms its input, the cosine similarity will be lower, resulting in a higher BI score. Layers with higher BI scores are considered more important because they’re doing more “work” in transforming the representations. When pruning, layers with the lowest BI scores are removed first, as they’re considered least important to the model’s performance.</p><p>The authors note that BI can be computed in a single forward pass, giving it a significant speed advantage over PPL-based importance. Additionally, the authors also mention that BI can be extended to estimate the importance of several contiguous layers at the same time.</p><h4>Obtaining the Pruned Model</h4><p>As summarised in the beginning obtaining pruning model involves multiple stages.</p><p><strong>Importance Ranking</strong>: For each axis (depth, width, embedding), elements are ranked according to their computed importance. This has been explained in great details in previous section.</p><p><strong>Direct Trimming</strong>: The authors perform trimming (reshaping) of the corresponding weight matrices directly based on the importance rankings. Specifically:</p><ul><li>For <strong>neuron</strong> pruning: MLP layer weights are trimmed.</li><li>For <strong>head</strong> pruning: MHA (Multi-Head Attention) layer weights are trimmed.</li><li>For <strong>embedding</strong> channels: The embedding dimension of weight matrices in MLP, MHA, and LayerNorm layers are trimmed.</li></ul><p><strong>Attention Head Pruning</strong>: For attention head pruning, authors introduce an additional step:</p><ul><li>The residual information from pruned heads is added back into the remaining heads.</li><li>This is done to preserve relevant knowledge from the pruned heads.</li><li>For the 𝑖ᵗʰ ℎ𝑒𝑎𝑑 (where 𝑖∈[𝐾−(𝐿−𝐾) ), the new ℎ𝑒𝑎𝑑 becomes: ℎ𝑒𝑎𝑑ᵢ+(ℎ𝑒𝑎𝑑ᵢ−ℎ𝑒𝑎𝑑₂ₖ₋ᵢ₊₁). So basically for each remaining head i, it adds a difference term to itself. This difference is between the current head and a symmetrically opposite head (2K-i+1).</li><li>K is the number of heads after pruning, and L is the original number of heads.</li><li>By adding this difference, it’s essentially incorporating information from the pruned heads into the remaining ones. The symmetric nature of the formula (2K-i+1) suggests it’s trying to balance the information across the remaining heads.</li><li>For grouped query attention, this strategy is applied only to the query heads.</li></ul><p><strong>Lightweight Neural Architecture Search: </strong>Following figures provides an overview of the search strategy for finding optimal architecture configurations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*55At2OLf-5dH4XnsiTJW0w.png" /><figcaption>Figure 3: Overview of our neural architecture search algorithm. RT refers to retraining. Source [2]</figcaption></figure><ul><li>Enumerates all feasible architectures meeting a given parameter budget to find a set of <strong><em>feasible candidates</em></strong>.</li><li>These candidates then undergo lightweight retraining (about 1.8B tokens in this work).</li><li>The best performing candidate after this retraining is selected for further training.</li></ul><p>Authors observe that parameter-efficient fine-tuning techniques such as LoRA can also be applied at this stage.</p><h3>2. Retraining with Knowledge Distillation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QXl1EoQL8nYg0-6BAgIPgg.png" /><figcaption>Figure 4: Overview of Distillation. A student model with N layers is distilled from a teacher model with M layers. The student learns by minimizing a combination of embedding output loss, logit loss and transformer encoder specific losses mapped across student block S and teacher block T.</figcaption></figure><p><strong>Teacher-Student Setup: </strong>The unpruned model serves as the teacher model and the pruned model which we got from previous steps acts as the student model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*1a3YF9HNml-juzcyMd-78g.png" /><figcaption>Overview of Distillation: If the original training data is unavailable, a slight fine-tuning of the teacher model is recommended. Distillation is then performed by minimizing KL divergence on the logits, with the original model as the teacher and the pruned model as the student. source: [4]</figcaption></figure><p><strong>Probability Distribution: </strong>Authors compute the output probability distribution for a token xᵢ using the softmax equation given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*iZDKoCM8b4-5S6HVd0vJWg.png" /><figcaption>Probability distribution for the token xᵢ. τ is the softmax temperature and |V| is the vocabulary size.</figcaption></figure><p><strong>Logit-based KD Loss: </strong>Logit-based KD loss across the sequence of all output tokens is represented as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*6Ulln7Eekwzu4iNu1cVEAw.png" /><figcaption>Logit-based KD loss across the sequence of all output tokens. l is the sequence length. 𝑝ₜ and 𝑝ₛ are the student and teacher probability distribution respectively.</figcaption></figure><p><strong>Intermediate State-based KD Loss: </strong>For distillation, authors explore various loss functions, and several combinations of intermediate states and mappings across the Transformer model as the loss components, along with their respective trade-offs. This is illustrated in Figure 4. The intermediate state-based KD loss across a sequence of Transformer-specific hidden states is represented as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*KUNoywEy9t29deZFI4wZ1A.png" /><figcaption>Intermediate State-based KD Loss. H is the set of chosen intermediate states. ℎᵏⁱₜ and ℎᵏⁱₛ represent the 𝑘ᵗʰ teacher and student hidden state for the 𝑖ᵗʰ token.</figcaption></figure><p>To handle mismatches between teacher and student hidden states, learned linear transformation is used to <strong><em>upscale</em></strong> the student hidden state to the teacher hidden state dimension and the hidden states used are always <strong><em>post LayerNorm</em></strong>.</p><p><strong>Total Loss: </strong>Total loss L is computed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*RtG_JaXpr_xBz88BhpwrPg.png" /><figcaption>L_CLM is the student cross-entropy loss against ground truth labels. Lᵢₛ is the intermediate state-based KD loss discussed in the previous subsection. α is a weighting coefficient, computed dynamically as L₍ₗₒgᵢₜₛ₎ / Lᵢₛ.</figcaption></figure><p><strong>Loss Functions: </strong>Authors experimented with various loss functions including KL divergence (KLD), MSE, cosine similarity, and reverse KLD (R-KLD). KLD is found to be the best choice for base model training.</p><h3>3. Structured Compression Best Practices</h3><p>Authors proposed and summarized several best practices for the structured compression backed by various empirical and experimental results in this paper. Those best practices can be summarized as follows:</p><p>1. To train a family of LLMs, train the largest one and prune+distill iteratively to smaller LLMs.</p><p>2. Use (batch=L2, seq=mean) importance estimation for width axes and PPL/BI for depth.</p><p>3. Use single-shot importance estimation; iterative provides no benefit.</p><p>4. Prefer width pruning over depth for the model scales we consider (≤ 15B).</p><p>5. Retrain <strong><em>exclusively with distillation loss using KLD</em></strong> instead of conventional training.</p><p>6. Use (logit+intermediate state+embedding) distillation when depth is reduced significantly.</p><p>7. Use logit-only distillation when depth isn’t reduced significantly.</p><p>8. Prune a model closest to the target size.</p><p>9. Perform lightweight retraining to stabilize the rankings of searched pruned candidates.</p><p>10. If the largest model is trained using a multi-phase training strategy, it is best to prune and retrain the model obtained from the final stage of training.</p><h3>4. Conclusion</h3><p>This paper presents a comprehensive approach to compressing large language models through structured pruning and efficient retraining. The authors develop a set of best practices for pruning across multiple axes, including depth, width, attention heads, and embedding channels. They introduce activation-based importance metrics for efficient pruning of large models and explore various retraining strategies using knowledge distillation. This work contributes valuable insights into efficient model compression and retraining techniques, potentially making the development and deployment of large language models more accessible and cost-effective.</p><h3>5. Appendix</h3><h4>A.1 Understanding Perplexity</h4><p>Perplexity in NLP literature is used to measure the performance a language model. It quantifies how well the model predicts a given sequence of words or tokens. Lower perplexity indicates better prediction. A perplexity of k means the model is as confused as if it had to <em>choose uniformly and independently from k possibilities for each word</em>. You can have a look at <a href="https://medium.com/nlplanet/two-minutes-nlp-perplexity-explained-with-simple-probabilities-6cdc46884584">this</a>[8] short blog for understanding it in detail. I have used that blog to write this part of the article. You can skip it and directly go there if you want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jFGW-HuyqW77w2o8ZwyaTw.png" /><figcaption>An example of a toy language model taken from [<a href="https://medium.com/nlplanet/two-minutes-nlp-perplexity-explained-with-simple-probabilities-6cdc46884584">8</a>] for understanding perplexity in detail.</figcaption></figure><p>Consider a very small language model with probability distribution shown above and with a vocabulary of only six words: “a”,<em> “the”</em>,<em> “red”</em>, <em>“fox”</em>,<em> “dog”</em>,<em> </em>and <em>“.” . W</em>e want to compute the probability of the sentence W, <em>i.e</em> “<em>a red fox.</em>” using that language model.</p><p>p(W) = p(w1, w2, w3, w4)</p><pre>p(“a red fox.”) = p(“a”, “red”, “fox”, “.”) = p(“a”) * p(“red” | “a”) * p(“fox” | “a red”) * p(“.” | “a red fox”)</pre><p>From the above image, we can see the distribution predicted by our language model. p(“a”)=0.4; p(“red” | “a”)=0.27; p(“fox” | “a red”)=0.55; p(“.” | “a red fox”) = 0.79.</p><p>Thus, p(“a red fox.”) = 0.4 * 0.27 * 0.55 * 0.79 = 0.0469. Since p(W) is the product of probabilities it’s value will become very less for larger sentence. In order to solve this we can normalize them using geometric mean. So normalized probability of p(W) = p(“a red fox.”) can be written as pₙₒᵣₘ = 𝑃(𝑊)^{1/N}, where N is the number of words in sentence. For our example we can pₙₒᵣₘ = (0.0469)^{1/4} = 0.465. This can be used to compare probabilities of sentence. The higher is this number the better is the language model</p><blockquote>Perplexity is defined as the reciprocal of this number. So, perplexity can be defined as the reciprocal of the geometric mean of product of the probabilities of all words in sentence.</blockquote><p>More formally,</p><p>Perplexity(W) = PPL(W) = 1/pₙₒᵣₘ = p(𝑊)^{-1/N}</p><p>Hence for our example, perplexity is 1/0.465 = 2.15. Perplexity value of 1 indicate and ideal language model. Now let’s try to prove this statement stated above:</p><blockquote>A perplexity of k means the model is as confused as if it had to <em>choose uniformly and independently from k possibilities for each word</em>.</blockquote><p>Since our example language model can predict only 6 words, hence the probability of each word is 1/6. Hence, p(W) = p(“a red fox.”) = (1/6) ^ 4 = 0.00077. PPL(W) = p(W)^(-1/N) = ((1/6)⁴)^(-1/4) = 6.</p><p>In the more general form, from a set of k words if we are predicting the probability of W, which contains N words. Then, PPL(W) = ((1/k)^N)^(-1/N) = k</p><h4>Entropy and Perplexity</h4><p>From this equation we can figure out the relationship between the Shannon’s entropy and perplexity which is more widely accepted definition of perplexity. Note that <a href="https://en.wikipedia.org/wiki/Entropy_(information_theory)">Shannon entropy</a> of a language model for sentence W is given by the H(W) = 𝔼[-log p(W)]. We can prove that,</p><blockquote>Perplexity = 2ᴴ⁽ᵂ⁾</blockquote><p>H(W) = -1/N*∑(log₂ p(xᵢ)) where xᵢ denotes the probability of the prediction of 𝑖ᵗʰ token. For example in the previous example let’s consider:</p><p>p(“a”) = p( 𝑥₁)</p><p>p(“red” | “a”) = p(𝑥₂)</p><p>p(“fox” | “a red”) = p(𝑥₃)</p><p>p(“.” | “a red fox”) = p(𝑥₄)</p><p>Then, H(W) can be written as ∑log₂ {p(xᵢ)^(-1/N)} = ∏(p(xᵢ) )^(-1/N) = P(W)^{-1/N} = Perplexity</p><p>Note that ∏(p(xᵢ) )^(-1/N) is the inverse of the geometric mean predicted probabilities of all words in W, which is perplexity.</p><h3>References</h3><ol><li>A Survey on Knowledge Distillation of Large<br>Language Models: <a href="https://arxiv.org/abs/2402.13116">https://arxiv.org/abs/2402.13116</a></li><li>Compact Language Models via Pruning and<br>Knowledge Distillation: <a href="https://arxiv.org/abs/2407.14679">https://arxiv.org/abs/2407.14679</a></li><li><a href="https://developer.nvidia.com/blog/how-to-prune-and-distill-llama-3-1-8b-to-an-nvidia-llama-3-1-minitron-4b-model/">https://developer.nvidia.com/blog/how-to-prune-and-distill-llama-3-1-8b-to-an-nvidia-llama-3-1-minitron-4b-model/</a></li><li>LLM Pruning and Distillation in Practice: The<br>Minitron Approach: <a href="https://arxiv.org/pdf/2408.11796v1">https://arxiv.org/pdf/2408.11796v1</a></li><li>Pruning vs Quantization: <a href="https://openreview.net/pdf?id=0OU1ZXXxs5">https://openreview.net/pdf?id=0OU1ZXXxs5</a></li><li>Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods: <a href="https://arxiv.org/abs/2402.02834">https://arxiv.org/abs/2402.02834</a></li><li>LaCo: Large Language Model Pruning via Layer Collapse: <a href="https://arxiv.org/abs/2402.11187">https://arxiv.org/abs/2402.11187</a></li><li><a href="https://medium.com/nlplanet/two-minutes-nlp-perplexity-explained-with-simple-probabilities-6cdc46884584">https://medium.com/nlplanet/two-minutes-nlp-perplexity-explained-with-simple-probabilities-6cdc46884584</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b5b7407c8c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding NDCG]]></title>
            <link>https://medium.com/@mukulranjan/understanding-ndcg-885656321b3b?source=rss-abdbc773a16e------2</link>
            <guid isPermaLink="false">https://medium.com/p/885656321b3b</guid>
            <dc:creator><![CDATA[Mukul Ranjan]]></dc:creator>
            <pubDate>Tue, 30 Apr 2024 19:58:23 GMT</pubDate>
            <atom:updated>2025-07-18T11:45:18.132Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>NDCG </strong>is probably the most used term in recommender systems. In this blog, we will try to understand the intuitive and mathematical logic behind it.</p><h4>What is it?</h4><p>NDCG (normalized discounted cumulative gain) measures the effectiveness and quality of product ranking in the search, feed ranking, and document retrieval tasks. The higher the NDCG, the better the ranking.</p><p>Before going into the details of this metric, let’s consider the following scenario: You have generated two lists of recommendations for a user, each consisting of the same five items but in a different order. In the first list, the user ordered the first item, while in the second list, the user ordered the fifth item. Which of these two recommendations would be better, according to you?</p><p>Obviously, the first recommendation is better. We will calculate the NDCG of a similar example and try to verify this using this measure. So, NDCG is based on the assumption that if something is the best, it should appear before the better one, which should appear before the good one. More formally, the following are the assumptions on which NDCG is based:</p><ol><li>Highly relevant items are more useful when appearing earlier.</li><li>Highly relevant items are more useful than marginally relevant items, which are, in turn, more useful than non-relevant items.</li></ol><h4>What is relevance?</h4><p>It is the subjective part of this topic. What is relevant depends on the use cases. It is determined mainly by the user inputs and our definition. There can be binary relevance, where we say that if an item is clicked, it is relevant ( a relevance of 1); otherwise (a relevance of 0), it is not relevant.</p><p>We can define relevance in non-binary fashion as well. For example consider the a scenario in e-commerce: we define a relevance label of 5 if an user order an item; if the user add the same item to cart then it has a relevance of 4; if they wishlist the item then it has a relevance of 3; if they share it with a friend then it has a relevance of 2, and if it is just clicked, then it will have relevance of 1. Otherwise, it has a relevance of 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bbnu6Rrm0LHr9OyexwIglQ.png" /><figcaption>Different user actions and relevance label for an e-commerce platform</figcaption></figure><p>Relevance definition can vary based on the use case. For example in case of social media or video sharing platform, like, comment, or share can have high relevance or it can also be binned based on view/browse time. For an online news platform relevance can be defined based on the time spent by the user on a news page.</p><p>To understand how ndcg score is calculated we will consider two different examples. One for binary relevance and the other for non-binary relevance.</p><h4>Binary Relevance</h4><p>For understanding binary let’s consider following search query sessions and product ids recommended for that search query. We will search for the queries Red Saree , Bedsheet , and sports shoes for men on the e-commerce platform Meesho. We will consider click as relevance for this. If the user has clicked on a product then we will assign relevance label 1 for that product. Binary relevance value for this user and query pair is shown in the table 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-VuR-US5BI64oJMvxGGxeA.png" /><figcaption>Search Result for query `Red Saree` on Meesho</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o4x7pTWw_ihR1dllpDDnaQ.png" /><figcaption>Search Result for query Bedsheet on Meesho</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vXx5gofKdpjBuIeF4wmDoA.png" /><figcaption>Search Result for query `sports shoes for men` on Meesho</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GN_RBh2wxKVkO3UcPChpIA.png" /><figcaption>Table 1: Defining binary relevance on user click on the products shown for different queries at different position</figcaption></figure><h4>Non-Binary Relevance</h4><p>[Going ahead we will drop query and product name and only keep their ids from the above examples]. Consider the same examples and let’s assume that different scenarios for our user:</p><ol><li>for query `red saree` (q1), user</li></ol><ul><li>ordered item p14</li><li>added to cart p11</li></ul><p>2. For query `bedsheet`, user</p><ul><li>ordered item p22</li><li>Shared item p24</li></ul><p>3. For query Sports Shoes for Men, user</p><ul><li>wishlisted item p34.</li></ul><p>We can now create another example based on the proposed definition of multi-label relevance above as shown in table 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*1SNSftljFDc0UxIJpp7RCA.png" /><figcaption>Table 2: Example of Multi-label relevance</figcaption></figure><h3>How to calculate it?</h3><p>NDCG is calculated in such a way that it considers the following assumptions:</p><blockquote>If something is the best, it should be at the top in any ranking, or recommender systems. An e-commerce ranking model will have the best NDCG if it can rank all the ordered products in a session at the top(assuming order has the highest relevance label).</blockquote><p>So, it must considers position(or rank) of all the recommended items, which means it should penalize the relevance based on the position of the recommended item. More formally:</p><ol><li>NDCG ∝ 1/position; [inversely proportional to position, or some monotonic function of position]</li><li>NDCG ∝ relevance; [directly proportional to relevance]</li><li>Along with the above two things, as the name suggest it should be normalized, so that its values lies between 0 and 1.</li></ol><p>Based on the above three assumptions we can write a following rough formula for the NDCG:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*0Y7hJhWNXeWhx_ZZNKmpbA.png" /><figcaption>Rough formula for NDCG@p; here β is the normalization factor to satisfy 3rd condition mentioned above; f(i) is some monotonic function of position i</figcaption></figure><p>We will add proper definition later but before that we should also learn about the predecessors of NDCG and some of their limitations.</p><h3>1) Cumulative Gain(CG)</h3><p>Cumulative gain is simply the sum of all the relevance score(or relevance label or gain) for each query. CG at position p is the sum of all the relevance score from position 1 to position p in the feed. More formally it can be written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/109/0*PEmR4V5bUC5BMclA" /><figcaption>source: <a href="https://en.wikipedia.org/wiki/Discounted_cumulative_gain">wikipedia</a></figcaption></figure><p>For example above in the case of binary relevance, CG@2 for q1 will be equal to 1. Let’s calculate CG values for both binary and non-binary relevance examples.</p><h4>Binary Relevance</h4><p><strong>For query q1(</strong><strong>Red Saree):</strong></p><p>CG@1 = 1</p><p>CG@2 = 1+0 = 1</p><p>CG@3 = 1+0+0 = 1</p><p>CG@4 = 1+0+0+1 = 2</p><p><strong>For query q2(</strong><strong>Bedsheet):</strong></p><p>CG@1 = 0</p><p>CG@2 = 0+1 = 1</p><p>CG@3 = 0+1+0 = 1</p><p>CG@4 = 0+1+0+1 = 2</p><p><strong>For query q3(</strong><strong>sports shoes for men):</strong></p><p>CG@1 = 0</p><p>CG@2 = 0+0 = 0</p><p>CG@3 = 0+0+0 = 0</p><p>CG@4 = 0+0+0+1 = 1</p><p>Across the queries we can report the average at different value of p.</p><p>CG@1 = (1+0+0)/3 = 0.33, CG@2 = (1+1+0)/3 = 0.66, CG@4 = (2+2+1)/3 = 1.66</p><h4>Non-Binary Relevance</h4><p><strong>For query q1(</strong><strong>Red Saree):</strong></p><p>CG@1 = 4</p><p>CG@2 = 4+0 = 4</p><p>CG@3 = 4+0+0 = 4</p><p>CG@4 = 4+0+0+5 = 9</p><p><strong>For query q2(</strong><strong>Bedsheet):</strong></p><p>CG@1 = 0</p><p>CG@2 = 0+5 = 5</p><p>CG@3 = 0+5+0 = 5</p><p>CG@4 = 0+5+0+2 = 7</p><p><strong>For query q3(</strong><strong>sports shoes for men):</strong></p><p>CG@1 = 0</p><p>CG@2 = 0+0 = 0</p><p>CG@3 = 0+0+0 = 0</p><p>CG@4 = 0+0+0+3 = 3</p><p>Similarly for this case as well, across the queries we can report the average at different value of p.</p><p>CG@1 = (4+0+0)/3 = 1.33, CG@2 = (4+5+0)/3 = 3.0, CG@4 = (9+7+3)/3 = 6.33</p><h4>Limitations of Cumulative Gain(CG)</h4><p>In the above example for binary relevance case we can see that for q1 relevant items is at the first and the 4th position however for q2 it is at 2nd and the 4th position but still they have same CG, but clearly q1 has better ranking that q2. Hence, CG is not a proper measure of the ranking systems because:</p><ol><li>It does not consider the rank of items</li><li>It is not normalized as it does not consider total number of example for a query(or session).</li></ol><h3>2) Discounted Cumulative Gain(DCG)</h3><p>DCG solves the above mentioned two limitations of CG. It normalizes the relevance by dividing it with the log of positions instead of directly taking the sum of relevance. It makes sure that highly relevant items appearing in the bottom should be penalized more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*BhGDNaczbXcRMo2H7NiKfw.png" /><figcaption>source: <a href="https://en.wikipedia.org/wiki/Discounted_cumulative_gain">wikipedia</a></figcaption></figure><p>Let’s calculate DCG for the above examples.</p><h4>Binary Relevance</h4><p><strong>For query q1(</strong><strong>Red Saree)</strong>:</p><p>DCG@1 = 1/log(1+1) = 1/1 = 1</p><p>DCG@2 = 1/log(2)+0/log(3) = 1</p><p>DCG@3 = 1/log(2)+0/log(3)+0/log(4) = 1</p><p>DCG@4 = 1/log(2)+0/log(3)+0/log(4) + 1/log(5) = 1.431</p><p><strong>For quer q2(</strong><strong>Bedsheet)</strong>:</p><p>DCG@1 = 0/log(1+1) = 0/1 = 0</p><p>DCG@2 = 0/log(2)+1/log(3) = 0.631</p><p>DCG@3 = 0/log(2)+1/log(3)+0/log(4) = 0.631</p><p>DCG@4 = 0/log(2)+1/log(3)+0/log(4) + 1/log(5) = 1.061606312</p><p><strong>For query q3(</strong><strong>sports shoes for men)</strong>:</p><p>DCG@1 = 0/log(1+1) = 0/1 = 0</p><p>DCG@2 = 0/log(2)+0/log(3) = 0</p><p>DCG@3 = 0/log(2)+0/log(3)+0/log(4) = 0</p><p>DCG@4 = 0/log(2)+0/log(3)+0/log(4) + 1/log(5) = 0.4306765581</p><h4>Non-Binary Relevance</h4><p>Similarly we can calculate DCG for non-binary relevance example as well.</p><p><strong>For query q1(</strong><strong>Red Saree)</strong>:</p><p>DCG@1 = 4/log(1+1) = 4/1 = 4</p><p>DCG@2 = 4/log(2)+0/log(3) = 4</p><p>DCG@3 = 4/log(2)+0/log(3)+0/log(4) = 4</p><p>DCG@4 = 4/log(2)+0/log(3)+0/log(4) + 5/log(5) = 4+2.15 = 6.15</p><p><strong>For quer q2(</strong><strong>Bedsheet)</strong>:</p><p>DCG@1 = 0/log(1+1) = 0/1 = 0</p><p>DCG@2 = 0/log(2)+5/log(3) = 3.15</p><p>DCG@3 = 0/log(2)+5/log(3)+0/log(4) = 3.15</p><p>DCG@4 = 0/log(2)+5/log(3)+0/log(4) + 2/log(5) = 3.15+0.86 = 4.01</p><p><strong>For query q3(</strong><strong>sports shoes for men)</strong>:</p><p>DCG@1 = 0/log(1+1) = 0/1 = 0</p><p>DCG@2 = 0/log(2)+0/log(3) = 0</p><p>DCG@3 = 0/log(2)+0/log(3)+0/log(4) = 0</p><p>DCG@4 = 0/log(2)+0/log(3)+0/log(4) + 3/log(5) = 1.29</p><h3>3) Normalized Discounted Cumulative Gain(NDCG)</h3><p>One of the major drawback of DCG is that the it is not normalized, for this reason it can not be used to compare the performance of a ranker or retrieval systems across the queries. For example, for q3 there is only 1 relevant item and query 1 and 2 has 2 relevant items. So, it is not fair to compare the relevance of these queries. To solve this issue we divide the <strong>DCG</strong> with the <strong>Ideal DCG(IDCG) </strong>for each queries. But now, how to define IDCG? How can we know what an ideal relevance score is?</p><p>As explained in the beginning, in any ideal ranking system, most relevant item should come first. So we can just sort the relevance score in descending order and then calculate DCG for that. Let’s calculate IDCG and then NDCG for the non-binary case of query q1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rqtu01DE5ilV6PMUo_x__w.png" /><figcaption>q1 data with sorted relevance</figcaption></figure><p>IDCG@1 = 5/log(1+1) = 5/1 = 5</p><p>IDCG@2 = 5/log(2)+4/log(3) = 5+ 2.52=7.52</p><p>IDCG@3 = 5/log(2)+4/log(3)+0/log(4) = 7.52</p><p>IDCG@4 = 5/log(2)+4/log(3)+0/log(4) + 0/log(5) = 4+2.15 = 7.52</p><p>We had already calculated the DCG for this example previously, we will use that to calculate NDCG.</p><p>NDCG@1 = 4/5 = 0.8<br>NDCG@2 = 4/7.52 = 0.53<br>NDCG@3 = 4/7.52 = 0.53<br>NDCG@4 = 6.15/7.52 = 0.82</p><p>So, finally NDCG can be defined as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BNHkYb8gAh947Oh6xnKGdQ.png" /><figcaption>Final Definition of NDCG where i’ represents the sorted order of relevance as explained above</figcaption></figure><p>We will write code for calculating NDCG and other ranking metrics in the next set of blogs.</p><h3>References</h3><ol><li><a href="https://en.wikipedia.org/wiki/Discounted_cumulative_gain">https://en.wikipedia.org/wiki/Discounted_cumulative_gain</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ndcg_score.html">https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ndcg_score.html</a></li><li><a href="https://scikit-learn.org/stable/modules/model_evaluation.html">https://scikit-learn.org/stable/modules/model_evaluation.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=885656321b3b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Product Embeddings for Shopping Recommendations at Pinterest]]></title>
            <link>https://medium.com/@mukulranjan/paper-review-itemsage-pinterest-cc1dbff5dc52?source=rss-abdbc773a16e------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc1dbff5dc52</guid>
            <dc:creator><![CDATA[Mukul Ranjan]]></dc:creator>
            <pubDate>Sat, 27 May 2023 18:54:12 GMT</pubDate>
            <atom:updated>2024-08-28T19:09:14.683Z</atom:updated>
            <content:encoded><![CDATA[<h4>[Paper Summary] ItemSage Pinterest</h4><blockquote>While most prior work focuses on building product embeddings from features coming from a single modality, we introduce a transformer-based architecture capable of aggregating information from both text and image modalities and show that it significantly outperforms single modality baselines.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QkjNiQO68jcRV5WFJwJifw.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2205.11728.pdf">ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest (arxiv.org)</a></figcaption></figure><p>ItemSage has been used on Pinterest:</p><ul><li>For generating candidates to upstream ranking systems via approximate nearest neighbor search</li><li>As featured in the ranking models responsible for determining the final ordering of the products shown to users</li><li>As signals in classification models aimed at inferring missing information from the shopping catalog (e.g., the category or the gender affinity for a specific product).</li></ul><h3>Difference from Existing Approaches</h3><h4>1. Multi-modal features:</h4><p><strong>It has both textual and image features</strong></p><ul><li>Introduce a transformer-based architecture capable of combining features from these different modalities.</li></ul><h4>2. Multi-modal vertical recommendations</h4><p>ItemSage can provide relevant candidates via an approximate neighbor search for all the surfaces and, therefore, in response to queries formulated in each of these modalities.</p><p>Pinterest has three surfaces:</p><ul><li><strong>Home surface:</strong> users are provided with recommendations based on their past activity. <br>The query here is a <strong>sequence of pins.</strong></li><li><strong>Closeup surface:</strong> Provide similar recommendations to a pin the user sees. <br>Query here is a <strong>single-pin</strong></li><li><strong>Search surface:</strong> Provide recommendations in response to a query string that the user has typed.<br> The query is a <strong>text string</strong></li></ul><h4>3. Multi-task Learning</h4><p>Optimize for several objectives: purchases, add-to-cart actions, saves, and clicks.</p><ul><li>Traditionally, recommendation systems tackle multi-objective optimization in the ranking phase by building models capable of computing a score for every engagement type under consideration conditioned on the query context and the ranked candidate.</li><li><strong>Ranking and Candidate Generation are at the same stage.</strong></li><li>Learn embeddings that additionally optimize the candidate generation phase for all engagement types under consideration, leading to an overall more efficient recommendation funnel.</li><li>Authors show that in cases where the labels are sparse, multi-task learning leads to improved results over models specialized only on sparse objectives.</li><li>For objectives where labeled data is plentiful, authors show that <strong>they</strong> <strong>can optimize their embeddings for new objectives with little or no performance penalty on existing objectives.</strong></li><li>The embeddings are learned in a multi-task regime that supports cross-modal candidate retrieval and joint optimization for several engagement types.</li></ul><h3>Features and Labels</h3><ul><li>A product consists of a list of images depicting the product from different angles or in different contexts and a list of text features. Authors truncate the list of images to at most 20 (99.7% of products in their catalog have less than or equal to 20 images). <br> — Each image is represented by its pre-trained PinSage embedding<br> — A zero embedding represents each empty slot for products with fewer than 20 images.</li><li>12 text features as input to the model<br>1. Title<br>2. Description<br>3. Merchant domain<br>4. Product links<br>5. Google product category(<em>The google product category represents the node from a standard taxonomy that merchants may tag their products with. The taxonomy can be found </em><a href="https://www.google.com/basepages/producttype/taxonomy.en-US.txt"><em>here</em> </a>)<br>6. Product types (<em>The product types are HTML breadcrumbs scraped from the merchant product page, e.g., Home Decor &gt; New Arrivals.</em>)<br>7. Brand<br>8. Colors<br>9. Materials<br>10. Patterns<br>11. Size<br>12. Size type</li><li>In cases where a product may have several values for a particular feature (e.g., links, colors, etc.), these strings are concatenated into one string.</li><li>Standard word-level tokenization and lowercasing are applied to all text features.</li><li>Each processed string is represented as a bag of word unigrams, bigrams, and <a href="https://posenhuang.github.io/papers/cikm2013_DSSM_fullversion.pdf">character trigrams</a>.</li><li>The tokens are mapped to numerical IDs using a vocabulary <strong><em>V</em></strong> of the most frequent 200,000-word unigrams, 1,000,000-word bigrams, and 64,000-character trigrams, and out-of-vocabulary tokens are discarded.</li></ul><h4>Constructing Training Dataset</h4><ul><li>Collect labels from the Closeup and Search engagement logs.</li><li>Each positive example is a <strong>pair of query and engaged products</strong> where the <strong>query represents either an image for examples mined from Closeup logs or a text string for search logs.</strong></li><li>Select four engagement types to train their models:<br>- <strong>clicks </strong>and <strong>saves</strong> are the main actions that users can take on any Pinterest content<br>- <strong>add-to-cart</strong> actions and <strong>checkouts</strong> are actions that express shopping intent.</li><li><strong>Loss uses random negative labels sampled randomly from the shopping catalog.<br>- </strong>The negative labels are joined with the features offline and streamed into the model trainer through a separate data loader. <br>- The trainer alternates between <strong>consuming a batch of positive and negative labels, which are then concatenated into a single training batch.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GJfdKWj05Gyqu20BwAVoSQ.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2205.11728.pdf">ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest (arxiv.org)</a></figcaption></figure><h3>Model Architecture and Loss Function</h3><p>Uses a transformer encoder as the basic building block for learning product embeddings.</p><h4>Input</h4><p>A sequence of 32 embeddings represents each product’s image and text features.</p><ul><li>The pre-trained PinSage model generates the image embeddings, while the text embeddings are learned jointly with the encoder.</li><li>To deal with the extensive vocabulary size, they apply the hash embedding trick from <a href="https://arxiv.org/abs/1709.03933">this</a> paper to learn a compact embedding table.</li></ul><p><strong>The Hash Embedding</strong></p><ul><li>The hash embedder consists of an embedding table <strong>E</strong> of size <em>100, 000 × 256</em> and an importance weight table <strong>W</strong> of size <strong><em>|V | × 2</em></strong>. They use two hashing functions <strong><em>ℎ1, ℎ2</em></strong> to map each token ID <em>i = 1, . . . , |V |</em> into two slots in the embedding table <strong><em>ℎ1 (i), ℎ2 (i)</em></strong>. The embedding of the token with ID ‘i’<em> </em>is then the weighted interpolation of the two embeddings:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*qhZ67HhDDHrVORhfiEp2-Q.png" /></figure><ul><li>The final embedding of a feature string is the summation of all its token embeddings.</li></ul><h4>Model Structure</h4><ul><li>Apply a linear transformation with output size 512 for each group of feature embeddings to relax the requirement that all input features must have the same dimension before passing it to the transformer Encoder(Figure 3).</li><li>It uses a global token [CLS] to aggregate the information from the input sequence. The transformed embeddings are concatenated with the global token and then passed through a one-layer transformer block consisting of a self-attention module with eight heads and a feed-forward module with one hidden layer(Figure 3).</li><li>The output corresponding to the global token then goes through a two-layer MLP head to <strong>produce the 256-dimension product embedding</strong>. When performing an ANN search, the final ItemSage embedding is L2-normalized for easier computation of cosine similarity with query embeddings.</li></ul><h4>Loss</h4><p>The goal is to predict the product from the catalog that the user will engage with next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3lFylcBBXu42sMgkzgwlTA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*caKc3MjtNv4wfgrqrJBvCg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jg-XuTFeRdVfULJdj3702A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Ach3TSj2CCXv5_2d7Vl5A.png" /></figure><ul><li>Obtain better results separating the loss terms of the two negative sampling approaches.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4QZK7OyiW-fU2g5n-qI5CQ.png" /></figure><h3>Inference and Serving</h3><p>Figure 5 illustrates how ItemSage embeddings are deployed to power Pinterest shopping recommendations.</p><p>The <strong>embeddings are inferred in a batch workflow</strong>, <strong>which ensures that the model inference latency does not impact the end-to-end latency of the vertical recommendation systems</strong>.</p><p>The <strong>inference workflow runs daily to update the embeddings based on the latest features and to extend the coverage to newly ingested products</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pafS6AQuTO4PAvmN4dk4ng.png" /></figure><ul><li>Each new set of embeddings creates an HNSW index for candidate generation and is also pushed to the online feature store for ranking applications.</li><li>The different vertical systems reuse the HNSW index and the feature set for Home, Closeup, and Search recommendations.</li><li>The Home and Closeup surfaces use precomputed PinSage embeddings fetched from the feature store as queries. In contrast, in Search, the query embeddings are inferred on the fly to support the long tail of previously unseen queries.</li></ul><h3>References</h3><ol><li><a href="https://www.google.com/basepages/producttype/taxonomy.en-US.txt">https://www.google.com/basepages/producttype/taxonomy.en-US.txt</a></li><li>Learning Deep Structured Semantic Models for Web Search using Clickthrough Data: <a href="https://posenhuang.github.io/papers/cikm2013_DSSM_fullversion.pdf">https://posenhuang.github.io/papers/cikm2013_DSSM_fullversion.pdf</a></li><li>Hash Embeddings for Efficient Word Representations: <a href="https://arxiv.org/abs/1709.03933">https://arxiv.org/abs/1709.03933</a></li><li>ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest: <a href="https://ar5iv.labs.arxiv.org/html/2205.11728">https://ar5iv.labs.arxiv.org/html/2205.11728</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc1dbff5dc52" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>