<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anvee Naik on Medium]]></title>
        <description><![CDATA[Stories by Anvee Naik on Medium]]></description>
        <link>https://medium.com/@AnveeNaik?source=rss-b754131c5a9b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Aj8SOZYL_d1TtCs0kCa4zw.jpeg</url>
            <title>Stories by Anvee Naik on Medium</title>
            <link>https://medium.com/@AnveeNaik?source=rss-b754131c5a9b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:31:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@AnveeNaik/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Making LLMs Lighter: A deep dive into LLM quantization with Code]]></title>
            <link>https://medium.com/data-science-collective/making-llms-lighter-a-deep-dive-into-llm-quantization-with-code-70aff50993dc?source=rss-b754131c5a9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/70aff50993dc</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[quantization]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gptq]]></category>
            <dc:creator><![CDATA[Anvee Naik]]></dc:creator>
            <pubDate>Mon, 24 Feb 2025 17:31:57 GMT</pubDate>
            <atom:updated>2025-02-24T17:31:57.804Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understand how LLM quantization methods such as LLM.int8() and GPTQ works</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*peNFyVsRIHwBWfU7" /><figcaption>Create by Dalle</figcaption></figure><p>The “Large language model” itself says Large so needless to say these model have large number of model parameters. In my <a href="https://medium.com/towards-artificial-intelligence/from-fp32-to-int8-the-science-of-shrinking-ai-models-514741f81eec">previous blog</a>, we explored different data types for representing numbers and some basic quantization technique such as Absmax and Zero-point. In this blog, I will introduce more nuanced approaches which are commonly employed for quantizing LLMs. For this blog I will assume that readers have basic understanding of quantization, if not I recommend going through my <a href="https://medium.com/towards-artificial-intelligence/from-fp32-to-int8-the-science-of-shrinking-ai-models-514741f81eec">previous blog</a> first. Before we delve into the algorithms, let’s first understand the problem of outlier.</p><h3>Outlier</h3><p>Imagine that you have a vector with the following values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*Pvq4ChbcnyuBZZaB6TBFNA.png" /><figcaption>Image by author</figcaption></figure><p>Here, 200 is clearly an outlier. With quantization algorithms, such as absmax and zero-point, where in you divide by the maximum value we can see how such an outlier can disproportionately scale down the entire vector, effectively pushing most values to zero. This is an issue since a single outlier can reduce the precision for all other values. But discarding these outlier features is not an option, as they may contain important information crucial for the model’s performance. Why am I bringing the topic of outlier now? Because it has been observed that outlier problem become prominent when we deal with larger models. So when quantzing LLMs we need to take care of these outliers as well to maintain model performance.</p><h3>LLM.int8()</h3><p>This method is one of the easiest and most frequently used method for quantizing a LLM. This approach relies on two key-techniques 1) vector-wise quantization 2) mixed precision decomposition. Let’s understand vector wise quantization first. In Absmax quantization we calculate maximum value from the entire tensor and use it as scaling factor to quantize all the elements, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/0*ZzdLaKdhnkDfrcSm.png" /><figcaption>Image by Author: Absmax Quantization</figcaption></figure><p>However, as we discussed, this approach struggles with <strong>outliers</strong> — a single large value can shrink the precision of all other elements. In vector wise quantization instead of taking maximum from entire tensor, we take maximum from each row. Each row is then quantized using its corresponding maximum value. This vector wise quantization handles the problem of outlier partially by ensuring that one extreme value present in a row do not affect scaling of other rows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*Iow_mgg_qQkZmk9WFa17Qw.png" /><figcaption>Image from <a href="https://arxiv.org/pdf/2208.07339">https://arxiv.org/pdf/2208.07339</a></figcaption></figure><p>Cₓ represents row-wise scale factor for input likewise Cᵧ represent column-wise scale factor for weights. We first calculate Cₓ and Cᵧ and use them to quantize the input and weight respectively. Then we perform matrix multiplication, which results the output in INT32 format. Why INT32 if both the inputs are INT8? Because during matrix multiplication, we multiply values and then take sum, so to ensure that overflow do not happen we use INT32 format. After multiplication, we de-quantize the output by using Cₓ and Cᵧ.</p><p>Apart from vector wise quantization, authors of LLM.int8() paper noticed that some of the input features have large magnitude, are crucial for model performance and require high precision quantization. To handle such features, the matrix multiplication for such features happen in 16-bit floating operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*_CdYkYI6RWuyJfBon5hijA.png" /><figcaption>Image from <a href="https://arxiv.org/pdf/2208.07339">https://arxiv.org/pdf/2208.07339</a></figcaption></figure><p>So as you can see from the figure above, we first decompose small and large magnitude features present in input. For small magnitude, we perform vector-wise quantization and for larger magnitude we perform calculation in 16 bit. Finally we combine both the features by doing the sum.</p><h4>Code Implementation</h4><p>Following is the implementation of what we discussed so far.</p><pre>import torch<br><br>def llm_int8_quantization(input_tensor: torch.Tensor, <br>                          weights: torch.Tensor, <br>                          alpha: float) -&gt; torch.Tensor:<br>    &quot;&quot;&quot;<br>    Quantizes the input and weight matrices using a mixed-precision scheme:<br>      - Small magnitude features (columns where the max absolute value is &lt;= alpha)<br>        are quantized using vector-wise (row-wise for input and column-wise for weights)<br>        quantization.<br>      - Large magnitude features (columns with max abs &gt; alpha) are computed in float16.<br>    The results are then summed to obtain the final output.<br>    <br>    Args:<br>        input_tensor (torch.Tensor): Input tensor of shape (sequence_length, feature_dim)<br>        weights (torch.Tensor): Weight matrix of shape (feature_dim, output_dim)<br>        alpha (float): Threshold to distinguish between small and large magnitude columns.<br>        <br>    Returns:<br>        torch.Tensor: Output tensor of shape (sequence_length, output_dim)<br>    &quot;&quot;&quot;<br>    # Determine which columns (features) are &quot;small&quot; vs. &quot;large&quot;<br>    max_abs_per_col = torch.max(torch.abs(input_tensor), dim=0).values<br>    small_mask = max_abs_per_col &lt;= alpha<br>    large_mask = max_abs_per_col &gt; alpha<br>    print(small_mask)<br>    print(large_mask)<br><br>    # Split the input and weight matrices accordingly<br>    input_small = input_tensor[:, small_mask]         <br>    weights_small = weights[small_mask, :]            <br><br>    input_large = input_tensor[:, large_mask]         <br>    weights_large = weights[large_mask, :]            <br>    <br>    # For the input: perform row-wise quantization.<br>    # Compute the row-wise (per sample) scale factor for the small features.<br>    scale_input = torch.max(torch.abs(input_small), dim=1, keepdim=True).values  # Shape: (sequence_length, 1)<br>    print(f&quot;Scale input: {scale_input.shape}&quot;)<br>    scale_input[scale_input == 0] = 1.0<br>    input_small_quant = torch.round((input_small / scale_input) * 127).to(torch.int8)<br>    print(f&quot;Input Small: {input_small_quant.shape}&quot;)<br><br>    # For the weights: perform column-wise quantization.<br>    scale_weights = torch.max(torch.abs(weights_small), dim=0, keepdim=True).values  # Shape: (1, output_dim)<br>    scale_weights[scale_weights == 0] = 1.0<br>    weights_small_quant = torch.round((weights_small / scale_weights) * 127).to(torch.int8)<br>    print(f&quot;Weight Small: {weights_small_quant.shape}&quot;)<br><br>    # Matrix multiplication with quantized values.<br>    result_small_int32 = torch.matmul(input_small_quant.to(torch.int32),<br>                                        weights_small_quant.to(torch.int32))<br>    # Dequantize: the effective scale is the product of the input and weight scales,<br>    result_small = result_small_int32.to(torch.float32) * (scale_input * scale_weights / (127 * 127))<br>    print(f&quot;Result Small: {result_small.shape}&quot;)<br>    <br>    # Processing for Large Features <br>    # For columns with large magnitude, we use float16 for higher precision.<br>    input_large_fp16 = input_large.to(torch.float16)<br>    print(f&quot;Input large: {input_large_fp16.shape}&quot;)<br>    weights_large_fp16 = weights_large.to(torch.float16)<br>    print(f&quot;Weight large: {weights_large_fp16.shape}&quot;)<br>    result_large = torch.matmul(input_large_fp16, weights_large_fp16).to(torch.float32)<br>    <br>    # Combine the Results <br>    output = result_small + result_large<br>    return output<br><br>if __name__ == &quot;__main__&quot;:<br>    # Dummy data: a sequence_length of 4 with 10 features and a weight matrix for 5 output neurons.<br>    sequence_length, feature_dim, output_dim = 4, 10, 5<br>    torch.manual_seed(0)<br>    input_tensor = torch.randn(sequence_length, feature_dim)<br>    weights = torch.randn(feature_dim, output_dim)<br>    alpha = 1.5  # Threshold for distinguishing large vs. small magnitude features<br>    output = llm_int8_quantization(input_tensor, weights, alpha)<br>    output_complete = torch.matmul(input_tensor, weights)<br>    #compute MSE b/w output and output_compelete<br>    mse = torch.mean((output - output_complete) ** 2)<br>    print(&quot;MSE: &quot;, mse)</pre><pre>#output for above<br>tensor([ True, False,  True, False,  True,  True,  True, False, False, False])<br>tensor([False,  True, False,  True, False, False, False,  True,  True,  True])<br>Scale input: torch.Size([4, 1])<br>Input Small: torch.Size([4, 5])<br>Weight Small: torch.Size([5, 5])<br>Result Small: torch.Size([4, 5])<br>Input large: torch.Size([4, 5])<br>Weight large: torch.Size([5, 5])<br>MSE:  tensor(1.5778e-05)</pre><p>In practise, you would not use this implementation. It was just for understanding. For practical purposes, bitsandbytes library has come to save everyone, it takes two argement load_in_8_bit and alpha. I have printed out the memory requirement for both the model and you can see how the requirement dropped to 1/3rd. (it’s not 1/4 since we are keeping some weights in original format as well).</p><pre>from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig<br>import torch<br>torch.manual_seed(0)<br><br>device = &#39;cpu&#39;<br><br># Load model and tokenizer<br>model_id = &#39;gpt2&#39;<br>model = AutoModelForCausalLM.from_pretrained(model_id).to(device)<br>tokenizer = AutoTokenizer.from_pretrained(model_id)<br><br># Print model size<br>print(f&quot;Model size: {model.get_memory_footprint():,} bytes&quot;)<br><br>device = torch.device(&#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39;)<br><br>quantization_config = BitsAndBytesConfig(load_in_8bit=True, llm_int8_threshold=6.0)<br><br>model_int8 = AutoModelForCausalLM.from_pretrained(model_id,<br>                                             device_map=&#39;auto&#39;,<br>                                            quantization_config=quantization_config<br>                                             )<br>print(f&quot;Model size Quantized: {model_int8.get_memory_footprint():,} bytes&quot;)</pre><pre>Model size: 510,342,192 bytes<br>Model size Quantized: 176,527,896 bytes</pre><h3>GPTQ Algorithm</h3><p>So far our focus was to convert model weights into INT8 format. With GPTQ, we take it a step further and aim to represent weights in INT4 format. In GPTQ, we do layer-by-layer quantization, with zero-point quantization. However, instead of blindly applying zero-point quantization to all weights present in layer, we optimize the following formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*BjHFVQfjHnDRUXiY2lFIYQ.png" /><figcaption>Image from <a href="https://arxiv.org/pdf/2210.17323">https://arxiv.org/pdf/2210.17323</a></figcaption></figure><p>Where w is original model parameter and ŵ represents quantized weights. So essentially we are trying to minimize the error between the output of layer with quantized and original weights.</p><h4>Hessian Matrix</h4><p>For this minimization, we compute hessian matrix, which is simply second order derivative of this error function. Once we have the hessian, we calculate the inverse of hessian which is used to determine how sensitive model parameters (weights) are to perturbations. How? Well if Hessian is large then loss function is highly curved which means small weight changes cause a large loss increase. Hence if Hessian Inverse is large, then loss function is flat in that direction so change in weights will have less effects. If Hessian Inverse is small, then loss function is not flat in that direction so change in weights will have more impact. Ideally we would want to change those weights more which have large hessian inverse compared to weight with smaller hessian inverse.</p><p>In GPTQ, hessian is computed using an approximation. It approximates the Hessian using a low-rank approximation technique. The key idea is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*TTxZ700psn7BAglD-O39fw.png" /><figcaption>Image by Author</figcaption></figure><p>where X​ is activation vector i.e. output of the layer. I have written below sample code to compute hessian as per GPTQ. If you want to look at their original implementation check <a href="https://github.com/IST-DASLab/gptq/blob/main/gptq.py#L32">this</a>.</p><p><strong>Code Implementation</strong></p><pre>import torch<br><br>class HessianApproximator:<br>    def __init__(self, hidden_dim):<br>        &quot;&quot;&quot;<br>        Initialize Hessian approximation matrix H.<br>        Args:<br>        - hidden_dim (int): Size of the input activation vectors.<br>        &quot;&quot;&quot;<br>        self.H = torch.zeros(hidden_dim, hidden_dim)  # Initialize Hessian approx<br>        self.nsamples = 0  # Track number of samples seen<br><br>    def update(self, activations):<br>        &quot;&quot;&quot;<br>        Update the Hessian approximation using a batch of activations.<br>        Args:<br>        - activations (torch.Tensor): Shape (batch_size, hidden_dim)<br>        &quot;&quot;&quot;<br>        batch_size = activations.shape[0]<br><br>        # Normalize activation input<br>        scale = torch.sqrt(torch.tensor(2.0 / (self.nsamples + batch_size)))<br>        activations = scale * activations.float()<br><br>        # Compute outer product<br>        self.H *= self.nsamples / (self.nsamples + batch_size)  # Decay old information<br>        self.H += activations.T @ activations  # Accumulate new information<br><br>        # Update sample count<br>        self.nsamples += batch_size<br><br>    def get_hessian(self):<br>        &quot;&quot;&quot;<br>        Returns the approximated Hessian matrix.<br>        &quot;&quot;&quot;<br>        return self.H<br><br># Example Usage<br>hidden_dim = 128  # Example hidden size<br>hessian_approx = HessianApproximator(hidden_dim)<br><br># Simulated activations (batch of size 32)<br>for _ in range(10):  # Simulate 10 batches<br>    activations = torch.randn(32, hidden_dim)  # Random activations<br>    hessian_approx.update(activations)<br>print(hessian_approx.get_hessian())</pre><h4>Quantization</h4><p>With this background, let’s understand how quantization is done. Assume we have a weight matrix (<strong>W</strong>) of shape [10, 100]. We will first quantize the first column (w₁) of the weight using zero-point quantization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*4-eVP4awRpCu28kUt6dsrA.png" /><figcaption>Image by Author</figcaption></figure><p>where i = 1, when we are quantizing first columns. Then we will compute the quantization error (e₁) by subtracting the weight and quantized output (w₁-q₁) and divide it with the corresponding inverse hessian value (d) for that direction i.e. column vector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*HFykPJ3gTjJERWhEpMkpig.png" /><figcaption>Image by Author</figcaption></figure><p>Once we have quantization error, we update the rest of the weights i.e. second column onwards by adding the product of Hessian inverse value of that column and quantization error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*XRXLNsTDGgli93tUmA4Erw.png" /><figcaption>Image By Author</figcaption></figure><p>This way, we are focusing on two things. 1) During quantizing the column, if inverse hessian is small i.e. the weights are important in that case we would want that the error corresponding to them should be zeroed out during further quantization. 2) While updating the rest of the weight, if their hessian inverse value is small we would want to update them less.</p><p><strong>Code Implementation</strong></p><p>This is a raw implementation of what we discussed. This is not something which is generally used for practical purposes, but it’s good to see the code to gauge a better understanding of things.</p><pre>def iterative_quantization(layer, H, device=&#39;cpu&#39;):<br>    &quot;&quot;&quot;<br>    Args:<br>        layer (torch.nn.Module): The layer to be quantized. Its weight is expected to be 2D <br>                                 with shape (out_features, in_features).<br>        H (torch.Tensor): Approximated Hessian matrix (in_features x in_features).<br>        device (str or torch.device): Device for computations (e.g., &#39;cpu&#39; or &#39;cuda&#39;).<br>    <br>    Returns:<br>        torch.nn.Module: The same layer with its weights quantized.<br>    &quot;&quot;&quot;<br>    # Clone and convert the weight matrix to float.<br>    W = layer.weight.data.clone().float()  # Shape: (out_features, in_features)<br>    # Handle potential &quot;dead&quot; weights by zeroing out columns where the Hessian diagonal is zero.<br>    dead = torch.diag(H) == 0<br>    H[dead, dead] = 1<br>    W[:, dead] = 0<br>    # Compute the inverse Hessian for error compensation.<br>    Hinv = compute_hessian_inverse(H)<br>    <br>    # Prepare an empty tensor for storing quantized weights.<br>    Q = torch.zeros_like(W)<br>    <br>    # Number of columns (i.e., in_features)<br>    n_cols = W.shape[1]<br>    <br>    # Iteratively quantize each column.<br>    for i in range(n_cols):<br>        # Get the current weight column.<br>        w = W[:, i]<br>        <br>        # Quantize the column using the provided quantizer.<br>        q = zero_point_quantization(w)<br>        Q[:, i] = q<br>        <br>        # Use the Hessian inverse to compute a sensitivity factor.<br>        d = Hinv[i, i]<br>        <br>        # Compute the quantization error (scaled by the sensitivity).<br>        error = (w - q) / d<br>        <br>        # Adjust subsequent columns based on error compensation.<br>        if i &lt; n_cols - 1:<br>            # Hinv[i, i+1:] is a row vector containing sensitivity info for later columns.<br>            compensation = error * Hinv[i, i+1:]<br>            # Subtract the compensation from all subsequent columns.<br>            W[:, i+1:] = W[:, i+1:] - compensation.unsqueeze(0)<br>    <br>    # Print the quantized weight shape for verification.<br>    print(&quot;Quantized weight shape:&quot;, Q.shape)</pre><p>I would not say that we have covered entire GPTQ algorithm so far, as there are more optimization which are done in the paper. But explaination so far cover it’s core working. You can read about additional optimization in <a href="https://arxiv.org/pdf/2210.17323">paper</a>.</p><p><strong>Code Implementation</strong></p><p>Now we will see, how you can implement the same using transformers library which is commonly preferred for practical use cases.</p><pre>import random<br><br>from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig<br>from datasets import load_dataset<br>import torch<br>from transformers import AutoTokenizer<br><br><br># Define base model and output directory<br>model_id = &quot;gpt2&quot;<br>out_dir = model_id + &quot;-GPTQ&quot;<br><br># Load quantize config, model and tokenizer<br>quantize_config = BaseQuantizeConfig(<br>    bits=4,<br>    group_size=128,<br>    damp_percent=0.01,<br>    desc_act=False,<br>)<br>model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_config)<br>tokenizer = AutoTokenizer.from_pretrained(model_id)<br><br><br># Load data and tokenize examples<br>n_samples = 1024<br>data = load_dataset(&quot;allenai/c4&quot;, data_files=&quot;en/c4-train.00001-of-01024.json.gz&quot;, split=f&quot;train[:{n_samples*5}]&quot;)<br>tokenized_data = tokenizer(&quot;\n\n&quot;.join(data[&#39;text&#39;]), return_tensors=&#39;pt&#39;)<br><br># Format tokenized examples<br>examples_ids = []<br>for _ in range(n_samples):<br>    i = random.randint(0, tokenized_data.input_ids.shape[1] - tokenizer.model_max_length - 1)<br>    j = i + tokenizer.model_max_length<br>    input_ids = tokenized_data.input_ids[:, i:j]<br>    attention_mask = torch.ones_like(input_ids)<br>    examples_ids.append({&#39;input_ids&#39;: input_ids, &#39;attention_mask&#39;: attention_mask})<br><br># Quantize with GPTQ<br>model.quantize(<br>    examples_ids,<br>    batch_size=1,<br>    use_triton=True,<br>)<br><br># Save model and tokenizer<br>model.save_quantized(out_dir, use_safetensors=True)<br>tokenizer.save_pretrained(out_dir)</pre><h4>Comparing the models</h4><p>Now that we have quantized the model, we would want to compare their performance as well. For the purpose of comparing the different models (in this case quantized LLMs), we will be using perplexity. Perplexity can be interpreted as “the average branching factor of the language model.” Lower perplexity means the model is more confident in its predictions, while higher perplexity means the model is more uncertain. So lower the better!</p><pre>def calculate_perplexity(model, text):<br>    # Encode the text<br>    encodings = tokenizer(text, return_tensors=&#39;pt&#39;).to(device)<br>    # Define input_ids and target_ids<br>    input_ids = encodings.input_ids<br>    target_ids = input_ids.clone()<br>    with torch.no_grad():<br>        outputs = model(input_ids, labels=target_ids)<br>    # Loss calculation<br>    neg_log_likelihood = outputs.loss<br>    # Perplexity calculation<br>    ppl = torch.exp(neg_log_likelihood)<br>    return ppl<br><br>ppl = calculate_perplexity(model, text)<br>ppl_gptq = calculate_perplexity(model_gptq, text_gptq)<br>ppl_int8 = calculate_perplexity(model_int8, text_int8)<br><br>print(f&quot;Original Model perplexity: {ppl.item():.2f}&quot;)<br>print(f&quot;INT8 perplexity:    {ppl_int8.item():.2f}&quot;)<br>print(f&quot;GPTQ perplexity:  {ppl_gptq.item():.2f}&quot;)</pre><pre>#output from above<br>Original Model perplexity: 16.70<br>INT8 perplexity:    18.70<br>GPTQ perplexity:  21.27</pre><p>As you can see from above, that perplexity of quantized model is higher compared to normal model. Denoting the degrade in quality however thought, the increase is not a lot indicating that quality is not very poor.</p><h3>Conclusion</h3><p>Quantization is a powerful technique that makes LLMs more efficient by reducing memory usage and speeding up inference while keeping accuracy intact. Throughout this blog and <a href="https://medium.com/towards-artificial-intelligence/from-fp32-to-int8-the-science-of-shrinking-ai-models-514741f81eec">previous blog</a>, we saw how different quantization methods — from simple Absmax to more advanced techniques like LLM.int8() and GPTQ — handle challenges like outliers and precision loss. I walked through the intuition behind these methods, highlighting how GPTQ leverages Hessian-based optimization to push weight compression even further while keeping performance degradation minimal. You can find the code using transformers library in <a href="https://colab.research.google.com/drive/1rLTguLJf9OSq5A0rXK2EDC76zzWkBlL9?usp=sharing">this colab notebook</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70aff50993dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/making-llms-lighter-a-deep-dive-into-llm-quantization-with-code-70aff50993dc">Making LLMs Lighter: A deep dive into LLM quantization with Code</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From FP32 to INT8: The Science of Shrinking AI Models]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pub.towardsai.net/from-fp32-to-int8-the-science-of-shrinking-ai-models-514741f81eec?source=rss-b754131c5a9b------2"><img src="https://cdn-images-1.medium.com/max/1976/1*fW1nhQeOSLTHuYt6aW93Pw.png" width="1976"></a></p><p class="medium-feed-snippet">Understanding quantization of neural network along with their implementation</p><p class="medium-feed-link"><a href="https://pub.towardsai.net/from-fp32-to-int8-the-science-of-shrinking-ai-models-514741f81eec?source=rss-b754131c5a9b------2">Continue reading on Towards AI »</a></p></div>]]></description>
            <link>https://pub.towardsai.net/from-fp32-to-int8-the-science-of-shrinking-ai-models-514741f81eec?source=rss-b754131c5a9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/514741f81eec</guid>
            <category><![CDATA[neural-network-pruning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[quantization]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Anvee Naik]]></dc:creator>
            <pubDate>Wed, 12 Feb 2025 22:01:39 GMT</pubDate>
            <atom:updated>2025-02-13T18:06:11.939Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Understanding Agentic RAG and How It’s Different From RAG With Code]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pub.towardsai.net/understanding-agentic-rag-and-how-its-different-from-rag-with-code-cf627a8f3ef3?source=rss-b754131c5a9b------2"><img src="https://cdn-images-1.medium.com/max/1024/0*DIeaXVsfY6q2Zl3S" width="1024"></a></p><p class="medium-feed-snippet">In the world of Large Language Models (LLMs), Retrieval Augmented Generation (RAG) has emerged as a game-changer. Traditional RAG, while&#x2026;</p><p class="medium-feed-link"><a href="https://pub.towardsai.net/understanding-agentic-rag-and-how-its-different-from-rag-with-code-cf627a8f3ef3?source=rss-b754131c5a9b------2">Continue reading on Towards AI »</a></p></div>]]></description>
            <link>https://pub.towardsai.net/understanding-agentic-rag-and-how-its-different-from-rag-with-code-cf627a8f3ef3?source=rss-b754131c5a9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf627a8f3ef3</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Anvee Naik]]></dc:creator>
            <pubDate>Tue, 28 Jan 2025 20:01:55 GMT</pubDate>
            <atom:updated>2025-02-13T18:08:18.962Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Diving Deeper into LLM Agents along with Code]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://levelup.gitconnected.com/diving-deeper-into-llm-agents-along-with-code-2e6f7a66395f?source=rss-b754131c5a9b------2"><img src="https://cdn-images-1.medium.com/max/2600/0*DcXIssfGAvx7soSU" width="4000"></a></p><p class="medium-feed-snippet">Many of us have seen Iron Man movie, where Jarvis was an AI assistant for the iron man conducting various tasks for him. Well with the&#x2026;</p><p class="medium-feed-link"><a href="https://levelup.gitconnected.com/diving-deeper-into-llm-agents-along-with-code-2e6f7a66395f?source=rss-b754131c5a9b------2">Continue reading on Level Up Coding »</a></p></div>]]></description>
            <link>https://levelup.gitconnected.com/diving-deeper-into-llm-agents-along-with-code-2e6f7a66395f?source=rss-b754131c5a9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e6f7a66395f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Anvee Naik]]></dc:creator>
            <pubDate>Sun, 15 Dec 2024 18:07:44 GMT</pubDate>
            <atom:updated>2025-02-13T18:09:19.789Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Extending Context Length of an LLM: Intuition, Implementation & Interview Questions]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pub.towardsai.net/extending-context-length-of-an-llm-intuition-implementation-interview-questions-d830034f1ac9?source=rss-b754131c5a9b------2"><img src="https://cdn-images-1.medium.com/max/2600/0*NnNTR4keleLKpl1y" width="6000"></a></p><p class="medium-feed-snippet">Understand multiple methods like Positional Interpolation, NTK awareness, and Dynamic NTK for extending context length.</p><p class="medium-feed-link"><a href="https://pub.towardsai.net/extending-context-length-of-an-llm-intuition-implementation-interview-questions-d830034f1ac9?source=rss-b754131c5a9b------2">Continue reading on Towards AI »</a></p></div>]]></description>
            <link>https://pub.towardsai.net/extending-context-length-of-an-llm-intuition-implementation-interview-questions-d830034f1ac9?source=rss-b754131c5a9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d830034f1ac9</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[rope]]></category>
            <dc:creator><![CDATA[Anvee Naik]]></dc:creator>
            <pubDate>Tue, 02 Jul 2024 16:02:53 GMT</pubDate>
            <atom:updated>2025-02-13T18:15:00.205Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Rotary Positional Embedding(RoPE): Motivation and Code Implementation]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pub.towardsai.net/rotary-positional-embedding-rope-motivation-and-implementation-ac221926e7df?source=rss-b754131c5a9b------2"><img src="https://cdn-images-1.medium.com/max/2600/0*fRIQBiesxWjn3zBx" width="4288"></a></p><p class="medium-feed-snippet">Delve deeper into RoPE along with its code to understand the positional embedding in LLMs better</p><p class="medium-feed-link"><a href="https://pub.towardsai.net/rotary-positional-embedding-rope-motivation-and-implementation-ac221926e7df?source=rss-b754131c5a9b------2">Continue reading on Towards AI »</a></p></div>]]></description>
            <link>https://pub.towardsai.net/rotary-positional-embedding-rope-motivation-and-implementation-ac221926e7df?source=rss-b754131c5a9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac221926e7df</guid>
            <category><![CDATA[rope]]></category>
            <category><![CDATA[positional-encoding]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Anvee Naik]]></dc:creator>
            <pubDate>Thu, 13 Jun 2024 12:19:18 GMT</pubDate>
            <atom:updated>2025-02-13T18:16:32.853Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Low-Rank Adaptation (LoRA): From Intuition to Implementation to Interview Questions]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pub.towardsai.net/low-rank-adaptation-lora-from-intuition-to-implementation-to-interview-questions-1461c6a81615?source=rss-b754131c5a9b------2"><img src="https://cdn-images-1.medium.com/max/2600/0*nJFY4bRxuSqiqugx" width="4100"></a></p><p class="medium-feed-snippet">Delving Deeper into LoRA for LLMs</p><p class="medium-feed-link"><a href="https://pub.towardsai.net/low-rank-adaptation-lora-from-intuition-to-implementation-to-interview-questions-1461c6a81615?source=rss-b754131c5a9b------2">Continue reading on Towards AI »</a></p></div>]]></description>
            <link>https://pub.towardsai.net/low-rank-adaptation-lora-from-intuition-to-implementation-to-interview-questions-1461c6a81615?source=rss-b754131c5a9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1461c6a81615</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[adapter]]></category>
            <dc:creator><![CDATA[Anvee Naik]]></dc:creator>
            <pubDate>Mon, 03 Jun 2024 15:56:42 GMT</pubDate>
            <atom:updated>2025-02-13T18:17:21.421Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Elevate Your Model Training with WandB: A Data Scientist’s Best Friend]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@AnveeNaik/elevate-your-model-training-with-wandb-a-data-scientists-best-friend-2f652fb3232c?source=rss-b754131c5a9b------2"><img src="https://cdn-images-1.medium.com/max/2600/0*i_02l530KUTyi2tY" width="3456"></a></p><p class="medium-feed-snippet">Learn how to incorporate WandB into all of your code</p><p class="medium-feed-link"><a href="https://medium.com/@AnveeNaik/elevate-your-model-training-with-wandb-a-data-scientists-best-friend-2f652fb3232c?source=rss-b754131c5a9b------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@AnveeNaik/elevate-your-model-training-with-wandb-a-data-scientists-best-friend-2f652fb3232c?source=rss-b754131c5a9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f652fb3232c</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science-training]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Anvee Naik]]></dc:creator>
            <pubDate>Mon, 24 Jul 2023 14:36:41 GMT</pubDate>
            <atom:updated>2025-02-13T18:18:05.259Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How AI can make your life better!]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/predict/how-ai-can-make-your-life-better-9c6d9782ba59?source=rss-b754131c5a9b------2"><img src="https://cdn-images-1.medium.com/max/2600/0*-4hEYKK8KRMltYWl" width="5184"></a></p><p class="medium-feed-snippet">AI (artificial intelligence) can be an intimidating word that has suddenly crept into our lives, so let&#x2019;s see how it affects us.</p><p class="medium-feed-link"><a href="https://medium.com/predict/how-ai-can-make-your-life-better-9c6d9782ba59?source=rss-b754131c5a9b------2">Continue reading on Predict »</a></p></div>]]></description>
            <link>https://medium.com/predict/how-ai-can-make-your-life-better-9c6d9782ba59?source=rss-b754131c5a9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c6d9782ba59</guid>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Anvee Naik]]></dc:creator>
            <pubDate>Tue, 01 Feb 2022 05:48:17 GMT</pubDate>
            <atom:updated>2022-02-05T00:02:41.083Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[A better way to code, as a data scientist]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/a-better-way-to-code-as-a-data-scientist-7e4376192441?source=rss-b754131c5a9b------2"><img src="https://cdn-images-1.medium.com/max/2494/0*73YTYsnqk-K9qhEN" width="2494"></a></p><p class="medium-feed-snippet">Even if you are just plotting a graph, a clean code is always appreciated, so try out these practices while coding</p><p class="medium-feed-link"><a href="https://medium.com/data-science/a-better-way-to-code-as-a-data-scientist-7e4376192441?source=rss-b754131c5a9b------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/a-better-way-to-code-as-a-data-scientist-7e4376192441?source=rss-b754131c5a9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e4376192441</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Anvee Naik]]></dc:creator>
            <pubDate>Wed, 15 Dec 2021 16:50:43 GMT</pubDate>
            <atom:updated>2021-12-15T18:28:37.952Z</atom:updated>
        </item>
    </channel>
</rss>