<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Drishti Sushma on Medium]]></title>
        <description><![CDATA[Stories by Drishti Sushma on Medium]]></description>
        <link>https://medium.com/@drishtisharma96505?source=rss-5dee8b22f042------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YZOE3mQyabKNdmdLG9QDpQ.jpeg</url>
            <title>Stories by Drishti Sushma on Medium</title>
            <link>https://medium.com/@drishtisharma96505?source=rss-5dee8b22f042------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 13:23:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@drishtisharma96505/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Accelerate Llama-2–7b Fine-tuning: Unsloth Outpaces Flash Attention-2]]></title>
            <link>https://medium.com/@drishtisharma96505/accelerate-llama-2-7b-fine-tuning-how-unsloth-outpaced-standard-and-flash-attention-ef2ba8a1131d?source=rss-5dee8b22f042------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef2ba8a1131d</guid>
            <dc:creator><![CDATA[Drishti Sushma]]></dc:creator>
            <pubDate>Sun, 14 Jan 2024 02:00:37 GMT</pubDate>
            <atom:updated>2024-01-25T19:37:10.610Z</atom:updated>
            <content:encoded><![CDATA[<h3>Objective of this Study</h3><p>This study evaluates the effectiveness of various training techniques i.e. <strong>i) Standard Attention, ii) Flash Attention-1, iii) Flash Attention-2, iv) and </strong><a href="https://github.com/unslothai/unsloth"><strong>Unsloth </strong></a>— on the Llama2–7b model using the <a href="https://huggingface.co/datasets/databricks/databricks-dolly-15k">databricks-dolly-15k</a> dataset. The key focus of this study was evaluating training efficiency, memory usage, and model performance to determine the most effective method for instruction generation tasks.</p><h3>Key Highlights</h3><ul><li><strong>Unsloth demonstrated exceptional memory efficiency</strong> by utilizing the least amount of peak memory at 18.727 GB and the lowest memory specifically for training at 14.622 GB, while also reserving the smallest percentage of GPU’s total capacity (47.333% reserved, 36.958% utilized for training), indicating superior memory management.</li><li>With a training time of 23.29 minutes, 🤗 + Unsloth significantly outperforms Standard Attention (57.93 minutes), Flash Attention 1 (40.69 minutes), and Flash Attention 2 (38.39 minutes), demonstrating a notable reduction in training duration.</li><li><strong>Unsloth is 1.65x faster than Flash Attention-2, 1.75x faster than Flash Attention-1, and 2.49x faster than Standard Attention.</strong></li><li><strong>Model performance remained nearly consistent across all techniques</strong>, with Unsloth and others showing similar training and validation losses, indicating that efficiency gains do not compromise learning ability.</li></ul><h3>Quick Overview of Unsloth</h3><p><a href="https://huggingface.co/blog/unsloth-trl">Unsloth</a>, a key focus of this study, is a lightweight tool developed by the Unsloth team (<a href="https://huggingface.co/danielhanchen">Daniel</a> and <a href="https://github.com/shimmyshimmer">Michael</a>) and the open-source community, for accelerated and memory-efficient fine-tuning of large language models (LLMs). <strong>The efficacy of Unsloth is derived from its approach of overwriting some parts of the modeling code with optimized operations. </strong>By manually deriving backpropagation steps and rewriting all Pytorch modules into Triton kernels, Unsloth can both reduce memory usage and make fine-tuning faster.</p><h3><strong>Dataset and Model Used</strong></h3><p>I utilized the <a href="https://huggingface.co/datasets/databricks/databricks-dolly-15k">databricks-dolly-15k</a> which is an open-source dataset of instruction-following records (15k) generated by thousands of Databricks employees in several of the behavioral categories outlined in the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization. Moreover, “<a href="https://huggingface.co/NousResearch/Llama-2-7b-hf">NousResearch/Llama-2–7b-hf</a>” model served as the foundational model for these experiments.</p><h3><strong>Common Experimental Setup for All Experiments</strong></h3><ol><li>All tests were conducted on 1 A100 GPU.</li><li>model_id = “<a href="https://huggingface.co/NousResearch/Llama-2-7b-hf">NousResearch/Llama-2–7b-hf</a>”</li><li>dataset = <a href="https://huggingface.co/datasets/databricks/databricks-dolly-15k">databricks-dolly-15k</a></li><li><strong>BitsAndBytesConfig Setup</strong></li></ol><p>Note: This is for Standard attention, Flash Attention 1, and Flash Attention 2, <strong>excluding Unsloth</strong> which has a distinct quantization approach.</p><pre># BitsAndBytesConfig int-4 config<br>bnb_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_use_double_quant=True,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_compute_dtype=torch.bfloat16<br>)py</pre><p>5. LoRA Config</p><pre># LoRA config based on QLoRA paper<br>peft_config = LoraConfig(<br>        lora_alpha=LORA_ALPHA,<br>        lora_dropout=LORA_DROPOUT,<br>        r=r,<br>        bias=&quot;none&quot;,<br>        task_type=&quot;CAUSAL_LM&quot;,<br>        target_modules=[&quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;]<br>)<br><br><br># prepare model for training<br>model = prepare_model_for_kbit_training(model)<br>model = get_peft_model(model, peft_config)</pre><p>6. Training Arguments</p><pre>args = TrainingArguments(<br>    output_dir= f&quot;llama2-7b-int4-dolly-15k-english-standard-attention-w-packing&quot;,<br>    num_train_epochs=3,<br>    per_device_train_batch_size=6,<br>    per_device_eval_batch_size=4,<br>    optim=&quot;paged_adamw_32bit&quot;,<br>    logging_steps=100,<br>    eval_steps=100,                # Evaluate every 100 steps<br>    save_strategy=&quot;epoch&quot;,<br>    evaluation_strategy=&quot;steps&quot;,   # Enable evaluation every &#39;eval_steps&#39;<br>    learning_rate=2e-4,<br>    fp16=True,<br>    lr_scheduler_type=&quot;cosine&quot;,<br>    group_by_length=True,<br>    save_safetensors=True,<br>    report_to=&quot;wandb&quot;,<br>    seed=42<br>)</pre><h3>Experiment-Specific Configurations</h3><p><strong>i) Experiment 1: Standard Attention</strong></p><p>This experiment utilized the common setup with no additional modifications.</p><p><strong>ii) Experiment 2: Flash Attention 1</strong></p><p><strong>Ad</strong>ditional dependency</p><pre>!pip install -U optimum</pre><p>Model Training Setup</p><pre># Setting up the trainer for Flash Attention 1<br><br>with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):<br>    trainer.train()</pre><p><strong>iii) Experiment 3: Flash Attention 2</strong></p><p>Additional Dependency</p><pre>!pip install -U flash-attn</pre><p>Load the model and enable the Flash Attention 2 feature by setting<em> </em><strong><em>use_flash_attention_2=True </em></strong>as follows:</p><pre>model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, use_cache=True, device_map=&quot;auto&quot;, use_flash_attention_2=True)</pre><p><strong>iv) Experiment 4: Unsloth</strong></p><p>Additional dependencies:</p><pre>major_version, minor_version = torch.cuda.get_device_capability()<br>if major_version &gt;= 8:<br>    # Use this for new GPUs like Ampere, Hopper GPUs (RTX 30xx, RTX 40xx, A100, H100, L40)<br>    !pip install &quot;unsloth[colab_ampere] @ git+https://github.com/unslothai/unsloth.git&quot;<br>else:<br>    # Use this for older GPUs (V100, Tesla T4, RTX 20xx)<br>    !pip install &quot;unsloth[colab] @ git+https://github.com/unslothai/unsloth.git&quot;<br>pass<br><br></pre><p>Model and Tokenizer Setup — Initializing FastLanguageModel with Custom Configuration</p><pre># Load model<br>max_seq_length = 2048 # Choose any! Auto supports RoPE Scaling internally!<br>dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+<br>load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.<br><br>model, tokenizer = FastLanguageModel.from_pretrained(<br>    model_name = model_id,<br>    max_seq_length = max_seq_length,<br>    dtype = dtype,<br>    load_in_4bit = load_in_4bit,<br>    device_map=&quot;auto&quot;,<br>)</pre><p><em>Performing model patching and adding fast LoRA weights while ensuring consistency in target modules and LoRA parameters with other experiments.</em></p><pre># Do model patching and add fast LoRA weights<br>model = FastLanguageModel.get_peft_model(<br>    model,<br>    r = r,<br>    target_modules = [&quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;],<br>    lora_alpha = LORA_ALPHA,<br>    lora_dropout = LORA_DROPOUT, # Dropout = 0 is currently optimized<br>    bias = &quot;none&quot;,    # Bias = &quot;none&quot; is currently optimized<br>    random_state = 42,<br>    max_seq_length = max_seq_length<br>)</pre><h3>Comparative Performance Evaluation of Training Optimization Techniques</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*-t5a4zZBx8ENtAxT2UZxlA.png" /><figcaption><strong>Table 1: Comparative Analysis of Performance Metrics for Llama-2–7b Model Using Various Optimization Techniques</strong></figcaption></figure><p>From the above table, we can interpret the following:</p><ol><li><strong>Initial Memory Efficiency:</strong> 🤗 + Unsloth begins with the lowest initial memory reservation of 4.105 GB, compared to 5.102 GB for both Standard and Flash Attention 1, and 5.092 GB for Flash Attention 2. This indicates a superior initial memory efficiency.</li><li><strong>Training Time:</strong> With a training time of 23.29 minutes, 🤗 + Unsloth significantly outperforms the Standard Attention (57.93 minutes), Flash Attention 1 (40.69 minutes), and Flash Attention 2 (38.39 minutes), demonstrating a notable reduction in training duration.</li><li><strong>Memory Utilization:</strong> In terms of peak memory usage, 🤗 + Unsloth shows the lowest at 18.727 GB, which is substantially lower than 32.838 GB for Standard Attention, 20.275 GB for Flash Attention 1, and 21.73 GB for Flash Attention 2. This trend is consistent in the memory specifically used for training, where Unsloth uses only 14.622 GB, evidencing its memory efficiency.</li><li><strong>Percentage of GPU Memory Usage: </strong>🤗 + Unsloth uses only 47.333% of the total available GPU memory and 36.958% for training. In contrast, Standard Attention uses up to 83.0% and 70.104% respectively, Flash Attention 1 uses 51.246% and 38.351%, and Flash Attention 2 uses 54.924% and 42.053%.</li><li><strong>Model Performance:</strong> The final training loss and validation loss across all methods are comparably close, with Unsloth recording a training loss of 1.1459 and a validation loss of 1.2197. These figures are similar to those of Standard Attention (1.1369 and 1.2201), Flash Attention 1 (1.1364 and 1.2200), and Flash Attention 2 (1.1368 and 1.2201), indicating that the efficiency gains of Unsloth do not come at the cost of model performance.</li></ol><h3>Future Work</h3><ol><li><strong>Scaling Experiments with Unsloth:</strong> Investigate the impact of Unsloth on larger-scale versions of the Llama-2–7b model to explore whether its advantages in efficiency and training speed are consistent or scale-dependent.</li><li><strong>Comprehensive Benchmarking Across Models:</strong> Broaden analysis to include various state-of-the-art models, comparing Unsloth and other methods to evaluate their general applicability.</li><li><strong>In-Depth Analysis of Training Parameters:</strong> Delve deeper into the training parameters and configurations to optimize and fine-tune their application across diverse models and datasets.</li></ol><h3>Conclusion</h3><p>From these comprehensive results, we can conclude that Unsloth is the most efficient training method in terms of both time and memory usage among the techniques tested. It provides a significant advantage in training speed and memory efficiency without a trade-off in model performance, as evidenced by the comparable training and validation loss figures. These gains make it ideal for large model training in limited-resource settings, potentially reducing costs and enabling the use of more complex models without hardware upgrades.</p><p><strong>Access the Trained Models on Hugging Face Hub:</strong> <a href="https://huggingface.co/collections/DrishtiSharma/multifaceted-attention-analysis-in-llama2-w-en-and-hi-dolly1-659f2bfe21c219062cdc58ec">here</a>!</p><h3><strong>References</strong></h3><ol><li><a href="https://huggingface.co/blog/unsloth-trl">https://huggingface.co/blog/unsloth-trl</a></li><li><a href="https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth">https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef2ba8a1131d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing the Impact of lora_alpha on Llama-2 Quantized with GPTQ]]></title>
            <link>https://medium.com/@drishtisharma96505/analyzing-the-impact-of-lora-alpha-on-llama-2-quantized-with-gptq-f01e8e8ed8fd?source=rss-5dee8b22f042------2</link>
            <guid isPermaLink="false">https://medium.com/p/f01e8e8ed8fd</guid>
            <dc:creator><![CDATA[Drishti Sushma]]></dc:creator>
            <pubDate>Thu, 14 Sep 2023 16:30:58 GMT</pubDate>
            <atom:updated>2024-01-26T12:47:37.240Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>|Updated: 26/01/24</strong></p><h3>Objective of the Study</h3><p>The key objective of this study was to analyze the effects of LoRA parameters namely r, lora_alpha, and lora_dropout on the performance of the Llama-2 with Flash Attention.</p><h3>Dataset and Model Used</h3><p>We utilized the <a href="https://huggingface.co/datasets/Abirate/english_quotes">Abirate/english_quotes</a> dataset from Hugging Face which has all the quotes retrieved from <a href="https://www.goodreads.com/quotes">goodreads quotes</a>. This dataset can be used for multi-label text classification and text generation. Moreover, we utilized “<a href="https://huggingface.co/TheBloke/Llama-2-7b-Chat-GPTQ">TheBloke/Llama-2–7b-Chat-GPTQ</a>” as the base model for performing experiments.</p><h3>LoRA</h3><p>LoRA (Low-Rank Adaptation) is a method employed for fine-tuning extensive pre-existing models. In the evolving landscape of machine learning with increasingly large models, LoRA offers an efficient mechanism to harness their capabilities, leveraging an adaptation matrix during fine-tuning. This technique preserves the foundational knowledge of the pre-trained model while enabling specific modifications for new datasets.</p><h3>Experimental Setup</h3><ul><li>All experiments were performed using A100 GPU.</li><li>LoRA Configuration: r=8, lora_alpha=64, and lora_dropout=0.10.</li><li>Target Modules: [&quot;k_proj&quot;,&quot;o_proj&quot;,&quot;q_proj&quot;,&quot;v_proj&quot;].</li><li>Training Batch Size: 1</li><li>Evaluation Batch Size: 1</li><li>Evaluation Strategy: Evaluated every 20 steps with a maximum of 100 steps.</li><li>Learning Rate: 2e-4</li><li>optim=”adamw_hf”</li><li>max_steps = 100</li></ul><h3>Evaluating the Impact of lora_alpha on Model Performance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*tEUbd14BprMVwuG9CudMUg.png" /></figure><ol><li><strong>Optimal lora_alpha Value:</strong>lora_alpha = 32 seems to provide the best balance in terms of training and validation loss. It offers a training loss of 3.8675 and validation loss of 4.2374 at step 100. These are among the lowest values for the metrics in this dataset, indicating optimal performance.</li><li><strong>Loss Fluctuations:</strong> As the value of lora_alpha increases from 16 to 32, there&#39;s a significant improvement in both training and validation loss. However, as we move beyond lora_alpha = 32, there&#39;s a tendency for the performance to decline. This could be due to overfitting or the model becoming too complex to adapt efficiently for the given dataset.</li></ol><ul><li>It’s also notable that for lora_alpha values 64 and 128, both training and validation losses remain identical, suggesting that the impact of this parameter plateaus at this range before deteriorating further.</li></ul><p><strong>3. Training Runtime:</strong> The average training runtime remains relatively consistent across varying lora_alpha values, with only slight variations. This suggests that, in terms of computational time, there isn&#39;t a significant penalty or benefit tied directly to the adjustment of the lora_alpha.</p><p><strong>4. Overall Training Loss:</strong> The overall training loss seems to be the most optimal for lora_alpha = 32 and starts to increase as we move to higher values. This could indicate that while the model might be adapting and capturing more nuances, it might also be getting overcomplicated or losing generalizability.</p><h3>Conclusion</h3><p>For the <a href="https://huggingface.co/datasets/Abirate/english_quotes">Abirate/english_quotes</a> dataset and <a href="https://huggingface.co/TheBloke/Llama-2-7b-Chat-GPTQ">TheBloke/Llama-2–7b-Chat-GPTQ</a>, lora_alpha at 32 yielded the lowest training (3.8675) and validation losses (4.2374) at step 100, demonstrating its superior performance. Notably, as lora_alpha rises beyond 32, performance tends to decline, potentially due to overfitting. Interestingly, identical losses were observed for lora_alpha values 64 and 128, indicating a performance plateau. Despite varying lora_alpha values, training runtimes remained consistent, suggesting no substantial computational impact. The upward trend in overall training loss at higher lora_alpha values further hints at reduced model generalizability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f01e8e8ed8fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing the Dual Impact: Batch Size and Mixed Precision on DistilBERT’s Performance in Language…]]></title>
            <link>https://medium.com/@drishtisharma96505/unraveling-the-dual-impact-batch-size-and-mixed-precision-on-distilberts-performance-in-language-56ab2d2b20b8?source=rss-5dee8b22f042------2</link>
            <guid isPermaLink="false">https://medium.com/p/56ab2d2b20b8</guid>
            <dc:creator><![CDATA[Drishti Sushma]]></dc:creator>
            <pubDate>Tue, 12 Sep 2023 01:45:18 GMT</pubDate>
            <atom:updated>2024-01-30T17:36:22.937Z</atom:updated>
            <content:encoded><![CDATA[<h3>Analyzing the Dual Impact: Batch Size and Mixed Precision on DistilBERT’s Performance in Language Detection</h3><h3>Introduction</h3><p>In deep learning, batch size and precision are two pivotal parameters that can significantly influence a model’s training speed and accuracy. In this study, we delve into understanding the combined and individual effects of batch size and <strong>mixed-precision training using both 16-bit (fp16) and 32-bit (fp32) floating-point types</strong> using DistilBERT (<a href="https://huggingface.co/distilbert-base-multilingual-cased">distilbert-base-multilingual-cased</a>) for a language detection task.</p><h3>Batch Size: What and Why?</h3><p>Batch size refers to the number of training samples used in one iteration. Its choice can have profound implications:</p><ul><li>Small Batch Sizes: Can lead to a more accurate model but at the cost of computational inefficiency.</li><li>Large Batch Sizes: Often speed up training but might compromise on model accuracy and can converge to sharp minimas, affecting generalization.</li></ul><h3>Mixed Precision (fp16): A Primer</h3><p>Mixed precision training primarily uses 16-bit floats (fp16) instead of the traditional 32-bit floats (fp32). Key benefits include:</p><ul><li>Speed: Faster mathematical operations and reduced memory usage.</li><li>Memory Efficiency: Allows for larger batch sizes or models.</li></ul><h3>Experimental Setup</h3><ol><li>All experiments were performed using T4 GPU.</li><li>Model: <a href="https://huggingface.co/distilbert-base-multilingual-cased">distilbert-base-multilingual-cased</a></li><li>Dataset: <a href="https://www.kaggle.com/lailaboullous/language-detection-dataset">language-detection-dataset</a></li><li>Task: Language Detection</li><li>BATCH_SIZE = [4,8,16,32,64,128]</li><li>MAX_INPUT_LENGTH = 144</li><li>LEARNING_RATE = 2e-5</li><li>NUM_OF_EPOCHS = 5</li></ol><p>We trained DistilBERT across different batch sizes, ranging from 4 to 128. Each model was trained using both fp16=True and fp16=False configurations to gauge the impact of mixed precision.</p><h3>The Dual Impact on DistilBERT</h3><h4>A) Training Metrics</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*00iduMgfZu8AeMIHBqS7XQ.png" /><figcaption>Training Metrics corresponding to different batch sizes and fp16=False</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*CdL0aCPmAAXA5Lvh_DWfrQ.png" /><figcaption>Training Metrics corresponding to different batch sizes and fp16= True</figcaption></figure><p>i) Training Loss (Epoch 5):</p><ul><li>As batch size increased, the training loss demonstrated a pattern of increase-decrease-increase in 32-bit precision.</li><li>In mixed precision, a similar trend was observed, though often with marginally lower values.</li></ul><p>ii) Validation Loss (Epoch 5):</p><ul><li>With pure 32-bit precision, an upward trend in validation loss was noticeable with rising batch sizes.</li><li>Mixed precision, however, showcased more consistency, especially in the mid-range batch sizes (32–64).</li></ul><p>iii) Accuracy (Epoch 5):</p><ul><li>For both precisions, accuracy was commendably high, with mixed precision showing a slight edge in specific batch sizes.</li></ul><p>iv) Training Time:</p><ul><li>Mixed precision consistently outperformed 32-bit precision in terms of speed, a benefit more pronounced with larger batch sizes.</li></ul><h4><strong>B) Evaluation Metrics</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*UF6fjtuV9uCcZemwweUzrg.png" /><figcaption>Evaluation Metrics corresponding to different batch sizes and fp16=False</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*odVsmNk7VTRdItlN9QLbgg.png" /><figcaption>Evaluation Metrics corresponding to different batch sizes and fp16=True</figcaption></figure><p>i) Evaluation Loss:</p><ul><li>32-bit precision showed an escalating loss with batch size, while mixed precision remained more stable, especially up to a batch size of 32.</li></ul><p>ii) Performance Metrics:</p><ul><li>Both precisions achieved near-perfect scores for smaller batch sizes, with only minor dips as batch size increased.</li></ul><p>iii) Evaluation Speed:</p><ul><li>Mixed precision’s benefits shone again, consistently surpassing 32-bit precision in speed, particularly with larger batch sizes.</li></ul><h3>Key Takeaways</h3><p><strong>i) Batch Size Sensitivity:</strong> DistilBERT’s performance, in terms of loss and accuracy, is sensitive to batch size, showcasing optimal performance at mid-range batch sizes.</p><p><strong>ii) Precision’s Interplay with Batch Size:</strong> Mixed precision amplifies the efficiency benefits as batch size increases, offering a blend of speed and performance. This synergy can be particularly beneficial in large-scale deployments where both time and accuracy are crucial.</p><h3>Conclusion</h3><p>The analysis reveals the nuanced interplay between batch size and training efficiency. Increasing batch size tends to expedite training time per epoch, owing to fewer forward and backward passes. However, performance metrics, such as training and validation losses, exhibit a subtle deterioration with very large batch sizes, suggesting possible coarse gradient approximations. Furthermore, utilizing fp16 enhances computational speed across different batch sizes while maintaining comparable accuracy metrics, showcasing its effectiveness in model optimization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56ab2d2b20b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comprehensive Evaluation of Various Transformer Models in Detecting Normal, Hate, and Offensive…]]></title>
            <link>https://medium.com/@drishtisharma96505/comprehensive-evaluation-of-various-transformer-models-in-detecting-normal-hate-and-offensive-a63c5c5226f?source=rss-5dee8b22f042------2</link>
            <guid isPermaLink="false">https://medium.com/p/a63c5c5226f</guid>
            <dc:creator><![CDATA[Drishti Sushma]]></dc:creator>
            <pubDate>Mon, 11 Sep 2023 23:06:18 GMT</pubDate>
            <atom:updated>2025-08-02T01:33:02.857Z</atom:updated>
            <content:encoded><![CDATA[<h3>Comprehensive Evaluation of Various Transformer Models in Detecting Normal, Hate, and Offensive Text</h3><h3>Objective of the Study</h3><p>With the increasing prevalence of social media platforms, detecting hate and offensive speeches has become imperative. This study leverages a comprehensive dataset from <a href="https://www.kaggle.com/datasets/subhajournal/normal-hate-and-offensive-speeches">Kaggle</a>, aiming to discern the performance of various transformer models in distinguishing between normal, hate, and offensive tweets.</p><h3>Transformer Models Under Examination</h3><p>The following transformer models were meticulously evaluated in this study:</p><ol><li><a href="https://huggingface.co/bert-base-uncased">bert-base-uncased</a></li><li><a href="https://huggingface.co/bert-large-uncased">bert-large-uncased</a></li><li><a href="https://huggingface.co/distilbert-base-uncased">distilbert-base-uncased</a></li><li><a href="https://huggingface.co/diptanu/fBERT">diptanu/fBERT</a></li><li><a href="https://huggingface.co/GroNLP/hateBERT">GroNLP/hateBERT</a></li><li><a href="https://huggingface.co/roberta-large">roberta-large</a></li></ol><h3>Dataset Overview</h3><p>The dataset, retrieved from <a href="https://www.kaggle.com/datasets/subhajournal/normal-hate-and-offensive-speeches">Kaggle</a>, is a combination of 12 individual datasets, each delineating tweets categorized as normal, hate, or offensive. All 12 datasets were merged based on relevant file labels to create one large CSV file having following distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sRIRrvRnDj7CaLaSxQ6Fhg.png" /><figcaption>Figure 1: i) Label Distribution ii) Text Length Distribution by Label</figcaption></figure><h3>Pre-processing the Dataset</h3><p>During pre-processing, we avoided converting text to lowercase and eliminating special characters for following reasons:</p><ol><li>There’s a possibility that the case of a word carries meaningful information (for instance, if capitalized words are used more frequently in certain types of speech in the dataset), then lowercasing should be avoided… that’s exactly what we did!</li><li>In certain contexts like hate speech or offensive language detection, users might obfuscate words with special characters to bypass content filters. In such cases, simply removing these characters might strip away essential information.</li></ol><p>We performed label-specific filtering to process the dataset.</p><pre>condition_hate = (data[&#39;label&#39;] == &#39;Hate&#39;) &amp; (data[&#39;text_length&#39;] &lt;= 62)<br>condition_offensive = (data[&#39;label&#39;] == &#39;Offensive&#39;) &amp; (data[&#39;text_length&#39;] &lt;= 60)<br>condition_normal = (data[&#39;label&#39;] == &#39;Normal&#39;) &amp; (data[&#39;text_length&#39;] &lt;= 88)<br><br>data = data[condition_hate | condition_offensive | condition_normal]</pre><h3>Handling Mild Class Imbalance</h3><p>Also, from label distribution we can see that there was a mild class imbalance which we tried to handle with the help of CustomTrainer as follows:</p><pre>total_samples = 1200 + 966 + 895<br>weights = torch.tensor([total_samples/1200, total_samples/966, total_samples/895])<br><br># Normalize the weights for numerical stability<br>weights = weights / weights.sum()<br><br>weights = weights.to(DEVICE)<br><br><br>class CustomTrainer(Trainer):<br>    def compute_loss(self, model, inputs, return_outputs=False):<br>        labels = inputs.get(&quot;labels&quot;)<br>        outputs = model(**inputs)<br>        logits = outputs.get(&quot;logits&quot;)<br>        loss_fct = torch.nn.CrossEntropyLoss(weight=weights)<br>        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))<br>        return (loss, outputs) if return_outputs else loss</pre><p>Then each individual model was trained over a span of 5 epochs.</p><h3>Experimental Setup</h3><ol><li>All experiments were conducted using A100 GPU.</li><li>NUM_OF_EPOCHS = 5</li><li>BATCH_SIZE = 16</li><li>LEARNING_RATE = 2e-5</li><li>MAX_LENGTH = int(88 * 1.3)</li></ol><h3>Results</h3><p><strong>I) Training Metrics for Different BERT-based Models:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D3MtEnZBsvk6uYowk1EmvA.png" /><figcaption>Training Metrics of Different Models</figcaption></figure><ol><li><strong>Accuracy &amp; Micro Metrics:</strong> All models, except roberta-large and GroNLP/hateBERT, seem to perform identically in terms of accuracy and micro-level metrics (F1, Recall, Precision). The roberta-large model&#39;s performance, in terms of accuracy, is slightly lower compared to others, and GroNLP/hateBERT is just marginally lower than the rest but still performs quite well.</li><li><strong>Macro Metrics:</strong> The models still show similar performances, except for roberta-large, which has a noticeably lower Macro F1, Recall, and Precision compared to the rest. This could suggest that roberta-large might not be performing consistently across different classes.</li><li><strong>Training Time:</strong> The bert-large-uncased and roberta-large models take significantly longer to train, which is expected given their larger size. On the other hand, distilbert-base-uncased trains the fastest, indicating its suitability for tasks with limited computational resources or time constraints.</li></ol><p><strong>II) Evaluation Metrics for Different BERT-based Models:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*txp3llxBx5jk8vpXrmcnOA.png" /><figcaption>Evaluation Metrics for Different BERT-based Models</figcaption></figure><ol><li><strong>Macro F1 Score:</strong> All models have a Macro F1 score that exceeds 0.98. The models are quite competitive with each other, with bert-base-uncased, bert-large-uncased, distilbert-base, and fbert/hate essentially having the same score. The roberta-large model slightly lags behind, and hateBERT has a modest drop in performance compared to the top performers but still is above 0.99.</li><li>Eval Loss: bert-large-uncased has the lowest evaluation loss. Lower loss indicates better model generalization on the evaluation set. distilbert-base and fbert/hate have higher losses compared to bert-base-uncased having similar Macro F1 scores, suggesting that while they make mistakes, the mistakes may not be as severe in terms of class imbalances.</li><li>Eval Runtime: distilbert-base is the fastest, followed closely by both bert-base-uncased and fbert/hate. bert-large-uncased and roberta-large take significantly longer, which is expected given their size.</li></ol><h3>Recommendations</h3><ol><li><strong>Best Overall Model:</strong> If seeking a balance between performance and efficiency, bert-base-uncased seems to be the most well-rounded choice. It offers top-tier performance metrics and has a reasonably quick evaluation time.</li><li><strong>Efficiency:</strong> For those needing a model that trains and evaluates quickly, distilbert-base-uncased is an ideal choice. It has almost identical performance metrics as the other top models but trains and evaluates in a fraction of the time.</li><li><strong>Highest Performance without Time Constraint:</strong> For the highest performance regardless of training and evaluation time, bert-large-uncased might be the best choice, especially given its lowest evaluation loss.</li><li><strong>Models to Potentially Avoid for this Task:</strong> roberta-large, despite its larger size, does not seem to offer any significant advantage in terms of performance metrics for this specific task. It takes longer to train and evaluate but does not provide superior results. Similarly, GroNLP/hateBERT, while still good, doesn&#39;t seem to offer any clear advantage over the other models in these specific metrics.</li></ol><h3>Conclusion</h3><p>Assessing various transformer models for distinguishing between normal, hate, and offensive tweets, bert-base-uncased stands out as a top performer that balances efficiency and precision. Meanwhile, distilbert-base-uncased offers swift training and evaluation, ideal for time-constrained tasks. Despite the capabilities of the roberta-large and GroNLP/hateBERT, they don&#39;t provide a clear advantage in this context, making model selection critical based on specific requirements and computational resources.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a63c5c5226f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding the Impact of Weight Decay on MBart-large-50 for English-Spanish Translation]]></title>
            <link>https://medium.com/@drishtisharma96505/unveiling-the-impact-of-weight-decay-on-mbart-large-50-for-english-spanish-translation-1c2fe3fb854d?source=rss-5dee8b22f042------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c2fe3fb854d</guid>
            <dc:creator><![CDATA[Drishti Sushma]]></dc:creator>
            <pubDate>Mon, 11 Sep 2023 20:50:19 GMT</pubDate>
            <atom:updated>2024-01-30T17:36:45.570Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>The MBart-large-50 model has astonishing capabilities to cater to multiple languages. However, like any other neural network-based model, it also offers parameters that can be tuned to optimize performance. One such parameter is weight decay, which plays a pivotal role in controlling overfitting and generalization. In this study, we aimed to understand the influence of varying weight decay values on the MBART-50’s performance for English to Spanish translation.</p><h3>Methodology</h3><p>The MBART-50 model was trained for 4 epochs on “<a href="https://www.kaggle.com/datasets/hgultekin/paralel-translation-corpus-in-22-languages">paralel-translation-corpus-in-22-languages</a>” from Kaggle, with weight decay values varied in the set [0, 1e-1, 1e-2, 1e-3, 1e-4]. The main metrics of focus during this experiment were the training loss, validation loss, Bleu score, and Rouge score. Additionally, training time and evaluation time were also recorded to understand any computational differences.</p><h3>Experimental Setup</h3><ol><li>All experiments were conducted using A100 GPU.</li><li>Dataset: “<a href="https://www.kaggle.com/datasets/hgultekin/paralel-translation-corpus-in-22-languages">paralel-translation-corpus-in-22-languages</a>”</li><li>Model: <a href="https://huggingface.co/facebook/mbart-large-50">mbart-large-50</a></li><li>BATCH_SIZE = 8</li><li>LEARNING_RATE = 1e-5</li></ol><h3>Observations and Insights</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*p4EOGUkbrHcOglTD_5KvpA.png" /><figcaption>Results obtained after varying the value of weight decay from 0 to 1e-4</figcaption></figure><ol><li>The variation in weight decay doesn’t seem to significantly affect the metrics (BLEU, ROUGE) across the epochs. The results are relatively close to each other.</li><li>Training and evaluation times vary somewhat across different weight decay values. However, the difference is within a few minutes, so it might not be a significant factor for choosing one weight decay over another.</li><li>All configurations appear to have a slight increase in the metrics from Epoch 1 to Epoch 4.</li><li>The BLEU score and the Rouge metrics seem to plateau or slightly increase by the 4th epoch in all configurations.</li></ol><h3>Future Work</h3><ol><li><strong>Exploration of Weight Decay Interplay:</strong> Delving into how weight decay interacts with other hyperparameters in the model to understand synergistic effects.</li><li><strong>Different Language Pairs:</strong> Examining the impact of weight decay on translation tasks across various language pairs. It would be interesting to see if its effects remain consistent or vary across languages.</li><li><strong>Ensemble Approaches:</strong> Exploring if combining models trained with different weight decay values leads to any improvements in translation quality.</li></ol><h3>Conclusion</h3><p>Hyperparameter tuning remains pivotal in optimizing neural models. Weight decay, a mechanism to counteract overfitting by penalizing large weights, shows only a muted influence on the MBart-50 model’s English-Spanish translation performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c2fe3fb854d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing Llama-2’s Behavior with Varied Pretraining Temperature and Attention Mechanisms]]></title>
            <link>https://medium.com/@drishtisharma96505/dissecting-llama-2s-behavior-with-varied-pretraining-temperature-and-attention-mechanisms-18c47bd64dbf?source=rss-5dee8b22f042------2</link>
            <guid isPermaLink="false">https://medium.com/p/18c47bd64dbf</guid>
            <dc:creator><![CDATA[Drishti Sushma]]></dc:creator>
            <pubDate>Mon, 11 Sep 2023 14:26:55 GMT</pubDate>
            <atom:updated>2024-01-26T02:12:14.241Z</atom:updated>
            <content:encoded><![CDATA[<h3>Objective of the Study</h3><p>The key objective of this study was to analyze the effects of:</p><ul><li>Different values of pretraining_tp on the behavior of a model during its training and inference phases. (Tested training_tp=1 and training_tp=2)</li><li>The difference in model performance/behavior between using Flash Attention and the Traditional Attention mechanism for training_tp = 1 and 2.</li></ul><h3>Dataset Used</h3><p>We utilized the <a href="https://huggingface.co/datasets/tatsu-lab/alpaca">Alpaca dataset</a>, which is a dataset of 52,000 instructions and demonstrations generated by OpenAI’s text-davinci-003 engine. It’s intended to enhance instruction-following capabilities in language models. We structured examples into task descriptions using a formatting function. This ensures that the model receives instructions in a way it understands.</p><pre>def format_instruction(sample):<br> return f&quot;&quot;&quot;### Instruction:<br>Use the Input below to create an instruction, which could have been used to generate the input using an LLM.<br><br>### Input:<br>{sample[&#39;response&#39;]}<br><br>### Response:<br>{sample[&#39;instruction&#39;]}<br>&quot;&quot;&quot;</pre><h3>Model Used</h3><p><a href="https://huggingface.co/NousResearch/Llama-2-7b-hf">NousResearch/Llama-2–7b-hf</a></p><h3>Tools &amp; Techniques:</h3><h4><strong>1. QLoRA:</strong></h4><p>A technique to reduce the memory footprint of large language models during fine-tuning. It involves:</p><ul><li>Quantizing the pre-trained model to fewer bits and freezing it.</li><li>Attaching small, trainable adapter layers (LoRA).</li><li>Fine-tuning only the adapter layers, using the frozen quantized model for context.</li></ul><h4><strong>2. Flash Attention:</strong></h4><p>Flash Attention is a method that makes attention computation more efficient both in terms of speed and memory.</p><ul><li>Reorders the attention computation to enhance speed and reduce memory usage.</li><li>It has been claimed that it accelerates training by up to 3x by converting the memory usage from quadratic to linear based on sequence length. We will be testing this out with the help of our brief study!</li><li>Suitable only for specific GPUs (Ampere &amp; Hopper series).</li></ul><h3>Experimental Setup</h3><ol><li>All tests were conducted on an A100 GPU.</li><li>model_id = “NousResearch/Llama-2–7b-hf”</li></ol><p><strong>2. BitsAndBytesConfig Setup</strong></p><pre><br># BitsAndBytesConfig int-4 config<br>bnb_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_use_double_quant=True,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_compute_dtype=torch.bfloat16<br>)</pre><h4><strong>3. LoRA Config</strong></h4><pre># LoRA config based on QLoRA paper<br>peft_config = LoraConfig(<br>        lora_alpha=16,<br>        lora_dropout=0.1,<br>        r=64,<br>        bias=&quot;none&quot;,<br>        task_type=&quot;CAUSAL_LM&quot;,<br>)</pre><h4>4. Training Arguments</h4><pre>args = TrainingArguments(<br>    output_dir=&quot;llama-7-int4-alpac-flash-attention-tp-1&quot;,<br>    num_train_epochs=2,<br>    per_device_train_batch_size=6 if use_flash_attention else 4,<br>    optim=&quot;paged_adamw_32bit&quot;,<br>    logging_steps=200,<br>    save_strategy=&quot;epoch&quot;,<br>    learning_rate=2e-4,<br>    bf16=True,<br>    tf32=True,<br>    lr_scheduler_type=&quot;constant&quot;,<br>    disable_tqdm=True # disable tqdm since with packing values are in correct<br>)</pre><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*JVk8qsjY7NwsxXdWgtDI0w.png" /></figure><h3>Observations</h3><h4>1. Training Loss:</h4><ul><li>The training losses across all cases are very close to each other. There is a very minor decrease in training loss when using a pretraining_tp of 1 vs. 2 but the difference is negligible.</li><li>The attention type (Flash vs. Normal) does not seem to have a noticeable impact on the final training loss.</li></ul><h4>2. Training Time:</h4><ul><li><strong>Using Flash Attention significantly reduces training time, nearly by half as compared to using Normal Attention.</strong></li><li>The pretraining_tp value does not seem to significantly impact the training time.</li></ul><h4>3. Inference Time:</h4><ul><li>Flash Attention with pretraining_tp of 2 has the fastest inference time.</li><li>Interestingly, Normal Attention has similar inference times for both pretraining_tp values, and they&#39;re both comparable or slightly faster than Flash Attention with pretraining_tp of 1.</li></ul><h3>Future Work</h3><ol><li>Testing the models on a validation dataset to measure the generalization performance.</li><li>Experimenting with a wider range of pretraining_tp values.</li><li>Investigating why the Normal Attention method resulted in similar or slightly better inference times for these specific pretraining_tp values.</li></ol><h3>Conclusion</h3><ol><li>Flash Attention is significantly faster in training compared to Normal Attention, which is expected based on the stated advantages of Flash Attention.</li><li>The pretraining_tp values, either 1 or 2, do not drastically impact the model&#39;s performance or training/inference times in this experiment. However, using pretraining_tp of 2 slightly improves inference time when using Flash Attention.</li><li>The model’s performance, in terms of loss, is mostly consistent across all cases. Hence, other considerations like training time, inference time, and computational resources could be more important when deciding on the configurations to use.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18c47bd64dbf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparative Study: Training OPT-350M and GPT-2 on Anthropic’s HH-RLHF Dataset Using Reward-Based…]]></title>
            <link>https://medium.com/@drishtisharma96505/comparative-study-training-opt-350m-and-gpt-2-on-anthropics-hh-rlhf-dataset-using-reward-based-bd1050a5e6ac?source=rss-5dee8b22f042------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd1050a5e6ac</guid>
            <dc:creator><![CDATA[Drishti Sushma]]></dc:creator>
            <pubDate>Mon, 11 Sep 2023 12:19:03 GMT</pubDate>
            <atom:updated>2024-03-07T08:14:42.776Z</atom:updated>
            <content:encoded><![CDATA[<h3>Comparative Study: Training OPT-350M and GPT-2 on Anthropic’s HH-RLHF Dataset Using Reward-Based Training</h3><h3>Introduction</h3><p>In the rapidly evolving field of Natural Language Processing, assessing model performance on specific datasets is crucial to ascertain their adaptability and efficiency. This study focuses on comparing two renowned models: Facebook’s <a href="https://huggingface.co/facebook/opt-350m">opt-350m</a> and <a href="https://huggingface.co/gpt2">GPT-2</a>. Both models are trained and tested on the <a href="https://huggingface.co/datasets/Anthropic/hh-rlhf">Anthropic/hh-rlhf dataset</a> having 160800 rows, offering insights into their respective performances and potential strengths.</p><h3>Data Preprocessing and Reward Modeling</h3><p>The foundation of any ML experiment is accurate data processing. The Anthropic/hh-rlhf dataset comprises paired samples of &quot;chosen&quot; and &quot;rejected&quot; sequences. We will define a preprocessing function which will tokenize, truncate and format the sequences to maintain a uniform structure.</p><p>TRL supports custom reward modeling to perform reward modeling on dataset and model. The reward trainer expects a very specific format for the dataset. Since the model will be trained to predict which sentence is the most relevant, given two sentences.</p><p>The entries should be named</p><ul><li>input_ids_chosen</li><li>attention_mask_chosen</li><li>input_ids_rejected</li><li>attention_mask_rejected</li></ul><p>The j and k suffixes are used to denote the two sentences in the paired dataset.</p><pre>def preprocess_function(examples):<br>    new_examples = {<br>        &quot;input_ids_chosen&quot;: [],<br>        &quot;attention_mask_chosen&quot;: [],<br>        &quot;input_ids_rejected&quot;: [],<br>        &quot;attention_mask_rejected&quot;: [],<br>    }<br>    for chosen, rejected in zip(examples[&quot;chosen&quot;], examples[&quot;rejected&quot;]):<br>        tokenized_j = tokenizer(chosen, truncation=True)<br>        tokenized_k = tokenizer(rejected, truncation=True)<br><br>        new_examples[&quot;input_ids_chosen&quot;].append(tokenized_j[&quot;input_ids&quot;])<br>        new_examples[&quot;attention_mask_chosen&quot;].append(tokenized_j[&quot;attention_mask&quot;])<br>        new_examples[&quot;input_ids_rejected&quot;].append(tokenized_k[&quot;input_ids&quot;])<br>        new_examples[&quot;attention_mask_rejected&quot;].append(tokenized_k[&quot;attention_mask&quot;])<br><br>    return new_examples</pre><p>With the processed and formatted dataset, the RewardTrainer works similarly to Hugging Face’s Trainer. It’s best to use an AutoModelForSequenceClassification as the reward model which is trained on a dataset of paired examples, where each example is a tuple of two sequences, training it to identify the most relevant sequence in a given pair for a task at hand.</p><p>To ensure our models only consider sequences within a specific length, we applied a mapping and filtering:</p><pre>train_dataset = train_dataset.map(preprocess_function, batched=True, num_proc=4)<br>train_dataset = train_dataset.filter(lambda x: len(x[&quot;input_ids_chosen&quot;]) &lt;= 512 and len(x[&quot;input_ids_rejected&quot;]) &lt;= 512)</pre><h3>Quantizing the Model</h3><p>Quantization can reduce model size and inference time without a significant decrease in performance. For this experiment, the opt-350m and GPT-2 model was quantized to a 4-bit precision:</p><pre>from transformers import AutoModelForSequenceClassification, BitsAndBytesConfig<br><br>quantization_config = BitsAndBytesConfig(<br>    load_in_8bit=False,<br>    load_in_4bit=True<br>)<br><br>model = AutoModelForSequenceClassification.from_pretrained(<br>    &quot;gpt2&quot;,<br>    quantization_config=quantization_config,<br>    device_map={&quot;&quot;: 0},<br>    trust_remote_code=True,<br>    num_labels=1,<br>)<br>model.config.use_cache = False</pre><h3>Setting Up Reward-based Training</h3><p>The aim was to fine-tune the models using rewards. This is where the trl library comes into play. We used the RewardTrainer class:</p><pre>trainer = RewardTrainer(<br>    model=model,<br>    tokenizer=tokenizer,<br>    args=training_args,<br>    train_dataset=train_dataset,<br>    peft_config=peft_config,<br>    max_length=512,<br>)</pre><p>With the PEFT configuration, specific parameters like r, lora_alpha, and others were set to optimize the training process.</p><h3>Insights &amp; Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*X9fKNetGJzmC_zrkar7zPw.png" /><figcaption>Training Loss Comparison</figcaption></figure><ol><li><strong>Training Loss:</strong> From the results, it’s evident that both models converge towards a similar loss by the 10,000th step. However, the trajectory of the decline in loss differs. opt-350m starts with a lower initial loss, while GPT-2, despite starting with a higher initial loss, reduces it at a more gradual pace. This suggests that the OPT-350M might have an initial advantage in terms of understanding or adapting to the dataset, but both models eventually reach a similar level of performance.</li><li><strong>Training Time: </strong>GPT-2 demonstrated a quicker training speed, completing in 33:18, while opt-350m required 1:23:54. The difference in training duration might be attributed to the inherent architecture of each model. GPT-2’s design may be inherently more optimized for tasks resembling those in the Anthropic/hh-rlhf dataset. Alternatively, GPT-2 might be better suited or more compatible with the specific characteristics of the Anthropic/hh-rlhf dataset, leading to quicker convergence.</li></ol><h3>Future Exploration</h3><p>It would be worthwhile to delve deeper into why such a significant difference in training time exists, considering both models are prominent in the NLP field. This could lead to insights about model architecture, optimization, or even dataset-specific peculiarities.</p><h3>Conclusion</h3><p>In our study comparing opt-350m and GPT-2 on the Anthropic/hh-rlhf dataset, distinct training patterns emerged. While opt-350m experienced a rapid initial decline in loss, GPT-2 showed a steadier descent but trained faster overall. This difference underscores the unique efficiencies of each model and their interaction with the dataset. It’s essential to recognize that our observations depend on various factors, including dataset choice and preprocessing. Reward-based training emerges as a potent technique in language modeling, highlighting the need to match model strengths to specific tasks for optimal outcomes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd1050a5e6ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-tune 4-bit Llama-2–7B with Flash Attention Using DPO]]></title>
            <link>https://medium.com/@drishtisharma96505/fine-tune-llama-2-7b-with-flash-attention-using-dpo-f989e7e6bfa4?source=rss-5dee8b22f042------2</link>
            <guid isPermaLink="false">https://medium.com/p/f989e7e6bfa4</guid>
            <dc:creator><![CDATA[Drishti Sushma]]></dc:creator>
            <pubDate>Mon, 11 Sep 2023 02:36:01 GMT</pubDate>
            <atom:updated>2025-08-01T22:06:28.744Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>This is essentially a documentation of the training process of 4-bit llama-2–7b model which I was trying to fine-tune on Stack-exchange dataset using DPO, but for some reason, the training prematurely stopped in between and since I’m running low on compute hence I’m not able to retrain at the moment. However, I hope to get it trained as soon as possible.</p><p>Now let’s begin!</p><h3>What is DPO?</h3><p>DPO offers a streamlined method for optimizing human-derived preferences in LLMs, such as GPT-4 or Claude. Traditional models utilize reinforcement learning (RL) to train models based on human feedback. This process, known as Reinforcement Learning from Human Feedback (RLHF), involves building a good reward function and carefully training the model to produce sensible text that aligns with human expectations.</p><p>DPO simplifies the RLHF process. <strong>Instead of using RL and a reward model, DPO employs a direct binary cross-entropy loss to fine-tune models.</strong> This method is significantly more straightforward and eliminates many complexities associated with RLHF.</p><h3>How does DPO differ from PPO?</h3><p>While the traditional RLHF method uses an auxiliary reward model to fine-tune models, DPO skips the reward modeling step entirely. It uses an analytical mapping from the reward function to the optimal RL policy, allowing for direct optimization of the language model based on preference data. Essentially, DPO simplifies the optimization process by focusing on the reference model and omitting the need for RL-based optimization.</p><h3>Training with TRL’s DPO:</h3><p>The TRL library, which supports DPO, provides tools for the entire RLHF pipeline. However, with DPO, only supervised fine-tuning (SFT) and data annotation for preference labels are needed. The DPOTrainer in TRL optimizes the model using the preference data.</p><p>For example, to train with the Stack Exchange preference dataset, you’d need to format the data appropriately. After processing the data, you can use the DPOTrainer, which needs the base model from the SFT pipeline, a reference model, and other necessary parameters to begin training.</p><p>So in short, for training with TRL’s DPO we will need to do following three steps:</p><ol><li>a supervised fine-tuning (SFT) step</li><li>the process of annotating data with preference labels</li><li>provide the DPOTrainer in TRL with preference data from step 2 which has a very specific format</li></ol><p>Precisely, the data should be in dictionary format with the following three keys:</p><ul><li><strong>Context Prompt:</strong> Information provided to the model at inference time for text generation.</li><li><strong>Chosen Response:</strong> The preferred generated response corresponding to the prompt.</li><li><strong>Rejected Response: </strong>The response that is not preferred or should not be selected as the generated response for the given prompt.</li></ul><p>Now let’s begin with the training process!</p><h3>Setup and Preliminary Configuration</h3><p>The foundational tools and libraries for our experiment include accelerate, peft, bitsandbytes, transformers, and trl, among others.</p><pre>!pip install -q accelerate==0.21.0 peft==0.4.0 bitsandbytes==0.40.2 transformers==4.31.0 trl==0.5.0<br>!pip install -q sentencepiece</pre><p>Hardware checks ensure compatibility with Flash Attention, postulating a CUDA capability of 8 or more.</p><pre>!python -c &quot;import torch; assert torch.cuda.get_device_capability()[0] &gt;= 8, &#39;Hardware not supported for Flash Attention&#39;&quot;<br>!pip install ninja packaging<br>!MAX_JOBS=4 pip install flash-attn - no-build-isolation</pre><p>Now we will define a function that maps the dataset entries to return the desired dictionary.</p><pre>from typing import Dict, List, Union<br><br>def return_prompt_and_responses(samples) -&gt; Dict[str, Union[str, List[str]]]:<br>    return {<br>        &quot;prompt&quot;: [<br>            &quot;Question: &quot; + question + &quot;\n\nAnswer: &quot;<br>            for question in samples[&quot;question&quot;]<br>        ],<br>        &quot;chosen&quot;: samples[&quot;response_j&quot;],   # rated better than k<br>        &quot;rejected&quot;: samples[&quot;response_k&quot;], # rated worse than j<br>    }</pre><h3>Loading the Dataset and Processing it</h3><pre>from datasets import load_dataset<br>dataset = load_dataset(<br>    &quot;lvwerra/stack-exchange-paired&quot;,<br>    split=&quot;train&quot;,<br>    data_dir=&quot;data/rl&quot;<br>)</pre><p>We obtain the Stack-Exchange dataset, suitably formatted for our task. The data is shuffled, segmented, and processed to generate prompts and responses, and is further partitioned into training and test sets.</p><pre>original_columns = dataset.column_names<br>dataset = dataset.shuffle(seed=42).select(range(2000))<br>dataset</pre><pre>dataset.map(<br>    return_prompt_and_responses,<br>    batched=True,<br>    remove_columns=original_columns<br>)</pre><h3>Split the Dataset into Training and Testing Subsets</h3><pre>dataset = dataset.train_test_split(seed=42, shuffle=True, test_size=0.1)<br>print(dataset)</pre><h3>Model Loading and Flash Attention Implementation</h3><p>Llama-2–7b is prepared with considerations for Flash Attention.</p><pre>import torch<br>from transformers import AutoModelForCausalLM, BitsAndBytesConfig, AutoTokenizer<br>model_id = &quot;NousResearch/Llama-2–7b-hf&quot;</pre><pre>use_flash_attention = True<br><br># replace attention with flash attention<br>if torch.cuda.get_device_capability()[0] &gt;= 8:<br>  from utils.llama_patch import replace_attn_with_flash_attn<br>  print(&quot;Using flash attention&quot;)<br>  replace_attn_with_flash_attn()<br>  use_flash_attention = True</pre><p>Once it’s confirmed that it’s using Flash Attention, the model is loaded with 4-bit quantization settings.</p><pre># load the base model in 4-bit quantization<br>bnb_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_compute_dtype=torch.bfloat16,<br>)<br><br>base_model = AutoModelForCausalLM.from_pretrained(<br>    model_id,<br>    quantization_config=bnb_config,<br>    device_map={&quot;&quot;: 0},<br>    trust_remote_code=True,<br>    use_auth_token=True,<br>)<br>base_model.config.use_cache = False<br>base_model.config.pretraining_tp = 1<br><br><br># Validate that the model is using flash attention, by comparing doc strings<br>if use_flash_attention:<br>    from utils.llama_patch import forward<br>    assert base_model.model.layers[0].self_attn.forward.__doc__ == forward.__doc__, &quot;Model is not using flash attention&quot;<br><br><br><br>tokenizer = AutoTokenizer.from_pretrained(model_id)<br>tokenizer.pad_token = tokenizer.eos_token<br>tokenizer.padding_side = &quot;right&quot;</pre><h3>LoRA Integration and Model Preparation</h3><p>LoRA (Low-Rank Adaptation) is incorporated atop the quantized base model. Post-LoRA integration, the model undergoes additional configurations to make it training-ready.</p><pre>from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model<br><br># add LoRA layers on top of the quantized base model<br>peft_config = LoraConfig(<br>        lora_alpha=16,<br>        lora_dropout=0.1,<br>        r=64,<br>        bias=&quot;none&quot;,<br>        task_type=&quot;CAUSAL_LM&quot;,<br>)<br><br><br># prepare model for training<br>base_model = prepare_model_for_kbit_training(base_model)<br>base_model = get_peft_model(base_model, peft_config)</pre><h3>Extending Model Compatibility with Torch Bfloat16</h3><p>Now we will apply ‘upcast_layer_for_flash_attention’ to extend model compatibility with Torch Bfloat16</p><pre>from utils.llama_patch import upcast_layer_for_flash_attention<br>model = upcast_layer_for_flash_attention(base_model, torch.bfloat16)</pre><h3>Defining Training Arguments</h3><pre>from transformers import TrainingArguments<br><br>training_args = TrainingArguments(<br>    output_dir=&quot;llama-7-int4-stack-exchange&quot;,<br>    max_steps=100,<br>    per_device_train_batch_size=6 if use_flash_attention else 4,<br>    optim=&quot;paged_adamw_32bit&quot;,<br>    logging_steps=10,<br>    save_strategy=&quot;steps&quot;,<br>    learning_rate=2e-4,<br>    bf16=True,<br>    tf32=True,<br>    lr_scheduler_type=&quot;constant&quot;,<br>    disable_tqdm=True # disable tqdm since with packing values are in correct<br>)</pre><h3>Setting Up the SFTTrainer</h3><pre>from trl import SFTTrainer<br><br>max_seq_length = 2048 # max sequence length for model and packing of the dataset<br><br>trainer = SFTTrainer(<br>    model=base_model,<br>    train_dataset=dataset[&quot;train&quot;],<br>    eval_dataset=dataset[&quot;test&quot;],<br>    peft_config=peft_config,<br>    max_seq_length=max_seq_length,<br>    tokenizer=tokenizer,<br>    packing=True,<br>    formatting_func=return_prompt_and_responses,<br>    args=training_args,<br>)</pre><h3>Training and Saving the Model</h3><pre>trainer.train()<br><br># save model<br>trainer.save_model()</pre><h3>DPO (Differential Privacy Optimization) Training</h3><p>Subsequent to the primary training phase, the model is subjected to DPO training for enhanced privacy considerations. This involves a separate trainer and an extended training process, followed by model saving.</p><pre>model = AutoPeftModelForCausalLM.from_pretrained(<br>    script_args.model_name_or_path, # location of saved SFT model<br>    low_cpu_mem_usage=True,<br>    torch_dtype=torch.float16,<br>    load_in_4bit=True,<br>    is_trainable=True,<br>)<br>model_ref = AutoPeftModelForCausalLM.from_pretrained(<br>    script_args.model_name_or_path,  # same model as the main one<br>    low_cpu_mem_usage=True,<br>    torch_dtype=torch.float16,<br>    load_in_4bit=True,<br>)<br>...<br>dpo_trainer = DPOTrainer(<br>    model,<br>    model_ref,<br>    args=training_args,<br>    beta=0.1,<br>    train_dataset=dataset[&quot;train&quot;],<br>    eval_dataset=dataset[&quot;test&quot;],<br>    tokenizer=tokenizer,<br>    peft_config=peft_config,<br>)<br>dpo_trainer.train()<br>dpo_trainer.save_model()</pre><h3>Conclusion</h3><p>Our endeavor to train the 4-bit llama-2–7b model using DPO faced challenges, but the potential of DPO is evident. DPO offers a streamlined approach to fine-tuning language models, sidestepping traditional complexities. The TRL library further simplifies this by providing essential tools for the process. As we embark on the training journey, it’s vital to understand the steps involved and the ultimate goal: a model that aligns with human preferences.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f989e7e6bfa4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparative Analysis of Fine-tuned BERT-based Models for Detecting Hate Speech in Social Media]]></title>
            <link>https://medium.com/@drishtisharma96505/comparative-analysis-of-fine-tuned-bert-based-models-for-detecting-hate-speech-in-social-media-7b0dd251a669?source=rss-5dee8b22f042------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b0dd251a669</guid>
            <dc:creator><![CDATA[Drishti Sushma]]></dc:creator>
            <pubDate>Thu, 07 Sep 2023 23:49:43 GMT</pubDate>
            <atom:updated>2023-09-08T10:56:12.155Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DuT-2Hm2eVbqzMm6" /><figcaption>Source: <a href="https://cjp.org.in/">https://cjp.org.in/</a></figcaption></figure><h3>Abstract</h3><p>Hate speech detection in social media is a pressing concern due to its negative impact on online communities. This research explores the effectiveness of various pre-trained transformer models for detecting normal, hate, and offensive speech in social media text. We fine-tuned various BERT-based models namely “bert-base-uncased”, “bert-large-uncased”, “distilbert-base-uncased”, “diptanu/fBERT”, “GroNLP/hateBERT”, “roberta-large”, and assessed model performance based on multiple evaluation metrics. The results provide valuable insights into the capabilities of different transformer models in addressing this critical issue.</p><h3>Dataset Overview and Insights</h3><p>We utilized a publicly available <a href="https://www.kaggle.com/datasets/subhajournal/normal-hate-and-offensive-speeches">dataset</a> containing tweets categorized into three classes: normal, hate, and offensive. The dataset consisted of 12 CSV files, each containing data for these categories. Further, we concatenated data from various CSV files into a single DataFrame. This step helped us prepare the data for further analysis and model training.</p><p>We visualized the distribution of labels (Normal, Hate, Offensive) to understand the class distribution within the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*aQsQOGxzXgkGp-FNvHoUww.png" /><figcaption>Label Distribution</figcaption></figure><p>To gain insights into the most frequent words within each class, we created word clouds for Normal, Hate, and Offensive texts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XGIDQUcHhGe47opLxOx_Wg.png" /><figcaption>Word Clouds for Offensive, hate, and normal texts</figcaption></figure><p>Understanding the distribution of text lengths was crucial for our analysis. Hence, we visualized the distribution of text lengths for each class with the help of histogram and boxplot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Kz5YLJgVN-WdmNNbBlGWA.png" /><figcaption>Text Distribution by Label: i) Histogram, ii) Boxplot</figcaption></figure><h3>Data Pre-processing</h3><p>For the experiments, we avoided converting text to lowercase and eliminating special characters in the preprocessing step due to following reasons:</p><ol><li>There’s a possibility that the case of a word carries meaningful information (for instance, if capitalized words are used more frequently in certain types of speech in the dataset).</li><li>In contexts like hate speech or offensive language, users might obfuscate words with special characters to bypass content filters. In such cases, simply removing these characters might strip away essential information.</li></ol><p>Moreover, we performed label-specific threshold filtering to ensure that text lengths are within reasonable limits for each class.</p><pre>condition_hate = (data[&#39;label&#39;] == &#39;Hate&#39;) &amp; (data[&#39;text_length&#39;] &lt;= 62)<br>condition_offensive = (data[&#39;label&#39;] == &#39;Offensive&#39;) &amp; (data[&#39;text_length&#39;] &lt;= 60)<br>condition_normal = (data[&#39;label&#39;] == &#39;Normal&#39;) &amp; (data[&#39;text_length&#39;] &lt;= 88)<br><br>data = data[condition_hate | condition_offensive | condition_normal]</pre><h3>Experimental Setup</h3><p>In all experiments, we maintained consistent hyperparameters:</p><ul><li>Number of training epochs (NUM_OF_EPOCHS): 5</li><li>Batch size (BATCH_SIZE): 16</li><li>Learning rate (LEARNING_RATE): 2e-5</li><li>Maximum sequence length (MAX_LENGTH): Set to 1.3 times the maximum text length in the dataset (88 words).</li></ul><h3>Handling Mild Class Imbalance</h3><p>The mild class imbalance was handled by assigning different weights to each class during the training process to ensure the model’s robustness in handling underrepresented classes.</p><pre>total_samples = 1200 + 966 + 895<br>weights = torch.tensor([total_samples/1200, total_samples/966, total_samples/895])<br><br># Normalize the weights for numerical stability<br>weights = weights / weights.sum()</pre><p>To effectively incorporate the class weights into the training process, we created a custom trainer. This custom trainer was responsible for computing the loss using the weights we just defined and ensuring that the model’s training aligns with our objectives.</p><pre>class CustomTrainer(Trainer):<br>    def compute_loss(self, model, inputs, return_outputs=False):<br>        labels = inputs.get(&quot;labels&quot;)<br>        outputs = model(**inputs)<br>        logits = outputs.get(&quot;logits&quot;)<br>        loss_fct = torch.nn.CrossEntropyLoss(weight=weights)<br>        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))<br>        return (loss, outputs) if return_outputs else loss</pre><p>By applying these weights to the CrossEntropyLoss, we emphasize the importance of each class according to its weight. This approach ensures that the model pays more attention to the underrepresented classes, “Hate” and “Offensive,” leading to a more balanced and effective hate speech detection model.</p><h3>Model Evaluation Metrics</h3><p>To assess the performance of the pre-trained transformer models, we employed a range of evaluation metrics, including <strong>Accuracy, Weighted F1-score, Weighted recall, Weighted precision, Micro F1-score, Micro recall, Micro precision, Macro F1-score, Macro recall, and Macro precision.</strong></p><h3>Models and Results</h3><p>We conducted experiments using the following pre-trained transformer models:</p><ol><li>bert-base-uncased</li><li>bert-large-uncased</li><li>distilbert-base-uncased</li><li>diptanu/fBERT</li><li>GroNLP/hateBERT</li><li>roberta-large</li></ol><p>Here’s a tabular comparison of their results, including training and evaluation times:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yawq7NQ2ZO9CQL6UlA95Ig.png" /><figcaption>Tabular Comparison of Results</figcaption></figure><h3>Analysis</h3><ol><li>All fine-tuned transformer models achieved high accuracy and performance in identifying hate speech. bert-base-uncased, bert-large-uncased, and distilbert-base-uncased exhibited nearly identical performance, suggesting that smaller models may suffice for this task.</li><li>diptanu/fBERT and GroNLP/hateBERT also performed exceptionally well, with GroNLP/hateBERT offering a slightly lower accuracy but still maintaining high precision and recall.</li><li>roberta-large achieved a good accuracy of 98.37% but exhibited slightly lower macro-level performance compared to other models.</li><li>Training time varies significantly among models, with roberta-large having the longest training time (281.49 seconds) and distilbert-base-uncased the shortest (46.27 seconds).</li><li>Evaluation time varies as well, with bert-large-uncased and roberta-large having the longest times (approximately 1.74 seconds) and distilbert-base-uncased the shortest (365 milliseconds).</li></ol><h3>Future Directions</h3><ol><li><strong>Multimodal Approaches: </strong>Integrating image and text data for hate speech detection could be explored for a more comprehensive solution.</li><li><strong>Bias Mitigation:</strong> Investigating methods to reduce model bias, especially concerning underrepresented groups.</li></ol><h3>Conclusion</h3><p>This research presented a comparative analysis of fine-tuned BERT-based transformer models for hate speech detection in social media text. The findings suggest that several models offer high accuracy and performance, with potential trade-offs between model size and performance. Future work will focus on further improving and deploying these models in real-world hate speech detection systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b0dd251a669" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>