<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sushama Rangarajan on Medium]]></title>
        <description><![CDATA[Stories by Sushama Rangarajan on Medium]]></description>
        <link>https://medium.com/@sushamarangarajan?source=rss-ca18f5a16d62------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fvSVRcw-c04HwIzH4MrV0A.png</url>
            <title>Stories by Sushama Rangarajan on Medium</title>
            <link>https://medium.com/@sushamarangarajan?source=rss-ca18f5a16d62------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:37:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sushamarangarajan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[BERT fine-tuning for News Classification]]></title>
            <link>https://sushamarangarajan.medium.com/fine-tuning-distilbert-for-news-classification-7d3f06fe2c97?source=rss-ca18f5a16d62------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d3f06fe2c97</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Sushama Rangarajan]]></dc:creator>
            <pubDate>Fri, 27 Feb 2026 21:04:30 GMT</pubDate>
            <atom:updated>2026-02-27T21:31:37.056Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How I built an end-to-end NLP pipeline with PyTorch, HuggingFace Transformers, and Gradio — and what the model’s mistakes taught me</em></p><h3><strong>TLDR</strong></h3><p>I fine-tuned DistilBERT on 120,000 news articles using PyTorch and HuggingFace Transformers, achieving 94.33% test accuracy across 4 categories. The findings were that 51% of all errors came from a single ambiguous boundary between Business and Sci/Tech news. The model is live on HuggingFace Spaces. This article covers the full pipeline: data, model, training, evaluation, and deployment.</p><h3>Why This Project</h3><p>I wanted to learn Pytorch and wanted to build something end-to-end — from raw data to a live, publicly accessible demo — and understand every step along the way. This project helped me understand how much of free resources are available to use &amp; how we can leverage platforms like Kaggle to code and run with GPUs and Hugging face - push models that can be easily accessible to users &amp; spaces to host a UI.</p><p>This article walks through how I fine-tuned DistilBERT on the AG News dataset to classify news articles into four categories: World, Sports, Business, and Sci/Tech. The model achieved <strong>94.33% test accuracy</strong> — within 1–2% of state of the art — and is deployed as a live Gradio demo on HuggingFace Spaces.</p><p>Here’s everything I learned.</p><h4>The Dataset</h4><p>AG News is a benchmark NLP dataset containing 120,000 training and 7,600 test news articles across four perfectly balanced categories. The balance matters — it means accuracy is a reliable metric and you don’t need to worry about class weighting.</p><p>Each sample has a title and description. I combined them using a separator token:</p><pre>text = &quot;[TITLE]&quot; + title + &quot;[SEP]&quot; + description</pre><p>Before writing a single line of model code, I ran a quick token length analysis on the combined text:</p><pre>lengths = train_dataset[&quot;text&quot;].apply(<br>    lambda x: len(tokenizer(x, truncation=False)[&quot;input_ids&quot;])<br>)<br>print(f&quot;Max: {lengths.max()}&quot;)       # 383<br>print(f&quot;Mean: {lengths.mean():.1f}&quot;) # 57.2<br>print(f&quot;95th percentile: {lengths.quantile(0.95):.0f}&quot;)  # 86</pre><p>The mean was just 57 tokens, with 99.9% of samples fitting within 128 tokens. The handful of samples over 200 tokens turned out to be legitimate long articles — but with only 144 out of 120,000 samples (0.12%) exceeding that threshold, setting max_length=128 seem to be sufficient given that we wouldn’t be losing a lot of data. One can still increase it to 256 also &amp; see how the results differ. But for this use-case I chose to keep it 128 &amp; it covers virtually all data while keeping training fast.</p><h4>Why DistilBERT?</h4><p>DistilBERT is a distilled (compressed) version of BERT — 40% smaller, 60% faster, but retaining 97% of BERT’s language understanding. Read more about DistilBERT <a href="https://huggingface.co/distilbert/distilbert-base-uncased">here</a>.</p><p>The core idea behind fine-tuning is simple: DistilBERT already understands language from pretraining on billions of words. You don’t retrain it from scratch — you add a small classification layer on top and train <em>that</em> on your labeled data. The pretrained knowledge transfers.</p><h4>Building the PyTorch Pipeline</h4><h4>The Dataset Class</h4><p>PyTorch requires you to wrap your data in a Dataset class with three methods:</p><pre>from torch.utils.data import Dataset<br>import torch<br>class AGNewsDataset(Dataset):<br>    def __init__(self, dataframe, tokenizer, max_length=128):<br>        self.data = dataframe<br>        self.tokenizer = tokenizer<br>        self.max_length = max_length<br>    def __len__(self):<br>        return len(self.data)<br>    def __getitem__(self, idx):<br>        text = self.data.iloc[idx][&quot;text&quot;]<br>        label = self.data.iloc[idx][&quot;label&quot;] - 1  # convert 1-4 to 0-3<br>        encoding = self.tokenizer(<br>            text,<br>            max_length=self.max_length,<br>            padding=&quot;max_length&quot;,<br>            truncation=True,<br>            return_tensors=&quot;pt&quot;<br>        )<br>        return {<br>            &quot;input_ids&quot;: encoding[&quot;input_ids&quot;].squeeze(),<br>            &quot;attention_mask&quot;: encoding[&quot;attention_mask&quot;].squeeze(),<br>            &quot;label&quot;: torch.tensor(label, dtype=torch.long)<br>        }</pre><p>A few things worth noting:</p><ul><li>Labels are <strong>1-indexed in AG News</strong> (1–4) but PyTorch’s CrossEntropyLoss expects 0-indexed. Subtracting 1 is easy to miss and causes confusing errors. If your dataset already has the range from 0–3, then remove this step.</li><li>.squeeze() removes the batch dimension the tokenizer adds — the DataLoader handles batching separately.</li><li>dtype=torch.long is required by CrossEntropyLoss, which uses labels as indices.</li></ul><h4>The Model</h4><p>The classifier is a thin wrapper around DistilBERT:</p><pre>from transformers import AutoModel<br>import torch.nn as nn<br>class AGNewsClassifier(nn.Module):<br>    def __init__(self, num_classes=4, dropout=0.3):<br>        super().__init__()<br>        self.bert = AutoModel.from_pretrained(&quot;distilbert-base-uncased&quot;)<br>        self.dropout = nn.Dropout(dropout)<br>        self.classifier = nn.Linear(768, num_classes)<br>    def forward(self, input_ids, attention_mask):<br>        outputs = self.bert(input_ids=input_ids, attention_mask=attention_mask)<br>        cls_output = outputs.last_hidden_state[:, 0, :]  # [CLS] token<br>        dropped = self.dropout(cls_output)<br>        return self.classifier(dropped)</pre><p>The [CLS] token at position 0 of DistilBERT&#39;s output is a 768-dimensional vector that represents the meaning of the entire input sequence. A single linear layer maps this to 4 class scores. That&#39;s 66 million parameters from DistilBERT, and just 3,076 from the classification head.</p><h4>The Training Loop</h4><p>PyTorch’s training loop follows a consistent 4-step pattern every time:</p><pre>for epoch in range(EPOCHS):<br>    model.train()<br>    for batch in tqdm(train_loader):<br>        input_ids = batch[&quot;input_ids&quot;].to(device)<br>        attention_mask = batch[&quot;attention_mask&quot;].to(device)<br>        labels = batch[&quot;label&quot;].to(device)<br>        <br>        optimizer.zero_grad()                          # 1. clear gradients<br>        logits = model(input_ids, attention_mask)      # 2. forward pass<br>        loss = criterion(logits, labels)               # 3. compute loss<br>        loss.backward()                                # 4. backward pass<br>        optimizer.step()                               # 5. update weights</pre><p>Key hyperparameter choices:</p><ul><li><strong>AdamW optimizer</strong> with lr=2e-5 — i.e., <em>0.00002 is the standard learning rate for fine-tuning BERT-based model. </em>A small learning rate preserves the pretrained weights.</li><li><strong>3 epochs</strong> — enough for convergence on a balanced 120k dataset without overfitting.</li><li><strong>Batch size 64</strong> — larger batches on GPU for stable gradient estimates.</li></ul><p>Training ran on Kaggle’s free T4 x2 GPU, taking roughly 15–20 minutes per epoch. Highly recommend trying it out — this was my first project using Kaggle’s GPU too.</p><h4>Results</h4><p>The model achieved <strong>94.33% test accuracy</strong> across all 4 classes. Breaking it down by category:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CJyo2wTApjH94bjmRkjZhQ.png" /><figcaption>Confusion Matrix</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SmZ7joolUvTh-4WQmxegGw.png" /><figcaption>Classification Metrics</figcaption></figure><p>Sports is by far the strongest category — sports language is distinctive and unambiguous. Business is the weakest, and the confusion matrix reveals exactly why.</p><h4>What the Confusion Matrix Revealed</h4><p>This is the most interesting finding from the project. Of the 432 total misclassifications:</p><ul><li><strong>145 Business articles</strong> were predicted as Sci/Tech</li><li><strong>76 Sci/Tech articles</strong> were predicted as Business</li></ul><p>Together that’s 221 errors — <strong>51% of all mistakes</strong> — concentrated in a single category pair.</p><p>This isn’t a model failure. It’s a reflection of a genuine ambiguity in the data. Consider these headlines:</p><ul><li><em>“Apple announces record quarterly revenue”</em> — Business or Sci/Tech?</li><li><em>“Google’s AI division reports breakthrough in protein folding” </em>— Sci/Tech or Business?</li><li><em>“Tesla stock surges after autonomous driving update”</em> — Business or Sci/Tech?</li></ul><p>The model is doing the right thing by being uncertain on genuinely ambiguous cases. The fact that it’s confident and correct on 94.33% of everything else is all that matters.</p><p>Now we know why the model mis-behaves on certain outlier cases.</p><h4>Deployment</h4><p>The trained model is hosted on HuggingFace Hub and served through a Gradio interface on HuggingFace Spaces. The entire inference pipeline in production:</p><pre>def news_classification(text):<br>    # check for empty texts<br>    if not text.strip():<br>        return {}<br>    <br>    #if not empty, proceed<br>    tokens = tokenizer(<br>        text,<br>        max_length = 128,<br>        padding=&#39;max_length&#39;,<br>        truncation = True,<br>        return_tensors = &#39;pt&#39;<br>    )<br><br><br>    with torch.no_grad():<br>        #torch.no_grad disables the gradient calculation operation temporarily<br>        logits = model(tokens[&#39;input_ids&#39;],tokens[&#39;attention_mask&#39;])<br>    <br>    probs = F.softmax(logits, dim = 1).squeeze()<br>    <br>    #return dict with labels and probability scores<br>    return {LABELS[i]:float(probs[i]) for i in range(4)}</pre><p>torch.no_grad() disables gradient calculation temporarily. This saves memory and speeds up inference.</p><p>The Gradio interface returns probability scores for all four categories, not just the top prediction. This lets users see model confidence and understand borderline cases.</p><p><strong>Try it yourself:</strong> <a href="https://huggingface.co/spaces/sushamarangarajan/ag-news-classifier">huggingface.co/spaces/sushamarangarajan/ag-news-classifier</a></p><p>I hope this project was interesting to you as much as I enjoyed working on this.</p><h4>Resources</h4><ul><li>Kaggle Notebook: <a href="https://www.kaggle.com/code/sushamarangarajan/agnewsclassifier-pytorch">https://www.kaggle.com/code/sushamarangarajan/agnewsclassifier-pytorch</a></li><li>Model on HuggingFace: <a href="https://huggingface.co/sushamarangarajan/ag-news-classifier">huggingface.co/sushamarangarajan/ag-news-classifier</a></li><li>Live Demo: <a href="https://huggingface.co/spaces/sushamarangarajan/ag-news-classifier">huggingface.co/spaces/sushamarangarajan/ag-news-classifier</a></li><li>Dataset: AG News via HuggingFace Datasets</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d3f06fe2c97" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>