<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pratik Sharma on Medium]]></title>
        <description><![CDATA[Stories by Pratik Sharma on Medium]]></description>
        <link>https://medium.com/@pratikacharya527?source=rss-aafa75b21250------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*imOfHiujXsnzzOsxDYFBZw.jpeg</url>
            <title>Stories by Pratik Sharma on Medium</title>
            <link>https://medium.com/@pratikacharya527?source=rss-aafa75b21250------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:49:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pratikacharya527/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building HeritageNet: An On-Device Image Classifier and Open Dataset for Nepali Heritage Sites]]></title>
            <link>https://medium.com/@pratikacharya527/building-heritagenet-an-on-device-image-classifier-and-open-dataset-for-nepali-heritage-sites-a2e2745d3f4b?source=rss-aafa75b21250------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2e2745d3f4b</guid>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Pratik Sharma]]></dc:creator>
            <pubDate>Sat, 29 Aug 2026 16:22:30 GMT</pubDate>
            <atom:updated>2026-08-29T16:22:30.896Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A technical write-up of how I built a 26-class heritage-site classifier with transfer learning, added open-set rejection so it can say “I don’t know,” and shipped it to run entirely in the browser — plus the experiment that reframed the whole project.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cMpQpLIbH_E48i_p4XIIrQ.jpeg" /></figure><p>Modern image models recognize the Taj Mahal instantly but have never seen a local Nepali temple, because that heritage was never in their training data. This post is the engineering story of closing that gap: collecting a dataset from scratch, training a lightweight classifier with transfer learning, giving it the ability to reject out-of-distribution inputs, and deploying it on-device via ONNX Runtime Web.</p><p>I’ll focus on the technical decisions and the reasoning behind them. Where I show code, it’s representative of the actual implementation — the real repository has the full pipeline, but these snippets capture the important pieces faithfully.</p><blockquote><strong><em>Stack:</em></strong><em> PyTorch + torchvision · MobileNetV3-Small · ONNX / ONNX Runtime Web · trained on a single RTX 3050 (6 GB).</em></blockquote><h3>The problem, stated technically</h3><p>I wanted a classifier over 26 Nepali heritage sites with three hard constraints:</p><ol><li><strong>Small data.</strong> No existing dataset — I’d be collecting ~1,000 images total, which is tiny by deep-learning standards.</li><li><strong>On-device deployment.</strong> It had to run in a browser on a phone, so model size and CPU latency mattered as much as accuracy.</li><li><strong>Open-world robustness.</strong> A deployed classifier will be shown things that aren’t heritage sites at all; it needed a way to decline rather than confidently mislabel a photo of a person as a temple.</li></ol><p>Each constraint drove a concrete decision, below.</p><h3>Data: collection, structure, and the split that keeps you honest</h3><p>We photographed the sites ourselves and organized them in the canonical image-folder layout — one directory per class:</p><pre>dataset/<br>├── Ashok_stupa_Chaitya/<br>│   ├── img001.jpg<br>│   └── ...<br>├── Badrinath_Temple/<br>├── MahaKaleshwor_Shiva_Temple/<br>└── ...  (26 classes)</pre><p>This layout loads directly with torchvision.datasets.ImageFolder, and the folder name becomes the label — no separate annotation file to maintain.</p><p>The dataset is small (~1,000 images) and imbalanced: the best-represented site has 87 images, the thinnest has ~22. With numbers that small, a naive random split can accidentally leave a class with almost no test examples, making its metrics meaningless. So I used a <strong>stratified split</strong> that preserves per-class proportions, with a fixed seed for reproducibility:</p><pre>from sklearn.model_selection import train_test_split</pre><pre># paths: list of image paths; labels: parallel list of class indices<br>train_paths, temp_paths, train_lbls, temp_lbls = train_test_split(<br>    paths, labels, test_size=0.30, stratify=labels, random_state=42<br>)<br>val_paths, test_paths, val_lbls, test_lbls = train_test_split(<br>    temp_paths, temp_lbls, test_size=0.55, stratify=temp_lbls, random_state=42<br>)<br># ~720 train / ~145 val / ~179 test</pre><p>One reproducibility detail worth calling out: because the dataset is meant to <em>grow</em>, I assign each image to a split using a hash of its content rather than a purely random shuffle, so adding new images never reshuffles the ones already assigned. That keeps results comparable across dataset versions.</p><h3>Preprocessing</h3><p>Standard ImageNet preprocessing, because we’re using an ImageNet-pretrained backbone and the input statistics need to match what it was trained on:</p><pre>from torchvision import transforms</pre><pre>IMAGENET_MEAN = [0.485, 0.456, 0.406]<br>IMAGENET_STD  = [0.229, 0.224, 0.225]</pre><pre>train_tf = transforms.Compose([<br>    transforms.Resize((256, 256)),<br>    transforms.RandomResizedCrop(224, scale=(0.8, 1.0)),<br>    transforms.RandomHorizontalFlip(),<br>    transforms.ColorJitter(0.2, 0.2, 0.2),<br>    transforms.ToTensor(),<br>    transforms.Normalize(IMAGENET_MEAN, IMAGENET_STD),<br>])</pre><pre>eval_tf = transforms.Compose([<br>    transforms.Resize((256, 256)),<br>    transforms.CenterCrop(224),<br>    transforms.ToTensor(),<br>    transforms.Normalize(IMAGENET_MEAN, IMAGENET_STD),<br>])</pre><p>The augmentations (crop, flip, jitter) matter more than usual here precisely <em>because</em> the dataset is small — they’re a cheap way to synthetically increase variety and fight overfitting. Critically, the <strong>eval transform is deterministic</strong> (center crop, no augmentation), and — this bites people — the browser deployment must replicate this exact pipeline, or you get train/serve skew where the model behaves differently in production than in your notebook.</p><h3>Model: why MobileNetV3-Small, and transfer learning</h3><p>The deployment constraint drove the architecture choice. I didn’t need the highest possible accuracy on a benchmark; I needed something that runs in ~milliseconds on a phone CPU and is small enough to download into a browser. MobileNetV3-Small fits: a few megabytes, designed for edge inference.</p><p>We load it pretrained and swap the classifier head for our 26 classes:</p><pre>import torch<br>import torch.nn as nn<br>from torchvision.models import mobilenet_v3_small, MobileNet_V3_Small_Weights</pre><pre>def build_model(num_classes=26, dropout=0.2):<br>    model = mobilenet_v3_small(weights=MobileNet_V3_Small_Weights.IMAGENET1K_V1)<br>    in_features = model.classifier[3].in_features   # 1024 for V3-Small<br>    model.classifier[3] = nn.Linear(in_features, num_classes)<br>    # dropout already present in classifier[2]; set p explicitly<br>    model.classifier[2] = nn.Dropout(p=dropout, inplace=True)<br>    return model</pre><p>Training from scratch was never an option — ~1,000 images cannot teach general vision. Transfer learning lets us reuse features learned from millions of ImageNet images and only adapt them to our domain.</p><p>The interesting question was <em>how much</em> to adapt.</p><h3>The core experiment: freeze vs. fine-tune</h3><p>I compared three strategies, controlling how much of the backbone was trainable.</p><p><strong>1. Feature extraction (frozen backbone).</strong> Freeze everything, train only the new head:</p><pre>def freeze_backbone(model):<br>    for p in model.parameters():<br>        p.requires_grad = False<br>    for p in model.classifier[3].parameters():   # new head only<br>        p.requires_grad = True<br>    return model</pre><p><strong>2. Partial fine-tuning.</strong> Unfreeze the last two feature blocks plus the head:</p><pre>def set_partial_trainable(model, unfreeze_last_n=2):<br>    for p in model.parameters():<br>        p.requires_grad = False<br>    # unfreeze the last N blocks of the features module<br>    for block in model.features[-unfreeze_last_n:]:<br>        for p in block.parameters():<br>            p.requires_grad = True<br>    for p in model.classifier.parameters():<br>        p.requires_grad = True<br>    return model</pre><p><strong>3. Full fine-tuning.</strong> Everything trainable, with a <em>low</em> learning rate so we adapt gently rather than destroying pretrained features:</p><pre>model = build_model(num_classes=26, dropout=0.2)<br>for p in model.parameters():<br>    p.requires_grad = True</pre><pre>optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)   # low LR is deliberate<br>criterion = nn.CrossEntropyLoss()</pre><p>The training loop is standard, with <strong>early stopping on validation macro-F1</strong> — I keep the weights from the best validation epoch, not the last one, which is the main guard against overfitting:</p><pre>best_val_f1, best_state, patience, wait = 0.0, None, 8, 0<br>for epoch in range(max_epochs):<br>    train_one_epoch(model, train_loader, optimizer, criterion, device)<br>    val_f1 = evaluate_macro_f1(model, val_loader, device)<br>    if val_f1 &gt; best_val_f1:<br>        best_val_f1, best_state, wait = val_f1, copy.deepcopy(model.state_dict()), 0<br>    else:<br>        wait += 1<br>        if wait &gt;= patience:<br>            break<br>model.load_state_dict(best_state)   # restore best, not last</pre><h3>Results</h3><p>Evaluated on the held-out test set:</p><p>Strategy Trainable params Test accuracy Macro-F1 Frozen (feature extraction) ~2% 79.9% 78.4% Partial fine-tuning ~63% 82.7% 80.6% <strong>Full fine-tuning</strong> <strong>100%</strong> <strong>86.6%</strong> <strong>85.8%</strong></p><p>Accuracy rises monotonically with how much of the network is allowed to adapt. The reason is intuitive: frozen ImageNet features are generic and can’t capture the fine architectural details that distinguish one temple from a near-identical one. Full fine-tuning lets the convolutional filters specialize.</p><p>The clearest evidence of that: one look-alike temple (Narmadeshwor) had a per-class F1 of just <strong>0.30</strong> under feature extraction; under full fine-tuning it reached <strong>0.89</strong>. That’s fine-grained discrimination emerging as the backbone adapts.</p><p>I report <strong>macro-F1</strong> as the headline metric, not accuracy, because the dataset is imbalanced — macro-F1 averages per-class F1 equally, so a strong performance on big classes can’t hide failures on small ones:</p><pre>from sklearn.metrics import classification_report, f1_score</pre><pre>y_true, y_pred, y_prob = predict(model, test_loader, device)<br>macro_f1 = f1_score(y_true, y_pred, average=&quot;macro&quot;)<br>print(classification_report(y_true, y_pred, target_names=classes, digits=3))</pre><h3>The experiment that reframed the project</h3><p>Sitting at 86.6%, I assumed the path forward was better regularization or a better learning-rate schedule — classic “the model is overfitting, tune it harder” thinking. So I ran controlled experiments.</p><p>Configuration Test accuracy Macro-F1 Champion (dropout 0.2, lr 1e-4) 86.6% <strong>85.8%</strong> Dropout 0.3 87.2% 85.2% Learning rate 5e-5 84.4% 82.9%</p><p>None of them improved macro-F1. Higher dropout nudged raw accuracy up but macro-F1 down (noise); a lower LR hurt both.</p><p>That’s the tell. <strong>If the model were overfitting for lack of regularization, more regularization should have helped — and it didn’t.</strong> Combined with the fact that my weakest classes were always my <em>smallest</em> classes, the conclusion was clear: this is a <strong>data-limited</strong> regime, not a capacity- or regularization-limited one. The bottleneck isn’t the model. It’s that some sites only have ~20 photos.</p><p>This reframed everything. I stopped trying to be cleverer with the model and accepted that the highest-leverage work is collecting more data. That’s an unglamorous finding, but it’s the correct one, and it directly shapes the roadmap.</p><h3>Open-set recognition: teaching it to say “I don’t know”</h3><p>A softmax classifier always outputs a distribution over its known classes — it <em>cannot</em> represent “none of these.” Show it a person and it’ll still pick a temple, often confidently, because softmax normalizes over the 26 classes no matter what.</p><p>I confirmed the confidence signal is <em>useful but insufficient</em>: on the test set, mean top-prediction confidence was 91.8% for correct predictions vs 59.9% for wrong ones. So low confidence correlates with errors — but it can’t catch a confidently-wrong out-of-distribution input.</p><p>So I added a second gate operating in feature space. The idea: compute a <strong>prototype</strong> (mean embedding) for each class from the training set, then at inference measure the cosine distance from an input’s embedding to the nearest prototype. If it’s too far from <em>everything</em> known, reject it.</p><p>First, expose the embedding (the 1024-d vector before the final Linear layer):</p><pre>class HeritageNetWithEmbedding(nn.Module):<br>    &quot;&quot;&quot;Wraps MobileNetV3-Small to return both logits and the penultimate embedding.&quot;&quot;&quot;<br>    def __init__(self, base):<br>        super().__init__()<br>        self.features = base.features<br>        self.avgpool  = base.avgpool<br>        # classifier[:3] = Linear(576-&gt;1024) + Hardswish + Dropout; [3] = Linear(1024-&gt;26)<br>        self.pre_head = base.classifier[:3]<br>        self.head     = base.classifier[3]</pre><pre>    def forward(self, x):<br>        x = self.features(x)<br>        x = self.avgpool(x)<br>        x = torch.flatten(x, 1)<br>        emb = self.pre_head(x)        # 1024-d embedding<br>        logits = self.head(emb)       # 26 class logits<br>        return logits, emb</pre><p>Build one prototype per class by averaging (L2-normalized) training embeddings:</p><pre>import torch.nn.functional as F</pre><pre>@torch.no_grad()<br>def build_prototypes(model, loader, num_classes, device):<br>    sums = torch.zeros(num_classes, 1024, device=device)<br>    counts = torch.zeros(num_classes, device=device)<br>    model.eval()<br>    for x, y in loader:<br>        x = x.to(device)<br>        _, emb = model(x)<br>        emb = F.normalize(emb, dim=1)          # normalize so cosine == dot product<br>        for c in range(num_classes):<br>            mask = (y == c)<br>            if mask.any():<br>                sums[c]  += emb[mask.to(device)].sum(0)<br>                counts[c]+= mask.sum()<br>    prototypes = F.normalize(sums / counts.unsqueeze(1), dim=1)<br>    return prototypes    # (26, 1024)</pre><p>At inference, the <strong>dual gate</strong>: accept only if confidence is high enough <em>and</em> the nearest-prototype distance is small enough.</p><pre>@torch.no_grad()<br>def predict_openset(model, x, prototypes, conf_thresh=0.55, dist_cutoff=0.57):<br>    logits, emb = model(x)<br>    emb = F.normalize(emb, dim=1)<br>    probs = torch.softmax(logits, dim=1)<br>    conf, pred = probs.max(1)</pre><pre>    cos_sim = emb @ prototypes.T              # (batch, 26)<br>    nearest_sim, _ = cos_sim.max(1)<br>    nearest_dist = 1.0 - nearest_sim          # cosine distance to closest prototype</pre><pre>    known = (conf &gt;= conf_thresh) &amp; (nearest_dist &lt;= dist_cutoff)<br>    # where not known -&gt; reject (&quot;not a recognized heritage site&quot;)<br>    return pred, conf, nearest_dist, known</pre><p>I calibrated dist_cutoff empirically from the distribution of nearest-prototype distances on known test images (min 0.21, mean 0.42, 90th percentile 0.52, max 0.70), choosing 0.57 to retain ~95% of genuine sites.</p><p><strong>Honest limitation:</strong> this reliably rejects inputs dissimilar to everything known (people, objects), but it <em>cannot</em> reliably reject inputs that genuinely resemble the heritage category — an unlisted temple, or an AI-generated heritage image — because those land in the same region of feature space as real sites. Distance-based OOD detection has this inherent ceiling, and I don’t claim otherwise.</p><h3>Deployment: ONNX in the browser</h3><p>For on-device inference I export to ONNX, keeping both outputs (logits and embedding) so the open-set gate can run client-side too:</p><pre>model.eval()<br>dummy = torch.randn(1, 3, 224, 224)<br>torch.onnx.export(<br>    model, dummy, &quot;heritagenet.onnx&quot;,<br>    input_names=[&quot;input&quot;],<br>    output_names=[&quot;logits&quot;, &quot;embedding&quot;],<br>    dynamic_axes={&quot;input&quot;: {0: &quot;batch&quot;}},<br>    opset_version=17,<br>)</pre><p>The exported model is ~6 MB. On the browser side, ONNX Runtime Web runs it via WebAssembly — no server, the image never leaves the device:</p><pre>import * as ort from &quot;onnxruntime-web&quot;;</pre><pre>const session = await ort.InferenceSession.create(&quot;heritagenet.onnx&quot;);<br>// preprocess() must replicate the Python eval transform exactly:<br>// resize 256, center-crop 224, scale to [0,1], normalize with ImageNet mean/std, NCHW<br>const input = preprocess(imageElement);<br>const { logits, embedding } = await session.run({ input });<br>// then apply softmax + the same dual-gate logic client-side</pre><p>Measured latency: <strong>~6.3 ms/image on GPU, ~13.1 ms on CPU.</strong> Since the browser runs on CPU, ~13 ms is the number that matters — comfortably real-time for a point-and-shoot experience.</p><p>The single most common deployment bug I hit wasn’t in the model — it was <strong>preprocessing mismatch</strong> (getting the resize/crop/normalize subtly different in JS than in Python) and <strong>aggressive browser caching</strong> of the .onnx/.js files during iteration. Hard-refresh, cache-busting, and a session.outputNames sanity check ([&#39;logits&#39;,&#39;embedding&#39;]) saved a lot of confusion.</p><h3>What I’d do differently / next</h3><p>The data-limitation finding sets the roadmap: <strong>more data beats more model.</strong> Concretely — expand well past 26 sites, add images to the thin classes, and build a contribution pipeline so others can add sites by dropping in a new folder (the hash-based split makes this painless). Few-shot methods are worth exploring for the long tail. And the open-set gate deserves a stronger method than distance-thresholding for near-distribution unknowns.</p><h3>Takeaways for anyone building a small-data, on-device classifier</h3><ul><li><strong>Transfer learning + full fine-tuning at a low LR</strong> is a strong default when you have little data; freeze only if you’re severely compute- or data-constrained.</li><li><strong>Judge by macro-F1, not accuracy,</strong> the moment your classes are imbalanced.</li><li><strong>Run controlled experiments before assuming you’re overfitting</strong> — you may be data-limited, and that changes what you should work on.</li><li><strong>Softmax confidence is not OOD detection.</strong> If you deploy to the open world, add a feature-space gate — and be honest about what it can’t catch.</li><li><strong>Preprocessing parity between training and deployment</strong> is where on-device projects quietly break. Replicate the eval transform exactly.</li></ul><p>The dataset and the running demo are open. The model scoring 86.6% is nice, but the point was never the number — it was building an open, reproducible foundation for documenting heritage that the rest of us can extend.</p><p><em>The HeritageNet dataset is openly licensed (CC BY 4.0) and the classifier runs in-browser, on-device. Code, dataset, and demo links are in the repository. Contributions — especially photographs of under-represented sites — are welcome.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2e2745d3f4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Can Recognize the Lumbini. It Had Never Seen My Neighborhood Temple.]]></title>
            <link>https://medium.com/@pratikacharya527/ai-can-recognize-the-lumbini-it-had-never-seen-my-neighborhood-temple-cc1187e53507?source=rss-aafa75b21250------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc1187e53507</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Pratik Sharma]]></dc:creator>
            <pubDate>Fri, 28 Aug 2026 17:05:42 GMT</pubDate>
            <atom:updated>2026-08-28T17:05:42.854Z</atom:updated>
            <content:encoded><![CDATA[<p><em>So I spent months teaching one to — and learned that the hardest part of building AI isn’t the AI.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oxN8e94iJD4C1yPwvld2fQ.png" /></figure><p>Ask any modern image-recognition model what the Lumbini is, and it will tell you instantly. The Eiffel Tower, the Colosseum, the Statue of Liberty — no hesitation. These models have seen those places tens of thousands of times, in the vast datasets scraped from the internet that teach machines to see.</p><p>Now show that same model a small temple from a Nepali town — the kind with centuries of carvings, the kind people walk past every morning on their way to work, and it has, in a very literal sense, no idea what it’s looking at. Not because the temple is unimportant. Because, to the digital world, it may as well not exist.</p><p>That gap bothered me more the longer I thought about it. And eventually it turned into a project.</p><h3>The Problem Nobody Was Talking About</h3><p>In April 2015, a magnitude-7.8 earthquake struck Nepal. Alongside the terrible human toll, it damaged or destroyed a large number of the country’s cultural heritage sites — temples, stupas, and monasteries, some of them centuries old.</p><p>In the aftermath, as restoration began, a quieter problem surfaced. For many of the damaged sites, there was no detailed record to rebuild <em>from</em>. No careful documentation of the carvings, the proportions, the specific character of each place. People had loved these sites for generations. But loving something is not the same as recording it.</p><p>This is the first half of the problem: <strong>an enormous amount of local heritage exists with almost no proper digital record.</strong> The famous, protected monuments get attention. The thousands of smaller, local sites — woven into daily life across the country — mostly don’t.</p><p>The second half of the problem is subtler, and it’s the one that pulled me in as someone interested in AI.</p><p>Modern artificial intelligence sees the world through the data it was trained on. And that data overwhelmingly reflects the wealthy, heavily-photographed parts of the world. Landmarks that appear millions of times online become things AI knows intimately. Heritage that was never photographed at scale, never labeled, never uploaded — becomes invisible to these systems entirely.</p><p>So Nepal’s heritage sits in a strange double absence: under-documented in the real world, and almost entirely unseen by the tools we’re increasingly using to understand the world.</p><p>I couldn’t stop thinking about that. If we don’t put these places into the record — a <em>machine-readable</em> record, the kind that can be searched, recognized, built upon — then in a very real sense, to the future, they might not be there at all.</p><h3>Why I Started Building This</h3><p>I didn’t start with a grand plan. I started with a nagging observation: <em>the tools are here, the heritage is here, and nobody has connected the two.</em></p><p>The pieces of modern computer vision — pre-trained models, open frameworks, cheap-enough hardware — are more accessible than they’ve ever been. And yet the specific, local, culturally-rich subject sitting right in front of me had no dataset, no model, no tool.</p><p>So the question became concrete: <strong>Could a small team, with no special resources, actually build a working system to recognize Nepali heritage sites — starting from nothing, not even data?</strong></p><p>That “not even data” part turned out to matter more than I expected. But I’m getting ahead of myself.</p><h3>The Big Idea</h3><p>Here is the whole idea in one sentence: <strong>photograph our heritage, turn those photographs into an open dataset anyone can use, and train an AI that can recognize the sites — then put that AI somewhere ordinary people can actually reach it.</strong></p><p>Notice the order. The dataset comes first. This wasn’t an accident.</p><p>It would have been easy to frame this as “I built an AI that recognizes temples.” But the AI is not the valuable part. Models come and go; a better one will exist next year. The <em>dataset</em> — a genuine, open, carefully-collected record of these specific places — is the thing that lasts, the thing others can build on, the thing that didn’t exist before. The model is really a demonstration: proof that once the heritage is documented, useful things become possible.</p><p>So we photographed the sites ourselves. Not scraped from the internet — photographed, in person, with ordinary cameras. This mattered for a practical reason (we needed the legal right to release the data openly) and a deeper one (a dataset you gather yourself is a dataset you actually understand). We ended up with roughly a thousand images across 26 heritage sites, and we released the whole thing under an open license, free for anyone.</p><p>Then came the model.</p><h3>How the System Works</h3><h3>The data</h3><p>Roughly 1,000+ images, 30 sites, one folder per site. We split them carefully into training, validation, and test sets, keeping each site proportionally represented, so that our final accuracy numbers would mean something honest rather than being accidentally inflated.</p><p>The dataset is also, frankly, <em>imbalanced</em>. Some sites we could photograph extensively; others we managed only a couple of dozen images. One site ended up with barely more than a dozen usable photos after we removed images we didn’t have the rights to share. Hold that detail — it comes back later, and it turned out to be the most important lesson of the whole project.</p><h3>The model, and why it’s small on purpose</h3><p>I used a model architecture called MobileNetV3-Small. The “small” is the point. I didn’t want a giant model that needs a server farm to run — I wanted something that could run on an ordinary phone, in a browser, with no server at all. If the goal is to make heritage recognition <em>accessible</em>, then a tool only researchers can run defeats the purpose.</p><p>I also didn’t train it from scratch. Training a vision model from zero needs millions of images — far beyond what any small team can collect. Instead I used <strong>transfer learning</strong>: I took a model that had already learned general vision from millions of everyday photos, and taught it to specialize in Nepali heritage.</p><p>The analogy I keep coming back to: it’s like hiring a photographer who already understands light and composition, then sending them to study one specific subject deeply. The general skill is already there. You’re just teaching the specialty.</p><h3>The moment it started to see</h3><p>Here’s where it got interesting.</p><p>At first, I kept the pre-trained model mostly “frozen” — I let it use its existing general knowledge and only trained a thin new layer on top. This got me to about 80% accuracy. Respectable. But it kept making a particular kind of mistake: it confused temples that looked alike.</p><p>One temple in particular was a disaster. It looked nearly identical to another site, and the model mixed them up constantly. Measured properly, it was getting that temple right barely a third of the time.</p><p>So I let the model adapt more deeply — “unfreezing” it, allowing it to reshape its own visual understanding around the specific details of Nepali architecture: the carvings, the rooflines, the proportions that a local eye would use to tell two temples apart.</p><p>That one temple went from being recognized correctly about <strong>30% of the time to nearly 90%.</strong></p><p>It’s hard to describe how satisfying that was. The model had stopped guessing based on rough overall shape and started attending to the fine details that actually distinguish one sacred place from another. It had, in a small way, learned to <em>see</em> like someone who grew up there.</p><p>Overall, the finished model correctly identifies the right site about <strong>87% of the time.</strong> And when it’s unsure, the correct answer is in its top three guesses <strong>94% of the time</strong> — meaning even its “mistakes” are usually near-misses.</p><h3>Teaching it to say “I don’t know”</h3><p>There’s a flaw in ordinary classifiers that doesn’t get talked about enough: <strong>they cannot express doubt.</strong></p><p>My model knows 26 heritage sites. But show it a photo of a person, or a car, or a plate of food, and it will still confidently announce which of its 26 temples it thinks that is — because picking one of its known answers is the only thing it knows how to do. It has no concept of “none of these.”</p><p>For a tool meant to be trusted, that’s a real problem. A confident wrong answer is worse than an honest “I’m not sure.”</p><p>So I added a second check. Alongside the model’s confidence, I gave it a way to measure how <em>similar</em> an incoming image is to anything it has actually learned. If a photo looks nothing like any heritage site in its memory, the system now declines to guess — it effectively says, <em>“I don’t recognize this.”</em></p><p>It’s not glamorous. But it’s the part of the project I’m most quietly proud of, because it reflects a choice I made early on: I would rather the system be trustworthy than appear impressive.</p><h3>Putting it where people can reach it</h3><p>Finally, I converted the model into a portable format and built a small web page around it. The whole thing — about 6 megabytes — runs directly in your browser, on your own device. Point your phone’s camera at a temple, and the recognition happens locally, in about the time it takes to blink. Your photo never leaves your device.</p><p>No app to install. No server. No account. Just a link.</p><h3>What Didn’t Work — and the Most Important Thing I Learned</h3><p>I could tell you the project went smoothly. It didn’t, and the ways it didn’t were the most educational part.</p><p>The biggest surprise was this: <strong>for a long time, I assumed my accuracy was limited by how I was tuning the model.</strong> So I did what you’re supposed to do — I experimented. I tried stronger regularization to reduce overfitting. I adjusted the learning rate. I ran controlled comparisons.</p><p>None of it meaningfully helped.</p><p>Increasing the regularization nudged one number up and another down, within the range of random variation. Lowering the learning rate made things slightly worse. No matter what knob I turned, I couldn’t push past a certain ceiling on the metric I cared about most.</p><p>That was frustrating — until I realized it was actually the finding.</p><p>If the model were failing because it was overfitting, then reducing overfitting <em>should</em> have helped. It didn’t. Which meant the bottleneck wasn’t the model at all. <strong>The bottleneck was the data.</strong> My weakest sites were, without exception, the ones with the fewest photographs. The model wasn’t under-tuned. It was under-fed.</p><p>This reframed the entire project for me. I had come in thinking of it as a modeling problem — pick the right architecture, tune it well. I left understanding it as a <em>data</em> problem. And that’s not a minor distinction; it’s the difference between “how do I make my model cleverer” and “how do we, collectively, document more of our heritage.”</p><p>The most important thing I built wasn’t a better model. It was a clearer understanding of what actually matters here.</p><h3>The Results, in Plain Terms</h3><p>Stripped of jargon:</p><ul><li>The system recognizes the correct heritage site about <strong>87 times out of 100</strong>, across 26 sites — trained on a small dataset, on a single consumer laptop.</li><li>When it’s wrong, it’s usually <em>close</em> — the right answer is in its top three guesses <strong>94% of the time.</strong></li><li>It runs <strong>on an ordinary phone, in a browser, offline, in a fraction of a second.</strong></li><li>It can decline to answer when shown something that isn’t a heritage site at all.</li><li>And the dataset behind all of it is <strong>open and free</strong> for anyone to use.</li></ul><p>None of these numbers are world-beating, and I want to be honest about that. This is a proof of concept, not a finished atlas. The dataset is small. The look-alike problem is reduced but not gone. The “I don’t know” feature reliably rejects obvious non-heritage images but can still be fooled by things that genuinely resemble a temple. Twenty-six sites is a beginning, not a national record.</p><p>I’m stating these plainly because a preservation tool that overpromises is worse than one that’s honest about its reach. The value here isn’t in pretending it’s finished. It’s in what it proves is <em>possible</em> — and in the foundation it lays.</p><h3>Why This Matters Beyond One Project</h3><p>It would be easy to file this under “a student ML project about temples.” But I think the underlying issue is bigger, and more general, than Nepal.</p><p>We are building a future increasingly mediated by AI systems — systems that recognize, recommend, translate, and describe the world. And those systems are only as inclusive as the data behind them. Whole cultures, languages, places, and forms of knowledge are quietly under-represented in that data, which means they risk being under-served, or simply unseen, by the tools everyone will soon depend on.</p><p>Nepal’s temples are one instance of a pattern that repeats everywhere the internet’s gaze is thin. The fix isn’t to wait for a big tech company to notice. It’s for people close to the knowledge — the ones who actually understand what matters and why — to document it themselves, openly, so it enters the record on their own terms.</p><p>That’s the part I’d want a reader far from Nepal to take away. <strong>This is a local problem with a global shape:</strong> if the communities who hold cultural knowledge don’t put it into the digital record, it may simply be absent from the future — not through malice, just through omission.</p><h3>What Comes Next</h3><p>The finding that this is a <em>data</em> problem, not a model problem, points clearly at what to do next: grow the record.</p><p>The immediate goal is to expand the dataset far beyond 26 sites, and to add many more photographs of the sites that are currently thin. But I don’t think that should be one team’s job, and here’s the part I’m most excited about: it doesn’t have to be.</p><p>Almost everyone now carries a capable camera in their pocket. The most natural way to build a comprehensive record of a country’s heritage is for the people who live near it to contribute — a photo of the temple down their street, the shrine in their village, the monastery their family has visited for generations. Citizen science, aimed at cultural memory.</p><p>Beyond that: better handling of the rare sites with few images, stronger methods for the “I don’t know” problem, and — because the dataset and model are open — whatever other people decide to build that I haven’t thought of. That last possibility is the whole point of releasing it openly.</p><h3>Final Thought</h3><p>When I started, I thought I was building a model that could recognize temples. What I actually built, I now realize, was the beginning of a memory.</p><p>The 2015 earthquake taught Nepal a hard lesson about how quickly the physical world can disappear, and how much is lost when it disappears unrecorded. I can’t do anything about earthquakes. But I can help make sure that the next time something is lost, there’s a record left behind — an open, growing, machine-readable record that anyone can add to and no disaster can erase.</p><p>The temples will not last forever. Nothing does. But the memory of them can — if we choose to write it down. This was my attempt to start.</p><p>And the most valuable thing about it isn’t the 87% on a test set. It’s that the record now exists, it’s open, and it’s waiting for the rest of us to help it grow.</p><p>GitHub Link : <a href="https://github.com/Pratik-sharma-py/HeritageNet_Main">https://github.com/Pratik-sharma-py/HeritageNet_Main</a></p><p>Try it : <a href="https://pratik-sharma-py.github.io/HeritageNet_Main/webapp/">https://pratik-sharma-py.github.io/HeritageNet_Main/webapp/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc1187e53507" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What 1,338 Strangers Taught Me About Predicting Medical Bills with Linear Regression]]></title>
            <link>https://medium.com/@pratikacharya527/what-1-338-strangers-taught-me-about-predicting-medical-bills-with-linear-regression-a13bf2d94992?source=rss-aafa75b21250------2</link>
            <guid isPermaLink="false">https://medium.com/p/a13bf2d94992</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Pratik Sharma]]></dc:creator>
            <pubDate>Thu, 04 Jun 2026 20:11:35 GMT</pubDate>
            <atom:updated>2026-06-04T20:11:35.923Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A beginner-friendly walk through my first end-to-end machine learning project — from a confusing scatter plot to a live web app.</em></p><h3>Starting with a simple question</h3><p>Imagine you work at an insurance company, and someone walks in. You know six things about them: their age, their sex, their BMI, how many children they have, whether they smoke, and which region they live in. The question your boss asks is brutally simple:</p><p><strong>How much will this person cost us this year?</strong></p><p>You could guess. Or you could find a few thousand past customers, look at what <em>they</em> actually cost, and learn the pattern. That second approach — turning history into a prediction — is exactly what machine learning is for. And the simplest, most honest tool for a question like this is <strong>linear regression</strong>.</p><p>This is the story of how I built that predictor, the one plot that changed how I thought about the whole problem, and the mistakes that taught me more than the successes.</p><h3>First, what is linear regression?</h3><p>Forget the intimidating name. Linear regression is just the idea of drawing the best possible straight line through a cloud of points.</p><p>Picture a scatter plot: years of experience on the x-axis, salary on the y-axis. The dots trend upward — more experience, more pay. Linear regression finds the single straight line that sits as close as possible to all those dots at once. Once you have that line, you can take any new x (say, 7 years of experience) and read off the predicted y (the salary).</p><p>The line has a formula you already know from school:</p><pre>y = m·x + b</pre><p>m is the slope (how steeply salary rises per year), b is where the line starts. Machine learning just extends this to many inputs at once:</p><pre>charges = w1·age + w2·bmi + w3·smoker + ... + b</pre><p>Each input gets its own weight (w). A <strong>big</strong> weight means that input strongly pushes the prediction up or down; a weight near zero means it barely matters.</p><p><strong>So how does the model “learn” those weights?</strong> It starts with random ones, makes predictions, and measures how wrong it is — the gap between predicted and actual values, squared so big mistakes hurt more. Then it nudges the weights slightly to shrink that error, and repeats, thousands of times, until the line fits as well as it can. That error-shrinking process is the “learning” in machine learning.</p><p>The beauty of linear regression is that it’s <em>interpretable</em>. After training, you can read the weights and understand exactly <strong>why</strong> it predicts what it does. No black box. That turned out to matter a lot for my project.</p><h3>The dataset: 1,338 strangers</h3><p>I worked with a well-known public insurance dataset: 1,338 anonymized records, each with those six inputs and one target — charges, the actual annual medical cost in dollars.</p><p>My first instinct, like every beginner, was to throw the data at a model immediately. I’m glad I didn’t. Because the most important lesson I’d learned from a previous project was this:</p><blockquote><strong><em>Look at your target first. Always.</em></strong></blockquote><p>The target is the thing you’re trying to predict. If you don’t understand <em>it</em> before you start, you’re flying blind.</p><h3>The target told me to take its logarithm</h3><p>So I plotted the distribution of charges before anything else. It was badly <strong>right-skewed</strong> — most people had modest bills clustered on the left, with a long tail of a few very expensive patients stretching far to the right. The skew measured 1.52 (a perfectly symmetric distribution is 0).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ckvjoKtWxj8uEV4YDhIPXw.png" /></figure><p>This matters for linear regression specifically. The model assumes its <em>errors</em> are roughly balanced and symmetric. A heavily skewed target tends to produce skewed errors, which makes the line fit poorly — especially at the expensive end.</p><p>The fix is an old trick: model the <strong>logarithm</strong> of charges instead of the raw dollars. The log compresses that long tail. When I re-plotted log(charges), the skew dropped to <strong>−0.09</strong> — almost perfectly symmetric. So I decided to train on the log, and convert predictions back to real dollars at the very end.</p><p>I would never have known to do this if I’d skipped looking at the target.</p><h3>The plot that changed everything</h3><p>Next I explored how each input related to the cost. Most were unremarkable — until I made one scatter plot: <strong>BMI on the x-axis, charges on the y-axis, with the dots colored by whether the person smokes.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IKGTYbFwlp8gy_t_Ltxo6w.png" /></figure><p>It was startling. The smokers and non-smokers didn’t blend together — they formed two completely separate bands. Non-smokers stayed in a low, tight cloud near the bottom regardless of BMI. Smokers sat in an entirely different, much higher zone. And within the smokers, higher BMI pushed charges even higher.</p><p>A correlation check confirmed it numerically. Smoking status correlated with charges at <strong>0.79</strong> — overwhelmingly the strongest signal in the entire dataset. Age was a distant second at 0.30. Sex and region were nearly worthless.</p><p>But the real insight was that <em>combination</em>. Look at the average charges I found:</p><p>Group Average annual charge Non-smoker, not obese ~$7,977 Non-smoker, obese ~$8,843 Smoker, not obese ~$21,363 <strong>Smoker, obese</strong> <strong>~$41,558</strong></p><p>Read that carefully. Being obese costs a non-smoker about $900 extra. But being obese costs a <em>smoker</em> about $20,000 extra. Obesity and smoking don’t just add up — they <strong>multiply</strong> each other.</p><h3>Teaching the model something it couldn’t see</h3><p>Here’s a limitation people don’t tell you about plain linear regression: <strong>it can’t discover interactions on its own.</strong> It treats every input separately and just adds up their effects. It has no way to learn “obesity is cheap <em>unless</em> you also smoke.” It can only say “obesity costs X” and “smoking costs Y,” full stop.</p><p>So I gave it a hand. I engineered a new feature — a simple flag that is 1 when a person both smokes <em>and</em> has a BMI of 30 or higher, and 0 otherwise:</p><pre>data[&#39;is_smoker_and_obese&#39;] = (<br>    (data[&#39;bmi&#39;] &gt;= 30) &amp; (data[&#39;smoker&#39;] == &#39;yes&#39;)<br>).astype(int)</pre><p>This single column hands the model the interaction it couldn’t find by itself. This is the heart of <strong>feature engineering</strong> — using your human understanding of the data to create inputs that make the model’s job possible.</p><h3>Building it the right way (and avoiding a silent trap)</h3><p>With the insights in hand, I built the model. The structure matters as much as the model itself, because of one sneaky danger: <strong>data leakage.</strong></p><p>Leakage is when information from your test data accidentally bleeds into training, giving you a falsely optimistic score that collapses in the real world. The classic way it happens: you scale or encode your data using statistics from the <em>entire</em> dataset before splitting. The cure is strict discipline — <strong>split first, then learn everything from the training set only.</strong></p><p>I used scikit-learn’s Pipeline and ColumnTransformer to enforce this automatically. The pipeline bundles every step — one-hot encoding the categories, scaling the numbers, and the regression itself — into one object that learns exclusively from training data:</p><pre>preprocess = ColumnTransformer([<br>    (&#39;cat&#39;, OneHotEncoder(drop=&#39;first&#39;), categorical),<br>    (&#39;num&#39;, StandardScaler(), numeric)<br>])</pre><pre>model = Pipeline([<br>    (&#39;prep&#39;, preprocess),<br>    (&#39;lr&#39;, LinearRegression())<br>])</pre><pre>model.fit(X_train, y_train)</pre><p>Before trusting any score, I also built a <strong>dummy baseline</strong> — a model that ignores all inputs and just predicts the average charge every time. If my real model couldn’t beat that, something was broken. The baseline scored an R² of essentially zero, exactly as a no-information model should. Now I had a floor to beat.</p><h3>The results — and what they confessed</h3><p>The trained model explained about <strong>84% of the variance</strong> in log-charges (R² = 0.844). In plain-dollar terms, the typical error (MAE) was around <strong>$3,800</strong>.</p><p>But one number told a deeper story. The RMSE — another error measure that punishes big mistakes harshly — was <strong>$7,740</strong>, roughly double the MAE. When RMSE towers over MAE like that, it’s confessing something: <em>most</em> predictions are good, but a <em>handful</em> are badly wrong. In this case, the model nailed the cheap majority and missed big on the most expensive patients — the high-cost smokers in that long tail. A real, honest limitation, surfaced by simply comparing two numbers.</p><p>Then came my favorite part. Because linear regression is interpretable, I pulled out the learned weights:</p><p>Feature Weight smoker (yes) <strong>1.21</strong> age 0.48 is_smoker_and_obese 0.20 children 0.12 bmi 0.02</p><p>The model agreed with my EDA completely. Smoking dominated. And look at bmi: on its own, its weight was a tiny 0.02 — almost nothing. But the interaction feature I engineered carried a weight of 0.20. The model was telling me, in its own language, that BMI barely matters <em>except</em> when combined with smoking. That one plot from the beginning had been right all along, and the math confirmed it.</p><h3>From notebook to something people can actually use</h3><p>A model trapped in a notebook helps no one. So I deployed it as a web app using <strong>Streamlit</strong>, where anyone can type in the six details and get an instant estimate.</p><p>The key to keeping the app simple was that I’d saved the <em>entire pipeline</em>, not just the model. All the encoding and scaling travel with it, so the app just feeds in raw inputs.</p><p>But there was one final trap, and it’s a classic real-world bug called <strong>training/serving skew.</strong> My model was trained on <em>seven</em> columns — the six raw inputs plus my engineered is_smoker_and_obese flag. But a user only types in <em>six</em>. If the app forgot to recreate that seventh feature, the model would receive data shaped differently from what it learned on, and quietly produce nonsense. So the app rebuilds the engineered feature from the user&#39;s input before predicting — making sure the data at serving time matches the data at training time, exactly.</p><h3>What I actually learned</h3><p>The model was never the hard part. Linear regression is a few lines of code. The real work — and the real learning — was everywhere else:</p><ul><li><strong>Understand the target before you model anything.</strong> The skew in charges dictated my entire approach.</li><li><strong>One good visualization beats ten metrics.</strong> The smoker scatter plot defined the whole project.</li><li><strong>Models can’t see interactions — you have to give them eyes.</strong> Feature engineering turned a limitation into the model’s strongest signal.</li><li><strong>Build to prevent leakage from the start,</strong> not as an afterthought.</li><li><strong>Let the model confess its weaknesses.</strong> The RMSE-vs-MAE gap was more useful than the headline accuracy.</li><li><strong>Shipping reveals problems modeling never does,</strong> like the serving-skew bug.</li></ul><p>I started this wanting to predict insurance charges. I finished understanding that machine learning is less about the algorithm and more about the conversation you have with your data along the way.</p><p>GITHUB LINK : https://github.com/Pratik-sharma-py/datascience_project/tree/main/Linear%20Regression</p><p><em>This was a learning project built on a small public dataset — a demonstration of the workflow, not a real medical or insurance tool.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a13bf2d94992" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sentiment Analysis with a Decision Tree: A Complete, Honest Walkthrough]]></title>
            <link>https://medium.com/@pratikacharya527/sentiment-analysis-with-a-decision-tree-a-complete-honest-walkthrough-c6fae77c6d13?source=rss-aafa75b21250------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6fae77c6d13</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Pratik Sharma]]></dc:creator>
            <pubDate>Sun, 31 May 2026 06:32:38 GMT</pubDate>
            <atom:updated>2026-06-02T13:26:51.270Z</atom:updated>
            <content:encoded><![CDATA[<p>From raw Amazon reviews to a deployed Flask app — with every mistake, metric, and line of code explained</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jRh46J62VWsMKykElCwCvQ.png" /></figure><p>When I started this project, the goal sounded simple: read an Amazon review and decide whether it’s <strong>positive</strong> or <strong>negative</strong>. I chose a <strong>Decision Tree</strong> because it’s one of the most intuitive models in machine learning — you can literally read the rules it learns.</p><p>What surprised me was that the model was the easy part. The real work was everything around it: cleaning the text, catching a silent data leak, handling a badly imbalanced dataset, and learning why my model’s “confidence” couldn’t be trusted. This post walks through the whole notebook, with the exact code I wrote.</p><h3>What Is a Decision Tree, and How Does It Work?</h3><p>A decision tree makes predictions by asking a series of yes/no questions. For text classification, the questions look like:</p><ul><li><em>Does the review contain the word “good”?</em> → if yes go one way, if no go the other.</li><li><em>Does it contain “worst”?</em> → split again.</li><li>…and so on, until a group is pure enough to make a decision.</li></ul><p>Each question is a <strong>node</strong>, each path is a <strong>branch</strong>, and the final boxes where the tree stops and makes a call are <strong>leaves</strong>.</p><p>How does it choose which question to ask first? It picks the split that best separates the two classes, measured by <strong>Gini impurity</strong> or <strong>entropy</strong> — both answer “after this split, how mixed are the resulting groups?” The tree greedily chooses the most informative question, then repeats on each branch.</p><p><strong>Why I picked it:</strong> This project is part of our learning journey and to implement learning into real project of Decision-Tree I initiated this project with my team-mate <a href="https://medium.com/u/d4e278a6b52e">Bishal Gurung</a>.</p><h3>Loading the Data</h3><p>The dataset is roughly 5,000 Amazon India reviews of the Apple iPhone XR — each row has the review text, a title, a star rating, a “found helpful” count, and some metadata.</p><pre>import numpy as np<br>import pandas as pd<br>import nltk<br>from nltk.corpus import stopwords<br>from nltk.stem import PorterStemmer<br>import seaborn as sns<br>import matplotlib.pyplot as plt<br>import re<br>import scipy.sparse as sp<br>from sklearn.model_selection import train_test_split, cross_val_score<br>from sklearn.feature_extraction.text import TfidfVectorizer<br>from sklearn.tree import DecisionTreeClassifier<br>from sklearn.dummy import DummyClassifier<br>from sklearn.metrics import classification_report, f1_score</pre><pre>df = pd.read_csv(&#39;apple_iphone_11_reviews.csv&#39;)<br>raw_data = df.copy()</pre><p>I always work on a .copy() so the original stays untouched if I need to start over.</p><h3>Part 3: Cleaning and Exploring the Data</h3><h3>Dropping nulls and duplicates</h3><p>First I removed empty rows — and crucially, <strong>duplicate reviews</strong>:</p><pre>raw_data = raw_data.dropna()<br>raw_data = raw_data.drop_duplicates(subset=[&#39;review_text&#39;])   # FIX: stop leakage</pre><p>That second line matters more than it looks. There were hundreds of repeated review texts in this dataset. If duplicates land in both the training and test sets, the model effectively “studies the answers before the exam” — it sees the same review twice and your reported score becomes a lie. This is <strong>data leakage</strong>, and removing duplicates <em>before</em> the split is the fix.</p><h3>Looking at the columns</h3><p>I checked the categorical columns and found several were constant — every review is the same product, same country:</p><pre>raw_data[&#39;product&#39;].value_counts()<br>raw_data[&#39;review_country&#39;].value_counts()</pre><p>Constant columns carry zero predictive information, so they get ignored as features.</p><h3>Date features</h3><pre>raw_data[&#39;reviewed_date&#39;] = pd.to_datetime(raw_data[&#39;reviewed_at&#39;])<br>raw_data[&#39;reviewed_year&#39;] = raw_data[&#39;reviewed_date&#39;].dt.year<br>raw_data[&#39;reviewed_year&#39;].value_counts().plot(kind=&#39;pie&#39;)</pre><pre>raw_data[&#39;reviewed_month&#39;] = raw_data[&#39;reviewed_date&#39;].dt.month<br>raw_data[&#39;reviewed_month&#39;].value_counts().plot(kind=&#39;bar&#39;)</pre><h3>The single most important finding: the data is imbalanced</h3><pre>raw_data[&#39;rating&#39;] = raw_data[&#39;review_rating&#39;].str.extract(r&#39;(\d+\.\d+|\d+)&#39;).astype(float)<br>raw_data[&#39;rating&#39;].value_counts().plot(kind=&#39;bar&#39;)</pre><p>About <strong>73% of reviews are 5-star</strong> and only ~2% are 2-star. This one fact shaped everything: a lazy model could score ~73% accuracy by guessing “positive” every time and learning nothing. So <strong>accuracy is a misleading metric here</strong> — I’d need to watch macro-F1 and per-class recall instead.</p><h3>Building the Target — From Stars to Sentiment</h3><p>I mapped star ratings to sentiment. My first instinct was three classes (positive / neutral / negative), but the neutral 3-star group was tiny and impossible to predict reliably — so I dropped it and made it a clean binary problem.</p><pre>def sentiment_label(value):<br>    if value &lt;= 2:<br>        return &quot;negative&quot;<br>    elif value == 3:<br>        return &quot;neutral&quot;<br>    else:<br>        return &#39;positive&#39;</pre><pre>raw_data[&#39;sentiment&#39;] = raw_data[&#39;rating&#39;].apply(sentiment_label)</pre><pre>raw_data = raw_data[raw_data[&#39;rating&#39;] != 3].copy()   # FIX: neutral can&#39;t be predicted</pre><h3>Cleaning the Text</h3><p>Raw review text is messy — capitals, punctuation, HTML, filler words. One function standardizes everything: lowercase it, strip non-letters, remove stopwords, and <strong>stem</strong> each word (so “amazing” and “amazingly” both collapse to “amaz”).</p><pre>nltk.download(&#39;stopwords&#39;)<br>stop_words = set(stopwords.words(&quot;english&quot;))<br>stemmer = PorterStemmer()</pre><pre>def clean_text(text):<br>    text = text.lower()                          # lowercase<br>    text = re.sub(r&quot;&lt;.*?&gt;&quot;, &quot; &quot;, text)           # remove HTML tags<br>    text = re.sub(r&quot;[^a-zA-Z]&quot;, &quot; &quot;, text)       # keep only letters<br>    words = text.split()                         # tokenize<br>    words = [stemmer.stem(w) for w in words if w not in stop_words]<br>    return &quot; &quot;.join(words)</pre><pre>raw_data[&#39;clean_text&#39;]  = raw_data[&#39;review_text&#39;].apply(clean_text)<br>raw_data[&#39;clean_title&#39;] = raw_data[&#39;review_title&#39;].apply(clean_text)</pre><h3>Extra features: length and helpfulness</h3><p>During EDA I found that negative reviews tend to be <strong>longer</strong>, and lower-rated reviews get <strong>more “helpful” votes</strong> (people upvote complaints). Both became numeric features.</p><pre>raw_data[&#39;word_count&#39;] = raw_data[&#39;review_text&#39;].astype(str).str.split().str.len()</pre><pre>def parse_help(s):<br>    m = re.search(r&#39;([\d,]+)&#39;, str(s))<br>    return int(m.group(1).replace(&#39;,&#39;, &#39;&#39;)) if m else 0<br>raw_data[&#39;helpful_num&#39;] = raw_data[&#39;helpful_count&#39;].apply(parse_help)</pre><h3>Word clouds — a preview of what the model will learn</h3><pre>from wordcloud import WordCloud, STOPWORDS</pre><pre>def cloud_text(df_subset):<br>    txt = &quot; &quot;.join(df_subset[&#39;clean_text&#39;].dropna().astype(str)).lower()<br>    txt = re.sub(r&#39;[^\w\s]&#39;, &#39;&#39;, txt)<br>    return &quot; &quot;.join(w for w in txt.split() if w not in STOPWORDS)</pre><pre>pos = cloud_text(raw_data[raw_data[&#39;rating&#39;] &gt;= 4])<br>neg = cloud_text(raw_data[raw_data[&#39;rating&#39;] &lt;= 2])</pre><p>Splitting the clouds into positive and negative shows exactly the kind of words — “good”, “best”, “camera” vs “bad”, “waste”, “problem” — that the tree later splits on.</p><h3>Part 6: Training the Model — Doing It Honestly</h3><p>This is where the care pays off. The golden rule of text modeling: <strong>split first, then fit the vectorizer on the training text only.</strong> If the vectorizer sees the test text, information leaks across the boundary.</p><pre>raw_data[&quot;text&quot;] = (raw_data[&quot;clean_title&quot;].fillna(&quot;&quot;) + &quot; &quot; +<br>                    raw_data[&quot;clean_text&quot;].fillna(&quot;&quot;))</pre><pre># split ROWS first<br>train, test = train_test_split(raw_data, test_size=0.2, random_state=42,<br>                               stratify=raw_data[&#39;sentiment&#39;])</pre><pre># TF-IDF on text only — FIT on train, TRANSFORM on test<br>vectorizer = TfidfVectorizer(max_features=3000, ngram_range=(1, 2), min_df=3)<br>Xtr_text = vectorizer.fit_transform(train[&#39;text&#39;])<br>Xte_text = vectorizer.transform(test[&#39;text&#39;])</pre><p>TF-IDF turns words into numbers, weighting words that are distinctive to a review over words that appear everywhere. The ngram_range=(1, 2) means it also captures two-word phrases like &quot;not good&quot; — important, because &quot;good&quot; and &quot;not good&quot; mean opposite things.</p><p>Then I glue the numeric features onto the text features:</p><pre>num = [&#39;helpful_num&#39;, &#39;word_count&#39;]<br>X_train = sp.hstack([Xtr_text, train[num].values])<br>X_test  = sp.hstack([Xte_text, test[num].values])<br>y_train, y_test = train[&#39;sentiment&#39;], test[&#39;sentiment&#39;]</pre><h3>Always compare to a baseline</h3><p>Before trusting any score, I check what blind guessing gets:</p><pre>dummy = DummyClassifier(strategy=&#39;most_frequent&#39;).fit(X_train, y_train)<br>print(&quot;Dummy baseline macro-F1:&quot;, round(f1_score(y_test, dummy.predict(X_test), average=&#39;macro&#39;), 3))<br># -&gt; 0.474</pre><h3>The model</h3><pre>dt_model = DecisionTreeClassifier(max_depth=12, min_samples_leaf=5,<br>                                  class_weight=&#39;balanced&#39;, random_state=42)</pre><pre>print(&quot;CV macro-F1:&quot;, round(cross_val_score(dt_model, X_train, y_train, cv=5, scoring=&#39;f1_macro&#39;).mean(), 3))<br>dt_model.fit(X_train, y_train)<br>print(classification_report(y_test, dt_model.predict(X_test), zero_division=0))</pre><p>Two settings do the heavy lifting: class_weight=&#39;balanced&#39; makes the rare negative class count more (without it, the tree just predicts &quot;positive&quot; for everything), and max_depth=12 stops the tree overfitting by memorizing the training set.</p><h3>The honest results</h3><pre>precision    recall  f1-score   support<br>    negative       0.28      0.80      0.41        79<br>    positive       0.97      0.77      0.86       710<br>    accuracy                           0.77       789<br>   macro avg       0.62      0.78      0.63       789</pre><p>Metric Value What it means Dummy baseline (macro-F1) 0.47 What blind guessing gets My tree (cross-validated macro-F1) 0.68 A real improvement over guessing Negative-class recall 0.80 Catches 4 out of 5 complaints</p><p>Notice the trap: overall <strong>accuracy is 0.77</strong> — but the dummy that always guesses “positive” gets nearly that too. The number that <em>proves</em> the model learned something is <strong>macro-F1 of 0.68 against a 0.47 baseline</strong>, plus the 0.80 recall on negatives. A real product team scanning for complaints would care most about that last number, and the model catches 4 out of 5.</p><h3>Predicting on New Reviews</h3><p>A model is useless if you can’t feed it new text. The key rule: apply the <strong>same</strong> cleaning and use vectorizer.transform() (never re-fit) so a new review maps onto the exact vocabulary the model was trained on.</p><pre>def predict_review(review, helpful_votes=0):<br>    &quot;&quot;&quot;Predict sentiment of any new review using the trained model.&quot;&quot;&quot;<br>    text = clean_text(review)                       # same cleaning as training<br>    word_count = len(str(review).split())           # same numeric feature<br>    tfidf = vectorizer.transform([text])            # use the FITTED vectorizer (don&#39;t refit!)<br>    X = sp.hstack([tfidf, [[helpful_votes, word_count]]])<br>    label = dt_model.predict(X)[0]<br>    confidence = round(max(dt_model.predict_proba(X)[0]), 2)<br>    return label, confidence</pre><pre>reviews = [<br>    &quot;This phone is amazing, best purchase ever, camera is fantastic!&quot;,<br>    &quot;Terrible product. Battery died in a week and screen cracked.&quot;,<br>    &quot;worst phone, totally disappointed, do not buy&quot;,<br>]<br>for r in reviews:<br>    label, conf = predict_review(r)<br>    print(f&quot;[{label} | {conf}]  {r}&quot;)</pre><p>The clearly positive review comes back positive, the angry ones come back negative.</p><h3>The most interesting thing I learned: “100% confidence” is a lie</h3><p>When I fed it a <strong>mixed</strong> review — <em>“the camera is fantastic but the battery is terrible”</em> — it reported <strong>100% positive</strong>. That felt wrong, and digging into why taught me the most.</p><p>A single decision tree’s “confidence” is just the makeup of the leaf the review lands in. Because the tree grows until most leaves are <em>pure</em> (all one class), it almost always reports 100% — not because it’s truly certain, but because of how leaves are structured. <strong>For a single tree, the confidence number is essentially meaningless.</strong> A Random Forest (hundreds of trees voting) would smooth this into a believable probability like 88%. I kept the single tree here because the project is about <em>understanding</em> trees — but knowing why the confidence is unreliable is the real takeaway.</p><h3>Part 8: Saving the Model for a Web App</h3><p>To use the model outside the notebook — in a small Flask website — I saved the trained vectorizer and model together in one file:</p><pre>import joblib<br>joblib.dump({&quot;vectorizer&quot;: vectorizer, &quot;model&quot;: dt_model}, &quot;sentiment_model.joblib&quot;)<br>print(&quot;saved!&quot;)</pre><p>A separate app.py loads this file and serves a page where you paste a review and get a green (positive) or red (negative) result. The important idea: <strong>you train once and save; the app just loads and predicts.</strong> That separation is how real ML products work.</p><h3>Limitations (Being Honest)</h3><ul><li><strong>Binary only.</strong> Neutral 3-star reviews were dropped, so the model is forced to label everything positive or negative.</li><li><strong>Narrow domain.</strong> Trained on iPhone XR / Amazon India reviews from 2018–2020; it won’t generalize well to other products or newer slang.</li><li><strong>Single tree.</strong> Great for interpretability, weaker than a forest — and, as we saw, useless for confidence scores.</li></ul><h3>What I Took Away</h3><p>The lessons that stuck were the ones tutorials skip:</p><ol><li><strong>Accuracy can lie.</strong> On imbalanced data, always compare to a baseline and read per-class metrics.</li><li><strong>Data leakage is silent.</strong> Duplicates crossing the train/test boundary inflate your score with no error message.</li><li><strong>Understand your metrics.</strong> A “100% confident” prediction taught me more about how decision trees work than any lecture.</li><li><strong>Ship it.</strong> Saving the model and wrapping it in an app forced me to understand the difference between training once and predicting many times.</li></ol><p><em>Thanks for reading. The full code is on my GitHub. I’d love to hear how you’d approach any part of this differently</em></p><p><strong>Git-Hub Link </strong>: <a href="https://github.com/Pratik-sharma-py/datascience_project/tree/main/Decision%20Tree">https://github.com/Pratik-sharma-py/datascience_project/tree/main/Decision%20Tree</a></p><p>It was really great learning while working with <a href="https://medium.com/u/d4e278a6b52e">Bishal Gurung</a> on this project, we brain-storm the idea together and implement what we have learn throughout the journey and there is always room to improvement and we will keep working on improving our project in the future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6fae77c6d13" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Email Spam Detector with Naive Bayes: A Complete Guide]]></title>
            <link>https://medium.com/@pratikacharya527/building-an-email-spam-detector-with-naive-bayes-a-complete-guide-200f36190a58?source=rss-aafa75b21250------2</link>
            <guid isPermaLink="false">https://medium.com/p/200f36190a58</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Pratik Sharma]]></dc:creator>
            <pubDate>Tue, 26 May 2026 11:23:42 GMT</pubDate>
            <atom:updated>2026-05-26T11:27:58.013Z</atom:updated>
            <content:encoded><![CDATA[<p>Complete Guide to Naive Bayes: Text Classification and Spam Detection</p><h3>Introduction</h3><p>In today’s digital world, email spam is a persistent problem. Every day, millions of spam emails flood inboxes, wasting time and potentially posing security risks. But what if we could automatically detect and filter these unwanted emails using machine learning?</p><p>In this comprehensive guide, I’ll walk you through how I built a <strong>98% accurate email spam detector</strong> using the <strong>Naive Bayes algorithm</strong>, TF-IDF vectorization, and Flask. Whether you’re new to machine learning or looking to deepen your understanding of Naive Bayes, this article covers everything from theory to implementation.</p><h3>Table of Contents</h3><ol><li>What is Naive Bayes?</li><li>Why Naive Bayes for Spam Detection?</li><li>How Naive Bayes Works</li><li>The Dataset</li><li>Data Preprocessing &amp; Feature Engineering</li><li>Building the Model</li><li>Results &amp; Performance</li><li>Building a Web App</li></ol><h4><strong>1. What is Naive Bayes?</strong></h4><p>Naive Bayes is a probabilistic machine learning algorithm based on <strong>Bayes’ Theorem</strong>, a fundamental principle in probability theory. It’s called “naive” because it makes a strong simplifying assumption: all features are <strong>conditionally independent</strong> given the class label.</p><h3>Bayes’ Theorem Formula</h3><pre>P(Class|Features) = P(Features|Class) × P(Class) / P(Features)</pre><p>Where:</p><ul><li><strong>P(Class|Features)</strong>: Probability of the class given the features (what we want to find)</li><li><strong>P(Features|Class)</strong>: Probability of features given the class</li><li><strong>P(Class)</strong>: Prior probability of the class</li><li><strong>P(Features)</strong>: Total probability of the features</li></ul><h3>Why “Naive”?</h3><p>The naive assumption is that all features are independent. In reality, this is often not true (words in emails are not independent), but surprisingly, <strong>Naive Bayes works very well in practice</strong>, especially for text classification!</p><h3>Types of Naive Bayes</h3><ol><li><strong>Multinomial Naive Bayes</strong>: Best for text classification (what we used!)</li><li><strong>Gaussian Naive Bayes</strong>: For continuous data with normal distribution</li><li><strong>Bernoulli Naive Bayes</strong>: For binary features</li></ol><h3>2. Why Naive Bayes for Spam Detection?</h3><h3>Advantages of Naive Bayes:</h3><p><strong>Fast</strong>: Trains quickly even on large datasets</p><p><strong>Efficient</strong>: Makes predictions in real-time</p><p><strong>Scalable</strong>: Works well with high-dimensional text data</p><p><strong>Interpretable</strong>: Easy to understand why a prediction was made</p><p><strong>Works Well with Imbalanced Data</strong>: When used with proper weighting</p><p><strong>Requires Less Training Data</strong>: Compared to deep learning models</p><p><strong>Proven</strong>: Widely used in production spam filters (Gmail, Outlook, etc.)</p><h3>Real-World Applications:</h3><ul><li><strong>Email Spam Filtering</strong>: Gmail, Outlook, Yahoo Mail</li><li><strong>Sentiment Analysis</strong>: Twitter, Facebook sentiment detection</li><li><strong>Document Classification</strong>: News categorization</li><li><strong>Medical Diagnosis</strong>: Disease probability assessment</li></ul><h3>3. How Naive Bayes Works: Step-by-Step</h3><h3>Example: Is this email spam?</h3><p>Email text: “<strong>Free money click here now!</strong>”</p><h4>Step 1: Extract Words (Tokenization)</h4><pre>Words: [&quot;free&quot;, &quot;money&quot;, &quot;click&quot;, &quot;here&quot;, &quot;now&quot;]</pre><h4>Step 2: Calculate Probability of Each Word in SPAM Class</h4><pre>P(&quot;free&quot;|SPAM) = 0.45    (45% of spam emails contain &quot;free&quot;)<br>P(&quot;money&quot;|SPAM) = 0.38   (38% of spam emails contain &quot;money&quot;)<br>P(&quot;click&quot;|SPAM) = 0.52   (52% of spam emails contain &quot;click&quot;)<br>P(&quot;here&quot;|SPAM) = 0.41    (41% of spam emails contain &quot;here&quot;)<br>P(&quot;now&quot;|SPAM) = 0.35     (35% of spam emails contain &quot;now&quot;)</pre><h4>Step 3: Calculate Probability of Each Word in HAM Class</h4><pre>P(&quot;free&quot;|HAM) = 0.02     (2% of legitimate emails contain &quot;free&quot;)<br>P(&quot;money&quot;|HAM) = 0.05    (5% of legitimate emails contain &quot;money&quot;)<br>P(&quot;click&quot;|HAM) = 0.08    (8% of legitimate emails contain &quot;click&quot;)<br>P(&quot;here&quot;|HAM) = 0.15     (15% of legitimate emails contain &quot;here&quot;)<br>P(&quot;now&quot;|HAM) = 0.12      (12% of legitimate emails contain &quot;now&quot;)</pre><h4>Step 4: Apply Bayes’ Theorem</h4><pre>P(SPAM|Email) = P(Email|SPAM) × P(SPAM) / P(Email)<br>P(SPAM|Email) = (0.45 × 0.38 × 0.52 × 0.41 × 0.35) × 0.147 / P(Email)<br>P(SPAM|Email) ≈ 0.92  (92% probability it&#39;s spam)</pre><pre>P(HAM|Email) = P(Email|HAM) × P(HAM) / P(Email)<br>P(HAM|Email) ≈ 0.08   (8% probability it&#39;s legitimate)</pre><h4>Step 5: Make Decision</h4><pre>Since P(SPAM|Email) &gt; P(HAM|Email)<br>→ Email is classified as SPAM </pre><h3>4. The Dataset</h3><h3>Dataset Overview:</h3><ul><li><strong>Source</strong>: UCI Machine Learning Repository</li><li><strong>Total Emails</strong>: 5,572</li><li><strong>Training Set</strong>: 4,457 emails (80%)</li><li><strong>Test Set</strong>: 1,115 emails (20%)</li></ul><h3>Class Distribution:</h3><pre>Spam:      747 emails (13.4%)<br>Legitimate: 4,825 emails (86.6%)</pre><p><strong>Important Note</strong>: The dataset is imbalanced. We have many more legitimate emails than spam. This is realistic but requires special handling (which we did with class weights).</p><h3>Sample Data:</h3><p><strong>Spam Email Example:</strong></p><pre>&quot;Free entry in 2 a wkly comp to win fa cup final tickets&quot;</pre><p><strong>Legitimate Email Example:</strong></p><pre>&quot;Hi, I wanted to check if we can meet tomorrow at 2pm for the project discussion&quot;</pre><h3>5. Data Preprocessing &amp; Feature Engineering</h3><h3>Step 1: Data Cleaning</h3><pre>import re<br>import pandas as pd<br>from nltk.stem.porter import PorterStemmer</pre><pre>ps = PorterStemmer()</pre><pre>def clean_and_stem(text):<br>    # Convert to lowercase<br>    text = str(text).lower()<br>    <br>    # Remove extra spaces<br>    text = re.sub(r&quot;\s+&quot;, &quot; &quot;, text).strip()<br>    <br>    # Tokenize into words<br>    words = text.split()<br>    <br>    # Stem each word (reduce to root form)<br>    stemmed = [ps.stem(word) for word in words]<br>    <br>    return &quot; &quot;.join(stemmed)</pre><p><strong>Why these steps?</strong></p><ul><li><strong>Lowercase</strong>: “Email”, “EMAIL”, “email” should be treated the same</li><li><strong>Remove extra spaces</strong>: Normalize formatting</li><li><strong>Stemming</strong>: “running”, “runs”, “ran” → “run” (reduces feature space)</li></ul><h3>Step 2: Feature Engineering with TF-IDF</h3><p><strong>What is TF-IDF?</strong></p><p>TF-IDF (Term Frequency-Inverse Document Frequency) converts text into numerical features by measuring how important each word is.</p><p><strong>Formula:</strong></p><pre>TF-IDF = TF × IDF</pre><pre>TF (Term Frequency) = (Number of times word appears in document) / (Total words in document)<br>IDF (Inverse Document Frequency) = log(Total documents / Documents containing word)</pre><p><strong>Why TF-IDF and not CountVectorizer?</strong></p><p>Feature CountVectorizer, TF-IDF What it counts: Word frequency ,Word importance ,Common words like “the” High value (problematic), Low value Spam-specific words like “free” Low value High value Best for spam detection</p><pre>from sklearn.feature_extraction.text import TfidfVectorizer</pre><pre>cv = TfidfVectorizer(<br>    stop_words=&#39;english&#39;,           # Remove common words (the, is, a)<br>    min_df=2,                        # Word must appear in 2+ documents<br>    max_features=5000,               # Use top 5000 words<br>    ngram_range=(1, 2)               # Use 1-word and 2-word combinations<br>)</pre><pre>X = cv.fit_transform(raw_text[&#39;message&#39;])</pre><h3>6. Building the Model</h3><h3>Step 1: Split Data</h3><pre>from sklearn.model_selection import train_test_split</pre><pre>X_train, X_test, y_train, y_test = train_test_split(<br>    X, y, <br>    test_size=0.2,      # 20% for testing<br>    random_state=42     # For reproducibility<br>)</pre><h3>Step 2: Handle Class Imbalance</h3><p>Since we have more legitimate emails (86.6%) than spam (13.4%), the model could just predict “legitimate” for everything and get 86.6% accuracy. We need to give more weight to spam:</p><pre>from sklearn.utils.class_weight import compute_class_weight<br>import numpy as np</pre><pre># Calculate weights to balance classes<br>class_weights = compute_class_weight(<br>    &#39;balanced&#39;,<br>    classes=np.unique(y_train),<br>    y=y_train<br>)</pre><pre># Apply weights during training<br>sample_weights = class_weights[y_train]</pre><h3>Step 3: Train Naive Bayes</h3><pre>from sklearn.naive_bayes import MultinomialNB</pre><pre>clf = MultinomialNB()</pre><pre># Train with class weights<br>clf.fit(X_train, y_train, sample_weight=sample_weights)</pre><h3>Step 4: Evaluate Performance</h3><pre>from sklearn.metrics import classification_report, confusion_matrix</pre><pre>y_pred = clf.predict(X_test)</pre><pre>print(&quot;Classification Report:&quot;)<br>print(classification_report(y_test, y_pred, target_names=[&#39;SPAM&#39;, &#39;HAM&#39;]))</pre><pre>print(&quot;\nConfusion Matrix:&quot;)<br>print(confusion_matrix(y_test, y_pred))</pre><h3>7. Results &amp; Performance</h3><h3>Model Accuracy:</h3><pre>Training Accuracy: 97.5%<br>Test Accuracy: 98.2%</pre><h3>Classification Report:</h3><pre>precision    recall  f1-score   support</pre><pre>SPAM           0.94      0.92      0.93       154<br>HAM            0.99      0.99      0.99       961</pre><pre>accuracy                           0.98      1115</pre><h3>What This Means:</h3><ul><li><strong>Precision (Spam)</strong>: 94% — Of emails we marked as spam, 94% were actually spam</li><li><strong>Recall (Spam)</strong>: 92% — We caught 92% of actual spam emails</li><li><strong>F1-Score</strong>: Balanced measure of precision and recall</li><li><strong>Overall Accuracy</strong>: 98.2% — Correct predictions on 98.2% of emails</li></ul><h3>Confusion Matrix:</h3><pre>Predicted<br>           SPAM    HAM<br>Actual SPAM  142     12     (12 spam emails missed)<br>       HAM     8    953     (8 legitimate emails marked as spam)</pre><h3>Key Insights:</h3><p>Very low false positives (only 8 legitimate emails marked as spam)</p><p>Very low false negatives (only 12 spam emails missed)</p><p>Excellent balance between catching spam and not blocking legitimate emails</p><h3>8. Building a Web Application</h3><p>To make the model accessible, I built a Flask web application with a beautiful UI:</p><h3>Tech Stack:</h3><ul><li><strong>Backend</strong>: Flask (Python)</li><li><strong>Frontend</strong>: HTML + CSS + JavaScript</li><li><strong>Styling</strong>: Gradient purple background, smooth animations</li></ul><h3>Features:</h3><pre>from flask import Flask, render_template, request, jsonify<br>import pickle</pre><pre>app = Flask(__name__)</pre><pre># Load trained models<br>cv = pickle.load(open(&#39;cv.pkl&#39;, &#39;rb&#39;))<br>clf = pickle.load(open(&#39;clf.pkl&#39;, &#39;rb&#39;))</pre><pre>@app.route(&#39;/predict&#39;, methods=[&#39;POST&#39;])<br>def predict():<br>    email_text = request.json[&#39;email&#39;]<br>    <br>    # Process and vectorize<br>    vector = cv.transform([email_text])<br>    <br>    # Get prediction<br>    prediction = clf.predict(vector)[0]<br>    probability = clf.predict_proba(vector)[0]<br>    <br>    result = &quot;SPAM&quot; if prediction == 0 else &quot;HAM&quot;<br>    confidence = max(probability)<br>    <br>    return jsonify({<br>        &#39;result&#39;: result,<br>        &#39;confidence&#39;: f&quot;{confidence:.2%}&quot;<br>    })</pre><p><strong>Live Demo</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*vYr3Si9JJL5_rpB-IPe83Q.png" /></figure><h3>9. Lessons Learned &amp; Future Improvements</h3><h3>What Went Well:</h3><ol><li><strong>98.2% accuracy</strong> demonstrates the power of Naive Bayes for text classification</li><li><strong>Proper handling of class imbalance</strong> was crucial for good results</li><li><strong>TF-IDF vectorization</strong> worked much better than simple word counts</li><li><strong>Flask web app</strong> made the model accessible to non-technical users</li></ol><h3>Challenges Faced:</h3><p><strong>Initial accuracy issues</strong>: Early attempts showed 100% spam classification because the data was imbalanced</p><p><strong>. Solution</strong>: Applied class weights during training</p><p><strong>Feature selection</strong>: Too many features (common words) caused noise</p><ul><li><strong>Solution</strong>: Used stop words removal and TF-IDF weighting</li></ul><p><strong>Text preprocessing</strong>: Words needed standardization</p><ul><li><strong>Solution</strong>: Implemented stemming and lowercasing-Future Improvements:</li></ul><h3>Why This Project Matters</h3><h3>Real-World Impact:</h3><ol><li><strong>Security</strong>: Reduces phishing attacks and malicious emails</li><li><strong>Productivity</strong>: Saves time by filtering unwanted emails</li><li><strong>Cost Saving</strong>: Reduces manual moderation overhead</li><li><strong>Scalability</strong>: Handles millions of emails efficiently</li></ol><h3>Educational Value:</h3><p>This project teaches:</p><ul><li>Probability and Bayes’ Theorem</li><li>Text preprocessing and NLP basics</li><li>Feature engineering with TF-IDF</li><li>Handling imbalanced datasets</li><li>Model evaluation metrics</li><li>Building ML applications with Flask</li><li>Deployment and production considerations</li></ul><h3>Conclusion</h3><p>Naive Bayes proves to be an excellent choice for spam detection, achieving <strong>98.2% accuracy</strong> while remaining interpretable, fast, and scalable. Despite its “naive” assumption of feature independence, it outperforms more complex models on this task.</p><p>The combination of <strong>proper data preprocessing, feature engineering with TF-IDF, and handling class imbalance</strong> was key to success. And with Flask, we transformed a trained model into a practical application anyone can use.</p><h3>Key Takeaways:</h3><ol><li><strong>Naive Bayes is powerful for text classification</strong> despite its simplicity</li><li><strong>Data preprocessing and feature engineering matter more than complex algorithms</strong></li><li><strong>Handling imbalanced datasets is crucial</strong> for real-world applications</li><li><strong>Simple models can be production-ready</strong> when properly implemented</li></ol><h3>Resources</h3><h3>Code Repository:</h3><ul><li><a href="https://github.com/Pratik-sharma-py/datascience_project/tree/main/Naive%20Bayes">https://github.com/Pratik-sharma-py/datascience_project/tree/main/Naive%20Bayes</a></li></ul><h3>Libraries Used:</h3><ul><li>Scikit-learn: Machine Learning</li><li>Pandas: Data manipulation</li><li>NLTK: Text processing</li><li>Flask: Web framework</li><li>NumPy: Numerical computing</li></ul><h3>About the Author</h3><p>I’m a data scientist passionate about machine learning and building practical applications. This project combines my interest in NLP, statistics, and ML .</p><p><strong>Connect with me:</strong></p><ul><li>GitHub: <a href="https://github.com/Pratik-sharma-py">https://github.com/Pratik-sharma-py</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/pratik-sharma-854641279">https://www.linkedin.com/in/pratik-sharma-854641279</a></li><li>Email: pratikacharya527@gmail.com</li></ul><h3>Questions or Feedback?</h3><p>Feel free to reach out or check out the code on GitHub! I’d love to hear your thoughts on this project and discuss other machine learning applications.</p><p>This is published as part of my journey learning machine learning. This article was created to share knowledge and inspire others to build intelligent systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=200f36190a58" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Analysed 12,964 Drug Overdose Deaths with Python — Here’s What the Data Revealed]]></title>
            <link>https://medium.com/@pratikacharya527/i-analysed-12-964-drug-overdose-deaths-with-python-heres-what-the-data-revealed-46c474dc956b?source=rss-aafa75b21250------2</link>
            <guid isPermaLink="false">https://medium.com/p/46c474dc956b</guid>
            <category><![CDATA[mental-health]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Pratik Sharma]]></dc:creator>
            <pubDate>Sat, 23 May 2026 05:40:01 GMT</pubDate>
            <atom:updated>2026-05-23T14:28:40.962Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>From messy CSV to statistical proof — a step-by-step Python analysis of fentanyl, demographics, and a decade of preventable deaths</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wgQz-DFqBdbsdjjJkGyjZQ.png" /></figure><p><strong>Problem Domain</strong></p><p>Drug overdose deaths are a health problem in the England. Connecticut has seen a rise in accidental drug deaths since 2012.This increase is mostly due to synthetic opioids. To fight this problem, we need to know who is dying what drugs they are. When they are dying. This information will help us create plans to prevent and stop these deaths. This report looks at data from the Connecticut Office of the Chief Medical Examiner (OCME). The goal is to find patterns, risk factors and trends in overdose deaths from 2012 to 2024.We want to understand drug overdose deaths in Connecticut. The data will help us identify who is at risk of drug overdose deaths. Connecticut’s accidental overdose deaths are a concern. The OCME data will help us find ways to prevent drug overdose deaths. Fentanyl and synthetic opioids are driving the increase in drug deaths, in Connecticut. We need to use this information to stop drug overdose deaths.</p><p><strong>Dataset Description</strong></p><p>The data comes from investigations done by the Connecticut Office of the Chief Medical Examiner.</p><p>These investigations include:</p><p>Toxicology reports</p><p>Death certificates</p><p>Scene investigations</p><p>A ‘Y’ in the substance columns means that a particular drug was found in the person who died. The Connecticut OCME did these investigations to get the data. The data has information, on what drugs were detected in people who died.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*Xq7R780gRCnjwbiEvaVV_A.png" /></figure><p><strong>Key Columns Used</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*539zqSccS-Y8lqgky6Sm6w.png" /></figure><p><strong>Research Question and Hypotheses</strong></p><p><strong>Research Questions</strong></p><p>· <strong>RQ1: </strong>Has the number of accident drug overdose deaths in Connecticut’s significantly increased between 2012 and 2024?</p><p>· <strong>RQ2:</strong> Is there a statistically significant difference in the age of death between male and female victims?</p><p>· <strong>RQ3:</strong> Is fentanyl detection in overdose cases significantly associated with the year of death, has it replaced heroin over time?</p><p>· <strong>RQ4:</strong> Which age group is most at risk, and does that risk pattern differ between genders?</p><p><strong>Hypotheses:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*cSDpjqDsm9kT8YRbptQoKQ.png" /></figure><h4><strong>Solution Exploration</strong></h4><p><strong>Literature Review</strong></p><p>The following tools and technologies were evaluated for their suitability in analysing the drug overdose dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*c-0R3QL0NwgzZhwV718y7A.png" /></figure><p>People have dealt with issues in public health by using similar methods. The CDC Opioid Surveillance program uses statistics and looks at trends in national overdose data. McKinney in 2017 showed that the standard way to clean up data is by using Pandas. Waskom in 2021 showed that Seaborn is really good, at helping us understand health data by using pictures<strong>.</strong></p><p><strong>Methodological Approach</strong></p><p>The chosen approach follows a three-stage structured pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*gjJF5SAMmXlqHxLBNejgmw.png" /></figure><p>This way of doing things makes sure that what we find is based on things we can see and on numbers that back it up. We do not just rely on one or the other. The steps we take to do this work are similar, to what Virtanen et al. Said to do in 2020 for using Python to do science work.</p><ol><li><strong>1</strong> <strong>Workflow Diagram</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*Nzd5I2WNFCINkP-YuXB_5w.png" /></figure><h4><strong>Solution Development</strong></h4><p><strong>Data Pre-processing</strong></p><p>The dataset required extensive cleaning before analysis. The following steps were applied systematically.</p><pre>data.head()        # first 5 rows<br>data.tail()        # last 5 rows<br>data.shape         # (12964, 49)<br>data.info()        # dtypes and non-null counts<br>data.dtypes        # column data types<br>data.isnull().sum() # null counts per column</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*kAz_PWMQ2GmqgmQk6JZ4Uw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*D_b2Q9NQtbXBSiG10tLMBA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*SmeEvOOvMrdHZ-zmDXW4Gw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*oEBPdK9gY10n6V6l3TOYHQ.png" /></figure><p><strong>Step 2 — Column Standardization and Deduplication</strong></p><pre>data.columns = data.columns.str.strip().str.lower().str.replace(&#39; &#39;, &#39;_&#39;)<br>data.drop_duplicates(inplace=True)</pre><p>Column names were standardization to lowercase with underscores. Duplicate rows were removed to prevent double-counting in analysis.</p><p><strong>Step 3 — Data Type Correction</strong></p><pre>data[&#39;date&#39;] = data[&#39;date&#39;].astype(&#39;datetime64[ns]&#39;)<br>data[&#39;age&#39;]  = data[&#39;age&#39;].fillna(data[&#39;age&#39;].mean()).astype(&#39;int&#39;)</pre><p>The date column was converted from string to datetime. Missing age values were filled with the column mean before casting to integer, preserving all records.</p><p><strong>Step 4 — Drug Detection Columns (90% + Missing)</strong></p><p>The drug detection columns had a lot of missing information. Over 90 percent was missing. Of getting rid of these important columns the missing values were filled in with ’N’ to mean ‘Not Detected’. This is the way the rest of the dataset works, where ‘Y’ means something was detected and ’N’ means it was not detected by the drug detection columns. The drug detection columns are important so replacing the missing values, in the drug detection columns made sense than removing the drug detection columns altogether.</p><pre>drug_columns = [&#39;heroin&#39;, &#39;cocaine&#39;, &#39;fentanyl&#39;, &#39;benzodiazepine&#39;, ...]<br>for col in drug_columns:<br>    data[col] = data[col].fillna(&#39;N&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*xY-9AoNKBzJGj80IQtqkkA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*MUcKdmCwesCDVSKRhwgHsw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*yLZdVZEhrOPWDsGtYcNSXQ.png" /></figure><p><strong>Step 5 — Standardization other_significant_conditons</strong></p><p>This column had some problems with the names of conditions. For example it had things like ‘Recent cocaine use’ and ‘RECENT COCAINE USE’. So a dictionary was used to make all the different names look the same. Now all the medical condition names are in a title-case format, which means the first letter of each word is capitalized. This makes the medical condition names, like ‘Recent cocaine use’ look consistent throughout.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*lq47WeBEpAfWICRMokQXNg.png" /></figure><p><strong>Step 5 — Race, Sex, and Ethnicity Cleaning</strong></p><pre># Standardise race labels<br>race_value = {<br>    &#39;Asian/Indian&#39;  : &#39;Asian Indian&#39;,<br>    &#39;Asian, Other&#39;  : &#39;Other Asian&#39;,<br>    &#39;Other&#39;         : &#39;Unknown&#39;,<br>    &#39;white&#39;         : &#39;White&#39;,<br>    &#39;Other Asian&#39;   : &#39;Asian&#39;<br>}<br>data[&#39;race&#39;] = data[&#39;race&#39;].replace(race_value)<br><br># Sex: replace Unknown/X with mode, then fill NaN<br>fill_sex = data[&#39;sex&#39;].replace(&#39;Unknown&#39;,&#39;X&#39;).mode()[0]<br>data[&#39;sex&#39;] = data[&#39;sex&#39;].replace([&#39;Unknown&#39;,&#39;X&#39;], fill_sex).fillna(fill_sex)<br><br># Race: fill NaN with mode<br>data[&#39;race&#39;] = data[&#39;race&#39;].fillna(data[&#39;race&#39;].mode()[0])<br><br># Ethnicity: standardise Yes/No prefix labels, fill NaN with &#39;Others&#39;<br>ethnicity_dict = {<br>    &#39;Yes, Cuban&#39;                         : &#39;Cuban&#39;,<br>    &#39;Unknown&#39;                            : &#39;Others&#39;,<br>    &#39;Yes, Puerto Rican&#39;                  : &#39;Puerto Rican&#39;,<br>    &#39;No, not Spanish/Hispanic/Latino&#39;    : &#39;Not Spanish/Hispanic/Latino&#39;,<br>    ...  # 6 mappings total<br>}<br>data[&#39;ethnicity&#39;] = data[&#39;ethnicity&#39;].replace(ethnicity_dict).fillna(&#39;Others&#39;)</pre><p><strong>Step 7 — Geographic and manner_of_death Columns</strong></p><pre># Geographic columns: fill NaN with &#39;Unknown&#39;<br>geo_columns = [&#39;residence_city&#39;,&#39;residence_county&#39;,&#39;injury_city&#39;, ...]<br>for col in geo_columns:<br>    data[col] = data[col].fillna(&#39;Unknown&#39;)<br><br># manner_of_death: fix case inconsistencies<br>death_manner = {&#39;accident&#39;:&#39;Accident&#39;,&#39;Acciddent&#39;:&#39;Accident&#39;,&#39;ACCIDENT&#39;:&#39;Accident&#39;}<br>data[&#39;manner_of_death&#39;] = data[&#39;manner_of_death&#39;].replace(death_manner).fillna(&#39;Accident&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*rrZDj6YydAts4w2H0aRthQ.png" /></figure><p><strong>Step 8 — Feature Engineering</strong></p><pre># Extract year and month<br>data[&#39;year&#39;]  = data[&#39;date&#39;].dt.year<br>data[&#39;month&#39;] = data[&#39;date&#39;].dt.month_name()<br><br># Create age groups<br>bins   = [0, 18, 30, 45, 60, 100]<br>labels = [&#39;0-18&#39;, &#39;19-30&#39;, &#39;31-45&#39;, &#39;46-60&#39;, &#39;60+&#39;]<br>data[&#39;age_group&#39;] = pd.cut(data[&#39;age&#39;], bins=bins, labels=labels)</pre><p>Year and month were taken from the date column for analysis over time. An age group feature was made using bins that make sense. This resulted in 52 features after working on them.</p><p><strong>Missing Value Handling Summary</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*xupotesDEbddEiEP19mhYQ.png" /></figure><p><strong>1.1</strong> Exploratory Data Analysis — Descriptive Statistics</p><p>Before visualizing the data, key statistical properties were examined to understand the dataset’s structure and distributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*_y_IuA3tLx3msnl6gkIh2w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*z2xmurSY-z_wb8-twKRI_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*W2QqSoeP1Ogmy0t9hDm5tQ.png" /></figure><p><strong>1.1</strong> <strong>Data Visualization</strong></p><p>They help to explore patterns over time demographics, geography and drug types. These visualizations cover time demographics, geography and drug types. They were created with Matplotlib and Seaborn.</p><p><strong>Drug-Overdose Deaths Trend (2012–2024)</strong></p><p>The number of deaths went up steadily from 357 in 2012 to over 1,500 by 2023. This is an increase in deaths over 12 years. The deaths just kept going up and up. There was no drop in the number of deaths. The deaths more than tripled over 12 years. The line chart, with gradient fill shows this. Deaths rose consistently over the years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*tDMBfhtOJOWQtzAZuoLgNQ.png" /></figure><p><strong>Deaths by Age Group</strong></p><p>The number of deaths went up steadily from 357 deaths in 2012 to over 1,500 deaths by 2023. This is an increase in deaths over 12 years. The deaths more than tripled. There was no drop in the number of deaths, over this time period. The deaths just kept going up and up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*pJY_-15N73-ezOrXCUQFFw.png" /></figure><p><strong>Gender Distribution</strong></p><p>The Donut chart shows that males make up about 70 percent of all deaths. This means that males are dying from overdoses a lot more than females. The Donut chart also shows that this has been the case for years with males significantly outnumbering female victims every year. The Donut chart is really clear about one thing: males are more likely to die from an overdose, than females. This is a trend that we see in the Donut chart every year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*YZoLofzQ5M8QBqvQDzeKxg.png" /></figure><p><strong>Tope 10 Cities with Highest Overdose Deaths</strong></p><p>Hartford, Bridgeport and New Haven are the cities, with the risk. These cities, Hartford, Bridgeport and New Haven always have the overdose deaths every year. Hartford, Bridgeport and New Haven are the ones that consistently have the numbers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*0XhHpRpXWinhxqw7kUvC6A.png" /></figure><p><strong>Most Detected Drugs</strong></p><p>The Horizontal bar chart says that Any Opioid and Fentanyl are the substances that people find the most. Fentanyl is a problem. The number of times Fentanyl is found has gone up a lot. This is not the case with heroin. Heroin is found often now than it was in 2016. The Horizontal bar chart for Fentanyl and heroin is very different from what it was, in 2016.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fh9232f6wYPF_LBw7EDAPA.png" /></figure><p><strong>Deaths by Race (Top 8)</strong></p><p>If we look at the people who were victims the ones who are white make up most of the deaths that we know about. This makes sense because of the kinds of people who live in the area and where most of the cases happened which’s, in Connecticut. The bar chart shows that white victims are the group of recorded deaths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*YJAf75UVKMmd8SvMr67hvw.png" /></figure><p><strong>Monthly Deaths Heatmap by Year</strong></p><p>What I notice is that no single month has high counts all the time. However, if I look at the years like 2020, 2021 2022 and 2023 I see that these years have higher counts for every month. The Seaborn heatmap for these years 2020, 2021 2022 and 2023 really stands out because the counts are higher, across all the months.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*nlKd-x_y48WxLCocnTBEhQ.png" /></figure><p><strong>Age Distribution by Gender (KDE)</strong></p><p>The male victims are most likely to be thirty eight to forty two years old. On the hand the female victims are a bit older and the age range is wider. This is what the KDE density plot shows us about the female victims and it also supports what we found out from the T-Test about the male and female victims.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*zOrBlLi0BgK4tJJ8Ju8P0w.png" /></figure><p><strong>Death Location Analysis</strong></p><p>Most overdose deaths happen at home. In fact home is the common place where overdose deaths occur making up the majority of all cases. This shows that most overdoses happen in private not in places and are a personal crisis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*oI7ZqV2BWZhu_fSoOHn1aw.png" /></figure><p><strong>Fentanyl vs Heroin vs Cocaine Over Time</strong></p><p>Fentanyl cases went up a lot from zero in 2012. By 2016 Fentanyl was the drug. Heroin cases dropped sharply. Cocaine cases stayed about the same. This directly answers the question. It is, about Fentanyl, heroin and cocaine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*8PBT_umdjuyu4mUWi4waEQ.png" /></figure><p><strong>Age Group x Gender Heatmap</strong></p><p>The cell for 31–45 year males shows the highest count. The pattern of risk is different for males and females, in all age groups. This supports H4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*NL6OPieYGnYYfD0gJK6ygw.png" /></figure><h4><strong>Statistical Hypothesis Testing</strong></h4><p>Four tests were done to check if the patterns we saw during EDA were real. We used a significance level of 0.05 for all tests. We did these tests to make sure our observations were correct. The tests helped us understand if what we saw was, by chance or not. All tests used alpha = 0.05.</p><h4><strong>Test 1-Independent T-Test: Gender Difference in Age at Death</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*tcOfQ21qsfX23oHi3eF-kw.png" /></figure><pre>male_ages   = data[data[&#39;sex&#39;] == &#39;Male&#39;][&#39;age&#39;].dropna()<br>female_ages = data[data[&#39;sex&#39;] == &#39;Female&#39;][&#39;age&#39;].dropna()<br>t_stat, p_value = stats.ttest_ind(male_ages, female_ages)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*G-hX-eiW5b_UO2t2g2-lyQ.png" /></figure><p>The result shows that the p-value is than 0.05. This means we reject the idea that there is no difference. We found out that there is a difference in the average age at which males and females die. Females who are victims tend to die at an average age than males who are victims. The average age at death for females is a bit higher than, for males.</p><h4><strong>Test 2 — Chi-Square Test: Fentanyl Detection vs Year</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*WjuK8-9rEGDXUaTwfbZ0cg.png" /></figure><pre>data[&#39;fentanyl_flag&#39;] = data[&#39;fentanyl&#39;].apply(lambda x: 1 if x == &#39;Y&#39; else 0)<br>contingency_table = pd.crosstab(data[&#39;year&#39;], data[&#39;fentanyl_flag&#39;])<br>chi2, p, dof, expected = stats.chi2_contingency(contingency_table)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*8tU2ENuHNSQ7ia3l83keeg.png" /></figure><p><strong>Result:</strong> The p-value is less, than 0.05 so we reject the null hypothesis. Fentanyl detection is strongly linked to the year of death. This confirms that fentanyl has become the drug causing overdose deaths since 2016 and its use has increased sharply.</p><h4>Test 3 —One-Way ANOVA: Age Differences Across Age Groups</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*xiuZdvA4jc7F8wCiJi8-Jw.png" /></figure><pre>groups = [grp[&#39;age&#39;].values for _, grp in data.groupby(&#39;age_group&#39;, observed=True)]<br>f_stat, p_anova = stats.f_oneway(*groups)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*y9iKG-7gzVnftc3Q5rQMvw.png" /></figure><p>The result shows that the p-value is, than 0.05, which means we reject the idea that there is no difference, known as H0. This tells us that the different age groups are actually very different when it comes to statistics. We can confirm that people of ages have different chances of facing certain risks and these differences are significant. The age groups are statistically distinct meaning that the risk profiles of demographic age groups are significantly different.</p><h4>Test 4 —Chi-Square Test: Age Group Risk vs Gender</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*eXPDngc0xgUXpGxbUUY1PQ.png" /></figure><pre>age_gender_table = pd.crosstab(data[&#39;age_group&#39;], data[&#39;sex&#39;])<br>chi2_ag, p_ag, dof_ag, _ = stats.chi2_contingency(age_gender_table)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*HTBwdri6AYaK8xVJtmNdmw.png" /></figure><p>The result shows that the p-value is than 0.05 so we reject the idea that there is no difference. This means that the number of deaths in different age groups is actually different for males and females. It is clear that gender plays a role in how old someone’s when they are at risk. The overdose deaths distribution is significantly different, between males and females which confirms that gender affects the risk of overdose at ages.</p><h4><strong>Answer to Research Question</strong></h4><p><strong>RQ1: Has the number of accidental drug overdose deaths significantly increased from 2012 to 2024?</strong></p><p>Yes. If we look at the yearly trend chart, we can see that accidental drug overdose deaths have risen from 357 in 2012 to over 1,500 by 2023. This is an increase. More than tripling over 12 years. The worst part is that there has been no sustained decline in accidental drug overdose deaths during this time.</p><p><strong>RQ2: Is there an age difference between male and female victims of accidental drug overdose deaths?</strong></p><p>Yes. We did a test. Found that there is a big difference in the mean age at death between males and females who are victims of accidental drug overdose deaths. If we look at the chart we can see that female victims of accidental drug overdose deaths are a bit older and the age range is broader.</p><p><strong>RQ3: Has fentanyl become more prevalent over the years in accidental drug overdose deaths?</strong></p><p>Yes it has. We did another test. Found that fentanyl detection is not random across years in accidental drug overdose deaths. If we look at the chart we can see that fentanyl started to become more common around 2016 and has continued to rise since then. In fact fentanyl has become the drug in accidental drug overdose deaths replacing heroin.</p><p><strong>RQ4: Does age group risk differ by gender in accidental drug overdose deaths?</strong></p><p>Yes it does. We did a test. Found that the risk patterns are different for males and females in different age groups. The 31–45 age group has the risk overall.. If we look at the chart we can see that males are at a higher risk in every age group. Accidental drug overdose deaths affect males more, than females in every age bracket.</p><p>Hi, I am Pratik Sharma (Delta) and I am currently learning Full/Stack AI/ML in one of the rising AI community of Nepal SocrateAI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46c474dc956b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You Are the Average of Your 5 Neighbors: Understanding KNN Through Olympic Data]]></title>
            <link>https://medium.com/@pratikacharya527/you-are-the-average-of-your-5-neighbors-understanding-knn-through-olympic-data-75b86f9045ca?source=rss-aafa75b21250------2</link>
            <guid isPermaLink="false">https://medium.com/p/75b86f9045ca</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[knn-algorithm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Pratik Sharma]]></dc:creator>
            <pubDate>Sat, 16 May 2026 18:43:46 GMT</pubDate>
            <atom:updated>2026-05-16T18:43:46.950Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Problem That Started My Machine Learning Journey</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eVXut8OGCrh4muT2UflmlA.png" /></figure><p>When I first encountered machine learning I did not just want to learn algorithms from textbooks. I wanted to solve a problem.</p><p>That’s when I started working with data about athletes. The question was easy: Can we guess an athletes weight based on their characteristics and background?</p><p>This was not an exercise. It was a challenge that helped me discover K-Nearest Neighbors (KNN) — and, in this blog I will walk you through how I solved it using KNN.</p><p>The Big Idea: <strong><em>You Are Like The People Around You</em></strong></p><p>“<strong><em>You are like the five people you hang out with the most</em></strong>.”. Jim Rohn</p><p>This is not something to motivate you. It is also how KNN works.</p><p>The basic idea is that if someone is tall and strong people they will probably be tall and strong too. If they are around gymnasts they will probably have physical traits. KNN uses this idea. To predict something about a person it looks at people who are most, like them and uses their traits to make a prediction.</p><h4><strong>How Does KNN Really Work?</strong></h4><p><strong>KNN is a type of machine learning that uses data that is already labeled to make predictions. This is how it works:</strong></p><p>1. Find The K People Like Them: Find The K similar examples in your data</p><p>2. Check Their Details: Look at what you want to know (like weight) of these people</p><p>3. Make A Prediction: Use the average of their details to predict what you do not know about the person</p><h4><strong>Our Mission: Predicting Athlete Weights</strong></h4><p>I wanted to try something so I made a KNN model using Olympic athlete data. The goal of Predicting Athlete Weights is to figure out how an athlete weighs based on things like Sex, Age, Height, Sport, Medal status and Year. Now I will tell you how I made Predicting Athlete Weights step by step.</p><h4><strong>Step 1: Set Up Your Environment</strong></h4><p>Every project that has to do with data science starts with one thing: getting the tools you need, for Predicting Athlete Weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*Vfo3keW2jsI5_n-yGdzYkA.png" /></figure><h4><strong>Step 2: Load and Explore Your Olympic Data</strong></h4><p>Next I loaded the Olympic dataset. Took a good look at the Olympic data. The Olympic data had 261,914 athletes from Olympic Games and the Olympic data had 20 different columns with information about each athlete in the Olympic data. I also found some values, in the Olympic data that I needed to work on.</p><p>The first rule of data science is to look at the Olympic data before you do anything to the Olympic data. So I made a copy of the Olympic data to keep it safe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*liQYjXYHo4zMHtV3pSQ_7A.png" /></figure><h4><strong>Step 3: Understand Your Data</strong></h4><p>Before I started cleaning the data I took a look at the dataset. I used the size command, the shape command, the info command and the describe command to get an understanding of the data. I wanted to know what the data was, like so I used these commands to examine the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*zcui_DdZbSSaet9ec6Gjng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*9d-AGDNDfdBao8lKZ4FnDQ.png" /></figure><h4><strong>Step 4: Standardize Column Names</strong></h4><p>I made all the column names lowercase so that the column names are consistent. I did this to make sure the column names are the throughout so I changed all the column names to lowercase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*sEkosikySMSDiXs-Bb0uGg.png" /></figure><h4><strong>Step 5: Handle Missing Values</strong></h4><p>I found values in the data and I decided to fill them with the median. This is a way to do it than using the mean. I think the median is more reliable, for handling missing values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*FPJus_ztpE6tFmZ7WG0lAQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*eAT1iOoYd5zGF6r80EmTOw.png" /></figure><h4><strong>Step 6: Feature Engineering Is Really Important</strong></h4><p>This is the part where working with data starts to feel like creating something. I took the categories. Made them into numbers and I also made new features that make sense like has_medal.</p><h4><strong>Step 7: Split Your Data Into Two Parts</strong></h4><p>You should never use the data to test and to train. I split my data into two parts, 80 percent for training and 20 percent for testing. I used random_state equal, to 42 so I can get the same result every time I do it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*Z5KAczzIqmQPYez8zD0ejg.png" /></figure><h4><strong>Step 8: Scale Your Features</strong></h4><p>I made sure all the features are on the level, by using StandardScaler. This is important because it helps to get distance calculations for the features. I used StandardScaler for this purpose to make the features fair.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*hjCfBZWFQZ_YMeYo3iPWsw.png" /></figure><h4><strong>Step 9: Build Your KNN Model</strong></h4><p>I made a KNN function that I can use again and again. This KNN function learns from the training data. Then it makes predictions using the test data.</p><h4><strong>Step 10: Find the Perfect K Value</strong></h4><p>I tried K values from 1, to 21 to see what works best for the KNN model. The KNN model works well when I use K=19 or K=21 for the KNN model. This is because K=19 or K=21 gives the balance for the KNN model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*a2CctCOxkGPik7fjpGvKog.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*g_B-6TQ9jG5NrszRJkGEhQ.png" /></figure><h4><strong>Step 11: Train Your Final Model</strong></h4><p>I used the K value to train the final model. Then I looked at how the final model did using the MAE metric, the RMSE metric and the R² metric. I wanted to see how the final model performed. The final model was evaluated with these metrics to see how it did. I used the MAE metric, the RMSE metric and the R² metric to evaluate the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*9TbW9jzGFqty37L7ZEhUAA.png" /></figure><h4><strong>Step 12: Make Predictions</strong></h4><p>The last thing to do is to use the model on new information that the model has never seen before. This is what we mean by using the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*7IVjdV0e1iH4Wb0pW7qp_A.png" /></figure><h3><strong>What I. What You Should Take Away</strong></h3><h4>Technical Lessons</h4><p>1. Cleaning the data is really important. The data you clean will affect how well your model works.</p><p>2. Making features is like an art. It is more important how you make these features than which method you use.</p><p>3. You should always divide your data into two parts. One for training and one for testing.</p><p>If you use the data for both you will get wrong results.</p><p>4. Making sure everything is on the scale is necessary. Especially when you are using methods like KNN that need distances.</p><p>5. You should try options for your model. Do not just guess, try them in a systematic way.</p><p>6. Using ways to measure how good your model is will give you the whole picture. Use MAE and RMSE and R² all together.</p><p><strong>You Can Do This Too</strong></p><p>If you are learning about machine learning do not just read about the methods.</p><p>Find a set of data that you think is interesting.</p><p>Ask a question, about this data.</p><p>Then build a model to answer this question.</p><p>You will run into problems.. That is where you will really learn something.</p><p>The whole process I showed you. From getting the data to making predictions with a model. Is what every machine learning project is based on no matter how complicated it is.</p><h4>Key Takeaways</h4><p>The KNN system is based on an idea: the KNN system says that you are like the people around you the KNN system says that you are the average of your K nearest neighbors. Always split your information into two groups: the group you use to teach the KNN system, which’s most of your information and the group you use to test the KNN system, which is the rest of your information. It is very important to make sure your information is good and to create features from your information. This is more important than the type of KNN system you choose.</p><p>You need to make sure all your features are on the scale so the KNN system can calculate the distances correctly.</p><p>When you are setting up your KNN system you should try different options to see what works best. Do not just guess what you think will work.</p><p>Use different ways to measure how well your KNN system is working, such as the mean absolute error the root mean squared error and the coefficient of determination.</p><p>Always use your KNN system to make predictions on information. This is when you will find out if your KNN system really works.</p><h4>Questions?</h4><p>If you found this helpful or have questions, about the KNN system leave a comment. I would love to hear about what you’re learning about machine learning.</p><p>Check out My Git-Hub for Code and Data-Set : <a href="https://github.com/Pratik-sharma-py/datascience_project/tree/main/KNN">https://github.com/Pratik-sharma-py/datascience_project/tree/main/KNN</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75b86f9045ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why ‘Diesel’ Breaks Your Machine Learning Model — And How to Fix It]]></title>
            <link>https://medium.com/@pratikacharya527/why-diesel-breaks-your-machine-learning-model-and-how-to-fix-it-587e7aa8bb64?source=rss-aafa75b21250------2</link>
            <guid isPermaLink="false">https://medium.com/p/587e7aa8bb64</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Pratik Sharma]]></dc:creator>
            <pubDate>Tue, 12 May 2026 14:53:48 GMT</pubDate>
            <atom:updated>2026-05-12T14:55:08.100Z</atom:updated>
            <content:encoded><![CDATA[<p>A complete blog on Ordinal Encoding, Label Encoding, and One-Hot Encoding with real Python code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Co4GjjupCNF859qU3ubkIw.png" /></figure><h3>Types of Data</h3><p>Before we jump into encoding, we need to understand the two main types of data we work with in machine learning:</p><ul><li><strong>Numerical:</strong> Weight, height, age, salary — any data that is already a number.</li><li><strong>Categorical:</strong> Gender, nationality, education level — data that represents categories or groups.</li></ul><p>Inside categorical data, there are two further types:</p><p><strong>i) Nominal</strong> — Data that has no natural order. You cannot say one value is higher or lower than another. Example: fuel type (CNG, Diesel, LPG, Petrol).</p><p><strong>ii) Ordinal</strong> — Data that has a meaningful order. Values can be ranked as higher or lower. Example: education level (School &lt; Undergraduate &lt; Postgraduate).</p><h3>Why Do We Need Categorical Encoding?</h3><p>Machine learning models understand only numbers. Categorical columns store data in textual form — words like “Good”, “Diesel”, or “Yes”. To feed this data into a model, we must convert it into numeric form.</p><p>The three main methods to do this are:</p><ol><li>Ordinal Encoding — especially for ordinal data</li><li>One-Hot Encoding — for nominal data</li><li>Label Encoding — for binary or target columns</li></ol><h3>I. Ordinal Encoding</h3><p>Suppose we have a column in the dataset named <strong>Education</strong> containing values: School, Undergraduate, Postgraduate. We know these have a natural order:</p><p>School &lt; Undergraduate &lt; Postgraduate</p><p>So we assign numbers that preserve this order:</p><ul><li>School → 0</li><li>Undergraduate → 1</li><li>Postgraduate → 2</li></ul><p>This way, School gets the lowest value and Postgraduate gets the highest — exactly matching the real-world relationship.</p><p>Let’s take a real dataset and see how it actually works.</p><p>We load the <strong>customer.csv</strong> dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*BwSQCwUU9ef6Dyzyu1d3Ig.png" /><figcaption><em>Customer dataset loaded into a DataFrame</em></figcaption></figure><p>Now we take a sample of 5 rows from the whole dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*xhpUXVa1oBUYWawNOvSPgA.png" /><figcaption><em>5 random sample rows — review, education, and purchased columns contain text values</em></figcaption></figure><p>We are doing Ordinal Encoding, so we will only work with three columns: <strong>review</strong>, <strong>education</strong>, and <strong>purchased</strong>.</p><p>Using Sklearn, we split the dataset into training and test sets — 80% for training and 20% for testing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*yj1-RxeOI3Tcur6dw1aToA.png" /><figcaption><em>Ordinal categories explicitly defined — Poor/Average/Good and School/UG/PG in order</em></figcaption></figure><p><strong>Note:</strong> Without explicitly defining the categories, the Ordinal Encoder can randomly assign the order of values — which will mislead your model.</p><p>We import OrdinalEncoder from sklearn and create the categories — assigning lower values to Poor and School, and higher values to Good and PG respectively:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ksbt-uG8BQVHbByvuud3cQ.png" /><figcaption><em>Dataset split 80/20, OrdinalEncoder fitted on the training data</em></figcaption></figure><p>We fit our training dataset to the Ordinal Encoder:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*s9LUpkv2tsek0kznQT7pSg.png" /><figcaption><em>OrdinalEncoder fitted on training columns</em></figcaption></figure><p>Now we transform X_train and X_test using the Ordinal Encoder. When transforming, the output becomes a NumPy array, so we convert it back to a DataFrame using pd.DataFrame():</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eATtkX32-RLEYzH3SXru2A.png" /><figcaption><em>review and education columns are now converted to numeric values</em></figcaption></figure><p>We can see that our <strong>review</strong> and <strong>education</strong> columns have been converted into numeric form using the Ordinal Encoder.</p><p>Now, for the <strong>purchased</strong> column which contains Yes and No — this is nominal data. We cannot assign a higher or lower order to Yes and No, so Label Encoder comes into play.</p><h3>II. Label Encoding</h3><p>To apply Label Encoding, we import LabelEncoder from sklearn, initialize it as le, and fit it on the y_train training data which contains the purchased column:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*GcnP5-La74GjBE2NDmQrWw.png" /><figcaption><em>LabelEncoder imported and initialized</em></figcaption></figure><p>Now we transform y_train and y_test for the purchased column:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*Q0C8kSS0oWrEj2c80tsjCw.png" /><figcaption><em>y_train transformed — output is an array of 1s and 0s</em></figcaption></figure><p>We can see the output of y_train_enc has been converted into 1s and 0s. Since the transformation returns a NumPy array, we convert it back to the original DataFrame form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*4UZO1Pahg5K5-4wnJvKdZA.png" /><figcaption><em>Purchased column successfully converted to numeric form — Yes=1, No=0</em></figcaption></figure><p>We can see the output after applying the Label Encoder to the purchased column — it is now in numeric form.</p><h3>III. One-Hot Encoding</h3><p>When we have nominal data, we cannot perform Ordinal Encoding — it will mislead our machine learning model. Suppose you have a dataset with a <strong>fuel type</strong> column. We cannot directly transform fuel types into numeric form with ordinal encoding, because no fuel type is higher or lower than another. Instead, we apply One-Hot Encoding to get the numeric form as a vector.</p><p>Let’s apply One-Hot Encoding to a dataset and see how it really works.</p><p>First, we load the dataset and take a look at it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NUpL9CqsbPg-iu-syC-GLg.png" /><figcaption><em>Car dataset with fuel, owner, brand, and km_driven columns</em></figcaption></figure><p>Using Pandas, we can apply One-Hot Encoding directly like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JqOdqbVbZ5d0f8qmCsvoww.png" /><figcaption><em>One-Hot Encoding applied directly using pd.get_dummies — each category becomes its own binary column</em></figcaption></figure><p>We can apply One-Hot Encoding directly with Pandas, but let’s understand how it works internally by breaking down the method.</p><h3>Before One-Hot Encoding — Two Concepts to Understand</h3><ul><li><strong>Corpus</strong> — The combination of all words in the dataset column. In our dataset, all values in the fuel and owner columns together form the corpus.</li><li><strong>Vocabulary</strong> — How many unique words the corpus is made up of.</li></ul><p>In the <strong>fuel</strong> column we have 4 types: CNG, Diesel, LPG, Petrol. So vocabulary = 4.</p><p>In the <strong>owner</strong> column we have 5 types: First, Second, Third, Fourth &amp; Above, Test Drive. So vocabulary = 5.</p><p>One-Hot Encoding converts each word in your dataset into a vector of size V (vocabulary). Internally, pd.get_dummies creates one column per unique value in the vocabulary.</p><h3>The Dummy Variable Trap</h3><p>While performing One-Hot Encoding, a <strong>multicollinearity</strong> problem can arise — a relationship between the input columns. You can observe that the sum of each encoded row equals 1, which can create problems when performing linear regression or logistic regression models.</p><p>To solve this, we apply the <strong>K-1 method</strong> — we drop the first column of the dummy variables (the variables created during One-Hot Encoding):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C20s4k23N12B9hDffzXZ8A.png" /><figcaption><em>First dummy variable column dropped to eliminate multicollinearity</em></figcaption></figure><p><strong>Note:</strong> If you have N categories, we only keep N-1 columns. N-1 columns can still perfectly represent all N categories.</p><p>Doing this manually for a large dataset is time-consuming. Instead, we use sklearn’s OneHotEncoder:</p><p>First, we split the data into train and test using sklearn:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wvUazmHwx5JpCJ29NgomLw.png" /><figcaption><em>80% training, 20% testing split for the car dataset</em></figcaption></figure><p>This is our raw data. 80% goes to training and 20% goes to testing.</p><p>We apply One-Hot Encoding to our nominal columns — fuel and owner. We import OneHotEncoder and initialize it as ohe:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*0xxFW3JIiwNWDv64BQWPdA.png" /><figcaption><em>OneHotEncoder imported from sklearn and initialized</em></figcaption></figure><p>We transform X_train and X_test on these two columns. After transformation, the One-Hot Encoder expands 2 columns into 9 columns, which we can confirm from the shape:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5At3_wb6nssORhwRuMnhRQ.png" /><figcaption><em>fuel and owner columns expanded from 2 to 9 columns after One-Hot Encoding</em></figcaption></figure><p>Now we have the encoded columns separately. We join them with the remaining columns — brand and km_driven — using np.hstack (which joins data horizontally). The final shape becomes 11 columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fV8Wtswltx8fuBl0AjvwvQ.png" /><figcaption><em>All columns combined horizontally — 9 encoded + brand + km_driven = 11 total</em></figcaption></figure><h3>Applying One-Hot Encoding to the Brand Column — Top Categories</h3><p>We can also apply One-Hot Encoding to the brand column using only the top categories. Let&#39;s see how:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-LIxvUYre86ps3aV3ZOE1A.png" /><figcaption><em>All unique brands in the dataset — many rare brands with very few entries</em></figcaption></figure><p>We can see the whole vocabulary of the brand column. To manage this, we group brands with fewer than 100 entries under a single Others category by setting the threshold to 100:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HwdBtz0O_h95axSJ2g9b9Q.png" /><figcaption><em>Rare brands replaced with ‘Others’ — vocabulary reduced to top brands only</em></figcaption></figure><p>Here we can see that One-Hot Encoding works properly, converting words in our data into V-dimension vectors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7SsvQNZ_RO6EDdehLFph_A.png" /><figcaption><em>Brand column successfully One-Hot Encoded using top categories approach</em></figcaption></figure><h3>Quick Reference:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*-HPdO9QkkrkPHr-4Pck5vw.png" /></figure><h3>Conclusion</h3><p>Categorical encoding is not just a preprocessing checkbox — it is the communication layer between your raw data and your machine learning model. Choosing the wrong method can silently corrupt what your model learns.</p><p>To summarize the decision:</p><ul><li>Does the data have a natural order? → <strong>Ordinal Encoding</strong></li><li>Is it a binary target column (Yes/No)? → <strong>Label Encoding</strong></li><li>Are the categories unordered? → <strong>One-Hot Encoding</strong></li><li>Using a linear model with One-Hot data? → Always <strong>drop one dummy column</strong></li></ul><p>Master these three techniques and you will be equipped to handle almost every categorical column you encounter in real-world datasets.</p><p>The raw code used in this blog : <a href="https://github.com/Pratik-sharma-py/datascience_project">https://github.com/Pratik-sharma-py/datascience_project</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=587e7aa8bb64" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Principal Component Analysis (PCA): Simplifying High-Dimensional Data]]></title>
            <link>https://medium.com/@pratikacharya527/principal-component-analysis-pca-simplifying-high-dimensional-data-5a754eaf85ae?source=rss-aafa75b21250------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a754eaf85ae</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[apc]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[pca-analysis]]></category>
            <dc:creator><![CDATA[Pratik Sharma]]></dc:creator>
            <pubDate>Fri, 08 May 2026 12:04:44 GMT</pubDate>
            <atom:updated>2026-05-08T12:04:44.511Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XxuQt9QpCeNR2BAhoaLPwQ.png" /></figure><h3>Introduction</h3><p>In modern data visualisation and machine learning, datasets often contain <strong>high dimensions</strong> — many features or columns. While more features can mean more information, they also introduce challenges known as the <strong>curse of dimensionality</strong>:</p><ul><li><strong>Sparsity</strong>: Many features contain zero or irrelevant values.</li><li><strong>Performance issues</strong>: More features can confuse algorithms and reduce accuracy.</li><li><strong>Computational complexity</strong>: High-dimensional data is time-consuming to process and harder to interpret.</li></ul><p>In short:</p><p><strong>More dimensions → more space → more empty space.</strong></p><p>This is where <strong>Principal Component Analysis (PCA)</strong> comes in.</p><h3>What is PCA?</h3><p>PCA is a <strong>feature extraction technique</strong> designed to reduce dimensionality while preserving the essence of the data. It transforms high-dimensional data into a lower-dimensional form, making it easier to analize, visualise, and process.</p><h3>Benefits of PCA</h3><ul><li><strong>Faster algorithms</strong>: Lower dimensions mean quicker computations.</li><li><strong>Better visualisation</strong>: Reduced dimensions allow clearer, more insightful plots.</li></ul><h3>Geometric Intuition: Feature Selection</h3><p>Imagine a dataset with three features:</p><ul><li>Number of rooms</li><li>Number of grocery shops nearby</li><li>Price of the house</li></ul><p>When plotting the data, suppose the variance along the <strong>x-axis (rooms)</strong> is much higher than along the <strong>y-axis (grocery shops)</strong>. This suggests that <strong>rooms</strong> are more important in determining house price than grocery shops.</p><p>But what if the features are the number<strong> of rooms</strong> and <strong>number of washrooms</strong>? Both are equally important, and domain knowledge tells us that <strong>square footage</strong> might be a better single feature to collect. PCA helps us uncover such relationships by finding the most informative directions in the data.</p><h3>How PCA Works (Step by Step)</h3><h3>Step 1: Standardization</h3><p>Different features often have different scales (e.g., salary vs. age vs. height). Without scaling, large-valued features dominate.</p><ul><li><strong>Standardization formula</strong>:</li></ul><p>x′=x−mean-standard deviation</p><ul><li>After scaling: mean = 0, standard deviation = 1.</li></ul><h3>Step 2: Covariance Matrix</h3><p>Covariance tells us how features vary together.</p><ul><li><strong>Diagonal elements</strong> → variance of each feature.</li><li><strong>Off-diagonal elements</strong> → covariance between features.</li></ul><p>PCA asks: <em>In which direction does the data spread the most?</em></p><p>That direction becomes our <strong>first principal component (PC1)</strong>.</p><h3>Step 3: Eigenvalues and Eigenvectors</h3><p>From the covariance matrix, we compute eigenvalues and eigenvectors.</p><ul><li><strong>Eigenvector (v)</strong> → direction in feature space.</li><li><strong>Eigenvalue (λ)</strong> → amount of variance in that direction.</li><li><strong>PC1</strong>: direction with maximum variance.</li><li><strong>PC2</strong>: perpendicular direction with second-most variance.</li></ul><p>Not all directions are equally useful — large eigenvalues indicate more important components.</p><h3>Step 4: Choosing the Number of Components</h3><p>We decide how many principal components to keep using <strong>explained variance</strong>.</p><ul><li>Example: Original dataset has 1000 features.</li><li>PCA reduces it to 100 features while retaining <strong>90–95% of variance</strong>.</li></ul><p>This balance ensures efficiency without losing critical information.</p><h3>Step 5: Transforming the Data</h3><p>Finally, we project the original data onto the new axes (principal components).</p><p>Z=X⋅W</p><ul><li>X = standardized data</li><li>W = projection matrix (top eigenvectors)</li><li>Z = transformed lower-dimensional data</li></ul><p>Result: A compact dataset with fewer, more meaningful features.</p><h3>Worked Example: House Price Prediction</h3><p>Suppose we have 5 features:</p><ul><li>Rooms, Washrooms, Square Footage, Grocery Shops, Distance to City Center.</li></ul><p>After PCA:</p><ul><li><strong>PC1</strong> might capture variance from Rooms + Washrooms + Square Footage (structural features).</li><li><strong>PC2</strong> might capture variance from Grocery Shops + Distance (location features).</li></ul><p>Instead of analyzing 5 separate features, we now work with just <strong>2 principal components</strong> that summarize the dataset effectively.</p><h3>Conclusion</h3><p>PCA is a powerful tool to combat the curse of dimensionality. By reducing features while retaining variance, it:</p><ul><li>Speeds up algorithms,</li><li>Improves visualization,</li><li>Helps uncover hidden relationships in data.</li></ul><p>In essence, PCA transforms complexity into clarity — making high-dimensional data manageable and insightful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a754eaf85ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Math We Forgot to Teach: Bridging the Gap Between Nepali Classrooms and the AI Revolution]]></title>
            <link>https://medium.com/@pratikacharya527/the-math-we-forgot-to-teach-bridging-the-gap-between-nepali-classrooms-and-the-ai-revolution-464928a8860d?source=rss-aafa75b21250------2</link>
            <guid isPermaLink="false">https://medium.com/p/464928a8860d</guid>
            <category><![CDATA[ai-in-math]]></category>
            <category><![CDATA[ai-in-nepal]]></category>
            <category><![CDATA[gap-of-ai-and-maths]]></category>
            <dc:creator><![CDATA[Pratik Sharma]]></dc:creator>
            <pubDate>Sat, 18 Apr 2026 15:10:20 GMT</pubDate>
            <atom:updated>2026-05-08T12:11:02.075Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Personal Realization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*56GvgyPOD7uH_XL9OpqVrA.png" /></figure><p>I didn’t fall in love with math in school. I tolerated it.</p><p>Like many students from small-town Nepal, I memorized formulas, solved repetitive problems, and chased marks. Mathematics was never presented to me as a language of the universe — it was an obstacle course. You solved enough questions, you passed. You passed, you moved on.</p><p>But something strange happened when I started studying Artificial Intelligence and Machine Learning.</p><p>Suddenly, the same math I once found meaningless — linear algebra, probability, calculus — became alive. Concepts that once felt abstract now powered real systems: recommendation engines, facial recognition, predictive models. That “boring” equation on the classroom blackboard was now deciding what videos billions of people watch, how diseases are predicted, and how self-driving cars make decisions.</p><p>And that’s when it hit me:</p><p>We were never taught math. We were taught how to survive math exams.</p><h4><strong>The Problem: A Culture of Blind Study</strong></h4><p>Walk into a typical school in a town outside Kathmandu and you’ll notice a pattern. Students are busy, teachers are working hard, and yet something fundamental is missing.</p><p><strong>We have built a system of “blind study.”</strong></p><p>Maths, especially from Class 8 to 12, is treated as a collection of abstract puzzles. Students are trained to recognize patterns in questions, apply memorized steps, and produce answers quickly. The focus is not understanding — it is replication.</p><p>Ask a student why a formula works, and silence follows. Ask where it is used in the real world, and you’ll often get a guess — or nothing at all.</p><p>At the same time, students are burdened with “extra subjects” that promise marks but rarely deliver real-world relevance. They memorize definitions, write exams, and forget everything soon after. This overload creates a dangerous trade-off: depth is sacrificed for breadth.</p><p><strong>The result?</strong></p><p>- Students fear maths instead of exploring it<br>- Curiosity is replaced by compliance<br>- Creativity is suppressed by exam pressure</p><p>And perhaps most critically:</p><p>We produce graduates who can solve equations but cannot connect them to reality.</p><h4><strong>The Gap: From Classroom to AI</strong></h4><p>The world is moving fast. AI is not a distant concept anymore — it is already shaping economies, jobs, and global power structures.</p><p>Yet, there is a massive disconnect between what Nepali students learn and what the world demands.</p><p><strong>Let’s look at this gap more concretely:</strong></p><p>School Math Concept| What Students Think It Is| AI/ML Reality<br>Linear Equations| Solve for x| Neural network weights adjusting predictions<br>Matrices| Arrange numbers in rows| Core of deep learning computations<br>Probability| Exam questions with dice| Predicting user behavior, risk modeling<br>Derivatives| Slopes of curves| Optimization in training AI models<br>Statistics| Mean, median, mode| Data analysis and decision-making systems</p><p>This table reveals the truth:</p><p>The maths is not wrong. The context is missing.</p><p>We are teaching the tools without explaining the purpose.</p><h4><strong>The Solution: A “Bridge Handbook”</strong></h4><p>If we want to fix this gap, we don’t need to rewrite the entire curriculum overnight. What we need is a bridge — something that connects what students already learn with what the modern world actually uses.</p><p>I propose a simple but powerful idea:</p><p><strong>The “Bridge Handbook”</strong></p><p>A supplementary guide designed for students from Class 8 to 12 that translates traditional maths concepts into real-world AI and technology applications.</p><p>This handbook would not replace textbooks. Instead, it would answer one crucial question that our system ignores:</p><p><strong>“Why does this matter?”</strong></p><p>What Would It Include?</p><p>- Concept Translation<br> <br> — Linear Equations → How Netflix recommends movies<br> — Probability → How spam filters work<br> — Vectors → How Google Maps calculates directions</p><p>- Visual Explanations<br> Diagrams showing how equations interact inside real systems.</p><p>- Mini Projects<br> Simple exercises like:<br> <br> — Predicting exam scores using basic regression<br> — Understanding bias in data</p><p><strong>- Local Context<br> </strong><br> — Predicting crop yield<br> — Traffic flow in cities like Butwal or Kathmandu</p><p>The goal is not to turn every student into an AI engineer.</p><p>The goal is to make them see maths differently.</p><p><strong>Teacher Advocacy: The Real Backbone</strong></p><p>It’s easy to criticize the system. It’s harder — and more honest — to understand its constraints.</p><p>Teachers in rural and semi-urban Nepal are not the problem.</p><p>In fact, many of them are doing incredible work with limited resources, outdated materials, and overwhelming workloads.</p><p>If students are stuck in “blind study,” it’s not because teachers lack dedication — it’s because they lack support.</p><p><strong>What Do Teachers Need?</strong></p><p><strong>1. Access to Modern Resources</strong><br> Updated materials that connect curriculum with real-world applications.</p><p><strong>2. Training in a Research-Oriented Mindset</strong><br> Instead of teaching fixed answers, teachers should be empowered to encourage exploration:<br> <br> — Asking “what if?”<br> — Encouraging experimentation<br> — Connecting subjects across disciplines</p><p><strong>3. Community and Collaboration</strong><br> Platforms where teachers can share ideas, lesson plans, and innovations.</p><p><strong>4. Recognition and Incentives</strong><br> Rewarding teachers who innovate, not just those who complete the syllabus.</p><p>If we expect teachers to inspire curiosity, we must first equip them with the tools to do so.</p><h4><strong>Rethinking Education: From “How” to “Why”</strong></h4><p>Our current system is obsessed with “how.”</p><p>- How to solve this equation<br>- How to get full marks<br>- How to complete the syllabus</p><p><strong>But the future belongs to those who understand “why.”</strong></p><p>- Why does this model work?<br>- Why does this pattern exist?<br>- Why does this solution matter?</p><p><strong>When students understand why, they don’t just memorize — they internalize.</strong></p><p><strong>And once they internalize, they innovate.</strong></p><p><strong>A Patriotic Responsibility</strong></p><p>This is not just an educational issue. It is a national one.</p><p>Nepal stands at a crossroads. We have a young population, increasing internet access, and growing exposure to global technology. But without reform, we risk becoming consumers of innovation rather than creators of it.</p><p>Do we want Nepali students to only use AI tools built elsewhere?</p><p>Or do we want them to build the next generation of those tools?</p><p>The answer depends on how we teach math today.</p><h4><strong>A Call to Action</strong></h4><p>To the Ministry of Education, local school boards, and policymakers:</p><p>This is not a call for radical disruption. It is a call for intentional evolution.</p><p>- Introduce supplementary resources like the Bridge Handbook<br>- Train teachers to connect theory with application<br>- Encourage project-based learning alongside exams<br>- Reduce emphasis on unnecessary “extra subjects” that dilute focus</p><p><strong>Most importantly:</strong></p><p>Shift the system from “What is the answer?” to “Why does this matter?”</p><p>Because the truth is simple:</p><p>We did not fail to teach math.</p><p>We failed to teach its meaning.</p><p>And in an age where math powers intelligence — both human and artificial — that failure is no longer acceptable.</p><h4><strong>Final Thought</strong></h4><p>Somewhere in a classroom in Butwal, a student is staring at an equation, wondering why it exists.</p><p>If we do nothing, they will memorize it, pass the exam, and forget it forever.</p><p>But if we act — if we bridge the gap — they might see something else:</p><p>Not just numbers on a page, but a language that can shape the future.</p><p>And maybe, just maybe, they’ll fall in love with math before AI forces them to.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=464928a8860d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>