<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yubrajparajuli on Medium]]></title>
        <description><![CDATA[Stories by Yubrajparajuli on Medium]]></description>
        <link>https://medium.com/@yubrajparajuli?source=rss-411455780bff------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IX04nqXoFXMC3n7jm9hQYQ.jpeg</url>
            <title>Stories by Yubrajparajuli on Medium</title>
            <link>https://medium.com/@yubrajparajuli?source=rss-411455780bff------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:35:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yubrajparajuli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[7 Changes, 1 Winner: What Actually Fixes an Overfitting CNN]]></title>
            <link>https://medium.com/@yubrajparajuli/7-changes-1-winner-what-actually-fixes-an-overfitting-cnn-7b2cb208932b?source=rss-411455780bff------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b2cb208932b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yubrajparajuli]]></dc:creator>
            <pubDate>Sat, 05 Sep 2026 05:49:58 GMT</pubDate>
            <atom:updated>2026-09-05T05:49:58.491Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lGmOmkTG23vJxmwE0MgoSw.png" /></figure><p>By epoch seven, my model hit 87.5 percent training accuracy. Impressive, until you look closer.</p><p>Validation accuracy had already peaked at 75.25 percent, three epochs earlier, and was now falling.</p><p>It wasn’t learning cats and dogs. It was memorizing photographs.</p><p>Seven experiments later, I found the one change that actually mattered.</p><h3>The Setup</h3><p>This project started as a course assignment. Build a cat versus dog image classifier using <strong>TensorFlow</strong> and <strong>Keras</strong> , trained on <strong>Google Colab </strong>with GPU access<strong>.</strong></p><p>Simple enough on paper.</p><p>But my tutor, Angat Sitaula, pushed it further than just training a model and submitting it.</p><p>We were asked to treat this as a real experiment. Change one thing at a time, padding, stride, kernel size, color mode, data augmentation, and compare the results properly before picking a final model.</p><p>The dataset was the <a href="https://www.kaggle.com/datasets/tongpython/cat-and-dog"><strong>Cat and Dog dataset</strong></a><strong> </strong>from Kaggle, roughly eight thousand labeled images split into training and test sets.</p><p><em>Before writing a single line of model code, the data itself deserved a closer look.</em></p><h3>Exploring the Data First</h3><p>Before touching any model code, I ran a proper exploratory data analysis, or <strong>EDA</strong>.</p><p>Partly because it is good practice. Partly because it ended up predicting one of my results before I even trained a single model.</p><h4>A first look at the dataset</h4><p>Before any numbers, it helps to actually see what the model would be looking at.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QC9jk0EerlxfDAGeJDiPCg.png" /><figcaption>A random sample of cat images from the training set, varied backgrounds, poses, and lighting.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9rIkIla80UuNTpX5Pp9w2Q.png" /><figcaption>A random sample of dog images from the training set, same variety, same real world messiness.</figcaption></figure><p>Nothing unusual here. Real photos, real backgrounds, some blurry, some perfectly framed. Exactly the kind of variety a model needs to learn from if it is going to generalize.</p><h4>The split</h4><p>The full dataset totaled just over ten thousand images. I split it into three parts.</p><p>6,805 images for training. 1,200 for validation. 2,023 for testing, kept completely untouched until the very end.</p><p>That last part matters more than it sounds. More on why, shortly.</p><h4>Class balance</h4><p>Roughly even. About 3,400 cats and 3,405 dogs in training, with a similar balance in validation and test. No special handling needed for imbalanced classes.</p><h4>Image sizes</h4><p>Wildly inconsistent. Sizes ranged from 41 by 57 pixels all the way up to 1023 by 768, with a strange spike around 500 pixels, likely leftover from however the dataset was originally compiled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pgdy-o8m-QZ92rjOIJQwXg.png" /><figcaption>Width and height distributions across the training set, wide spread with a hard spike near 500 pixels.</figcaption></figure><p>This confirmed one thing early. Every image would need to be resized to a fixed size before training.</p><p>I settled on <strong>150 by 150 </strong>pixels, small enough to train quickly, large enough to preserve real detail.</p><h4>Pixel intensity</h4><p>This is where things got interesting.</p><p>I plotted the pixel value distributions for cats and dogs separately, expecting maybe some difference in brightness or contrast.</p><p>Instead, the two distributions were almost identical. Same range, nearly the same average value.</p><blockquote>If cats and dogs share nearly the same pixel intensity distribution, color itself is probably not what separates them. Shape and texture likely matter more.</blockquote><p>That hunch became a real experiment later in the project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GDnhM9jqIp579zIBRJxTQQ.png" /><figcaption>Pixel intensity distributions for cats versus dogs, nearly identical across the full 0 to 255 range.</figcaption></figure><h4>A quick look at augmentation</h4><p>I also previewed what <strong>data augmentation</strong> actually looks like on a real photo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t32vce1QERq-HDHGLxXYmg.png" /><figcaption>The same cat photo run through six augmentation types: original, horizontal flip, rotation, zoom, contrast, and translation.</figcaption></figure><p>Seeing each transformation labeled and separated made it obvious why augmentation works, long before I saw a single number back it up.</p><p><em>With the data understood, it was time to build something and watch it fail.</em></p><h3>The Baseline: A Textbook Case of Overfitting</h3><p>With the data ready, I built the simplest reasonable architecture. Three blocks of <strong>Conv2D</strong>, <strong>ReLU</strong>, and <strong>MaxPooling</strong>, filters growing from 32 to 64 to 128 as the network went deeper.</p><p>After the three blocks, a <strong>Flatten</strong> layer, then a single Dense neuron with a <strong>sigmoid</strong> activation to output one probability. Cat or dog.</p><p>About 135,000 parameters in total. Nothing fancy.</p><pre>early_stop_baseline = keras.callbacks.EarlyStopping(<br>    monitor=&#39;val_loss&#39;,<br>    patience=3,<br>    restore_best_weights=True<br>)<br><br>baseline_model = build_model(padding=&#39;same&#39;, stride=1, kernel_size=3)<br><br>history_baseline = baseline_model.fit(<br>    train_ds,<br>    validation_data=val_ds,<br>    epochs=25,<br>    callbacks=[early_stop_baseline]<br>)</pre><p>I trained this with <strong>early stopping</strong> watching validation loss, with a <strong>patience</strong> of three epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*45cqxbGrx-kEiEBNYZaMGw.png" /><figcaption>Training accuracy climbing past 87 percent while validation accuracy peaks early at epoch four, then slides backward as validation loss climbs.</figcaption></figure><p>Training ran for seven epochs before stopping itself. The best result actually came earlier, at epoch four.</p><p>The pattern was still clear as overfitting, just less extreme than it would have been left unchecked.</p><p>Training accuracy climbed steadily, 58 percent, then 70, then 79, all the way to 87.5 percent by epoch seven.</p><p>Validation accuracy told a different story.</p><p>It peaked around epoch four at 75.25 percent, then started slipping backward even as training accuracy kept climbing.</p><p>Validation loss made the pattern unmistakable. It bottomed out at 0.51 on epoch four, then rose steadily to 0.72 by epoch seven, even as training loss kept falling.</p><blockquote>Rising validation loss while training loss keeps falling is the clearest possible sign a model has stopped learning and started memorizing.</blockquote><p>This became the number to beat.</p><p>If you have reached this point now you can officially say that you know how Convolutional Layers work.</p><p>Every experiment from here on was really one question asked seven different ways.</p><p>Could I close this gap?</p><p><em>Before running a single experiment, I needed rules. Otherwise, comparing results would mean nothing.</em></p><h3>The Rules of the Experiment</h3><p>This is going to be a long post, so feel free to take a coffee break now. Still with me?</p><p>Awesome. Let’s get into how the experiments were actually run.</p><p>Before running anything else, I set a few strict rules. Otherwise, comparing results across experiments would mean nothing.</p><p><strong>One variable at a time</strong></p><p>Every experiment changed exactly one thing, padding, stride, kernel size, color, or augmentation.</p><p>Everything else stayed locked to the same baseline settings. Same padding, same stride, same kernel size, same batch size of 32.</p><p>Change two things at once, and you can never be sure which one actually caused a result to shift.</p><p><strong>Early stopping, every time</strong></p><p>Every single run used <strong>early stopping</strong>, watching validation loss with a patience of three to five epochs, restoring the best weights once training stopped.</p><p>This mattered more than it might seem. Without it, comparing two models trained for a fixed number of epochs risks comparing one model at its best moment against another model well past its peak, already sliding into overfitting.</p><p>Comparing models fairly means comparing each one at its best moment, not at whatever epoch happens to be last.</p><p><strong>The validation set decides, the test set stays untouched</strong></p><p>All seven experiments were judged using the <strong>validation set</strong> (a portion of data held back from training, used to check performance during experimentation).</p><p>The <strong>test set</strong> (2,023 images, completely separate from anything used during training or comparison) was saved for exactly one moment. The very end, on the single model that won.</p><p>Here is why that separation matters. If you keep checking your test set while comparing experiments, you are quietly letting that data influence your choices.</p><p>By the time you report a final number, it is no longer a fair, unseen measurement.</p><p><em>With the rules set, it was time to start changing things one at a time and see what actually moved the needle.</em></p><h3>Testing the Architecture</h3><p>With the rules set, I ran through four architectural changes first. Padding, stride, kernel size, and color mode.</p><p>None of them came close to fixing the overfitting gap. But each one taught something worth knowing.</p><h4>Padding: same versus valid</h4><p><strong>Padding</strong> controls what happens at the edges of an image during convolution.</p><p><strong>Same</strong> padding adds a border of zeros so the output stays the same size as the input.</p><p><strong>Valid</strong> padding adds nothing, so the output shrinks slightly with every convolution.</p><p>Switching from same to valid barely moved the numbers. <strong>75.83</strong> percent validation accuracy against the baseline’s <strong>75.25</strong> percent, with validation loss actually improving slightly, <strong>0.503</strong> against<strong> 0.511</strong>.</p><p>With only three convolutional layers, the difference in how border pixels get handled just was not large enough to matter.</p><p>I would expect this gap to widen in a much deeper network, where the effect compounds across many more layers.</p><h4><strong>Stride: 1 versus 2</strong></h4><p><strong>Stride</strong> controls how far the kernel moves between scans. A stride of 1 checks nearly every position. A stride of 2 skips every other position, scanning more coarsely.</p><p>Stride 2 matched baseline accuracy almost exactly, <strong>76.00</strong> percent, with a nearly identical validation loss of <strong>0.508</strong> against the baseline’s <strong>0.51</strong>1, while using about <strong>30</strong> percent fewer parameters, 93,761 compared to the baseline’s 134,721.</p><p>That is a genuinely useful trade. Same accuracy, same loss, smaller and faster model. Worth remembering for anyone working with limited compute.</p><h4>Kernel size: 3x3 versus 5x5</h4><p>A <strong>kernel</strong> is the small grid of numbers sliding across the image, looking for patterns. A larger kernel sees a wider neighborhood of pixels per scan, at the cost of more parameters to learn.</p><p>Doubling the kernel size to 5x5 more than doubled the parameter count, <strong>300,097</strong> against the baseline’s <strong>134,721</strong>.</p><p>Neither accuracy nor loss improved. <strong>75.50</strong> percent accuracy, essentially tied with baseline, and a validation loss of <strong>0.518</strong>, slightly worse than the baseline’s <strong>0.511</strong>.</p><p>For 150 by 150 images of subjects that mostly fill the frame, the wider receptive field just did not add anything useful. More compute, no real gain, and if anything, a marginally worse fit.</p><h4>Color: RGB versus greyscale</h4><p>This is the one the earlier pixel intensity analysis predicted.</p><p>Converting images to <strong>greyscale</strong>, stripping color information and keeping only brightness, actually outperformed the RGB baseline on both counts. <strong>77.25</strong> percent accuracy versus <strong>75.25 </strong>percent, and a lower validation loss too, <strong>0.498</strong> against <strong>0.511</strong>, using nearly the same number of parameters.</p><blockquote>The model was learning from shape and texture all along, not color. The data had already hinted at this before a single greyscale image was ever trained on.</blockquote><p>A satisfying result. The hypothesis formed during EDA held up under an actual experiment, and held up on both metrics, not just one.</p><h4>The scoreboard so far</h4><p>Here is how all four architectural experiments stacked up against the baseline.</p><pre>Experiment                        Val Accuracy   Val Loss   Parameters<br>======================================================================<br>Baseline (same, stride 1, RGB)    75.25%         0.511      134,721<br>Padding = valid                   75.83%         0.503      130,241<br>Stride = 2                        76.00%         0.508      93,761<br>Greyscale                         77.25%         0.498      134,145</pre><p>Every number sitting within about <strong>two</strong> percentage points of baseline. Statistically, this is closer to noise than to a real effect, aside from the greyscale result, which at least had a hypothesis behind it.</p><p><em>Four architectural changes down, and none of them meaningfully closed the overfitting gap. The next experiment did.</em></p><h3>The Real Winner: Data Augmentation</h3><p>Four experiments in, and nothing had meaningfully closed the overfitting gap from the baseline.</p><p>This one did.</p><p><strong>Data augmentation</strong> works by showing the model a slightly different version of each training image every epoch, a random flip here, a rotation there, a shift in zoom or contrast.</p><p>The underlying photo never changes, but the model never sees the exact same pixels twice.</p><pre>data_augmentation = keras.Sequential([<br>    layers.RandomFlip(&quot;horizontal&quot;),<br>    layers.RandomRotation(0.15),<br>    layers.RandomZoom(0.15),<br>    layers.RandomContrast(0.15),<br>])</pre><p>This layer sits right at the front of the model, applied only during training, never during validation or testing.</p><p>The result was the biggest jump of the entire project. Validation accuracy climbed from the baseline’s <strong>75.25 </strong>percent to <strong>81.83</strong> percent, a gain no architectural tweak came close to matching.</p><p>Validation loss told the same story. Down to <strong>0.401</strong>, the lowest of every experiment run so far.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q1o5E9mB31rZAh9HHOsLIg.png" /><figcaption>Training and validation accuracy tracking each other closely throughout training, instead of splitting apart like the baseline did.</figcaption></figure><p>Look at how close the two lines stay for almost the entire run. That is the real story here, not just a higher number, but a model that stopped memorizing and started generalizing.</p><blockquote>Augmentation did not change how the network processes images. It changed what the network was allowed to memorize in the first place.</blockquote><p>Here is the part that surprised me most. Training accuracy under augmentation actually ended up lower than the baseline’s training accuracy.</p><p>That sounds like a bad thing. It is not.</p><p>The baseline hit high training accuracy by memorizing specific photos. Augmentation made memorization impossible, since every epoch showed a slightly different version of every image.</p><p>Lower training accuracy here was not a weakness, it was proof the model could no longer take shortcuts.</p><p>Validation accuracy, the number that actually reflects real world performance, is what improved. That is the number that matters.</p><p><em>One clear winner so far. But one more experiment was still left, and it did not go the way I expected.</em></p><h3>The Surprising Loser: GlobalAveragePooling</h3><p>With augmentation identified as the clear winner, I wanted to test one more idea on top of it.</p><p>Every model so far ended with a <strong>Flatten</strong> layer followed by a Dense neuron. Flatten takes the final feature grid and unrolls it into one long list of numbers, keeping every single value, including exactly where in the image each feature was detected.</p><p><strong>GlobalAveragePooling</strong>, or <strong>GAP</strong>, does something more aggressive.</p><p>Instead of keeping every value, it averages each entire feature map down to a single number. Used in well known architectures like ResNet, it is often praised for being lighter and more efficient than Flatten.</p><pre>x = layers.GlobalAveragePooling2D()(x) <br>outputs = layers.Dense(1, activation=&#39;sigmoid&#39;)(x)</pre><p>The efficiency claim held up immediately.</p><p>Swapping Flatten for GAP cut the parameter count by about <strong>30</strong> percent, down to <strong>93,377</strong> from the augmentation model’s <strong>134,721</strong>.</p><p>The accuracy claim did not hold up at all.</p><p>Validation accuracy dropped to <strong>70.83</strong> percent, the worst result of all seven experiments, well below even the plain augmentation run it was built on top of.</p><p>Validation loss finished at <strong>0.556</strong>, the highest final loss of any experiment, more than a full point worse than augmentation’s <strong>0.401.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M__nFXQT9QS-7jmaTb9KCg.png" /><figcaption>Training and validation accuracy both climbing, but plateauing far lower than the Flatten based augmentation model.</figcaption></figure><p>Fewer parameters is not the same thing as a better model. It depends entirely on what information those parameters were protecting.</p><p>My best explanation comes down to network depth.</p><p>GAP works well in architectures with many convolutional layers, where features have already become highly abstract by the time pooling is applied.</p><p>This network only had three convolutional blocks.</p><p>Apparently, at that shallow depth, the model still needed to know where a feature was located in the image, not just how much of it was present on average.</p><p>GAP threw away exactly the information the final Dense layer needed.</p><p>A good reminder that techniques used in famous architectures are not automatically the right choice for every architecture.</p><blockquote>Context matters more than reputation.</blockquote><p><em>Six experiments run, one clear winner identified. Only one thing was left to do, put it to the real test.</em></p><h3>Final Test: The Number That Actually Counts</h3><p>Every experiment so far had been judged on the validation set, deliberately.</p><p>Seven configurations compared fairly, without the test set ever entering the picture</p><p>Now, with augmentation confirmed as the winner, came the one moment the test set was built for.</p><p>2,023 images. Never seen during training. Never checked during any of the six experiments. Used exactly once, right here.</p><pre>test_loss, test_accuracy = model_aug.evaluate(test_ds)<br>print(f&quot;Test accuracy: {test_accuracy:.4f}&quot;)<br><br>y_pred = (model_aug.predict(test_ds).flatten() &gt; 0.5).astype(int)<br>print(classification_report(y_true, y_pred, target_names=[&#39;cats&#39;, &#39;dogs&#39;]))</pre><p>One detail worth explaining.</p><p>The model does not output “cat” or “dog” directly. It outputs a single probability between 0 and 1, thanks to the <strong>sigmoid</strong> activation on that final neuron.</p><p>To turn that probability into an actual label, I applied a <strong>threshold</strong> (the cutoff point used to convert a probability into a final decision) of 0.5.</p><p>Anything above 0.5 got classified as a dog, anything below as a cat.</p><p>A simple, standard choice, and a reasonable one here given the classes were balanced.</p><p>In situations with imbalanced classes or where one type of mistake matters more than the other, that threshold is often worth tuning instead of leaving at the default midpoint.</p><p>The result. 83.44 percent test accuracy.</p><p>Slightly higher than the 81.83 percent seen on validation, which is a genuinely good sign.</p><p>It means the model was not just tuned to look good on validation data, it actually generalizes to images it has never encountered in any form.</p><p><strong>Precision, recall, and what they actually mean</strong></p><p>Accuracy alone does not tell the whole story, so I also checked <strong>precision</strong> and <strong>recall</strong> for each class.</p><p><strong>Precision</strong> answers one question. Of everything the model called a cat, how many actually were cats.</p><p><strong>Recall</strong> answers a different question. Of all the actual cats in the test set, how many did the model correctly catch.</p><pre>              precision    recall  f1-score   support<br>------------------------------------------------------<br>cats             0.84        0.82      0.83      1011<br>dogs             0.83        0.84      0.84      1012<br>------------------------------------------------------<br>accuracy                               0.83      2023</pre><p>Both scores landed in the 0.82 to 0.84 range for each class, cats and dogs performing almost identically.</p><p>No hidden bias toward one class over the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*tfbjbViXb30sYGZkL1Z5LA.png" /><figcaption>834 cats and 854 dogs correctly identified, with 177 cats and 158 dogs misclassified, a balanced error rate across both classes.</figcaption></figure><p>The confusion matrix makes the balance obvious at a glance.</p><p>Errors happened, but they happened roughly equally in both directions, not concentrated on one class the model struggled with.</p><p><em>Seven experiments, one clear winner, and a final number that held up under real testing. Here is what the whole process actually taught me.</em></p><h3>What I Actually Learned</h3><p>Numbers aside, a few lessons from this project stuck with me more than the results themselves.</p><p><strong>Training accuracy can lie to you</strong></p><p>The baseline’s climbing training accuracy looked impressive in isolation. It meant almost nothing.</p><blockquote>Validation and test performance are what actually reflect a model’s real ability.</blockquote><p><strong>Data beats architecture, at least here</strong></p><p>None of the architectural tweaks, padding, stride, kernel size, came close to what augmentation achieved. None of them addressed the actual problem. Augmentation did.</p><p><strong>Exploration is not just a formality</strong></p><p>The pixel intensity analysis predicted the greyscale result before a single training epoch ran.</p><p>Exploring data properly is not a box to check before the real work starts, it often is the real work.</p><p><strong>Modern is not automatically better</strong></p><p>GlobalAveragePooling is a legitimate, widely used technique. Blindly swapping it in still hurt performance here.</p><p>Context, network depth, dataset size, matters more than following a trend uncritically.</p><p>Here is the complete picture, every experiment run across this project, side by side.</p><pre>Experiment                              Val Accuracy   Val Loss   Parameters<br>==============================================================================<br>Baseline (same, stride 1, RGB)          75.25%         0.511      134,721<br>Padding = valid                         75.83%         0.503      130,241<br>Stride = 2                              76.00%         0.508      93,761<br>Greyscale                               77.25%         0.498      134,145<br>Data augmentation                       81.83%         0.401      134,721<br>Kernel size = 5x5                       75.50%         0.518      300,097<br>GlobalAveragePooling + augmentation     70.83%         0.556      93,377</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gINiQqUOAngGFyho4Kx6cA.png" /><figcaption>Validation accuracy across all seven experiments. Augmentation stands well above every other configuration, while GlobalAveragePooling falls well below.</figcaption></figure><p>No project is without gaps, and this one is worth being honest about</p><h3>Limitations</h3><p>A few honest caveats worth stating clearly.</p><ol><li>Each experiment was a single training run. Small differences, like padding same versus valid sitting within about one percentage point, are plausibly just random variation from weight initialization, not necessarily a real effect.</li><li>Only the augmentation result was large enough to be a confident finding. A jump of five to six percentage points on a single run is hard to dismiss as noise.</li><li>No cross validation was used, mainly due to time constraints, a natural next step if I revisit this project.</li></ol><p><em>That covers the project itself, worth reading regardless</em>.</p><h3>Full Project</h3><p>Every experiment, the full notebook, and all results are available on <a href="https://github.com/yubrajparajuli/cat-dog-cnn"><strong>GitHub</strong></a>.</p><p>A few thank yous before wrapping up.</p><h3>Acknowledgement</h3><p>I am currently learning AI and Data Science under the guidance of Angat Sitaula. This project was built as part of my learning journey. Thank you for the direction, motivation and support.</p><h3>About the Author</h3><p>I am Yubraj Parajuli, a Computer Engineer from Nepal building things with AI and Data Science.</p><blockquote>I write about real projects, real data and real mistakes I made along the way, so you don’t have to.</blockquote><p>Connect on <a href="http://www.linkedin.com/in/yubraj-parajuli">LinkedIn</a> or follow on <a href="https://github.com/yubrajparajuli">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b2cb208932b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Opened the Black Box. Built Linear Regression from Scratch on Real Housing Data.]]></title>
            <link>https://medium.com/@yubrajparajuli/i-opened-the-black-box-built-linear-regression-from-scratch-on-real-housing-data-f69d46086685?source=rss-411455780bff------2</link>
            <guid isPermaLink="false">https://medium.com/p/f69d46086685</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yubrajparajuli]]></dc:creator>
            <pubDate>Fri, 05 Jun 2026 05:53:36 GMT</pubDate>
            <atom:updated>2026-06-05T06:45:26.394Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ydkRo2jA8XaIDGNPoQDezA.png" /></figure><blockquote>Every ML tutorial tells you to import sklearn and call .fit(). Nobody tells you what happens inside.</blockquote><p>That bothered me.</p><p>So I took 21,613 real house sales from King County, Washington and built Linear Regression completely from scratch using only NumPy.</p><p><strong>No sklearn for modeling. Just math, matrices and clean Python.</strong></p><p>Then I compared my scratch model against sklearn to see if I actually understood what I was doing.</p><p>Here is everything I did, from raw data to a deployed web app.</p><h3>The Dataset</h3><p>I used the King County House Sales dataset from <a href="https://www.kaggle.com/datasets/harlfoxem/housesalesprediction">Kaggle</a>.</p><p>21,613 real house sales. 21 features. Washington state. 2014 to 2015.</p><p>Features cover everything that drives house value. Size, location, condition, grade, waterfront access, number of floors and more.</p><p><strong>Target variable</strong>: Price ranging from $75,000 to $7.7 million.</p><p>This is real world data. Not synthetic. Not cleaned up for a tutorial. That means outliers, skewed distributions and features that need engineering.</p><p>Exactly the kind of data worth learning on.</p><p>Before we could model anything, the data needed work. Here is what we found.</p><h3>Data Cleaning</h3><p>Raw data is never ready. Here is what we found and fixed.</p><p>First I inspected the data. 21,613 rows, 21 columns, zero missing values and zero duplicates. A clean start.</p><p>But several columns needed attention.</p><p><strong>id </strong>was just an index. <strong>zipcode</strong> had 70 unique values with no ordinal meaning. Both dropped immediately.</p><p><strong>date</strong> came as a string like 20141013T000000. We extracted <strong>sale_year</strong> and <strong>sale_month</strong> from it and dropped the original column.</p><p>One row had 33 bedrooms. The dataset mean is 3.37. That is a typo, not a house. Removed.</p><p>All column names were lowercased, stripped and spaces replaced with underscores. Small detail. Big difference when you have 20 features to type repeatedly.</p><p>Here is the full cleaning pipeline in one block:</p><pre># drop unnecessary columns<br>df.drop(columns=[&#39;id&#39;, &#39;zipcode&#39;], inplace=True)<br><br># extract date features<br>df[&#39;sale_year&#39;]  = df[&#39;date&#39;].str[:4].astype(int)<br>df[&#39;sale_month&#39;] = df[&#39;date&#39;].str[4:6].astype(int)<br>df.drop(columns=[&#39;date&#39;], inplace=True)<br><br># remove bedrooms outlier<br>df = df[df[&#39;bedrooms&#39;] != 33]<br><br># engineer house_age and is_renovated<br>df[&#39;house_age&#39;]    = 2015 - df[&#39;yr_built&#39;]<br>df[&#39;is_renovated&#39;] = (df[&#39;yr_renovated&#39;] != 0).astype(int)<br>df.drop(columns=[&#39;yr_built&#39;, &#39;yr_renovated&#39;], inplace=True)<br><br># clean column names<br>df.columns = df.columns.str.lower().str.strip().str.replace(&#39; &#39;, &#39;_&#39;)</pre><p>Result: 21,612 rows. 20 clean features. Ready for exploration.</p><p>With the data clean, it was time to understand what it was telling us.</p><h3>Exploratory Data Analysis</h3><p>Before building any model, we need to understand the data. EDA is not optional. It is where intuition is built.</p><h4>Price Distribution</h4><p>First thing we always check, how is the target variable distributed?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qacFpcfl5P5bBSGFXCaiNg.png" /><figcaption>Raw price is right skewed. Log transform brings it to a near normal distribution.</figcaption></figure><p>Raw price is strongly right skewed. Most houses sit between $200k and $800k with a long tail stretching to $7.7 million.</p><p>The log transform brings it to a near normal bell curve.</p><p><strong>We trained the model on log price, not raw price</strong>. This stabilizes variance and significantly improves model performance. At prediction time we simply reverse it with np.expm1().</p><p><strong>What Actually Drives Price?</strong></p><h4>Correlation Heatmap</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WZ0opwYVjIegwW2Jk1U7HA.png" /><figcaption>Living area and grade are the strongest predictors of price at 0.70 and 0.67.</figcaption></figure><p>The strongest correlations with price are <strong>living area</strong> at 0.70, <strong>grade</strong> at 0.67, <strong>above ground area</strong> at 0.61, <strong>neighbor living area </strong>at 0.59 and <strong>bathrooms</strong> at 0.53.</p><p>One red flag is <strong>living area </strong>and <strong>above ground area</strong> had a correlation of 0.88 with each other. That is <strong>multicollinearity</strong>.</p><p>We handled it in feature engineering.</p><blockquote>Size and quality dominate. Location follows. This is real estate in one sentence.</blockquote><h4>Price vs Features</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfsZv38qxpPbfNgI0pLCng.png" /><figcaption>Clear positive linear relationships between price and top features.</figcaption></figure><p><strong>Living area</strong> shows a clear positive linear trend with <strong>price</strong>. Exactly what Linear Regression needs.</p><p><strong>Grade</strong> shows a strong stepwise increase. Higher grade means significantly higher price at every level.</p><p><strong>Bathrooms</strong> and <strong>neighbor living are</strong>a follow the same pattern. The signal is strong and consistent.</p><h4>Outlier Check</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hs2G279Bw40WGz1VrVnNTA.png" /><figcaption>Price and lot size have long upper tails. All other features are well distributed.</figcaption></figure><p><strong>Price</strong> has a long upper tail extending to $7.7 million. We kept them. Luxury houses are valid data points, not errors.</p><p><strong>Lot size</strong> has extreme values up to 1.6 million sqft. Also valid as large land plots exist in King County.</p><p>The only true outlier we removed was the single house with 33 <strong>bedrooms</strong>.</p><h4>Location, Condition and Grade</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WC64UuS4C03r9lPOurVk7Q.png" /><figcaption>Waterfront properties command 3x higher median price. Grade shows the strongest stepwise effect.</figcaption></figure><p><strong>Waterfront</strong> properties have a median price of $1.4 million versus $450k for non-waterfront. A 3x difference from a single binary feature.</p><p><strong>Grade</strong> tells an even cleaner story. Every <strong>grade</strong> level up means a significant jump in <strong>price</strong>.</p><p><strong>Condition </strong>surprisingly shows little effect. Buyers care more about <strong>size</strong> and <strong>grade</strong> than <strong>condition</strong>.</p><h4>The Geography of Price</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9WdfGiEmn_r93mM3Lmq17Q.png" /><figcaption>Northern King County commands premium prices. Location shapes every prediction.</figcaption></figure><p>This is the most visually striking insight in the entire analysis.</p><p>Northern King County commands higher prices. Expensive houses cluster in specific pockets.</p><blockquote>Location is not just a feature. It is the entire story of this dataset.</blockquote><p><strong>Latitude ended up as the strongest coefficient in our final model. Geography wins.</strong></p><h4>Feature Relationships at a Glance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xyXH2kP53jUjno1xQTfXRQ.png" /><figcaption>Living area, grade and neighbor living area show the strongest pairwise relationships with price.</figcaption></figure><p>The pairplot confirms everything we saw individually. <strong>Living area</strong>, <strong>grade</strong> and <strong>neighbor living are</strong>a all show strong positive relationships with <strong>price</strong>.</p><p>The diagonal histograms show right skewed distributions across all features. Consistent with real world housing data.</p><p>EDA gave us a clear picture. We knew exactly what features to keep, what to engineer and what to drop.</p><p>Time to prepare the data for modeling.</p><h3>Feature Engineering</h3><p>Clean data is not always ready data. We needed to create features that tell a better story than the raw columns.</p><h4>Handling Multicollinearity</h4><p>The correlation heatmap revealed a problem.</p><p>Living area and above ground area had a correlation of 0.88 with each other. Keeping both in the model means feeding it the same information twice. The model gets confused. Coefficients become unreliable.</p><p>We dropped above ground area.</p><p>Lot size and neighbor lot size had a correlation of 0.72. Same problem. Neighbor lot size dropped.</p><blockquote>Multicollinearity does not break your model. It quietly makes it worse. That is harder to catch.</blockquote><h4>Creating New Features</h4><p>Four new features were engineered from existing columns.</p><p><strong>Total Rooms</strong>, bedrooms plus bathrooms. A single number capturing overall house capacity without separating two highly correlated features.</p><p><strong>Living Area Ratio</strong>, living area divided by neighbor living area. A house that is larger than its neighbors commands a premium. This feature captures that relationship explicitly.</p><p><strong>House Age</strong>, 2015 minus year built. More intuitive and meaningful than a raw year for a regression model. A house built in 1950 means more as 65 years old than as the number 1950.</p><p><strong>Is Renovated</strong>, a binary flag derived from year renovated. Either a house was renovated or it was not. Simple. Clean. Meaningful.</p><pre>df[&#39;total_rooms&#39;]       = df[&#39;bedrooms&#39;] + df[&#39;bathrooms&#39;]<br>df[&#39;living_area_ratio&#39;] = df[&#39;sqft_living&#39;] / df[&#39;sqft_living15&#39;]<br>df[&#39;house_age&#39;]         = 2015 - df[&#39;yr_built&#39;]<br>df[&#39;is_renovated&#39;]      = (df[&#39;yr_renovated&#39;] != 0).astype(int)</pre><h4><strong>Log Price Target</strong></h4><p>Price is right skewed. We confirmed this in EDA.</p><p>Training a Linear Regression model on a skewed target is like asking it to learn from a distorted ruler. The model struggles with the long tail of luxury properties.</p><p>The fix is simple:</p><pre>df[&#39;log_price&#39;] = np.log1p(df[&#39;price&#39;])</pre><p>Log transform compresses the tail and brings the distribution close to normal. The model learns better. At prediction time we reverse it:</p><pre>actual_price = np.expm1(predicted_log_price)</pre><blockquote>A well chosen target variable is as important as the model itself.</blockquote><h4>Train Test Split and Standardization</h4><p>We split the data 80/20 using a fixed random state for reproducibility.</p><pre>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</pre><p>17,289 training samples. 4,323 test samples.</p><p>Then we standardized from scratch. No sklearn StandardScaler.</p><pre>train_mean = np.mean(X_train, axis=0)<br>train_std  = np.std(X_train, axis=0)<br><br>X_train_scaled = (X_train - train_mean) / train_std<br>X_test_scaled  = (X_test  - train_mean) / train_std</pre><p>Two rules here that matter:</p><p>First: always compute mean and std from training data only. Never touch test data statistics. That is data leakage.</p><p>Second: apply the same training statistics to the test set. The test set must be scaled with what the model learned from training.</p><blockquote>Standardization does not change what the data says. It changes how loud each feature speaks.</blockquote><p>Finally we added a bias term, a column of ones prepended to every row.</p><pre>X_train_final = np.c_[np.ones(X_train_scaled.shape[0]), X_train_scaled]<br>X_test_final  = np.c_[np.ones(X_test_scaled.shape[0]),  X_test_scaled]</pre><p>This allows the model to learn an intercept. Without it the regression line is forced through the origin. Almost always wrong.</p><p>19 clean engineered features. Properly scaled. Bias term added.</p><p>The data was ready. Time to build the model.</p><h3>The Math Behind Linear Regression</h3><p>Before writing a single line of model code, we need to understand what Linear Regression actually does under the hood.</p><blockquote>Most people skip this part. That is exactly why they treat machine learning as a black box.</blockquote><p>We are not doing that here.</p><h4>The Core Idea</h4><p>Everyone learned this in school:</p><pre>y = mx + c</pre><p>One input. One output. One straight line.</p><p><strong>m</strong> is the slope: how much y changes when x changes.</p><p><strong>c</strong> is the intercept: where the line crosses the y axis when x is zero.</p><p>Linear Regression is the same idea extended to multiple inputs:</p><pre>ŷ = w₀ + w₁x₁ + w₂x₂ + ... + wₙxₙ</pre><p>The hat on<strong> ŷ </strong>means predicted value. Not the actual price. The model’s best guess.</p><p>Each <strong>w</strong> is a weight , the equivalent of slope m but for each individual feature. w₀ is the bias term, the equivalent of intercept c.</p><p>In our case this equation has 19 terms. One weight for living area. One for grade. One for latitude. One for each of our 19 features.</p><p>In matrix form this becomes elegantly simple:</p><pre>ŷ = Xw</pre><p><strong>X</strong> is our feature matrix. Every row is one house. Every column is one feature. In our project X has shape (17,289 × 20), meaning 17,289 training houses and 20 columns including the bias term.</p><p><strong>w</strong> is our weight vector of shape (20,). One number per feature. These are what the model learns during training.</p><p>To make a prediction for a new house we simply multiply its feature row by the weight vector and sum everything up. That is the entire prediction in one line.</p><blockquote>Linear Regression is not magic. It is a weighted sum. Understanding that changes everything.</blockquote><p>The bias term <strong>w₀</strong> deserves special attention. We manually added a column of ones to <strong>X</strong> before training:</p><pre>X_train_final = np.c_[np.ones(X_train_scaled.shape[0]), X_train_scaled]</pre><p>Without it the model has no intercept. The regression hyperplane is forced through the origin. In almost every real world problem that is wrong.</p><p>One column of ones. One extra weight. Complete model.</p><h4>The Loss Function</h4><p>How do we know if our weights are good or bad? We measure the gap between what the model predicted and what actually happened.</p><p>We use Mean Squared Error:</p><pre>MSE = (1/n) Σ (y - ŷ)²</pre><p>Break this down term by term.</p><p>(y — ŷ) is the residual: the difference between the actual price and the predicted price for one house</p><p>Squaring does two things. It makes all residuals positive so positive and negative errors do not cancel each other out. And it penalizes large mistakes far more than small ones.</p><p>(1/n) averages the error across all houses so the metric does not grow just because we have more data.</p><p>In our project y is log price and ŷ is the model’s predicted log price. We compute this for all 4,323 test houses and average.</p><blockquote>The loss function is not just a metric. It is the signal the model learns from.</blockquote><p>A model that is slightly wrong on every house is better than a model that is catastrophically wrong on a few. MSE captures that preference mathematically.</p><p>The entire training process has one goal. Make this number as small as possible.</p><p>But how?</p><p>You can solve it in one mathematical stroke. Or you can find it step by step, iteration by iteration.</p><p>Two methods. Same destination. Completely different journeys.</p><h3>Two Ways to Minimize the Loss</h3><p>There are two fundamentally different approaches to finding the optimal weights. One is exact. One is iterative. Both arrive at the same answer.</p><h4>Normal Equation</h4><p>The first approach solves the problem directly. No iterations. No learning rate. Just pure linear algebra.</p><p>The Normal Equation gives us the optimal weights in one shot:</p><pre>w = (XᵀX)⁻¹ Xᵀy</pre><p>Break this down.</p><p>Xᵀ is the transpose of our feature matrix. XᵀX produces a square matrix of shape (20 × 20). We invert it. Then multiply by Xᵀy.</p><p>The result is the exact weight vector that minimizes MSE. Mathematically guaranteed.</p><p>In NumPy this is one line:</p><pre>self.weights = np.linalg.inv(X.T @ X) @ X.T @ y</pre><p>No loops. No hyperparameters. No convergence to worry about.</p><blockquote>The Normal Equation is the mathematician’s solution. Exact. Elegant. Ruthlessly efficient on small datasets.</blockquote><p>But there is a catch.</p><p>Matrix inversion scales at O(n³) with respect to the number of features. For 20 features this is instant. For 10,000 features this becomes a serious bottleneck.</p><p>This is where the second approach becomes essential.</p><h4>Gradient Descent</h4><p>Instead of solving the problem in one shot, Gradient Descent finds the optimal weights iteratively. One small step at a time, in the direction that reduces the loss.</p><p>Imagine standing on a hill blindfolded. You cannot see the bottom. But you can feel which direction slopes downward. So you take one small step downhill. Then another. Eventually you reach the lowest point.</p><p>That is Gradient Descent.</p><p>The weight update rule is:</p><pre>w = w - α * (1/n) * Xᵀ(Xw - y)</pre><p>Break this down.</p><p>(Xw — y) is the vector of residuals. How wrong are our current predictions.</p><p>Xᵀ(Xw — y) is the gradient. The direction of steepest increase in loss. We want to go the opposite direction.</p><p>α is the learning rate. How big each step is. Too large and we overshoot. Too small and convergence is painfully slow.</p><p>(1/n) normalizes the gradient across all training samples.</p><p>In NumPy:</p><pre>for i in range(self.n_iterations):<br>    y_pred   = X @ self.weights<br>    error    = y_pred - y<br>    gradient = (1 / n_samples) * X.T @ error<br>    self.weights -= self.lr * gradient</pre><p>We ran this for 1000 iterations with a learning rate of 0.01. Here is what the loss looked like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s3M5Ie-jUcHgFgVF8lb9Eg.png" /><figcaption><em>Loss drops sharply in the first 200 iterations then flattens as the model converges.</em></figcaption></figure><p>The loss starts at 85.24 and converges to 0.031 by iteration 900. A textbook perfect convergence curve.</p><blockquote>Gradient Descent is the engineer’s solution. Scalable. Flexible. The foundation of every deep learning model ever built.</blockquote><p><strong>Why Both?</strong></p><p>Normal Equation and Gradient Descent are not competing methods. They are complementary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZP0ZR6jrrThqOHiUvn0yCQ.png" /><figcaption>Normal Equation vs Gradient Descent. Two methods, one answer.</figcaption></figure><p>For our dataset of 21,612 houses and 19 features the Normal Equation is the right choice. Fast and exact.</p><p>But we implemented both deliberately. Because understanding Gradient Descent means understanding how neural networks, deep learning and modern AI actually work at their core.</p><p>Both models trained on identical data. Both evaluated on identical test samples. The results were identical. That was the point.</p><blockquote>Master the simple version first. The complex version is just more of the same.</blockquote><p>With the math fully understood it was time to write the code.</p><h3>Building from Scratch</h3><p>This is the section most tutorials skip. We are not skipping it.</p><p>We built two complete Linear Regression implementations using only NumPy. No sklearn. No shortcuts.</p><h4>Normal Equation from Scratch</h4><pre>class LinearRegressionNormalEquation:<br>    <br>    def __init__(self):<br>        self.weights = None<br><br>    def fit(self, X, y):<br>        self.weights = np.linalg.inv(X.T @ X) @ X.T @ y<br>        return self<br><br>    def predict(self, X):<br>        return X @ self.weights</pre><p>Three methods. Few lines. A complete Linear Regression model.</p><p><strong>fit</strong> computes the exact optimal weights using the Normal Equation. <strong>predict</strong> multiplies the feature matrix by the learned weights. That is the entire model.</p><blockquote>Sklearn’s LinearRegression does the same thing under the hood. Now you know what is inside.</blockquote><h4>Gradient Descent from Scratch</h4><pre>class LinearRegressionGradientDescent:<br><br>    def __init__(self, lr=0.01, n_iterations=1000):<br>        self.lr           = lr<br>        self.n_iterations = n_iterations<br>        self.weights      = None<br>        self.loss_history = []<br><br>    def fit(self, X, y):<br>        n_samples, n_features = X.shape<br>        self.weights = np.zeros(n_features)<br><br>        for i in range(self.n_iterations):<br>            y_pred   = X @ self.weights<br>            error    = y_pred - y<br>            gradient = (1 / n_samples) * X.T @ error<br>            self.weights -= self.lr * gradient<br><br>            loss = (1 / (2 * n_samples)) * np.sum(error ** 2)<br>            self.loss_history.append(loss)<br><br>        return self<br><br>    def predict(self, X):<br>        return X @ self.weights</pre><p>Every line maps directly to the math we covered in the previous section.</p><p><strong>np.zeros(n_features)</strong> initializes weights at zero. The loop runs 1000 times. Each iteration computes the gradient and takes one step in the direction that reduces loss.</p><p>The loss history is stored so we can visualize convergence later.</p><blockquote>Two classes. Pure NumPy. Complete from scratch implementations that match sklearn exactly.</blockquote><p>With both models built it was time to train them and see the results.</p><h3>Training and Results</h3><p>With 17,289 training samples ready, scaled and bias term added, we trained all three models on identical data.</p><p>Normal Equation solved it in one matrix operation. No loops. No hyperparameters. Instant.</p><pre>ne_model = LinearRegressionNormalEquation()<br>ne_model.fit(X_train_final, y_train)</pre><p>Gradient Descent took 1000 iterations with a learning rate of 0.01. Loss dropped from 85.24 at iteration 0 to 0.031 by iteration 900.</p><pre>gd_model = LinearRegressionGradientDescent(lr=0.01, n_iterations=1000)<br>gd_model.fit(X_train_final, y_train)</pre><p>Sklearn was included purely as a reference. If our scratch implementation is correct it should match Sklearn exactly.</p><pre>sk_model = LinearRegression()<br>sk_model.fit(X_train_final, y_train)</pre><p>Three models. Same data. Same features. Same split. Now let the numbers speak.</p><p>All three models were evaluated on 4,323 unseen test houses using four metrics built from scratch.</p><pre>def mse(y_true, y_pred):<br>    return np.mean((y_true - y_pred) ** 2)<br><br>def rmse(y_true, y_pred):<br>    return np.sqrt(mse(y_true, y_pred))<br><br>def mae(y_true, y_pred):<br>    return np.mean(np.abs(y_true - y_pred))<br><br>def r2(y_true, y_pred):<br>    ss_res = np.sum((y_true - y_pred) ** 2)<br>    ss_tot = np.sum((y_true - np.mean(y_true)) ** 2)<br>    return 1 - (ss_res / ss_tot)</pre><pre>Model                  MSE      RMSE     MAE      R²<br>Normal Equation        0.0652   0.2554   0.1962   0.7719<br>Gradient Descent       0.0654   0.2558   0.1964   0.7711<br>Sklearn                0.0652   0.2553   0.1962   0.7719</pre><p>R² of 0.77 means our model explains 77% of the variance in log price. For a Linear Regression model on real world data with 19 features that is a strong result.</p><p>RMSE of 0.26 in log scale means predictions are within roughly 26% of the actual price on average.</p><blockquote>Our from scratch implementation matched Sklearn exactly. That is not luck. That is proof the math was right.</blockquote><p>Numbers tell one story. Visuals tell a deeper one.</p><h3>Model Evaluation</h3><p>Numbers alone do not tell the full story. We need to see how the model behaves visually.</p><h4>Actual vs Predicted</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dTSRuQdfIdFSCl3qd_Rr-A.png" /><figcaption>Points follow the diagonal closely. Both models perform identically across the full price range.</figcaption></figure><p>If the model were perfect every point would sit exactly on the red diagonal line. The closer the scatter is to that line the better the model.</p><p>Predictions follow the diagonal closely across the full price range. No systematic bias. The model performs equally well on affordable houses and expensive ones.</p><p>Slight spread increases at higher prices. Luxury houses are harder to predict with a linear model. Expected and honest.</p><h4>Residual Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wz-O767EAcUrg4B_OecxRw.png" /><figcaption>Residuals are evenly distributed around zero. No systematic pattern. Linear regression assumptions are met.</figcaption></figure><p>A residual is the difference between the actual price and the predicted price for one house.</p><p>A good model has residuals that are randomly scattered around zero with no pattern. That is exactly what we see here.</p><p>No systematic over or under prediction. No funnel shape. No curve. Just random scatter around zero.</p><blockquote>When residuals have no pattern the model has captured everything linear in the data. What remains is noise.</blockquote><p>A few outliers beyond ±1.0 are luxury properties that are genuinely difficult to predict with a linear model. Not a flaw. A limitation of linear modeling on extreme values.</p><h4>What the Model Learned</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*46PJ9I3VIsAHPxA5Kmm0ZQ.png" /><figcaption>Latitude and grade dominate. Total rooms coefficient is negative by design</figcaption></figure><p>Latitude is the strongest positive predictor. Northern King County commands premium prices. Geography is not just a feature. It is the foundation of every prediction.</p><p>Grade follows closely. Building quality directly translates to price at every level. A grade 10 house is not just better than a grade 7. It is in a completely different price bracket.</p><p>Total rooms carries a negative coefficient. Surprising at first. But when living area is held constant more rooms means smaller individual rooms. Buyers pay for space not count.</p><p>House age is also strongly positive. Older houses in established King County neighborhoods hold and grow their value. Age here signals location maturity not deterioration.</p><blockquote>The model did not just learn to predict price. It learned the economics of King County real estate.</blockquote><p>The model worked. But every tool has its limits. Before deploying it we need to be honest about when Linear Regression should not be your first choice.</p><h4>When Not to Use Linear Regression</h4><p>Linear Regression is powerful. But only when the problem fits the assumptions.</p><p>If the relationship between features and target is curved, exponential or highly complex the model will consistently underfit. No feature engineering fixes a fundamentally nonlinear problem.</p><p>Extreme outliers are another weakness. MSE squares the errors. One massive outlier can pull the entire regression line toward itself and distort every prediction in the dataset.</p><p>Multicollinearity makes coefficients unreliable. The model still predicts but the weights lose their meaning. We handled this carefully in feature engineering. In datasets with hundreds of correlated features it becomes much harder.</p><p>And if your target is not continuous, Linear Regression has no place. Predicting categories, binary outcomes or probabilities needs a different tool entirely.</p><blockquote>Linear Regression is not a universal tool. It is a precision instrument. Use it where it fits.</blockquote><p>Knowing the limits of a tool is what separates a practitioner from someone who just runs code.</p><h3>Key Takeaways</h3><p>This project was never just about predicting house prices. It was about understanding what happens inside the model.</p><p>Here is what building from scratch taught us that no tutorial ever could.</p><p><strong>The math is simpler than you think</strong>. Linear Regression is a weighted sum. The Normal Equation is three matrix operations. Gradient Descent is a loop with five lines of NumPy. The complexity lives in understanding why, not in the code itself.</p><p><strong>Sklearn is not magic</strong>. It is optimized, tested and production ready. But it does exactly what we did. Our scratch implementation matched it to six decimal places. That is the proof.</p><p><strong>Data preparation is 80% of the work</strong>. We spent more time on cleaning, feature engineering and standardization than on the model itself. The model rewarded that investment with an R² of 0.77.</p><p><strong>Log transforming the target matters</strong>. Training on raw price versus log price is not a small tweak. It fundamentally changes what the model learns. Always check your target distribution before training.</p><p><strong>Residuals tell the real story.</strong> Metrics give you a number. Residual plots tell you whether the model is actually behaving correctly. Never skip them.</p><blockquote>Understanding one algorithm deeply is worth more than knowing ten algorithms superficially.</blockquote><p>The model is built. The lessons are learned. One thing left to do.</p><h3>Deployment</h3><p>The full project is live and open source.</p><p>Try the model yourself, explore the code or run it locally with Docker.</p><p><strong>Live App</strong> <a href="https://yubrajparajuli-house-price-prediction-appstreamlit-app-cvdjo3.streamlit.app/">House Price Predictor</a></p><p><strong>Docker Image</strong> <a href="https://hub.docker.com/r/yubraj101/house-price-prediction">yubraj101/house-price-prediction</a></p><p><strong>GitHub Repository</strong> <a href="https://github.com/yubrajparajuli/house-price-prediction">github.com/yubrajparajuli/house-price-prediction</a></p><blockquote>A model that lives in a notebook helps no one. Ship it.</blockquote><h3>Acknowledgement</h3><p>I am currently learning AI and Data Science under the guidance of Angat Sitaula. This project was built as part of my learning journey. Thank you for the direction, motivation and support.</p><h3>About the Author</h3><p>I am Yubraj Parajuli, a Computer Engineer from Nepal building things with AI and Data Science.</p><blockquote><em>I write about real projects, real data and real mistakes I made along the way, so you don’t have to.</em></blockquote><p>Connect on <a href="http://www.linkedin.com/in/yubraj-parajuli">LinkedIn</a> or follow on <a href="https://github.com/yubrajparajuli">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f69d46086685" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Naive But Powerful: Building a Spam Detector from Scratch]]></title>
            <link>https://medium.com/@yubrajparajuli/naive-but-powerful-building-a-spam-detector-from-scratch-4b8520af529a?source=rss-411455780bff------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b8520af529a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yubrajparajuli]]></dc:creator>
            <pubDate>Mon, 25 May 2026 09:37:39 GMT</pubDate>
            <atom:updated>2026-05-25T09:37:39.325Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r8OPemDRMuZ_dnZ9RpaHIQ.png" /></figure><p>A step by step guide to Naive Bayes from math intuition to production ready spam detection in Python</p><p>Every time you open your inbox and it is clean, no “FREE PRIZE!!!” or “CLAIM YOUR £1000 NOW”, something smart happened behind the scenes.</p><p>Not deep learning. Not transformers. Not a massive neural network running on expensive GPUs.</p><p>Just probability. A 270 year old theorem. And a surprisingly bold assumption that somehow works.</p><p>That algorithm is Naive Bayes. And by the end of this blog, you will understand exactly how it works and build one yourself from scratch.</p><h3>What is Naive Bayes?</h3><p>Naive Bayes is a probabilistic classification algorithm.</p><p>Instead of just predicting a class, it calculates the probability of each class and picks the highest one.</p><p>It is built on Bayes Theorem, a 270 year old formula from statistics.</p><p>The word Naive comes from one bold assumption. Every word in a message is independent of every other word. Linguistically wrong. But in text classification, it works remarkably well.</p><h4>Why it still matters in 2025 ?</h4><p>Despite newer and more complex models, Naive Bayes is still widely used in production spam filters, sentiment analysis and document classification.</p><p>Fast. Interpretable. Works well even on small datasets.</p><p>But to truly understand why it works, we need to go back to where it all started. Bayes Theorem.</p><h3>Bayes Theorem</h3><h4>The Real Life Example</h4><p>You go to a doctor. You test positive for a rare disease that affects only 1% of the population. The test is 90% accurate.</p><p>Should you panic?</p><p>Most people say yes. But the math says otherwise.</p><p>Because the disease is so rare, even with a positive test, the actual probability you have it is only around 8%. The rarity of the disease pulls the probability down dramatically</p><p>This is Bayes Theorem. It updates your belief based on new evidence.</p><h4>The Formula</h4><pre>P(C | x₁, x₂, ..., xₙ) = P(x₁ | C) × P(x₂ | C) × ... × P(xₙ | C) × P(C) / P(x₁, x₂, ..., xₙ)</pre><p><strong>C : </strong>the class we want to predict, spam or ham</p><p><strong>x₁, x₂, …, xₙ</strong> : each individual word in the message</p><p><strong>P(C)</strong> : how often this class appears in our training data, also called prior probability</p><p><strong>P(xᵢ | C)</strong> : how likely a specific word appears in that class, also called likelihood</p><p><strong>P(C | x₁…xₙ): </strong>the final probability we calculate, how likely this message belongs to class C given all its words, also called posterior probability</p><p>Now that we understand the formula, let us look at the assumption that makes this algorithm both flawed and powerful at the same time.</p><h3>The Naive Assumption</h3><p>Naive Bayes assumes every word in a message is independent of every other word.</p><p>In reality this is never true. The word free appearing after win is not a coincidence. Words have context and relationships.</p><p>But here is the surprising part. Even with this wrong assumption, Naive Bayes performs remarkably well on text classification tasks.</p><p>Why? Because even if the independence assumption is violated, the relative probabilities still point in the right direction. Spam words cluster together regardless.</p><p>This is why it is called Naive.</p><blockquote>Not because it is simple. But because it makes a bold assumption and gets away with it.</blockquote><p>Now that we understand the assumption, let us look at which variant of Naive Bayes is best suited for text data.</p><h3>Types of Naive Bayes</h3><p>There are three main variants. Each is designed for a different type of data.</p><h4>Gaussian Naive Bayes</h4><p>Used when features are continuous numbers like height, weight or temperature. It assumes the data follows a normal distribution.</p><h4>Bernoulli Naive Bayes</h4><p>Used when features are binary. It only cares whether a word appears or not in a message. Does not care how many times.</p><h4>Multinomial Naive Bayes</h4><p>Used when features are word counts or frequencies. It cares about how many times each word appears. This is the one we use for spam detection.</p><h4>Why Multinomial for our case?</h4><p>TF-IDF gives us word frequencies, not binary values. Multinomial Naive Bayes is designed exactly for this. It extracts more signal from the data compared to Bernoulli.</p><blockquote>The right variant for the right data makes all the difference.</blockquote><p>But even with the right variant, there is one more problem we need to solve before building our model. The zero probability problem.</p><h3>Laplace Smoothing</h3><p>Consider a word that never appeared in spam during training. When the model sees it in a new message, it assigns a probability of zero.</p><p>Zero multiplied by anything is zero. One unseen word kills the entire prediction.</p><p>This is called the<strong> zero probability problem</strong>.</p><p>Laplace Smoothing fixes this by adding a small count to every word , even words never seen before.</p><pre>P(xᵢ | C) = (count(xᵢ, C) + α) / (count(C) + α × V)</pre><p><strong>α : </strong>smoothing parameter, usually 1.0 by default</p><p><strong>V</strong> :total vocabulary size</p><p><strong>count(xᵢ, C) : </strong>how many times word xᵢ appears in class C</p><p>With α = 1, no word ever gets zero probability. The model stays stable.</p><blockquote>In our project, hyperparameter tuning found α = 0.1 as the best value light smoothing, trusting the training data more.</blockquote><p>Now we have all the theory we need. Let us look at why Naive Bayes is specifically powerful for text before we jump into building.</p><h3>Why Naive Bayes for Text?</h3><p>A fair question. We have XGBoost, Random Forests, Transformers. Why reach for something this simple?</p><p>Because in production, simplicity wins. Naive Bayes trains in milliseconds, requires no hyperparameter marathon, and every prediction is fully explainable. You can open the model and see exactly which words pushed the decision.</p><p>For text classification with limited data, nothing beats that tradeoff.</p><blockquote>Fast to train. Easy to debug. Hard to beat on small text datasets.</blockquote><p>Now let us stop talking about theory and start building. Here is what I built.</p><h3>The Build</h3><h4>The Dataset</h4><p>I used the <a href="https://www.kaggle.com/datasets/uciml/sms-spam-collection-dataset">SMS Spam Collection dataset</a> from UCI Machine Learning Repository. 5572 real SMS messages, labeled as spam or ham.</p><p>After removing duplicates and cleaning, we are left with 5169 messages. 653 spam against 4516 ham.</p><p>Before any analysis, labels were encoded numerically. ham becomes 0 and spam becomes 1.</p><pre>df[&quot;label&quot;] = df[&quot;label&quot;].map({&quot;ham&quot;: 0, &quot;spam&quot;: 1})</pre><p>Now let the data speak.</p><h3>Exploratory Data Analysis</h3><h4>Class Distribution</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*igTiRkK_GLgBfB5ZypmiLA.png" /><figcaption>0 (ham): 4516 messages, 1 (spam): 653messages</figcaption></figure><p>The dataset is heavily imbalanced. 87.4% ham, 12.6% spam. A model that predicts everything as ham would still get 87% accuracy. This is why accuracy alone is not a reliable metric here. We need precision, recall and F1.</p><h4>Message Length</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6jixjcOxEU5IrQfFcJsX0w.png" /><figcaption>Spam messages are consistently longer. Ham messages vary widely.</figcaption></figure><p>Spam messages average 138 characters. Ham averages just 71. More interesting is the consistency. Spam has a very tight spread. Spammers follow a template. Ham is all over the place because real conversations vary.</p><h4>Word Count</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TH_-I9eMYWIYI6emFkUF2Q.png" /><figcaption>Spam averages 24 words. Ham averages just 14.</figcaption></figure><p>Same pattern as message length. Spam is consistent, ham is not. Spammers pack more words to convince you. Real messages are short and casual.</p><h4>What Spam and Ham Actually Look Like</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9dLoTPr3yDgALj85LlLP6A.png" /><figcaption>Most frequent words in spam messages</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*skYn4kTjgufTKqVH9MnGiA.png" /><figcaption>Most frequent words in ham messages</figcaption></figure><p>Two completely different vocabularies. Spam is full of free, call, reply, text, claim, urgent. Ham is full of ok, go, know, good, love, time. One is trying to sell you something. The other is just talking to you.</p><h4>Top 20 Words</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FJk5g8nVMYOOBY9NcEIK1A.png" /><figcaption>Top 20 most frequent words in spam messages</figcaption></figure><p>call appears around 315 times across 653 spam messages. That is once every two messages. Words like free, txt, text, claim, mobile, urgent dominate. Every word is designed to trigger action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mtq4KwcTj-q4w03ZMaX1uQ.png" /><figcaption>Top 20 most frequent words in ham messages</figcaption></figure><p>Completely different story. u, get, go, ok, know, good, love. Casual, personal, conversational. No urgency. No prizes. Just people talking.</p><h4>Feature Comparison</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X63YjVETGqn_j5veNbrRsQ.png" /><figcaption>Punctuation count, digit count and word count compared across spam and ham</figcaption></figure><p>Digit count is the most striking. Spam averages 15.4 digits per message. Ham averages just 0.3. Phone numbers, prize amounts, claim codes. Spam is packed with numbers. This single feature is a powerful spam signal.</p><h4>Correlation Heatmap</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pILqfOwxDdd1StsYIZJD1Q.png" /><figcaption>Correlation between all engineered features and the label</figcaption></figure><p>Digit count has a 0.84 correlation with the label. That is remarkably high for a single feature. Message length follows at 0.38. Also notice message length and word count are 0.97 correlated. They measure almost the same thing.</p><blockquote>One insight that stood out: digit count alone can predict spam with 84% correlation. No model needed.</blockquote><p>Now that we understand the data, it was time to clean the text before feeding it to the model.</p><h3><strong>Text Preprocessing</strong></h3><p>Raw SMS messages are messy. Punctuation, special characters, URLs, uppercase, extra spaces. All of this is noise. The model does not need it</p><p>Here is what I applied step by step:</p><pre>def preprocess(text):<br>    text = to_lowercase(text)<br>    text = strip_whitespace(text)<br>    text = remove_extra_spaces(text)<br>    text = remove_tabs_and_newlines(text)<br>    text = normalize_unicode(text)<br>    text = remove_html_tags(text)<br>    text = remove_urls(text)<br>    text = remove_emails(text)<br>    text = remove_hashtags_mentions(text)<br>    text = remove_special_characters(text)<br>    text = remove_stopwords(text)<br>    text = lemmatize(text)<br>    return text</pre><p>Each step has a purpose. Lowercasing ensures Free and free are treated as the same word. Stopword removal drops words like the, is, a that carry no meaning. Lemmatization reduces words to their root form, winning becomes win, entries becomes entry.</p><p>The result before and after:</p><p>Before: “FREE entry in 2 a wkly comp to win FA Cup final tkts!”</p><p>After: “free entry wkly comp win fa cup final tkt”</p><p>Clean, meaningful, ready for the model.</p><blockquote>I chose lemmatization over stemming. Stemming gives entri, crazi, avail. Lemmatization gives entry, crazy, available. Real words perform better with TF-IDF.</blockquote><p>Now the clean text needed to be converted into numbers.</p><h3>TF-IDF Vectorization</h3><p>Machine learning models cannot read text. They need numbers. TF-IDF converts each message into a numerical vector.</p><p>TF-IDF stands for Term Frequency Inverse Document Frequency.</p><p>Term Frequency : how often a word appears in one message.</p><p>Inverse Document Frequency : penalizes words that appear in every message.</p><p>So a word like the that appears everywhere gets a low score. A word like prize that appears mostly in spam gets a high score.</p><blockquote>TF-IDF does not just count words. It finds the words that actually matter.</blockquote><pre>tfidf = TfidfVectorizer(max_features=5000)<br>X_train_tfidf = tfidf.fit_transform(X_train)<br>X_test_tfidf = tfidf.transform(X_test)</pre><p>Each message is now represented as a row of 5000 numbers. One number per word in our vocabulary. The model can now read it.</p><p>Now it was time to train.</p><h3>Modeling</h3><h4><strong>The Pipeline</strong></h4><p>I used sklearn Pipeline, industry standard for keeping preprocessing and model together in one clean object.</p><pre>model = Pipeline([<br>    (&quot;tfidf&quot;, TfidfVectorizer(max_features=5000)),<br>    (&quot;clf&quot;, MultinomialNB())<br>])</pre><h4>Data Splitting</h4><p>I split the data 80/20 for train and test using stratified sampling, ensuring the same 12.6% spam ratio in both sets.</p><pre>X_train, X_test, y_train, y_test = train_test_split(<br>    X, y, test_size=0.2, random_state=42, stratify=y<br>)</pre><p>Without stratify=y, the spam ratio could differ between train and test sets leading to unreliable evaluation.</p><h4>Handling Class Imbalance</h4><p>With 4516 ham and only 653 spam, a naive model would just predict everything as ham and still get 87% accuracy. To fix this I used sample weights giving each spam message 3.7x more importance during training.</p><pre>sample_weights = compute_sample_weight(class_weight=&quot;balanced&quot;, y=y_train)</pre><h4>Hyperparameter Tuning</h4><p>The key parameter in Multinomial Naive Bayes is alpha , the Laplace smoothing value. I used GridSearchCV with 5 fold cross validation scored on F1 to find the best value.</p><pre>param_grid = {&quot;clf__alpha&quot;: [0.1, 0.5, 1.0, 2.0, 5.0]}<br>grid_search = GridSearchCV(model, param_grid, cv=5, scoring=&quot;f1&quot;, n_jobs=-1)<br>grid_search.fit(X_train, y_train, clf__sample_weight=sample_weights)</pre><p>Best alpha: 0.1 light smoothing, trusting the training data more. Best CV F1: 0.851</p><p>Now let us look at how the final model performed.</p><h3>Results</h3><h4>Metrics</h4><pre>Metric        Score<br>──────────────────<br>Accuracy   : 96.3%<br>Precision  : 81.6%<br>Recall     : 91.6%<br>F1 Score   : 86.3%<br>ROC AUC    : 99.2%<br>CV F1 Mean : 91.2%</pre><p>Accuracy looks impressive but remember the class imbalance. The real story is in recall and AUC. The model catches 91.6% of all spam. And with a 99.2% AUC it is near perfect at separating spam from ham.</p><h4>Confusion Matrix</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D5b9m1PJv59lzDUkEj9COw.png" /><figcaption>876 true negatives, 120 true positives, 27 false positives, 11 false negatives</figcaption></figure><p>876 ham messages correctly identified. 120 spam correctly caught. Only 11 spam messages slipped through. And 27 legitimate messages wrongly flagged as spam.</p><blockquote>Missing spam is worse than flagging a real email. The model prioritizes catching spam. Exactly the right tradeoff.</blockquote><h4>ROC Curve</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G8MK3NlRCQfssv_hpV5uoA.png" /><figcaption>AUC 0.992— near perfect separation between spam and ham</figcaption></figure><p>The curve hugs the top left corner. That means high true positive rate with very low false positive rate. AUC of 0.992 out of 1.0.</p><blockquote>No deep learning. No transformers. Just Bayes theorem and TF-IDF. 99.2% AUC.</blockquote><h4>What the Model Actually Learned</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*py4Kq5eNKDG67wsRkVjvCA.png" /><figcaption>Top 20 features driving spam (red) and ham (green) predictions</figcaption></figure><p>The model learned exactly what a human would flag. call, free, mobile, text, claim, urgent, prize push toward spam. im, ok, get, come, go, know, good push toward ham.</p><p>This is what makes Naive Bayes special. It is fully interpretable. You can open it and see exactly why it made each decision.</p><h4>Saving the Model</h4><p>The trained model is saved using joblib and is available in the GitHub repository.</p><pre>joblib.dump(best_model, &quot;outputs/model/spam_detector.joblib&quot;)</pre><p>Loading and predicting on a new message is straightforward:</p><pre>loaded_model = joblib.load(&quot;outputs/model/spam_detector.joblib&quot;)<br>loaded_model.predict([&quot;Free entry win £1000 now!&quot;])</pre><p>Now let us look at where it failed.</p><h3><strong>Error Analysis</strong></h3><p>No model is perfect. 38 messages out of 1034 were misclassified. Looking at these errors reveals exactly where the model struggles.</p><h4>False Positives — Ham flagged as Spam</h4><p>These are legitimate messages the model wrongly flagged:</p><ul><li>&quot;kthen special&quot; — too short, not enough context to decide</li><li>&quot;hiwhat think match&quot; — normal conversation flagged due to unusual words</li><li>&quot;free call sir waiting&quot; — contains spam trigger words free and call, model defaulted to spam</li></ul><p>The pattern is clear. Very short messages with spam trigger words confuse the model. Without enough context it defaults to the word probability.</p><h4>False Negatives — Spam that slipped through</h4><p>These are spam messages the model missed:</p><ul><li>&quot;hi ur lookin 4 saucy daytime fun wiv busty mar...&quot; — written casually like a real message</li><li>&quot;thanks vote sing along star karaoke mobile fre...&quot; — looks like a normal message but is spam</li><li>&quot;madamregret disturbancemight receive reference...&quot; — formal language completely fooled the model</li></ul><p>Spam that mimics casual conversation fools the model. When spammers avoid trigger words and write naturally, Naive Bayes loses its advantage.</p><blockquote>This is the fundamental limitation of bag of words models. They see words, not meaning. Context and intent are invisible to them.</blockquote><h3>When NOT to use Naive Bayes</h3><p>Naive Bayes is powerful but not universal.</p><p>When word order matters, it struggles. Sentences like “not good” and “good” look identical to it. For tasks where negation and context matter, transformers do a much better job.</p><p>When you have millions of examples, gradient boosting or deep learning will outperform it significantly. The naive independence assumption also breaks down when features are strongly correlated.</p><blockquote>If you need speed, interpretability and solid performance on small text data, Naive Bayes wins. If you need maximum accuracy on large data, look elsewhere.</blockquote><h3>Key Takeaways</h3><p>Naive Bayes is not naive in performance. A 270 year old probability theorem with TF-IDF achieves 99.2% AUC on real world spam data.</p><p>Class imbalance is real and must be handled. Without sample weights the model would ignore spam entirely.</p><p>Interpretability is underrated. You can open Naive Bayes and see exactly which words drive every prediction. No black box. No guessing.</p><p>Simple does not mean weak.</p><h3>GitHub Repository</h3><p>The complete project including modular source code, Jupyter notebook and all visualizations is available on <a href="https://github.com/yubrajparajuli/spam-detector-naive-bayes">GitHub.</a></p><h3>Docker</h3><p>Pull and run the complete pipeline without any setup:</p><pre>docker pull yubraj101/spam-detector<br>docker run --rm -v &quot;$(pwd)/outputs:/app/outputs&quot; yubraj101/spam-detector</pre><h3>Acknowledgement</h3><p>I am currently learning AI and Data Science under the guidance of Angat Sitaula. This project was built as part of my learning journey. Thank you for the direction, motivation and support.</p><h3>About the Author</h3><p>I am Yubraj Parajuli, a Computer Engineer from Nepal building things with AI and Data Science.</p><blockquote>I write about real projects, real data and real mistakes I made along the way, so you don’t have to.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b8520af529a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You Can’t Do ML Without This Math — Vectors Explained Visually]]></title>
            <link>https://medium.com/@yubrajparajuli/you-cant-do-ml-without-this-math-vectors-explained-visually-c40a58ed8218?source=rss-411455780bff------2</link>
            <guid isPermaLink="false">https://medium.com/p/c40a58ed8218</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Yubrajparajuli]]></dc:creator>
            <pubDate>Fri, 22 May 2026 03:47:07 GMT</pubDate>
            <atom:updated>2026-05-22T03:47:07.928Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QHd-eHxeC4Zaw5AGPOxS8g.png" /></figure><p>16 vector concepts, 16 visualizations, and the exact place each one appears in real Machine Learning algorithms.</p><p><strong>Do you know what happens inside a neural network every time it makes a prediction?</strong></p><p>It is not magic. It is not a black box. It is a dot product. Repeated millions of times.</p><p>Every recommendation Netflix makes. Every search result Google returns. Every word your LLM predicts next.</p><p>All of it traces back to one thing. Vectors.</p><blockquote>You don’t need to know all of mathematics. But you cannot do serious ML without understanding vectors.</blockquote><p>I learned this the hard way.</p><p>When my neural network wasn’t learning. When my PCA wasn’t working. When my RAG system kept returning irrelevant results.</p><p>Every answer led back to the same place.</p><p>So I went back to basics.</p><p>16 vector concepts. 16 visualizations. And the exact place each one shows up in real ML.</p><h3>The Foundation</h3><p>Before we touch any ML algorithm, we need to understand what a vector actually is.</p><p>Most people skip this part. That is exactly where the confusion starts.</p><h4>What is a Vector?</h4><p>A vector is anything with two properties. Magnitude and direction.</p><p>That is it. Nothing more.</p><p>In 2D space a vector looks like this.</p><p>v = [3, 2]</p><p>3 steps in x. 2 steps in y. That is your vector.</p><pre>import numpy as np<br><br>v = np.array([3, 2])<br>print(v)  # [3 2]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6GchPK_u6-ymkl0sLgTelA.png" /><figcaption>Same vector, different positions. Direction and magnitude never change.</figcaption></figure><p>Notice the right plot. The same vector appears three times from different starting points.</p><p>Position does not matter. Only direction and magnitude define a vector.</p><p><strong>ML Connection</strong></p><p>Every data point in your dataset is a vector.</p><p>A person with age=25, salary=50000, experience=3 becomes [25, 50000, 3].</p><p>Your entire dataset is a collection of vectors living in feature space.</p><h4>Basis Vectors</h4><p>Every vector is built from two fundamental building blocks.</p><p>i = [1, 0] and j = [0, 1]</p><p>These are called basis vectors.</p><p>Any vector you can imagine is just a combination of these two.</p><p>v = [3, 2] = 3·i + 2·j</p><pre>i = np.array([1, 0])<br>j = np.array([0, 1])<br>v = np.array([3, 2])<br><br># Any vector as combination of basis vectors<br>print(3*i + 2*j)  # [3 2]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eEN3CG4CJL4YnTlVqxt2CQ.png" /><figcaption>Every vector is a combination of basis vectors i and j.</figcaption></figure><p><strong>ML Connection</strong></p><p>Features in your dataset are basis vectors.</p><p>age, salary and experience each define one direction in feature space.</p><p>PCA finds entirely new basis vectors that better represent your data. That is literally what principal components are.</p><h4>Vector Space</h4><p>A vector space is the playground where vectors live.</p><p>It has one simple rule. Any operation you do on vectors must stay inside the space.</p><p>Add two vectors. Still in the space. Multiply by a scalar. Still in the space. That consistency is what makes ML math work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tzFWycb55-lL9SP4P2xPuA.png" /><figcaption>Vector space closure under addition and scalar multiplication.</figcaption></figure><p><strong>ML Connection</strong></p><p>Your entire dataset lives in a vector space.</p><p>Normalization, standardization, PCA — all of these are operations that stay inside the vector space.</p><p>That is why they work without breaking your data.</p><h3>The Operations</h3><p>Now that we know what vectors are, let us see what we can do with them.</p><h4>Vector Addition and Subtraction</h4><p>Adding vectors is like following directions one after another.</p><p>Walk 3 steps east and 2 steps north. Then walk 1 step east and 4 steps north. Where do you end up? That is vector addition.</p><pre>v1 = np.array([3, 2])<br>v2 = np.array([1, 4])<br><br>addition    = v1 + v2  # [4, 6]<br>subtraction = v1 - v2  # [2, -2]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D-F4hDqbpDK3WRU9lpY9Kg.png" /><figcaption>Left: Triangle law of addition. Right: Subtraction finds the difference between two vectors.</figcaption></figure><p><strong>ML Connection</strong></p><p>Subtraction measures the difference between two data points.</p><p>customer_B-customer_A tells you exactly how much they differ in every feature dimension.</p><p>This is the foundation of distance calculations, clustering and anomaly detection.</p><h4>Scalar Multiplication</h4><p>Multiplying a vector by a scalar does one thing.</p><p>It stretches or shrinks the vector.<br>Direction stays the same. Only magnitude changes.</p><p>Multiply by 2. Twice as long.<br>Multiply by 0.5. Half as long.<br>Multiply by -1. Direction flips completely.</p><pre>v = np.array([3, 2])<br><br>print(2   * v)   # [6, 4]   stretched<br>print(0.5 * v)   # [1.5, 1] shrunk<br>print(-1  * v)   # [-3, -2] flipped</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OVl9TjfLjr87iZFP6TfZoQ.png" /><figcaption>Scalar multiplication stretches, shrinks or flips a vector.</figcaption></figure><p><strong>ML Connection</strong></p><p>The learning rate in gradient descent is a scalar.</p><p>It controls how much we scale the gradient vector before updating model weights.</p><p>gradient = [0.5, 0.3, 0.8]<br>learning_rate = 0.01<br>update = 0.01 × gradient = [0.005, 0.003, 0.008]</p><p>Too large a scalar and your model overshoots.<br>Too small and it learns too slowly.</p><h3>The Power Tools</h3><p>These two concepts are the most important in all of ML.</p><p>If you understand nothing else from this blog, understand these.</p><h4>Dot Product</h4><p>The dot product answers one question.</p><p>How much do two vectors point in the same direction?</p><p>Positive result. Same direction.<br>Zero. Perpendicular. No similarity at all.<br>Negative. Opposite directions.</p><pre>v1 = np.array([3, 2])<br>v2 = np.array([1, 4])<br><br>dot = np.dot(v1, v2)  # 11</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f3lRarP7yjqGi0PpDoAItQ.png" /><figcaption>Positive, zero and negative dot products visualized.</figcaption></figure><p>The formula has two forms.</p><pre>v1 · v2 = x1·x2 + y1·y2<br>v1 · v2 = |v1| · |v2| · cos(θ)</pre><p>The second form reveals something powerful.<br>The dot product is zero when θ = 90°.<br>Maximum when θ = 0°.</p><p><strong>ML Connection</strong></p><p>Every single neuron in a neural network computes a dot product.</p><p>output = weights · inputs + bias<br>output = [w1, w2, w3] · [x1, x2, x3] + b</p><p>That is a dot product.</p><p>Every forward pass in a neural network is millions of dot products happening simultaneously.</p><h4>Cross Product</h4><p>The dot product gives you a number.<br>The cross product gives you a vector.</p><p>Specifically a vector perpendicular to both inputs.</p><p>Point your right hand fingers from v1 toward v2.<br>Your thumb points in the direction of the cross product.</p><p>That is the right hand rule.</p><pre>v1 x v2 = perpendicular to both v1 and v2<br><br>Key property:<br>v1 x v2 = -(v2 x v1)<br><br>Order matters!</pre><pre>v1 = np.array([1, 0, 0])<br>v2 = np.array([0, 1, 0])<br><br>cross = np.cross(v1, v2)<br>print(cross)           # [0, 0, 1]<br>print(np.cross(v2,v1)) # [0, 0, -1] order matters!<br><br># Verify perpendicularity<br>print(np.dot(cross, v1))  # 0<br>print(np.dot(cross, v2))  # 0</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ojDu_6mC6vvfLBL-6E_BGQ.png" /><figcaption>Left: Cross product perpendicular to both vectors. Right: Order matters — v1xv2 and v2xv1 point in opposite directions.</figcaption></figure><p>Notice the right plot.</p><p>v1 x v2 points up.<br>v2 x v1 points down.</p><p>Same vectors. Different order. Completely opposite result.</p><p><strong>ML Connection</strong></p><p>Cross product is less common in standard ML but critical in:</p><p>Computer Vision — calculating surface normals for 3D objects.<br>Robotics — computing rotation axes and torque.<br>3D Graphics — finding perpendicular directions for lighting.</p><p>The key insight is that cross product finds a direction completely orthogonal to your current space.<br>Useful whenever you need to escape into a new dimension.</p><h4>Cosine Similarity</h4><p>The dot product has one problem.</p><p>Longer vectors produce bigger dot products even if they point in the same direction.</p><p>Cosine similarity fixes this.<br>It measures the angle between vectors regardless of their length.</p><p>Result is always between -1 and 1.</p><p>1 means identical direction.<br>0 means perpendicular.<br>-1 means completely opposite.</p><pre>def cosine_similarity(a, b):<br>    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))<br><br>v_small = np.array([1, 1])<br>v_large = np.array([4, 4])<br><br>print(cosine_similarity(v_small, v_large))  # 1.0</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xWVGvMDdYWu5kXFAo5luqQ.png" /><figcaption>Left: Three cases of cosine similarity. Right: Magnitude does not affect the result.</figcaption></figure><p>Notice the right plot.</p><p>v_small and v_large have completely different magnitudes.<br>But their cosine similarity is exactly 1.0.</p><p>Because they point in the same direction.<br>That is the power of cosine similarity.</p><p><strong>ML Connection</strong></p><p>When you built a RAG system, cosine similarity was finding the most relevant document chunks for your query.</p><p>When Google returns search results, cosine similarity is comparing your query vector to millions of document vectors.</p><p>When Netflix recommends a movie, cosine similarity is finding users with similar taste vectors to yours.</p><p>One formula. Powering the entire modern AI stack.</p><h3>The Measurements</h3><p>How do you measure the size of a vector?</p><p>Turns out there are multiple ways. And each one matters in a different ML context.</p><h4>Magnitude and Unit Vector</h4><p>Magnitude is simply the length of a vector.</p><p>It is just Pythagoras theorem applied to vectors.</p><p>v = [3, 4]</p><pre>magnitude = sqrt(3*3 + 4*4) = sqrt(25) = 5.0</pre><p>A unit vector is that same vector divided by its magnitude.<br>Same direction. Length becomes exactly 1.</p><pre>v = np.array([3, 4])<br><br>magnitude   = np.linalg.norm(v)       # 5.0<br>unit_vector = v / magnitude            # [0.6, 0.8]<br><br>print(np.linalg.norm(unit_vector))    # 1.0 always</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lH8s3GGZaFJkbFwyGzRjxw.png" /><figcaption>Left: Magnitude using Pythagoras. Right: Unit vector lands exactly on the unit circle.</figcaption></figure><p><strong>ML Connection</strong></p><p>When you normalize features before training a model, you are creating unit vectors.</p><p>You remove the magnitude and keep only the direction.<br>This makes comparison between features fair and meaningful.</p><p>Text embeddings in modern LLMs are normalized to unit vectors before cosine similarity is computed.</p><h4>L1 and L2 Norm</h4><p>Magnitude is actually the L2 norm.</p><p>But there is another way to measure vector length.<br>The L1 norm.</p><p>Think of it this way.</p><p>L2 norm is the straight line distance. As the crow flies.<br>L1 norm is the city block distance. Walking on a grid.</p><p>Same destination. Different paths. Different distances.</p><pre>v = np.array([3, 4])<br><br>l1 = np.sum(np.abs(v))      # 7  (3 + 4)<br>l2 = np.linalg.norm(v)      # 5  (sqrt(9+16))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NfhRxQPpOVcHA7U8P0bD8Q.png" /><figcaption>Left: L1 city blocks vs L2 straight line. Right: L1 diamond vs L2 circle.</figcaption></figure><p>The right plot reveals something fascinating.</p><p>The L1 unit boundary is a diamond.<br>The L2 unit boundary is a circle.</p><p>The corners of the L1 diamond touch the axes.<br>That is not a coincidence.</p><p><strong>ML Connection</strong></p><p>L1 and L2 norms are the foundation of regularization in ML.</p><p>L1 regularization adds the L1 norm of weights to your loss function.<br>Solutions tend to hit the corners of the diamond where some weights become exactly zero.<br>This is how Lasso performs automatic feature selection.</p><p>L2 regularization adds the L2 norm of weights to your loss function.<br>Weights shrink toward zero but rarely reach it exactly.<br>This is how Ridge prevents overfitting.</p><p>The shape of the boundary is literally why they behave differently.</p><h3>The Advanced Concepts</h3><p>These three concepts are deeply connected.</p><p>They all answer the same fundamental question.<br>Are these vectors carrying unique information or redundant information?</p><h4>Orthogonality</h4><p>Two vectors are orthogonal when they are perfectly perpendicular.<br>The angle between them is exactly 90 degrees.</p><p>How do you check? One simple test.</p><pre>v1 · v2 = 0  →  orthogonal<br>v1 · v2 ≠ 0  →  not orthogonal</pre><pre>v1 = np.array([3, 2])<br>v2 = np.array([-2, 3])<br><br>print(np.dot(v1, v2))  # 0 — orthogonal!</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lAGHya7QCLj1lWMDWaB-Xg.png" /><figcaption>Orthogonal vectors have dot product exactly zero and angle of 90 degrees.</figcaption></figure><p><strong>ML Connection</strong></p><p>PCA finds orthogonal components.</p><p>Each principal component is perpendicular to every other.<br>This means they carry completely independent information.<br>No overlap. No redundancy.</p><p>That is why PCA works so well for dimensionality reduction.</p><h4>Linear Independence</h4><p>Two vectors are linearly independent when neither one can be made from the other.</p><p>Dependent vectors are just scaled versions of each other.<br>They carry the same information twice.</p><pre>v1_dep = np.array([2, 1])<br>v2_dep = np.array([4, 2])   # 2 * v1_dep — dependent!<br><br>v1_ind = np.array([2, 1])<br>v2_ind = np.array([1, 3])   # truly different — independent!</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ll5clF_sWytRTDeKDV3L5Q.png" /><figcaption>Dependent vectors collapse onto a line. Independent vectors span a full 2D plane.</figcaption></figure><p><strong>ML Connection</strong></p><p>height_cm and height_inches are dependent features.<br>They carry identical information. One is completely redundant.</p><p>age and salary are independent features.<br>They carry unique information. Both are useful.</p><p>Using dependent features in ML causes multicollinearity.<br>Your model gets confused and learns wrong patterns.</p><h4>Span of Vectors</h4><p>The span of a set of vectors is every point you can reach by combining those vectors.</p><p>One vector spans a line.<br>Two independent vectors span an entire 2D plane.</p><pre>v1 = np.array([2, 1])<br>v2 = np.array([1, 3])<br><br># Any point reachable by combining v1 and v2<br>point = 2.5*v1 + 1.3*v2  # [6.3, 6.4]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1RZwN2-RCaq1Bl4eWxuhqA.png" /><figcaption>One vector spans only a line. Two independent vectors span the full 2D plane.</figcaption></figure><p><strong>ML Connection</strong></p><p>PCA reduces dimensions by finding the minimum number of vectors that still span most of your original data space.</p><p>The explained variance ratio tells you how much of the original span is preserved after reduction.</p><p>When you keep 95% explained variance in PCA, you are keeping the vectors that span 95% of your original feature space.</p><h3>The Transformations</h3><p>This is where linear algebra becomes truly powerful.</p><p>Everything we have covered so far was about understanding vectors. Now we learn how to reshape entire spaces.</p><h4>Linear Transformation</h4><p>A linear transformation is what happens when a matrix acts on a vector.</p><p>It does not just move the vector.<br>It reshapes the entire space the vector lives in.</p><p>It can rotate, scale, shear or flip.</p><p>But one thing never changes.<br>Straight lines stay straight. Origin stays at origin.</p><pre>T(v) = A * v<br><br>Scale:    [[2, 0], [0, 2]]   stretches everything by 2<br>Rotate:   [[0,-1], [1, 0]]   rotates 90 degrees<br>Shear:    [[1, 1], [0, 1]]   slants the space<br>Reflect:  [[-1,0], [0, 1]]   flips horizontally</pre><pre>v = np.array([2, 1])<br><br>T_scale  = np.array([[2, 0], [0, 2]])<br>T_rotate = np.array([[0, -1], [1, 0]])<br><br>print(T_scale  @ v)   # [4, 2]<br>print(T_rotate @ v)   # [-1, 2]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ZKSNniyoDlK1x0QTovuZw.png" /><figcaption>Four transformations applied to the same grid. Scale, rotate, shear and reflect.</figcaption></figure><p>Look at the grid in each plot.</p><p>The entire space is being reshaped.<br>Not just the vector. The whole coordinate system.</p><p>That is what a matrix transformation does.</p><p><strong>ML Connection</strong></p><p>Every layer in a neural network is a linear transformation.</p><p>input → weight matrix → transformed output</p><p>The network learns the best transformation matrices to convert your input into the correct output.</p><p>That is all deep learning is.<br>A series of linear transformations followed by non-linear activation functions.</p><h4>Vector Projection</h4><p>Projection answers one simple question.</p><p>If you shine a light straight down on v1, what shadow does it cast on v2?</p><p>That shadow is the projection.</p><pre>proj(v1 onto v2) = (v1 * v2 / |v2|^2) * v2<br><br>scalar = dot product / squared magnitude of v2<br>projection = scalar * v2</pre><pre>v1 = np.array([3, 4])<br>v2 = np.array([5, 0])<br><br>scalar = np.dot(v1, v2) / np.dot(v2, v2)<br>proj   = scalar * v2<br><br>print(proj)    # [3. 0.]<br>print(v1-proj) # [0. 4.] perpendicular component</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p6MWrlSp85EnMAtbSc-5Yg.png" /><figcaption>Left: Projection as a shadow with light coming perpendicular. Right: Projection onto different directions.</figcaption></figure><p>Notice the verification.</p><p>proj + perpendicular = original vector exactly.<br>proj and perpendicular have dot product zero.</p><p>The projection captures what v1 shares with v2.<br>The perpendicular captures what v1 has that v2 does not.</p><p><strong>ML Connection</strong></p><p>Vector projection is the mathematical heart of PCA.</p><p>PCA projects your high dimensional data onto the directions of maximum variance.</p><p>Each principal component is a projection direction.<br>The projection captures the most important information.<br>The perpendicular component is discarded as noise.</p><p>When you reduced your dataset to 2D using PCA, every single data point was being projected onto the first two principal components using exactly this formula.</p><h3>The Big Picture</h3><p>We have covered 16 concepts so far.</p><p>Let us now see how they all connect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qr_u7B9GZwmrbxizRC1nZQ.png" /><figcaption>How every vector concept connects to a real ML algorithm.</figcaption></figure><p>Look at this diagram carefully.</p><p>Every concept we covered today is powering something real.<br>Not theoretical. Not academic. Actually running in production systems right now.</p><p>Dot product is inside every neural network making predictions.<br>Cosine similarity is inside every search engine returning results.<br>Vector projection is inside every PCA reducing dimensions.<br>L1 and L2 norms are inside every regularized model preventing overfitting.<br>Linear transformation is inside every deep learning layer learning features.</p><blockquote>Linear algebra is not a prerequisite for ML.<br>It is the language ML is written in.</blockquote><h3>Key Takeaways</h3><p>After going through all 16 concepts, here is what matters most.</p><p><strong>Every data point is a vector.</strong></p><p>The moment you load your dataset, you are working with vectors.<br>Understanding them is not optional.</p><p><strong>Dot product is everywhere.</strong></p><p>Neural networks, similarity search, recommendations.<br>All of it is dot products running at scale.</p><p><strong>Cosine similarity removes magnitude bias.</strong></p><p>When comparing vectors of different sizes, always use cosine similarity over raw dot product.</p><p><strong>L1 creates sparsity. L2 creates smoothness.</strong></p><p>Choose your regularization based on whether you want feature selection (L1) or weight shrinkage (L2).</p><p><strong>Projection is the heart of dimensionality reduction.</strong></p><p>Every time you run PCA, you are projecting your data onto new basis vectors. Now you know exactly what is happening.</p><p><strong>Linear transformations are what neural networks learn.</strong></p><p>Each layer learns a transformation matrix.<br>The network stacks these transformations to convert input into output.</p><h3>Final Thoughts</h3><p>Most ML courses teach you to use the tools.<br>Very few teach you to understand what the tools are doing.</p><p>That gap is what separates someone who uses ML from someone who understands ML.</p><p>Vectors are not just math.<br>They are the language your models speak.</p><blockquote>Learn the language. Understand the model.<br>Build things that actually work.</blockquote><p>If this blog helped you see the math differently, share it with someone who is still afraid of linear algebra. They need to read this.</p><h3>Full Code on GitHub</h3><p>Find the full project code here → <a href="https://github.com/yubrajparajuli/linear-algebra-ml">linear-algebra-ml</a></p><h3>Acknowledgement</h3><p>This project is part of my learning journey under the mentorship of <a href="https://www.linkedin.com/in/angatsitoula/">Angat Sitaula</a>. Thank you for teaching us to not just understand concepts but to build with them.</p><h3>About the Author</h3><p>I am Yubraj Parajuli, a Computer Engineering graduate from Nepal currently navigating the world of Artificial Intelligence and Data Science. I document my learning journey through hands-on projects and blogs, one concept at a time.</p><blockquote>I write about real projects, real data and real mistakes I made along the way, so you don’t have to.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c40a58ed8218" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Using One-Hot Encoding for Everything, Here’s What to Use Instead]]></title>
            <link>https://medium.com/@yubrajparajuli/stop-using-one-hot-encoding-for-everything-heres-what-to-use-instead-73e2b0e9c023?source=rss-411455780bff------2</link>
            <guid isPermaLink="false">https://medium.com/p/73e2b0e9c023</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[categorical-encoding]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Yubrajparajuli]]></dc:creator>
            <pubDate>Tue, 05 May 2026 11:09:20 GMT</pubDate>
            <atom:updated>2026-05-05T11:09:20.056Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7fURGETkirKg1yw94k95ow.png" /></figure><p>A hands-on guide to 7 categorical encoding techniques with real data, visuals, and a decision flowchart to help you choose the right one every time.</p><p>I used One-Hot Encoding on a column with 41 unique values.</p><p>My 15-column dataset exploded to 80+ columns overnight. My model slowed down. Memory usage spiked. Results got worse.</p><p>That was the day I stopped blindly using One-Hot Encoding for everything.</p><p>One-Hot Encoding is not bad. It’s just massively overused.</p><p>Most data scientists default to it without thinking. And it silently hurts their models every single time. In this blog, I’ll walk you through 7 categorical encoding techniques, what they are, when to use them, and most importantly, when NOT to use One-Hot Encoding.</p><p><strong>By the end, you’ll have a clear decision framework for choosing the right encoding every time.</strong></p><h3>Why Does Encoding Even Matter?</h3><p>Machine learning models speak numbers, not words. When your dataset has categories like “Private”, “Self-employed”, or “Government”, the model has absolutely no idea what to do with them.</p><p>Encoding converts these categories into numbers. Simple enough. But here is what most people miss. HOW you convert them changes everything.</p><blockquote>Wrong encoding → model learns false patterns → worse predictions.</blockquote><p>It’s not just a preprocessing step. It’s a decision that directly impacts your model’s accuracy, training speed, and memory usage.</p><h3>The Dataset</h3><p>For this project, I used the Adult Income Dataset, a classic real world dataset originally from the UCI Machine Learning Repository.</p><p>You can download it directly from Kaggle here → <a href="https://www.kaggle.com/datasets/uciml/adult-census-income">Adult Income Dataset</a></p><p>The dataset has 32,561 rows, 15 columns and 9 categorical columns. The target variable predicts whether a person earns more than $50K per year.</p><p>One interesting challenge I found was that the dataset contained hidden missing values disguised as “?” instead of standard NaN. After detecting and removing 2,399 affected rows, we were left with 30,162 clean rows ready for encoding.</p><p>Our 9 categorical columns ranged from binary columns like sex with just 2 unique values all the way to high cardinality columns like native_country with 41 unique values. This diversity is exactly what makes this dataset perfect for our encoding showdown.</p><h3>Understanding Cardinality</h3><p>Before jumping into encoding techniques, there is one concept you must understand. Cardinality.</p><p>Cardinality simply means the number of unique values in a categorical column. This single number determines which encoding technique you should use.</p><p>Look at our dataset. native_country has 41 unique values. sex has only 2. These two columns need completely different encoding strategies. Applying the same technique to both is where most people go wrong.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SxCNJaQZTmZeChXGxc7Xxg.png" /><figcaption>Cardinality per categorical column</figcaption></figure><p>From the chart we can clearly see three groups. High cardinality columns like native_country, education and occupation with more than 10 unique values. Medium cardinality columns like workclass, marital_status and relationship sitting between 5 and 10. And low cardinality columns like race, sex and income with less than 5 unique values.</p><p>Each group needs a different encoding strategy. Let’s go through all 7.</p><h3>The Encoding Techniques</h3><p>Now let’s get into the actual techniques. For each one I will show you the code, the output, when to use it and when to avoid it.</p><h4>1. Label Encoding</h4><p>Label Encoding is the simplest technique. It assigns a number to each unique category. Sklearn does this <strong>alphabetically</strong> by default.</p><pre>from sklearn.preprocessing import LabelEncoder<br><br>le = LabelEncoder()<br>df[&#39;marital_status_label&#39;] = le.fit_transform(df[&#39;marital_status&#39;])</pre><p>Applied on our marital_status column, here is what the mapping looks like.</p><pre>Divorced             → 0<br>Married-AF-spouse    → 1<br>Married-civ-spouse   → 2<br>Married-spouse-absent→ 3<br>Never-married        → 4<br>Separated            → 5<br>Widowed              → 6</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7GuRIN8z8Hwe1jJQi2je1A.png" /><figcaption>Label encoding mapping for marital status</figcaption></figure><p>Notice the problem. Sklearn assigned numbers alphabetically, not meaningfully. Your model might think Widowed (6) is greater than Divorced (0). That makes no logical sense for marital status.</p><p><strong>When to use it ?</strong></p><p>Label Encoding works well with tree based models like Decision Tree, Random Forest and XGBoost because these models split on values and do not interpret numbers as having order.</p><p><strong>When to avoid it ?</strong></p><p>Never use Label Encoding with linear models. They will treat the numbers as meaningful order and learn completely wrong patterns from your data.</p><h4>2. One-Hot Encoding</h4><p>One-Hot Encoding creates a new binary column for each unique category. If a column has 7 unique values, you get 7 new columns. Each row gets a 1 in its category column and 0 everywhere else.</p><pre>ohe_df = pd.get_dummies(df[[&#39;workclass&#39;, &#39;sex&#39;]], dtype=int)</pre><p>Applied on workclass and sex, our 2 columns became 9 binary columns. Each row has exactly one 1 per original column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3vwEO5jZzQtYqQjjMs21hw.png" /><figcaption>One-Hot Encoding applied on workclass and sex</figcaption></figure><p>Clean and interpretable. But here is the real problem. If we applied this same technique on native_country which has 41 unique values, we would instantly get 41 new columns. That is the dimensionality explosion that killed my model at the beginning of this blog.</p><p><strong>When to use it?</strong></p><p>One-Hot Encoding works best when your column has less than 10 unique values. It is the safest choice when you want full interpretability and no assumptions about order.</p><p><strong>When to avoid it?</strong></p><p>Never use it on high cardinality columns. 41 unique values means 41 new columns instantly. This is the<strong> Curse of Dimensionality</strong> in action.</p><h4>3. Dummy Encoding</h4><p>One-Hot Encoding has a hidden problem that most people never notice. It creates perfectly correlated columns. This is called <strong>multicollinearity</strong> and it silently destroys linear model performance.</p><p>Dummy Encoding fixes this with one simple trick. Drop one column.</p><pre>dummy_df = pd.get_dummies(df[[&#39;relationship&#39;]], drop_first=True, dtype=int)</pre><p>Applied on relationship, we dropped Husband as the reference category. When all 5 columns show 0, that row is automatically Husband. Nothing is lost. The model still gets all the information it needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Q08XMWV7yVuIYjFyIlbsg.png" /><figcaption>Dummy Encoding on relationship column. Husband is the reference category.</figcaption></figure><p>This solves a problem called multicollinearity, where columns are perfectly correlated with each other. Linear models are especially sensitive to this.</p><p><strong>When to use it?</strong></p><p>Linear and regression models. Anywhere multicollinearity is a concern.</p><p><strong>When to avoid it?</strong></p><p>Tree based models do not suffer from multicollinearity so dropping a column is unnecessary. Stick with One-Hot for those.</p><h4>4. Ordinal Encoding</h4><p>Label Encoding assigned numbers alphabetically. Widowed got 6. Divorced got 0. No logic behind it whatsoever.</p><p>Ordinal Encoding lets YOU define the order. And for education, the order is very clear. Preschool is not equal to Doctorate.</p><pre>from sklearn.preprocessing import OrdinalEncoder<br><br>education_order = [<br>    &#39;Preschool&#39;, &#39;1st-4th&#39;, &#39;5th-6th&#39;, &#39;7th-8th&#39;, &#39;9th&#39;,<br>    &#39;10th&#39;, &#39;11th&#39;, &#39;12th&#39;, &#39;HS-grad&#39;, &#39;Some-college&#39;,<br>    &#39;Assoc-voc&#39;, &#39;Assoc-acdm&#39;, &#39;Bachelors&#39;, &#39;Masters&#39;,<br>    &#39;Prof-school&#39;, &#39;Doctorate&#39;<br>]<br><br>oe = OrdinalEncoder(categories=[education_order])<br>df[&#39;education_ordinal&#39;] = oe.fit_transform(df[[&#39;education&#39;]])</pre><p>Preschool gets 0. Doctorate gets 15. The model now understands that higher means more educated. That is meaningful. That is correct.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AXESdXrJg0lPTJId0aXwBA.png" /><figcaption>Ordinal encoding on education. Higher value means more education.</figcaption></figure><p><strong>When to use it?</strong></p><p>Any column with a natural ranking. Education level, satisfaction ratings, size categories like Small, Medium and Large.</p><p><strong>When to avoid it?</strong></p><p>Nominal categories with no meaningful order. Forcing a ranking where none exists is just as misleading as Label Encoding.</p><h4>5. Frequency Encoding</h4><p>What if instead of creating new columns, we simply counted how many times each category appears in the dataset?</p><p>That is exactly what Frequency Encoding does. It replaces each category with its count. No new columns. No dimensionality explosion. Just one clean column.</p><pre>freq_map = df[&#39;native_country&#39;].value_counts()<br>df[&#39;native_country_freq&#39;] = df[&#39;native_country&#39;].map(freq_map)</pre><p>Applied on native_country with 41 unique values, here is what we get.</p><pre>United-States → 27504<br>Mexico        → 610<br>Philippines   → 188<br>Germany       → 128</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jZCoSnZqiYrtwLKAM0ZBIQ.png" /><figcaption>Frequency encoding on native_country. Top 15 countries by count.</figcaption></figure><p>41 unique values. Still one column. That is the power of Frequency Encoding.</p><p>But there is a catch. India and El-Salvador both appeared exactly 100 times in our dataset. They got the same encoded value even though they are completely different countries. This is called the <strong>collision problem</strong>.</p><p><strong>When to use it?</strong></p><p>High cardinality columns where you have no target variable available. native_country with 41 values is a perfect example.</p><p><strong>When to avoid it?</strong></p><p>When multiple categories have similar frequencies. The collision problem means your model cannot distinguish between them.</p><h4>6. Target Encoding</h4><p>Executive managers are 70 times more likely to earn over $50K than private house servants. Target Encoding found this automatically.</p><p>Instead of counts or arbitrary numbers, Target Encoding replaces each category with the mean of the target variable. In our case, the probability of earning more than $50K per year.</p><pre>df[&#39;income_binary&#39;] = (df[&#39;income&#39;] == &#39;&gt;50K&#39;).astype(int)<br>target_map = df.groupby(&#39;occupation&#39;)[&#39;income_binary&#39;].mean()<br>df[&#39;occupation_target&#39;] = df[&#39;occupation&#39;].map(target_map)</pre><p>Applied on occupation, here is what the encoding revealed.</p><pre>Exec-managerial  → 0.485  (48.5% earn &gt;50K)<br>Prof-specialty   → 0.448  (44.8% earn &gt;50K)<br>Priv-house-serv  → 0.007  (0.7% earn &gt;50K)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Ah-HCXBQW1-xZ24khb8mw.png" /><figcaption>Target encoding on occupation. Higher value means higher probability of earning over $50K.</figcaption></figure><p>This is not just encoding. This is your model learning the relationship between occupation and income before training even begins.</p><p>But there is one serious risk. Target Encoding uses the target variable to encode categories. If you are not careful, this leads to <strong>data leakage</strong>. Your model will perform brilliantly on training data and fail completely on unseen data. Always use cross validation when applying Target Encoding in production.</p><p>In simple terms, this is <strong>overfitting</strong> caused by data leakage. Your model memorizes the training data instead of learning from it.</p><p><strong>When to use it?</strong></p><p>High cardinality columns in supervised learning where a target variable is available. occupation and native_country are perfect examples.</p><p><strong>When to avoid it?</strong></p><p>Unsupervised learning where no target variable exists. Also avoid it without cross validation in production. Data leakage is a silent killer of ML models.</p><h4>7. Binary Encoding</h4><p>5 categories. 3 columns. That is all Binary Encoding needs. One-Hot would have created 5.</p><p>Binary Encoding converts each category to a number first, then represents that number in binary digits. It is the most space efficient encoding technique we have covered so far.</p><pre>import category_encoders as ce<br><br>be = ce.BinaryEncoder(cols=[&#39;race&#39;])<br>race_binary = be.fit_transform(df[[&#39;race&#39;]])</pre><p>Applied on race with 5 unique values, here is what we get.</p><pre>White               → 0  0  1<br>Black               → 0  1  0<br>Asian-Pac-Islander  → 0  1  1<br>Amer-Indian-Eskimo  → 1  0  1<br>Other               → 1  0  0</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zTvLBlPFjdPWsbyl2uBcuA.png" /><figcaption>Binary encoding on race. 5 categories represented in just 3 columns.</figcaption></figure><p>Each category gets a unique binary pattern. No false order. No dimensionality explosion. A clean middle ground between Label Encoding and One-Hot Encoding.</p><p><strong>When to use it?</strong></p><p>Medium to high cardinality columns where One-Hot creates too many columns but you still want to avoid the false order of Label Encoding. race with 5 values and native_country with 41 values are both good candidates.</p><p><strong>When to avoid it?</strong></p><p>Low cardinality columns with less than 4 unique values. Binary Encoding offers no advantage there. One-Hot is simpler and more interpretable.</p><h3>The Complete Comparison</h3><p>Now that we have covered all 7 techniques, let us see them side by side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*93RsRFHvnFGyeedY2oEvoA.png" /><figcaption>Complete comparison of all 7 encoding techniques.</figcaption></figure><p>The pattern is clear. When cardinality is low, One-Hot and Dummy are your safest choices. When cardinality is high, Target and Frequency Encoding keep your dataset lean and meaningful. Binary Encoding sits right in the middle as an efficient compromise.</p><h3>Before vs After Encoding</h3><p>Here is a visual look at how our column count changed after applying each technique on our dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R59xrkPBX0eHICNd-t2aIg.png" /><figcaption>Before vs after encoding. Column count comparison per technique.</figcaption></figure><p>One-Hot on workclass and sex alone created 9 columns from just 2. Meanwhile Target, Frequency, Label and Ordinal Encoding kept everything at 1 column. This is why encoding choice matters.</p><h3>So Which One Should You Use?</h3><p>This is the question every data scientist faces before preprocessing. There is no single right answer. But there is a clear decision framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jVK7ffu4f103kHbVA2I3pg.png" /><figcaption>Decision flowchart for choosing the right encoding technique.</figcaption></figure><p>Start with one question. Does your column have a natural order? That single answer sets your entire encoding path. Follow the flowchart above and you will never make a wrong encoding decision again.</p><h3>Key Takeaways</h3><p>After going through all 7 techniques on real data, here is what I want you to remember.</p><p><strong>One-Hot Encoding is not always wrong. It is just overused.</strong></p><p>Use it when cardinality is low. Avoid it when you have more than 10 unique values in a column.</p><p><strong>Cardinality is everything.</strong></p><p>Before encoding anything, always check how many unique values your column has. That single number guides your entire decision.</p><p><strong>Label Encoding is dangerous for linear models.</strong></p><p>It implies a false order that linear models will take seriously and learn completely wrong patterns from.</p><p><strong>Target Encoding is powerful but risky.</strong></p><p>Always use cross validation in production. Data leakage is a silent killer that will make your model look great on training data and fail completely on new data.</p><p><strong>There is no one size fits all encoding</strong>.</p><p>The right encoding depends on your column, your model type and whether you have a target variable. Use the decision flowchart above every time you are unsure.</p><h3>Final Thoughts</h3><p>Encoding is one of those topics that looks simple on the surface but hides a lot of depth underneath. Most people learn One-Hot Encoding and stop there. But as we saw today, that single decision can make or break your model.</p><p>The next time you reach for get_dummies(), pause for a second. Check your cardinality. Think about your model type. And choose the encoding that actually fits your data.</p><blockquote>Your model is only as good as the data you feed it. And encoding is how you prepare that data.</blockquote><p>I hope this guide helps you make better encoding decisions going forward. If you found this useful, feel free to share it with someone who might still be blindly using One-Hot Encoding for everything.</p><h3>Acknowledgement</h3><p>This project is part of my learning journey under the mentorship of <a href="https://www.linkedin.com/in/angatsitoula/">Angat Sitaula</a>. Thank you for teaching us to not just understand concepts but to build with them.</p><h3>About the Author</h3><p>I am Yubraj Parajuli, a Computer Engineering graduate from Nepal currently navigating the world of Artificial Intelligence and Data Science. I document my learning journey through hands-on projects and blogs, one concept at a time.</p><blockquote>I write about real projects, real data and real mistakes I made along the way, so you don’t have to.</blockquote><p>Find the full project code here → <a href="https://github.com/yubrajparajuli/categorical-encoding-showdown">categorical-encoding-showdown</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73e2b0e9c023" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Built PCA from Scratch in Python — Here’s What It Did to My Model]]></title>
            <link>https://medium.com/@yubrajparajuli/i-built-pca-from-scratch-in-python-heres-what-it-did-to-my-model-51363889f880?source=rss-411455780bff------2</link>
            <guid isPermaLink="false">https://medium.com/p/51363889f880</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[apc]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Yubrajparajuli]]></dc:creator>
            <pubDate>Mon, 27 Apr 2026 11:03:08 GMT</pubDate>
            <atom:updated>2026-04-27T11:03:08.294Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3BnC2L2f389mx8JB" /><figcaption>Photo by <a href="https://unsplash.com/@plutonicmedia?utm_source=medium&amp;utm_medium=referral">Loui Kiær</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A step by step guide to understanding Principal Component Analysis by building it from scratch with NumPy, verifying it with sklearn, and measuring its real impact on a Logistic Regression model.</p><p>Most data science tutorials teach you PCA like this:</p><p>from sklearn.decomposition import PCA<br>pca = PCA(n_components=2)<br>X_pca = pca.fit_transform(X)</p><p>Three lines. Done. Move on.</p><p>But here is the problem — if you cannot explain what just happened inside those three lines, you do not understand PCA. You are just using a black box.</p><p>So I did something different.</p><p>I built PCA completely from scratch using only NumPy — no sklearn, no shortcuts, no magic. Every single step by hand. Then I applied it to 1143 real wine samples and measured exactly what it did to my machine learning model.</p><p>The result surprised me.</p><p>Reducing 11 features down to 6 components caused only 0.44% accuracy drop — but made training 18 times faster.</p><p>Here is exactly how I did it.</p><h3>The Dataset</h3><p>I used the Wine Quality Dataset from Kaggle — 1143 red wine samples, each described by 11 chemical measurements including fixed acidity, volatile acidity, citric acid, alcohol and more.</p><p>The target variable is a quality score from 3 to 8 given by wine experts. Since extreme scores had very few samples — only 6 wines scored 3 and 16 scored 8 — the model could never learn from them properly.</p><p>I converted this into a binary problem — quality 5 and below becomes Bad Wine with 522 samples, quality 6 and above becomes Good Wine with 621 samples. Clean, balanced and ready for PCA.</p><p>Dataset: <a href="https://www.kaggle.com/datasets/yasserh/wine-quality-dataset">Wine Quality Dataset on Kaggle</a></p><h3>Exploratory Data Analysis</h3><p>Before applying PCA, I explored the data carefully. EDA is not optional — it tells you whether PCA will actually work on your dataset.</p><h4>Feature Distributions</h4><p>I plotted the distribution of all 11 features. The most important finding — features operate at completely different scales. Total sulfur dioxide reaches 289 while density varies by only 0.01.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XGH1Pn5dKMnPlAez-wqr_A.png" /><figcaption>Feature Distributions</figcaption></figure><p>This is exactly why standardization is mandatory before PCA. Without it, large scale features would dominate unfairly and PCA would give wrong results.</p><h4>Correlation Heatmap</h4><p>This is the most important EDA plot for PCA. If features are not correlated, PCA has nothing to compress. Correlated features carry redundant information — and PCA’s job is to find and remove that redundancy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iSuit8l95iChWIUitmgFZA.png" /><figcaption>Correlation Heatmap</figcaption></figure><p>Three strong correlations stood out immediately. Fixed acidity and citric acid share 0.67. Free and total sulfur dioxide share 0.66 — they are essentially measuring the same thing. Alcohol and density share -0.49. These correlations confirm PCA will work beautifully on this dataset.</p><h4>Quality Distribution</h4><p>The original quality scores range from 3 to 8 but are heavily imbalanced. Quality 3 has only 6 samples and quality 8 has only 16. A model can never learn properly from such few examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HCTwknZZu4p-ZyLrzhNynA.png" /><figcaption>Quality_distribution</figcaption></figure><p>I converted the scores into binary — Bad Wine for quality 5 and below, Good Wine for quality 6 and above. This gave 522 vs 621 samples — balanced and ready for modeling.</p><pre>df[&#39;target&#39;] = (df[&#39;quality&#39;] &gt;= 6).astype(int)</pre><h4>Boxplot — Bad Wine vs Good Wine</h4><p>I compared each feature between bad and good wines side by side. One finding dominated everything else.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NF7Evdsvh-1keNivIN1Uqg.png" /><figcaption>Feature Distribution: Bad Wine vs Good Wine</figcaption></figure><p>Alcohol is the strongest predictor of wine quality. Good wines have a median alcohol of 10.8% vs 9.5% for bad wines. Volatile acidity tells the opposite story — bad wines have higher acidity giving them an unpleasant vinegar taste. Features like residual sugar and pH showed almost no difference between good and bad wines at all.</p><h3>What is PCA?</h3><p>Principal Component Analysis is a dimensionality reduction technique. In simple terms — it takes many correlated features and compresses them into fewer, more meaningful ones called principal components.</p><p>Think of it like this. Imagine describing a person using 10 characteristics — height, weight, shoe size, arm length, leg length and more. Many of these are correlated — tall people tend to have longer legs, bigger feet, longer arms. PCA finds these relationships and compresses them into 2–3 key measurements that still describe the person well.</p><p>PCA does this in 5 steps:</p><p>Step 1 — Standardize the data<br>Step 2 — Compute the covariance matrix<br>Step 3 — Compute eigenvectors and eigenvalues<br>Step 4 — Sort components by importance<br>Step 5 — Project data onto top components</p><p>Each step has a clear mathematical purpose. Instead of just calling sklearn, I implemented every step from scratch using only NumPy. Here is exactly what I did.</p><h3>Building PCA from Scratch</h3><p>The first step before PCA is standardization. Every feature must have mean 0 and standard deviation 1. Without this, features with large values would dominate the results unfairly.</p><pre>scaler = StandardScaler()<br>X_scaled = scaler.fit_transform(X)</pre><h4>Step 1 — Center the Data</h4><p>Centering subtracts the mean from every feature. This moves the entire data cloud to the origin — a requirement for PCA to work correctly.</p><pre>mean = np.mean(X_scaled, axis=0)<br>X_centered = X_scaled - mean</pre><h4>Step 2 — Covariance Matrix</h4><p>The covariance matrix captures relationships between every pair of features. It is the mathematical version of the correlation heatmap we saw in EDA.</p><pre>n_samples = X_centered.shape[0]<br>cov_matrix = np.dot(X_centered.T, X_centered) / (n_samples - 1)</pre><p>The result is an 11×11 matrix — every feature compared against every other feature. Diagonal values are all 1.0 confirming equal variance after standardization.</p><h4>Step 3 — Eigenvectors and Eigenvalues</h4><p>This is the heart of PCA. Eigenvalues tell us how much variance each direction captures. Eigenvectors tell us the actual directions — the principal components.</p><pre>eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)<br>eigenvalues  = eigenvalues.real<br>eigenvectors = eigenvectors.real</pre><p>The largest eigenvalue was 3.16 — meaning PC1 alone captures 28.7% of all variance in the dataset. The smallest was 0.06 — essentially noise that can be safely discarded.</p><h4>Step 4 — Sort and Select Components</h4><p>We sort eigenvalues from largest to smallest so the most important components come first. Then we select only the components needed to retain 80% of variance.</p><pre>sorted_idx   = np.argsort(eigenvalues)[::-1]<br>eigenvalues  = eigenvalues[sorted_idx]<br>eigenvectors = eigenvectors[:, sorted_idx]<br><br>explained_variance_ratio = eigenvalues / eigenvalues.sum()<br>cumulative_variance      = np.cumsum(explained_variance_ratio)<br><br>n_components = np.argmax(cumulative_variance &gt;= 0.80) + 1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d-hnzqaYWcJ2eDrhNRPfZA.png" /><figcaption>Scree Plot</figcaption></figure><p>The scree plot tells the complete story. PC1 captures 28.7% alone. By PC6 we have retained 85.85% of all information. After PC6 the curve flattens — adding more components gives diminishing returns. We selected 6 components.</p><h4>Step 5 — Project the Data</h4><p>The final step projects our data from 11 dimensions down to 6. This is a simple matrix multiplication.</p><pre>W = eigenvectors[:, :n_components]<br>X_pca = np.dot(X_centered, W)</pre><p>One wine that was described by 11 chemical measurements is now represented by just 6 numbers — retaining 85.85% of all information. That is PCA.</p><h3>Verifying with Sklearn</h3><p>Building PCA from scratch is only valuable if the results are correct. I verified by comparing against sklearn’s PCA implementation — the industry standard.</p><pre>pca = PCA(n_components=6)<br>X_pca_sklearn = pca.fit_transform(X_scaled)</pre><p>The comparison was component by component — every single principal component matched exactly.</p><pre>Component    Sklearn      Scratch      Match<br>---------------------------------------------<br>PC1          28.69        28.69        OK<br>PC2          17.08        17.08        OK<br>PC3          14.31        14.31        OK<br>PC4          11.01        11.01        OK<br>PC5          8.71         8.71         OK<br>PC6          6.05         6.05         OK<br><br>Average difference: 0.000000</pre><p>The average difference between scratch and sklearn transformed values was literally zero. Not 0.001. Not 0.0001. Zero. This confirms that building PCA from scratch with NumPy produces mathematically identical results to sklearn — which is built and maintained by hundreds of professional engineers worldwide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3WCE30kJcmaifoDoFEAxWw.png" /><figcaption>PCA 2D Scatter Plot</figcaption></figure><p>Projecting onto the first two components shows a 2D view of our wine data. Red dots are bad wines, green are good. While they appear mixed in 2D, remember we are compressing 6 dimensions into 2 — some separation is naturally lost in the visualization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*53WJ6gtRz2H-Ksg3gIEgjQ.png" /><figcaption>PCA Biplot</figcaption></figure><p>The biplot reveals the chemistry of wine automatically. Free and total sulfur dioxide point in almost identical directions — confirming their 0.66 correlation. Fixed acidity and pH point in opposite directions — confirming wine chemistry. Alcohol points downward — opposite to sulfur dioxide — meaning high alcohol wines need less preservation. PCA discovered all of this on its own.</p><h3>Model Comparison — Before vs After PCA</h3><p>The real test of PCA is not mathematical — it is practical. Does reducing dimensions actually help or hurt your model? I trained the same Logistic Regression model twice — once on all 11 original features and once on 6 PCA components.</p><h4>Before PCA — 11 Features</h4><pre>Features:       11<br>Training time:  0.01381 seconds<br>Accuracy:       77.73%</pre><h4>After PCA — 6 Components</h4><pre>Features:       6<br>Training time:  0.00077 seconds<br>Accuracy:       77.29%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TsqpFEzqNxEMFVYA2PvaIQ.png" /><figcaption>Model Performance: Before vs After PCA</figcaption></figure><p>Three results stood out immediately.</p><p>First — the accuracy drop was only 0.44%. We removed 5 features and lost less than half a percent of accuracy. In real world applications this is completely acceptable.</p><p>Second — training speed improved 18 times. From 0.01381 seconds to 0.00077 seconds. On a small dataset this seems minor. But imagine running this on millions of records — the time saving becomes enormous.</p><p>Third — good wine detection remained completely unchanged at F1 score of 0.79. PCA retained all the information needed to identify good wines perfectly.</p><p>Keeping 85.85% of information preserved 99.4% of model performance. That is the power of PCA.</p><h3>Key Findings and Conclusion</h3><p>Here is everything this project taught me about PCA.</p><h4>What PCA Actually Does</h4><p>PCA does not remove features randomly. It finds the directions of maximum variance in your data and projects everything onto those directions. Correlated features get compressed into single components. Noise gets left behind.</p><h4>When to Use PCA</h4><p>Use PCA when your features are correlated — like ours were. The correlation heatmap is your best signal. If you see strong red and blue values, PCA will work. If everything is white and near zero, PCA has nothing to compress.</p><h4>The Numbers That Matter</h4><pre>Features reduced:     11 → 6 components<br>Variance retained:    85.85%<br>Accuracy before PCA:  77.73%<br>Accuracy after PCA:   77.29%<br>Accuracy drop:        0.44%<br>Speed improvement:    18x faster</pre><p>Building PCA from scratch was the best decision I made in this project. When you implement every step yourself — centering, covariance matrix, eigenvectors, projection — you stop seeing PCA as a black box and start seeing it as a tool you truly understand.</p><p>The sklearn verification proved it — zero difference between scratch and sklearn. Same math, same results, complete understanding.</p><p>If you want to understand any machine learning algorithm deeply — build it from scratch first. Then use the library.</p><h4>GitHub Repository</h4><p>The complete project including modular source code, Jupyter notebook and all visualizations is available on <a href="https://github.com/yubrajparajuli/pca-wine-quality">GitHub</a>.</p><h4>Acknowledgement</h4><p>I am currently learning AI and Data Science under the guidance of Angat Sitaula. This project was built as part of my learning journey. Thank you for the direction, motivation and support.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51363889f880" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>