<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sathwik Pinninti on Medium]]></title>
        <description><![CDATA[Stories by Sathwik Pinninti on Medium]]></description>
        <link>https://medium.com/@sathwikpinninti?source=rss-b2f0a7b28c4f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*UHDqElva89XwF1Ys</url>
            <title>Stories by Sathwik Pinninti on Medium</title>
            <link>https://medium.com/@sathwikpinninti?source=rss-b2f0a7b28c4f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:57:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sathwikpinninti/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[K-Means clustering from scratch: Building a Customer Support Ticketing system]]></title>
            <link>https://medium.com/@sathwikpinninti/k-means-clustering-from-scratch-building-a-customer-support-ticketing-system-e480156a56ae?source=rss-b2f0a7b28c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e480156a56ae</guid>
            <category><![CDATA[hnsw]]></category>
            <category><![CDATA[tfidf-vectorizer]]></category>
            <category><![CDATA[dbscan]]></category>
            <category><![CDATA[sentence-transformers]]></category>
            <category><![CDATA[k-means-clustering]]></category>
            <dc:creator><![CDATA[Sathwik Pinninti]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 12:46:11 GMT</pubDate>
            <atom:updated>2026-11-09T12:49:59.718Z</atom:updated>
            <content:encoded><![CDATA[<h3>K-Means clustering from scratch: Building a Production Grade Customer Ticketing Support system</h3><p><a href="https://github.com/Sathwik-007/ticket-intelligence.git">GitHub - Sathwik-007/ticket-intelligence</a></p><p>On our next algorithm in the list, I arrived at K-means clustering. An algorithm that’s different from all the regression but yet closely similar to the classification techniques we have encountered so far. “Closely-similar” is just because its an unsupervised learning, where we don’t have actual labels to classify against. We cluster similar datapoints together, so when new data arrives, we predict which cluster it can closely belong to and label it with that cluster.</p><p>If you think of how do we know what data points are together and to which clusters they might belong to, our go-to method would be calculating the straight line distance between them in a 2, 3 or multi-dimensional space. Let&#39;s take a small example.</p><p>For simplicity, consider a group of points lie on a 2D plane. If you pick any two points and try to calculate the distance between them, it would be straight line distance viz. <strong>Euclidean distance</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*e2nHtVM-KJCOdNBCilK3OA.png" /></figure><p>and that rings us bells because we came across this distance calculation in <a href="https://medium.com/@sathwikpinninti/k-nearest-neighbours-building-a-fraud-transaction-detection-model-from-scratch-f3ea61fde180">K-nearest neighbours</a> algorithm we unboxed before. Similarly, we find distance of all data points to a set of chosen centroids <em>(A centroid is a center of the cluster from which we calculate distance to other data points and decide whether they belong to this or a neighbouring cluster) </em>and mark which cluster they belong to.</p><h3>The Math behind …</h3><h4>Euclidean Distance (L2 Norm)</h4><p>In K-means every record is transformed into vectors in n-dimensional space. Calculating the distance between two vectors using Euclidean Distance is given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*onP-mkkk3c6CSZQJAQ6DfQ.png" /></figure><h4>J-inertia (Within Cluster Sum of Squares)</h4><p>K-Means defines the error as the mean of its squared distances to the assigned centroid <strong>||x — uk||².</strong></p><p>To measure the overall quality of model, we calculate the mean squared distances of all data points in a cluster. Then sum up all those clusters together. That gives us the J-inertia.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*sQT9cdPtJBJCpgmTfN394w.png" /></figure><p>For K-means to work we need to understand something. To know which data points belong together, we need to choose a center. To choose a center, we need to know which data points clump together so that every data point maps to a center. To solve this chicken-and-egg problem, We pick ‘<strong><em>k’ </em></strong>random points on the dataset and choose them to be our centers and freeze them in place. Then we run this on two loops. <strong><em>Expectation and Maximisation: Coordinate Descent</em></strong><em>.</em></p><h4>Expectation loop (Cluster assignment to data points)</h4><p>Once the centroids are frozen, we calculate the distance of every point to chosen centers and assign it to a center it is closest to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*vTZ6RNsAcmvcvhzvzAdckg.png" /></figure><pre>sq_diff = self._compute_distances(X)<br>self.labels = np.argmin(sq_diff, axis=1)</pre><h4>Maximisation loop (Update the centers)</h4><p>To find the perfect center, we isolate the inertia function to just one cluster. Our goal is to find the value of μk. In calculus, to find the exact minimum of a curve, you take the derivative with respect to the variable you want to change (in this case, our centroid μk​), and set that derivative equal to zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*tb8DAEawnr6E9lzYRx7EtQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*z0v1-1imiU88ctaQON4jMQ.png" /></figure><p><em>(Note: The negative sign appears because of the chain rule applied to −μk​ inside the parenthesis).</em></p><p>To find the absolute lowest point of the function, we set the derivative equal to zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*CUOHpEbdgGAaIKUJsuUBgA.png" /></figure><p>Now simplify,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*HqNnEP6VTnY7ZOH3MldyIw.png" /></figure><p>Distribute summation,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*SXZe-H-EsqKjA79Cio2Osg.png" /></figure><p>Now, if we look closely at <em>μk​ </em>term, the center of cluster doesn’t change to all the data points within that cluster. So if we add a constant N no.of times, then its simply N times that constant. In our case N is Nothing but <em>Ck​.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*HTbpdGNinKxw8gbx-RliaA.png" /></figure><p>Solve for <em>μk</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*2Pr3rSp4SnMMzbcToLse6w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*hiVr-KwPVR-9C9E5bePz-g.png" /></figure><p>The right side of the equation is exactly the formula for an average: add up all the points (∑xi​) and divide by the total number of points (Ck​).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*hiVr-KwPVR-9C9E5bePz-g.png" /></figure><pre># 4. Updation of centroids(Maximisation step)<br># Sum of data points with in the centroid / no.of data points in that centroid<br>new_centroids = np.zeros_like(self.centroids)<br>for label in range(self.k):<br>    members = X[self.labels == label]<br><br>    if len(members) &gt; 0:<br>        new_centroids[label] = np.mean(members, axis=0)<br><br>    # Edge case: if its an empty cluster, assign farthest point<br>    else:<br>        farthest_point = np.argmax(min_dists, axis=1)<br>        new_centroids[label] = X[farthest_point]<br><br># 5. Check for convergence<br>centroid_shift = np.linalg.norm(new_centroids - self.centroids)<br>self.centroids = new_centroids</pre><p>So, we freeze centers. Form clusters around every centroid (expectation: assignment). Then freeze assignments and update centroids (maximisation). We run this loop over ‘max-iters’ times untill there is no significant drop in J-inertia.</p><pre>class CustomKmeansPlusPlus:<br>    def __init__(self, k=8, max_iters=100, tol=1e-4, n_init=10, random_state=42):<br><br># Rest of the code</pre><p>To summarise in a note: k-means randomly chooses <em>‘k’</em> random points as centroids, calculates distances to all other points. Calculates J-inertia. Updates the centers by moving the centroids to the mean of its own clusters. Then re-runs the entire process over max-iters times and chooses the <em>‘k’</em> with lowest possible inertia to determine the value of <em>‘k’. </em>But … we have a problem.</p><h4>Problem</h4><p>WCSS, EM loop are good for calculating and minimising error. But they don’t actually guarantee global minimum. If you remember, we initialised centroids randomly during the beginning. Random assignments are prone to outliers. If a far most data point (noise) is treated as centroid, no vectors will be closer in its space. It can also place more than one centroid in a large cluster dividing it into 2 or more individual groups, getting trapped in a local minima. To address this, we have a more efficient way of choosing centroids which is introduced in <strong>K-means++</strong>.</p><h3>K-means++: Weighted Random Sampling</h3><p>K-means++ solves this problem by probabilistically choosing the centroids. By that it means, it picks one data point uniformly at random to be its first centroid. This is the only random step. To then choose the next centroid, it performs a probability distribution according the distance of all data points to the already chosen centroid. The vector that is actually farther away has higher probability to be chosen as the next centroid and the ones that are closer, will have lower probability to be chosen. This process is repeated untill all <em>‘k’ </em>centroids have been chosen.</p><p>The probability <strong>P(x)</strong> is given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*5gMp-ZUcUxe0GAzD4o9e3w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*c12GUvPLHJLIyG4KGiSOBA.png" /></figure><p>Now the algorithm hands over these intelligently chosen centroids to the EM loop we discussed earlier and it almost always converges to global minimum very faster than standard approach.</p><p>Before we move on to how to choose what is the optimal ‘k’ value, lets discuss on yet another topic TF-IDF that is later used in the project.</p><h3>Term Frequency — Inverse Document Frequency (TF-IDF)</h3><p>Computers donot understand words. They only comprehend numbers. and hence they can understand distance in a coordinate space. In order to turn words into numbers (vectors), a naive approach would be counting the no.of times a word appears in a given document. But there’s a fall. Consider you are counting the word “cancel” in a Customer support ticketing system. Then the common words like “The”, “is” will dominate the coordinate space and reduces the importance of important words like “cancel”, “refund” etc.</p><p>To fix this, TF-IDF penalizes commonly occuring words in the dataset and enhances the weight of rarely or important words by calculating the importance of a given word locally (within a ticket) and globally (within the document or database). It does so by follows:</p><h4>1. Term Frequency (TF)</h4><p>How many a times a given word appears locally within a ticket</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X149TnuBmljJB9BDscndDg.png" /></figure><p>why do this? If a customer writes a ten-word ticket and “refund” appears twice, the term frequency of word “refund” would be 2 / 10 = 0.2.</p><p>If another customer writes a 100 word ticket and “refund” is only used twice, the TF is 2 / 100 = 0.2. Dividing by the total words in a ticket count ensures the term doesn’t get massive score just simply because the ticket is a long essay.</p><h4>2. Inverse Document Frequency (IDF)</h4><p>Measure how important a word is across a given database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*_m6jdRSgS9REX_J6tMoQyg.png" /></figure><p><em>where:</em></p><ul><li><em>N : Total no.of tickets in your dataset</em></li><li><em>dft : No.of tickets that contain the word “t”</em></li></ul><h4>scikit-learn optimisation</h4><p>To prevent division by zero error, we add 1 to numerator and denominator inside the log (somehow if any word had a frequency of 0). We add 1 at the very end to ensure that even a word appearing in every single document doesn’t get a final score of exactly 0, preserving its basic TF value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*Lc5Igl8Lsa8sKYw2fNezew.png" /></figure><p>IDF(“refund”, D) = log((1 + 1000 / 1 + 197)) + 1</p><h4>3. TF x IDF</h4><p>Combine local frequency and global importance into a single score.</p><p>Let’s look at a concrete example using the smoothed formula. Assume a corpus of 3 tickets (N=3):</p><ol><li>“cancel my order”</li><li>“where is my order”</li><li>“cancel my account”</li></ol><p>Let’s calculate the coordinate for the word <strong>“account”</strong> in Ticket 3:</p><ul><li><strong>TF:</strong> “account” appears 1 time out of 3 total words. TF=0.333</li><li><strong>IDF:</strong> N=3, and “account” appears in 1 ticket (df=1).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FzSHysoNB6zot-C1O8pEbA.png" /></figure><h4>4. L2 Normalization (The Final Vector)</h4><p>Goal is to ensure that every vector’s length is exactly 1.0 in the coordinate space. To do this, we divide the every element in the vector with the entire vector’s Euclidean length (L2 Norm)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G6-jMpEPv0eUnAqSYqYp_g.png" /></figure><h3>Finding optimal ‘k’ value: Elbow method and Silhouette Scores</h3><p>Now, that’s all the math we needed to know to understand how K-means works behind the scenes. We also talked about TF-IDF to tokenize words into vector embeddings. But if we wind back to our last question, how do we know how many centroids we need to choose for cluster optimisation? We actually don’t. We simply run k-means over different k-values [3, 5, 7, 9, 11, 14, 17 …] etc and choose the optimal <em>k</em> from where the is no significant drop in J-inertia.</p><h3>Elbow method</h3><p>We can use Elbow method to visualise ‘k’ value vs J-inertia graph and check for a steep point after which the line tries to move as horizontal as possible. That steep point is our optimal k.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QbnYMtN-q4w7bRWaA8wvMQ.png" /></figure><pre>K=2 -&gt; Inertia: 1624.45<br>K=3 -&gt; Inertia: 1535.87<br>K=5 -&gt; Inertia: 1485.93<br>K=8 -&gt; Inertia: 1321.58<br>K=12 -&gt; Inertia: 1232.81<br>K=16 -&gt; Inertia: 1180.10<br>K=20 -&gt; Inertia: 1085.51</pre><h3>Silhouette Score</h3><p>Elbow method utilises J-inertial score which in turn calculates how tight a cluster is (cohesion) but ignores how separate the clusters are from each other (separation). Silhouette scores measures cohesion w.r.to separation. Silhouette score of a vector is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*fPZBSn5I6fZbWukLIdsI6w.png" /></figure><p><em>where</em>:</p><ul><li><strong>b(i) Separation:</strong> Measures separation. The average Euclidean distance of point i to all other points outside current cluster. <strong><em>(Higher is better)</em></strong></li><li><strong>a(i) Cohesion: </strong>It is the average Euclidean distance of point i to all other points in its current cluster. <strong><em>(Lower is better)</em></strong></li></ul><p><strong>Reading the score:</strong></p><ul><li><strong>Near -1,</strong> <strong><em>a(i) is larger than b(i):</em></strong><em> </em>if s(i) yields a score closer to -1, the point is wrongly assigned to a cluster <strong>(Incorrect cluster assignment).</strong></li><li><strong>Near 0:</strong> if s(i) yields a score closer to 0, the point is right in the boundary of two clusters.</li><li><strong>Near +1,</strong> <strong><em>b(i) is larger than a(i):</em></strong><em> </em>if s(i) yields a score closer to +1, the point is tightly packed in its own cluster and is far away from other clusters <strong>(Perfect clustering).</strong></li></ul><p>We calculate Silhouette scores of all data points and we take an average for the current cluster assignment. We do this over and over again on multiple ‘k’ values and choose the ‘k’ that yields the best maximum average Silhouette score across all the triaged ‘k’ values.</p><h3>Accuracy vs Adjusted Rand Index</h3><p>K-means creates clusters with names or index 0, 1, 2. Cluster 0 might contain tickets with “cancellation” and cluster 1 might have “refund” related tickets. But since names mismatch w.r.to our true labels and k-means generated labels, if we attempt to calculate raw accuracy, it might give 0.</p><p>The <strong>Adjusted Rand Index (ARI)</strong> is a statistical measure used in unsupervised machine learning and data science to evaluate the quality of cluster assignments. It measures how similar two data clusterings are to each other (such as comparing a clustering algorithm’s output against the true ground-truth labels). [<a href="https://www.geeksforgeeks.org/machine-learning/rand-index-in-machine-learning/">1</a>, <a href="https://www.sciencedirect.com/topics/computer-science/adjusted-rand-index">2</a>, <a href="https://www.youtube.com/watch?v=-UBChMttQOI">3</a>, <a href="https://www.youtube.com/watch?v=pU5MaLe8Mxs">4</a>]</p><p>How It Works: The Pair-Counting Foundation</p><p>The ARI builds upon the standard <strong>Rand Index (RI)</strong>, which evaluates clustering similarity by looking at all pairs of data points: [<a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_rand_score.html">1</a>, <a href="https://www.geeksforgeeks.org/machine-learning/rand-index-in-machine-learning/">2</a>]</p><ul><li><strong>True Positive (TP):</strong> Two points are in the <em>same</em> cluster in the prediction and the <em>same</em> class in the ground truth.</li><li><strong>True Negative (TN):</strong> Two points are in <em>different</em> clusters in the prediction and <em>different</em> classes in the ground truth.</li><li><strong>False Positive (FP):</strong> Two points are in the <em>same</em> cluster in the prediction, but <em>different</em> classes in the ground truth.</li><li><strong>False Negative (FN):</strong> Two points are in <em>different</em> clusters in the prediction, but the <em>same</em> class in the ground truth. [<a href="https://www.youtube.com/watch?v=_yahk6tQY2Q&amp;t=1">1</a>]</li></ul><p>The raw Rand Index calculates the proportion of correct decisions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*bsAs7KDuvLyzyLfdw133PQ.png" /></figure><p>Why “Adjusted”? (The Problem with Raw RI)</p><p>The raw Rand Index has a major flaw: <strong>it does not account for random chance.</strong> Even if you assign clusters completely at random, the raw RI score will often be well above 0 (sometimes 0.7 or higher, depending on the number of clusters and data size), making it difficult to know if a model is actually performing well or just benefiting from random groupings. [<a href="https://medium.com/@samson.sabu/rand-index-ri-vs-adjusted-rand-index-ari-in-k-means-clustering-06c364bdf1ef">1</a>, <a href="https://www.youtube.com/watch?v=pU5MaLe8Mxs">2</a>]</p><p>The <strong>Adjusted Rand Index</strong> fixes this by subtracting the expected score from random chance and scaling the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*DVRyzREXsxdqx3Z-_pAzKg.png" /></figure><h4>Score Range and Interpretation</h4><ul><li><strong>1.0:</strong> Complete and identical agreement between the two clusterings (ignoring mere label permutations).</li><li><strong>0.0:</strong> Agreement is no better than what you would expect from a random assignment.</li><li><strong>Negative values (down to about -0.5):</strong> Systematic disagreement or worse-than-random clustering.</li><li><strong>Python Implementation:</strong> You can easily calculate it using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_rand_score.html">scikit-learn 1.9.1 documentation</a></li></ul><pre>from sklearn.metrics import adjusted_rand_score<br><br># true_labels comes from the &#39;intent&#39; column in your dataset<br>ari_score = adjusted_rand_score(true_labels, km.labels)<br>print(f&quot;Adjusted Rand Index: {ari_score:.4f}&quot;)</pre><h3>Scaling to Production — Project: Customer support ticketing system</h3><p>Building a custom class mimicking k-means behaviour is okay when you have small datasets. In real world scenarios like customer service supporting systems, user queries come in way more complicated, noisy, unstructured and often with bad grammar. The following are the challenges.</p><ul><li>TF-IDF places “I want a refund” and “Give me my money back” vectors far away from each other because they share 0 words. But ideally both are seeking for a refund. TF-IDF creates sparse matrices (vectors with more 0s) that lack semantic meaning.</li><li>Kmeans++ is prone to bad centroid initialisation (note: we do run the entire algorithm n_init=10 times to prevent this. As a noise being a randomly chosen centroid would yield higher inertia anyways). It also forces every data point to be a part of cluster even if its noise. A noisy data point like “It is a lovely day” can easily be clustered into a “edit order” customer query which collapses cluster integrity.</li><li>(X[:, np.newaxis, :] - centroids[np.newaxis, :, :])**2 we are using leads to OOM (Out Of Memory) bottleneck. Broadcasted Shape = (27000, 8, 20000). This intermediate tensor contains 4.32 billion elements. Because Python defaults to float64 (8 bytes per element), this single operation demands <strong>34.5 GB of contiguous RAM</strong>.</li></ul><p>Which is why we are gonna use the following to tackle the above problems.</p><h4>Escaping the Sparse Matrix Trap</h4><p>TF-IDF relies on exact keyword matching, meaning it is blind to semantic intent. If a customer types “Where is my package?” and another types “Delivery is late,” TF-IDF plots them in completely different galaxies because they share zero vocabulary.</p><p>To capture true meaning, we transition the pipeline to <strong>Sentence Transformers</strong> (specifically the lightweight all-MiniLM-L6-v2 model). Instead of a sparse 20,000-dimensional array, the transformer projects every ticket into a dense, continuous 384-dimensional space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GifZ-fUkKEsZq1DzhGnVrg.png" /><figcaption><em>TF-IDF UMAP projection</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SS7Sii1-5HAa0FNh44jo6w.png" /><figcaption>sentence-transformers UMAP projection</figcaption></figure><p>By switching to dense embeddings, the spatial distance between vectors now mathematically mirrors their conceptual similarity. We can confidently feed this dense (27000, 384) matrix into Scikit-Learn’s highly optimized K-Means++ implementation, bypassing our pure Python memory bottlenecks via Scikit-Learn&#39;s compiled C extensions.</p><pre>from sentence_transformers import SentenceTransformer<br>import numpy as np<br><br><br>sentence_transformer = SentenceTransformer(&#39;all-MiniLM-L6-v2&#39;)<br><br>def embed_tickets(tickets: list[str]) -&gt; np.ndarray:<br>    embeddings = sentence_transformer.encode(<br>        tickets,<br>        batch_size=14,<br>        normalize_embeddings=True,<br>        show_progress_bar=True<br>    )<br><br>    return embeddings</pre><h4>The Automation Boundary: Why K-Means Needs HDBSCAN</h4><p>With Scikit-Learn handling the K-Means clustering, our system can successfully group tickets. But K-Means has a dangerous flaw for production routing: it forces <em>every single data point</em> into a cluster.</p><p>If a customer types random gibberish or a highly obscure multi-part complaint, K-Means will forcefully assign it to whichever centroid happens to be closest. In a live support environment, confidently misrouting ambiguous tickets degrades the customer experience.</p><p>To solve this, we introduce <strong>HDBSCAN</strong> (Hierarchical Density-Based Spatial Clustering of Applications with Noise) as our diagnostic layer. HDBSCAN does not guess a rigid KK number of clusters; it looks for dense neighborhoods of data.Crucially, if a ticket falls in the sparse ocean between clusters, HDBSCAN labels it as -1 (Noise).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*umyrrgTPnoUbdB7RN5NIxg.png" /></figure><p>This noise ratio dictates our automation boundary. The pure clusters are safely routed by the algorithm, while the -1 grey dots represent edge cases that must be kicked to a human triage queue.</p><pre>from hdbscan import HDBSCAN<br>from umap import UMAP<br><br>umap_reducer = UMAP(<br>    n_neighbors=15,<br>    n_components=2,<br>    min_dist=0.1,<br>    metric=&#39;cosine&#39;,<br>    random_state=42<br>)<br><br>X_umap = umap_reducer.fit_transform(X_embed)<br><br>clusterer = HDBSCAN(<br>    min_samples=5,<br>    min_cluster_size=15,<br>    metric=&#39;euclidean&#39;,<br>    cluster_selection_method=&#39;eom&#39;<br>)<br><br>hdbscan_labels = clusterer.fit_predict(X_umap)<br>hdbscan_labels</pre><h4>The FAISS Retrieval Layer &amp; Final Architecture</h4><p>Assigning a cluster is only half the battle. A true support intelligence system must instantly retrieve historical tickets to give agents context.</p><p>If we use a standard np.dot() linear scan to find the nearest neighbors across 27,000 dense vectors, the O(N)O(N) time complexity blocks the server thread and spikes API latency.</p><p>We solve this with <strong>FAISS (Facebook AI Similarity Search)</strong>. Instead of a linear scan, we index the dataset using an HNSW (Hierarchical Navigable Small World) graph. FAISS navigates this multi-layered graph to execute approximate nearest neighbour (ANN) retrieval.</p><pre>import faiss<br>import numpy as np<br><br>class TicketIndexer:<br>    def __init__(self, vector_dim: int = 384, use_hnsw: bool = True):<br>        &quot;&quot;&quot; Initialises the class whether to use hnsw &quot;&quot;&quot;<br>        self.dim = vector_dim<br>        if use_hnsw:<br>            # Approximate Nearest Neighbor (HNSW Graph)<br>            # 32 represents the number of bi-directional links per node<br>            self.index = faiss.IndexHNSWFlat(vector_dim, 32)<br>        else:<br>            # Brute force search<br>            self.index = faiss.IndexFlatL2(vector_dim)<br><br>    def build_index(self, embeddings: np.ndarray):<br>        &quot;&quot;&quot;Loads the dense vectors and builds the index.&quot;&quot;&quot;<br>        embeddings_flattened = np.ascontiguousarray(embeddings, dtype=np.float32)<br>        self.index.add(embeddings_flattened)<br><br>    def search(self, query_vector: np.ndarray, k: int = 5):<br>        &quot;&quot;&quot;Search the query against the built index&quot;&quot;&quot;<br>        query_flattened = np.ascontiguousarray(query_vector, dtype=np.float32)<br>        distances, indices = self.index.search(query_flattened, k)<br>        return distances[0], indices[0]</pre><p><strong>Retrieval Benchmark:</strong></p><ul><li><strong>Linear Scan (NumPy):</strong> Blocks the thread; scales linearly with dataset size.</li><li><strong>FAISS HNSW:</strong> Returns the top 5 semantically similar historical tickets out of 27,000 in <strong>&lt; 1 millisecond</strong>.</li></ul><h3>Conclusion</h3><p>When started uncovering these algorithms, I had already realised that the ML brain alone won’t be self explanatory unless something visually appealing catches the viewer’s attention. Hence wrapped the brain around a simple yet not so complex streamlit and fastapi backend. After completing decision trees, I realised something else. I wanted to pursue my learning in the path of modern LLM, RAG concepts so that I can familiarise myself with vectorisation and embeddings and move towards a production level, real world use case scenarios as close as possible.</p><p><a href="https://github.com/Sathwik-007/ticket-intelligence.git">GitHub - Sathwik-007/ticket-intelligence</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e480156a56ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision Trees with LightGBM style Quantile binning and Histogram optimisation: How I built…]]></title>
            <link>https://medium.com/@sathwikpinninti/decision-trees-with-lightgbm-style-quantile-binning-and-histogram-optimisation-how-i-built-4498c84465c6?source=rss-b2f0a7b28c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4498c84465c6</guid>
            <category><![CDATA[lightgbm]]></category>
            <category><![CDATA[gini-impurity]]></category>
            <category><![CDATA[rca]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[decision-tree-classifier]]></category>
            <dc:creator><![CDATA[Sathwik Pinninti]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 17:51:11 GMT</pubDate>
            <atom:updated>2026-09-14T17:51:11.394Z</atom:updated>
            <content:encoded><![CDATA[<h3>Decision Trees with LightGBM style Quantile binning and Histogram optimisation: How I built Automated Root cause Analysis engine from scratch.</h3><p><a href="https://github.com/Sathwik-007/Automated_RCA.git">GitHub - Sathwik-007/Automated_RCA</a></p><p>I think there were times when I encountered few challenges in terms of increasing complexity of whatever I start to explore, making me decide whether I want to go through the hard path or the easier one. And it doesn’t stop there. As I take the harder path, it starts from harder turns, steeper roads, to hills of upstreams and valleys of despair. But, I truly believe the unpleasant roads would teach me the geography of the place, make me familiar to the things I come across this difficult path just so I know how to still keep moving if I were to go through same path again or face similar challenges else where. And that they would toughen me up and tranform me into a less defeatable version.</p><p>When I started exploring decision trees, the standard version of for loop architecture involved, calculating unique values in a feature and checking each individual value a threshold to know whether they are resulting in a best split or not which was fine for smaller datasets with a few hundreds or thousands of rows. But most of the real world data like <strong>HuggingFace’s LEMMA RCA Cloud computing datasets</strong> are usually very complex, messy, unstructured and sometimes even have missing values and recorded failures. Apparently (as per the processed and extracted version of the dataset) it has 172K rows across &gt;1.1K features and every row has atleast one unique value. Imagine if a decision tree had to find the best split based on calculating unique values in a given feature. The following is the overview of the time complexity of the traditional approach.</p><h3>The True Complexity of Original Loop</h3><p>The original for-loop implementation was actually much worse than <strong>O(NlogN).</strong></p><p>If a node has N samples and F features, our naive approach did this:</p><ul><li>Used np.unique() to find thresholds: <strong>O(F⋅NlogN)</strong></li><li>Iterated through all N thresholds and applied a boolean mask (col &lt;= thresh) to count the left and right splits. Applying a mask to an array of size N takes <strong>O(N)</strong> time.</li><li>Checking N thresholds with an O(N) mask operation results in <strong>O(N²) time complexity per feature</strong>.</li></ul><p>Across 1,194 features and 172,000 rows, an <strong>O(N²)</strong> operation is mathematically paralysing. This is why your original notebook would have taken weeks to train a single tree.</p><blockquote>So how did I manage to build a tree that is capable of training in a few seconds which would typically take days to train?</blockquote><blockquote>Whether have I computed unique values for a dataset which has all unique rows for at least one feature?</blockquote><p>We will wind back to how we solved this problem. But lets pretty much for now deep dive into the mathematical foundations of the traditional approach that would build a solid foundation of designing these Quantile Decision trees.</p><h3>The Math behind …</h3><p>Before building decision trees, we need to understand 3 key concepts</p><ul><li>How do we measure chaos in a given mixture of data?</li><li>How do we decide what is the best split that best separates the mixture of data and attempts to make it pure as it continues to split?</li></ul><p>Inorder to measure chaos, we can either choose <strong>Gini Impurity</strong> or <strong>Entropy. </strong>I chose Gini Impurity as it doesn’t require logarithmic calculation but typically resolves to sum of squares</p><h4><strong>Gini Impurity</strong></h4><p>The algorithm needs to know if a group of data points is “pure” (all one class) or “mixed” (multiple classes). But computers need a single number to represent that state so they can minimize it.</p><p>Let’s assume a node contains data points belonging to <em>C</em> different classes. Let pi​ be the probability of randomly picking a data point that belongs to class <em>i</em>.</p><p>Because probabilities must represent the entire dataset, the sum of the probabilities of all classes must perfectly equal 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*0RCeURYQu2PCK9-44IdyPw.png" /></figure><p>To make a misclassification, two distinct, independent events must happen in sequence:</p><ol><li><strong>Event A (The Draw):</strong> You randomly draw a data point that actually belongs to class <em>i</em>. The probability of this happening is pᵢ.</li><li><strong>Event B (The Guess):</strong> You assign a label to it, but you guess wrong. You guess based on the node’s distribution, so you have a probability pⱼ​ of guessing class <em>j</em>. For your guess to be wrong, <em>j</em> must not equal <em>i</em>.</li></ol><p>Because these events are independent, the probability of <em>both</em> happening is their product:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z7BtKSB7-qXB7oMAJ--xtw.png" /></figure><p>What is the mathematical probability of “guessing anything other than <em>i</em>”?</p><p>Since the sum of all probabilities is 1, the probability of guessing <em>wrong</em> when the true class is <em>i</em> is simply 1 minus the probability of guessing <em>right</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*Nib8AyaZGI_ouXEOT8TN7A.png" /></figure><p>Now, substitute this back into our equation from Step 2. The probability of picking a data point of class <em>i</em> AND misclassifying it is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*EqD8T85d5EXNQNuCxxR96Q.png" /></figure><p>Errorᵢ only gives us the chance of making a mistake if we happen to draw class <em>i</em>. But our node contains <em>C</em> total classes.</p><p>To find the <strong>Total Expected Error</strong> for the entire node, we must sum this misclassification probability across every single class from <em>i=1</em> to <em>C</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*cpwGWxVsIF81MTolwpK-DQ.png" /></figure><p>This equation is the literal mathematical translation of our original question. The rest is just algebra.</p><p>We distribute the pᵢ term through the parentheses:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*BjCLfnXWKsDlknzzsMFHrw.png" /></figure><p>In mathematics, the summation of a difference is equal to the difference of the summations. We can split this into two separate terms:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*YNLmF3KhTqtSsKyLM60Hmg.png" /></figure><p>Now look closely at the first term. As we established in Step 1, the sum of all class probabilities in the node must always equal exactly 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*0RCeURYQu2PCK9-44IdyPw.png" /></figure><p>That’s how we arrive at the final equation used in the algorithm:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*7FUb6ERgQGTJMq_R1qH3BQ.png" /></figure><h4>Information Gain</h4><p>To derive Information Gain, we must shift our perspective. Gini and Entropy measure the chaos isolated inside a <em>single</em>bucket. But a Decision Tree’s job is to split data.</p><p>Information Gain is the mathematical formulation of a very simple question: <strong>“How much chaos did this specific split remove from the system?”</strong></p><p>Before we make any decisions, we have a starting bucket of data. We will call this the <strong>Parent Node</strong> (Nparent​). Using either Gini or Entropy, we calculate the chaos of this starting bucket. We will call this baseline value Iparent​.</p><p>The algorithm looks at a feature and splits the Parent Node into two new buckets: the <strong>Left Child</strong> (Nleft​) and the <strong>Right Child</strong> (Nright​).</p><p>We then calculate the individual chaos of both new buckets, giving us Ileft​ and Iright​.</p><p>The intuitive (but mathematically incorrect) way to measure the gain would be to simply subtract the children’s chaos from the parent’s chaos:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*-73NsQkgmtaprol3YShirQ.png" /></figure><p>Why is this wrong? <strong>Because it ignores the volume of data.</strong></p><p>Lets take an example of ticketing system. Imagine our parent node has <strong>100 tickets</strong>. We split it, but the feature we chose is terrible.</p><ul><li><strong>Left Child:</strong> Gets 99 tickets, still highly mixed (high chaos).</li><li><strong>Right Child:</strong> Gets exactly 1 ticket. Because a bucket with 1 ticket is perfectly pure, its chaos is 0.</li></ul><p>If we just averaged or subtracted the raw impurity scores, that one tiny, pure bucket would artificially make the split look highly successful.</p><p>To solve the volume problem, we must calculate the <strong>Expected Impurity</strong> after the split. We need to calculate an “Expected Value” by weighting each outcome by the probability of landing in that bucket.</p><ul><li>The probability of a data point going to the left bucket is the fraction of total samples it holds: <strong>(​Nleft/Nparent​​).</strong></li><li>The probability of a data point going to the right bucket is: (<strong>Nright​​/Nparent).</strong></li></ul><p>We multiply each child’s impurity by its weight, and add them together. This gives us the true, volume-adjusted chaos of the system <em>after</em> the split:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KQGR3hsKPV_u2ofPzEQi-A.png" /></figure><p><strong><em>Information Gain (IG)</em></strong> is strictly defined as the reduction in chaos. Therefore, we take our starting baseline (Information Gain on Parent) and subtract our volume-adjusted chaos (Information gain on Left child and Right child with volume adjustment).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*oZz8ZI1J3MIn4EE90I1QCA.png" /></figure><h3>Beyond import LightGBM: Engineering a Fast-Histogram Decision Tree from Scratch in Pure NumPy</h3><p>In modern cloud infrastructure, a targeted fault injection or memory leak can trigger cascading failures across hundreds of microservices. During a recent chaos engineering simulation using the Hugging Face Cloud LEMMA dataset, a manually injected pod defect took three hours of grueling log parsing to isolate. Three hours of downtime in production is catastrophic.</p><p>The solution is obvious: build a Root Cause Analysis (RCA) engine to ingest continuous system telemetry and isolate degrading metrics in milliseconds.</p><p>If you are a machine learning engineer reading this, your first instinct is likely to run from sklearn.tree import DecisionTreeClassifier or import lightgbm. It works, it’s highly optimized, and it takes three lines of code. So why did I spend weeks tearing open the black box of Gradient Boosting frameworks to build a high-performance decision tree from scratch in pure NumPy?</p><p>Because standard libraries abstract away the realities of memory and time complexity. To truly understand how enterprise AI scales, you need <strong>mechanical sympathy. A</strong>n understanding of how the algorithm interacts with the underlying hardware.</p><p>Here is how I reduced a 52.5-billion-operation computational bottleneck down to 206 million lookups, and built a real-time cloud anomaly detector without a single ML dependency.</p><h3>The O(N²) Trap and the Sorting Bottleneck</h3><p>When dealing with Kubernetes cluster telemetry, the data is inherently high-dimensional. After pivoting and flattening the hierarchical node and pod metrics, my training matrix sat at <strong>172,686 rows and 1,194 continuous floating-point features</strong>.</p><p>If you attempt to write a naive decision tree in Python, you will hit a mathematical wall almost immediately. To find the best split at any given node, a standard algorithm identifies every unique value in a feature column, tests it as a threshold, and applies a boolean mask (X &lt;= threshold) to calculate the Gini impurity.</p><p>Applying an O(N) boolean mask across <strong><em>N</em></strong> unique thresholds results in an <strong><em>O(N²)</em></strong> time complexity per feature. Across 1,194 columns and 172,000 rows, this requires roughly <strong>52.5 billion array evaluations</strong> per node. A pure Python loop would take weeks to train.</p><p>Production libraries like scikit-learn avoid this <strong><em>O(N²)</em></strong> trap by sorting the continuous arrays first (<strong><em>O(N \log N)</em></strong>) and walking linearly down them. But sorting 1,194 massive arrays of continuous floats at every single node is still a massive compute drain.</p><p>To achieve LightGBM-level speed, we need to eliminate the sorting step entirely.</p><h3>Optimisation 1: Quantile Binning (Destroying the Sort)</h3><p>The first C-level memory optimization is <strong>Quantile Binning</strong>. Instead of asking the tree to evaluate continuous float64numbers (e.g., 52.503948ms), we discretize the data before training.</p><p>By using NumPy’s percentile and searchsorted functions, the engine maps the continuous telemetry into 255 discrete integer bins.</p><pre># A simplified look at the binning transformation<br>percentiles = np.linspace(0, 100, 255)<br>bins = np.percentile(col, percentiles)<br>X_binned[:, feat] = np.searchsorted(bins, X[:, feat])</pre><p>The impact on memory and speed is immediate. We convert a massive matrix of heavy floats into an array of C-optimized uint32 integers (0 to 255). Because the possible values are strictly bounded, the algorithm <strong>never has to sort the data during the split-search phase again</strong>.</p><h3>Optimisation 2: Fast-Histogram Subtraction (O(N) to O(1))</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qemk_nYQfYE0lwiavw7ElQ.png" /></figure><p>Even with binned data, running a boolean mask across the array to count the left and right child nodes for every threshold takes <strong><em>O(N)</em></strong> time. This is where we replicate LightGBM’s crown jewel: <strong>Fast-Histogram Subtraction</strong>.</p><p>Instead of filtering the rows over and over, we scan the feature column exactly once using np.bincount to build a frequency histogram. Then, we use np.cumsum to instantly calculate the running totals for every possible threshold.</p><pre># The O(1) Split Engine<br>bin_counts = np.bincount(feat_data[y == 1], minlength=max_bins+1)<br>cum_counts = np.cumsum(bin_counts)<br><br>for thresh in range(max_bins):<br>    # O(1) mathematical lookup instead of O(N) array masking<br>    n_left_anomalies = cum_counts[thresh]<br>    n_right_anomalies = total_anomalies - n_left_anomalies</pre><p>By substituting an <strong><em>O(N)</em></strong> array evaluation with an <strong><em>O(1)</em></strong> index lookup, the time complexity drops off a cliff. Benchmarking this exact loop against the naive masking approach revealed a reduction from ~<strong>52.5 billion operations down to just ~206 million</strong>.</p><h3>The Precision-Recall Tradeoff: Tuning for the Real World</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s43Ew0T4qkSdWi3zkAVjcQ.png" /></figure><p>Algorithms don’t exist in a vacuum; they serve a business objective. In cloud infrastructure, anomalies are incredibly rare (massive class imbalance), and the cost of a False Negative (missing a critical server outage) is astronomically higher than a False Positive (waking an SRE up for a false alarm).</p><p>Standard accuracy metrics are useless here. To force the tree to care about the minority class, I customized the Gini impurity calculation with an anomaly_weight parameter.</p><p>By treating a single anomaly as if it carried the weight of 50 normal rows, the tree mathematically penalized itself for missing defects. The final metrics on the 172k dataset speak for themselves:</p><ul><li><strong>Recall:</strong> 1.0000 (72,287 / 72,287 critical defects caught)</li><li><strong>F1-Score:</strong> 0.9525</li><li><strong>Precision:</strong> 0.9094</li></ul><p>We deliberately traded a 9% drop in precision to guarantee that absolutely zero critical outages slipped through the cracks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ejcno6o0-rUJaH1fHYcAkQ.png" /></figure><h3>Native Explainability: Why SREs Don’t Want SHAP</h3><p>The AI industry is obsessed with post-hoc explainability tools like SHAP (SHapley Additive exPlanations). While SHAP is brilliant for forcing black-box deep learning models to make sense via game-theory approximations, it is computationally heavy and often abstract.</p><p>A Site Reliability Engineer debugging a live outage doesn’t want to read that “Latency contributed +0.4 to the anomaly score.” They need hard logic.</p><p>Because this engine is a highly optimized decision tree, it operates as a fully transparent “white-box.” I wrote a recursive traversal engine that reverses the quantile mapping, traces the exact physical memory path the data took through the nodes, and outputs human-readable RCA rules instantly.</p><pre>Analyzing Data Point at Index 724:<br>--- Prediction Explanation Path ---<br> -&gt; Moving [RIGHT] - Feature name: Cluster Level | Metric: Latency, Value: 61.0000 &gt; Threshold: 52.5000<br>     -&gt; Final Classification: ANOMALY</pre><h3>The Roadmap: What We Left on the Table</h3><p>Implementing Quantile Binning and Fast-Histogram Subtraction from scratch solves the most crippling bottleneck of decision tree training. For our specific RCA architecture. Processing 172,686 rows in milliseconds and adding class weight to anomaly class to achieve 100% anomaly recall, this engine is more than sufficient. We achieved our goal without bloating the codebase.</p><p>However, true framework performance parity means acknowledging the remaining arsenal of techniques that make LightGBM the industry standard. While we stopped at the histogram optimization, there are four major algorithmic leaps left on the table that I am deeply enthusiastic about implementing in the next iteration:</p><ol><li><strong><em>Leaf-Wise (Best-First) Tree Growth:</em></strong> Standard trees (like the one implemented here) grow <em>level-wise</em> (depth-first), splitting nodes across the same depth simultaneously. LightGBM grows <em>leaf-wise</em>, aggressively splitting only the single leaf that offers the maximum loss reduction, regardless of depth. This results in deeper, asymmetrical trees that converge faster and achieve higher accuracy on massive datasets.</li><li><strong><em>Gradient-based One-Side Sampling (GOSS):</em></strong> Instead of evaluating every row to find splits, GOSS intelligently drops rows that the model already predicts accurately (instances with small gradients) and focuses computational power exclusively on the “hard” examples (large gradients). It is a brilliant statistical shortcut we have yet to implement.</li><li><strong><em>Exclusive Feature Bundling (EFB):</em></strong> In high-dimensional cloud telemetry, many features are mutually exclusive (e.g., if one specific microservice is active, another might be perfectly idle/sparse). EFB mathematically bundles these non-overlapping sparse features together into a single feature, drastically compressing the 1,194-column matrix before training even begins.</li><li><strong><em>Native Missing Value Optimisation:</em></strong> Our current pipeline requires external handling or interpolation for NaNvalues (dropped or forward-filled metrics). LightGBM natively ignores missing values during histogram building and automatically learns whether sending NaNs to the left or right child node minimizes the loss function.</li></ol><p>Building these last four optimisations from scratch in pure NumPy is the ultimate test of mechanical sympathy. While the current engine already outpaces traditional standard algorithms by orders of magnitude, bridging this final gap is the next logical frontier.</p><p><a href="https://github.com/Sathwik-007/Automated_RCA.git">GitHub - Sathwik-007/Automated_RCA</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4498c84465c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[K-Nearest Neighbours: Building a Fraud transaction detection Model from scratch]]></title>
            <link>https://medium.com/@sathwikpinninti/k-nearest-neighbours-building-a-fraud-transaction-detection-model-from-scratch-f3ea61fde180?source=rss-b2f0a7b28c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3ea61fde180</guid>
            <category><![CDATA[hnsw]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[faiss]]></category>
            <category><![CDATA[euclidean-distance]]></category>
            <category><![CDATA[knn-algorithm]]></category>
            <dc:creator><![CDATA[Sathwik Pinninti]]></dc:creator>
            <pubDate>Tue, 28 Jul 2026 19:08:40 GMT</pubDate>
            <atom:updated>2026-08-03T05:20:00.741Z</atom:updated>
            <content:encoded><![CDATA[<p>I finally hit the next algorithm in my list which turned out to be quite a different one from all the ones we’ve learnt previously. Unlike regression or classification models we’ve built till this point, we have no loss estimation, we have no sigmoid/softmax function to transform our raw scores/probabilities, we have no for loop, no forward or backward pass techniques here, no updating gradients etc (<em>For it may have, in a different way of implementation. I’d rather give a disclaimer than sounding like an absolute idiot)</em>. It went with a simple Euclidean distance formula which we might have learnt during our schooling. It can as well go as complex as KD-trees/ball-trees or an <strong>ANN</strong> (<strong>Approximate Nearest Neighbour</strong>. Kindly donot confuse it with the term “Artificial Neural Network”. <em>I did tho</em>), <strong>HNSW (Heirachical Navigable Small World) a FAISS index </strong>implementation which we did as a part of this project. (note: I didn’t implement FAISS index from scratch, which I wanted to but rather used faiss-cpu to do the heavy lifting).</p><p>Okay, like always lets dive into the <strong>Math </strong>first<strong>.</strong></p><h3>The Starting Point: How Do We Measure “Near”?</h3><p>The entire premise of KNN is based on the idea that similar data points exist in close proximity to each other. To make a prediction for a new, unseen data point, we just need to find the K points in our dataset that are closest to it.</p><p>But how do we define “closest”? We need a mathematical definition of distance.</p><h4>The 2D Origin: Pythagorean Theorem</h4><p>If you have two points on a 2D plane, P=(x1​,y1​) and Q=(x2​,y2​), the shortest distance between them is a straight line. You likely remember calculating the hypotenuse of a right triangle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*WTb55HHSy1kIk1Cnwyc65w.png" /></figure><p>Solving for distance <em>(d)</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*deAEatQMFp8pFno3KY4B5g.png" /></figure><h4>Generalizing to N-Dimensions (Euclidean Distance)</h4><p>In machine learning, your datasets rarely have just two features. You might have 5, 20, or 100 features representing a single data point. We need to expand our distance formula from a 2D space to an N-dimensional vector space.</p><p>Let p and q be two data points (vectors) in an n-dimensional feature space:</p><p><strong><em>p=(p1​,p2​,…,pn​)</em></strong></p><p><strong><em>q=(q1​,q2​,…,qn​)</em></strong></p><p>The <strong>Euclidean Distance</strong> between these two multi-dimensional vectors is the starting point of our KNN math:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*5NzELbVZAHJLQc6lvUq8Lg.png" /></figure><p><strong>Expanded form:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rcg6v4F4wcMkTzkjXuijfg.png" /></figure><p>Yup, that sums up all the math we needed to know. Surprising isn’t it? Didn’t I give a headsup? Like I said, no more transforming raw scores, updating gradients etc. A Euclidean distance calculation is what that sits behind all this.</p><p>Now lets move on to our project implementation,</p><h3>Finding Fraud in the Desert: Rethinking Anomaly Detection in High-Dimensional Space</h3><p>If we spend enough time looking at machine learning tutorials, we start to believe that throwing a Random Forest or an XGBoost model at a dataset will magically solve any classification problem. But the real world is rarely that accommodating especially in fintech.</p><p>When confronted with a credit card fraud dataset of 284,807×29 (representing over 280,000 transactions across 29 PCA-transformed features), the classic supervised learning approaches felt brittle. Fraud is highly imbalanced; true anomalies are needles in a massive, noisy haystack.</p><p>Instead of trying to teach a model what fraud “looks like,” what if we only taught the model what <em>normal</em> behaviour looks like, and treated everything else with deep suspicion?</p><p>This led us down the rabbit hole of semi-supervised learning, distance mathematics, and eventually, modern vector search. Here is the journey of how we built it, the mathematical roadblocks we hit, and the epiphanies that changed our approach.</p><h3>Part I: The Mathematics of Isolation</h3><p>The core philosophy of our semi-supervised approach was simple: we trained our model <strong>exclusively on normal transactions</strong>. Our test set, however, contained both normal and fraudulent points.</p><p>The deciding factor for any new transaction was strictly geometrical:</p><ul><li><strong>Normal:</strong> The data point sits tightly packed among other normal historical data points.</li><li><strong>Fraud:</strong> The data point is located far away from the bounds of normal behavior.</li></ul><h4>The “Crowded Stadium” Analogy</h4><p>The core concept we are trying to explain is why we use a high number for k (like the 100th neighbor) instead of just looking at the single closest data point (k=1) when trying to detect an anomaly.</p><p>Imagine a massive, sold-out football stadium. Inside, tens of thousands of fans are packed tightly into the stands. This dense crowd represents our <strong>“Normal”</strong> transaction data.</p><p>Miles away, far outside the city, there is an empty desert. Out in this wasteland is a lone wanderer. This is our <strong>“Fraud”</strong>anomaly.</p><p>Our algorithm’s job is to figure out where a new, unknown person is located, based <em>only</em> on how far they have to walk to find other people.</p><p>Here is why picking the right <strong><em>‘k’</em></strong> makes all the difference:</p><h4>The Flaw of k=1 (The Local Trap)</h4><p>Suppose we configure our algorithm to check only the <strong>1st nearest neighbor</strong> (k=1). We drop a pin on our suspect in the desert and say, “Find the closest person.”</p><p>Here is the problem: What if our fraudster happens to be standing next to one other lost wanderer? Or maybe there’s a statistical glitch nearby (noise in the data). If we only measure the distance to that <em>single</em> closest person, the distance is just 2 feet.</p><p>The algorithm looks at that 2-foot distance and concludes, “This person is extremely close to someone else! They must be in the stadium!” The anomaly slips right through the cracks because we only looked at their immediate micro-environment.</p><h4>The Power of k=100 (Proving Density)</h4><p>To fix this, we change the rules. We don’t ask, “How far is your closest neighbor?” Instead, we ask, <strong>“How far out do you have to draw a circle to capture exactly 100 people?”</strong> (k=100).</p><p>Now, let’s test both locations again:</p><ul><li><strong>Scenario A: The Person in the Stadium (Normal)</strong> If you are standing in the middle of a packed stadium, finding 100 people is effortless. You only need to draw a circle with a radius of maybe 10 or 15 feet to capture 100 fans. The distance to your 100th neighbor is tiny.</li><li><strong>Scenario B: The Person in the Desert (Fraud)</strong> If you are the fraudster out in the desert, having one or two lost friends nearby doesn’t help you anymore. To find your <em>100th</em> closest neighbor, you have to draw a massive circle that stretches for miles, all the way back into the city and into the stadium. The distance to your 100th neighbor is enormous.</li></ul><h4>The Mathematical Takeaway</h4><p>By anchoring our logic to the k-th neighbor instead of the 1st, we aren’t just asking, “Are you alone?” We are mathematically proving, <strong>“Are you inside a dense cluster?”</strong></p><p>If the radius required to capture 100 neighbors is short, the transaction is safely packed inside the normal baseline. If the radius is miles wide, they are standing in the desert and the transaction is flagged as fraud.</p><h3>The Euclidean Engine</h3><p>Under the hood, this requires calculating the straight-line distance between vectors in a 29-dimensional space. We used the classic Euclidean distance metric. For an incoming test transaction p and a historical normal transaction q, the distance is calculated as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*VUoGoW-Qk77pq9FGAQE-DA.png" /></figure><p>To visualize this, imagine a tiny slice of our matrix with just 3 features (V1​,V2​,V3​):</p><ul><li><strong>Normal Baseline:</strong> [0.1,−0.2,0.05]</li><li><strong>Test Transaction A (Normal):</strong> [0.15,−0.1,0.02]</li><li><strong>Test Transaction B (Fraud):</strong> [4.5,−3.2,7.8]</li></ul><p>When we run the math, Transaction A yields a tiny Euclidean distance (it is inside the stadium). Transaction B’s massive variance squares exponentially, resulting in a huge distance metric. Transaction B is wandering in the desert.</p><h3>Part II: The Reality Check and the Curse of Dimensionality</h3><p>In our initial implementation, we took a subset of the dataset and computed these Euclidean distances purely using NumPy arrays. It worked conceptually, but the performance metric was a highly underwhelming baseline score of <strong>0.19</strong>. Furthermore, brute-forcing d(p,q) across hundreds of thousands of records was computationally agonizing.</p><p>We needed optimization.</p><p>Our first thought was to use a <strong>KD-Tree</strong> (k-dimensional tree), a classic data structure that partitions space to make neighbor searching faster. It sounded perfect in theory, but it led to our first major epiphany: <strong>The Curse of Dimensionality.</strong></p><p>KD-Trees work beautifully in 2D or 3D space. But in higher dimensions, the math breaks down. For a KD-Tree to effectively partition space without having to search almost every single node anyway, the number of data points (N) needs to be significantly larger than 2D (where D is the number of dimensions).</p><p>Our dataset had 29 dimensions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*4rnmL3vTKjXircgyJ7ZnZg.png" /></figure><p>Because our dataset size was vastly smaller than 536,870,912 (which is 229), the KD-Tree collapsed back into a gloriously inefficient brute-force search.</p><h3>Part III: The Highway to Optimization (HNSW &amp; FAISS)</h3><p>Realizing that exact mathematical precision was actively harming our system’s viability, we decided to adopt a more skeptical, pragmatic mindset. We didn’t need the <em>absolute</em> nearest neighbor; an <strong>Approximate Nearest Neighbor (ANN)</strong> would do the job just fine, provided it was lightning fast.</p><p>This led us to <strong>HNSW (Hierarchical Navigable Small World)</strong> graphs, implemented via Meta’s <strong>FAISS</strong> library. FAISS is famously used in modern Generative AI to look up vector embeddings for Large Language Models but it turns out, it is equally outperforming to credit card fraud.</p><h4>The Highway Analogy</h4><p>If brute-force Euclidean math is like driving down every single driveway in a city to find a specific house, HNSW is like using the interstate highway system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ErdE4MdcNtRTlecP3XML_Q.png" /></figure><p>When a new transaction enters the FAISS index, it doesn’t start at the bottom checking neighbours. It starts at the top layer of the hierarchy — the “highways.” It makes massive, long-distance jumps across the vector space to quickly get into the right general zip code. Once it is close, it takes an “off-ramp” to a lower layer (local roads), finally navigating the short Euclidean distances to find the nearest normal transactions.</p><p>By shifting from our brute-force NumPy implementation to the FAISS HNSW index, our performance baseline skyrocketed from <strong>0.19 to 0.50</strong>. We traded a negligible fraction of mathematical perfection for a massive leap in architectural efficiency.</p><pre>no_of_features = X_train_scaled.shape[1]<br>index = faiss.IndexHNSWFlat(no_of_features, 32)<br>index.add(X_train_scaled)<br><br># FORCE FAISS to be more accurate (Default is usually 16, let&#39;s make it 256)<br>index.hnsw.efSearch = 256<br><br># Increase &#39;k&#39; to account for the extreme density of 200,000 points<br>k = 100<br>distances, indices = index.search(X_test_scaled, k)<br><br>anomaly_scores = distances[:, k-1]</pre><h3>Part IV: Thresholds and Interpretability</h3><p>With the engine running efficiently, two critical challenges remained: deciding exactly where the “stadium” ended, and explaining our decisions.</p><h4>Maximising the F1 Score</h4><p>How far is <em>too far</em>? If you set the distance threshold too low, you decline legitimate customers (False Positives). If you set it too high, fraudsters walk away with the money (False Negatives).</p><p>We couldn’t just guess. We solved this by treating the threshold as a mathematical optimisation problem, calculating the precision and recall across a sliding scale of distances to maximise the F1​ score:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*JzKhVJjfwmNoct8nwRr3XA.png" /></figure><pre>import matplotlib.pyplot as plt<br>from sklearn.metrics import precision_recall_curve, auc<br><br># 1. Get your anomaly scores (Euclidean distances from your test points)<br># Assuming &#39;anomaly_scores&#39; contains the distance to the k-th neighbor<br># and &#39;y_test&#39; contains the actual labels (0 for normal, 1 for fraud)<br><br># 2. Compute Precision and Recall for all possible thresholds<br># Note: Since higher distances mean HIGHER probability of fraud, <br># we pass the raw anomaly scores directly.<br>precision, recall, thresholds = precision_recall_curve(y_test, anomaly_scores)<br><br># 3. Calculate the Area Under the Curve (AUPRC)<br>auprc_score = auc(recall, precision)<br>print(f&quot;Area Under the Precision-Recall Curve (AUPRC): {auprc_score:.4f}&quot;)<br><br># 4. Plot the AUPRC Curve<br>plt.figure(figsize=(8, 6))<br>plt.plot(recall, precision, label=f&#39;Inlined k-NN Anomaly Detector (AUPRC = {auprc_score:.4f})&#39;, linewidth=2)<br><br># Plot a baseline horizontal line representing a random classifier<br># A random classifier&#39;s precision is equal to the proportion of positive cases<br>baseline = np.sum(y_test == 1) / len(y_test)<br>plt.axhline(y=baseline, color=&#39;gray&#39;, linestyle=&#39;--&#39;, label=f&#39;Baseline (Random = {baseline:.4f})&#39;)<br><br>plt.xlabel(&#39;Recall (Fraction of Frauds Caught)&#39;)<br>plt.ylabel(&#39;Precision (Fraction of Flagged Transactions that are Actually Fraud)&#39;)<br>plt.title(&#39;Precision-Recall Curve for Anomaly Detection&#39;)<br>plt.legend(loc=&#39;lower left&#39;)<br>plt.grid(True, alpha=0.3)<br>plt.savefig(&#39;auprc_faiss.png&#39;)<br>plt.show()</pre><p>By plotting this curve, the math explicitly revealed the optimal cutoff point that balanced consumer friction with security risk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*5It_l2N-ZHSfvnCgA8hQTw.png" /></figure><h3>Opening the Black Box with SHAP</h3><p>Finally, simply telling a fraud investigator “The transaction was declined because its 29 value vector is far away” is unacceptable in the real world. They need to know <em>why</em>.</p><p>We integrated <strong>SHAP (SHapley Additive exPlanations)</strong> to bring model explainability to our pipeline. It allowed us to deconstruct the distance metric in real-time. We could instantly highlight exactly which PCA features (V4​,V11​, etc. even “Amount”) acted as propulsive forces that violently pushed the test point out of normal behavior and into the desert.</p><pre>feature_names = df.drop([&#39;Time&#39;, &#39;Class&#39;], axis=1).columns<br><br>def explain_anomaly(test_index, k=100):<br>    # grab the scaled test point and reshape the matrix to be 1 row and 28 column values (1, 28)<br>    test_point = X_test_scaled[test_index].reshape(1, -1)<br><br>    # grab the distances and indices of those test points that are nearer.<br>    distances, indices = index.search(test_point, k)<br><br>    # get the indices of k-1th col index so that we can grab it from X_train scaled class<br>    neighbour = indices[0][k-1]<br>    normal_neighbour = X_train_scaled[neighbour]<br><br>    squared_differences = (test_point[0] - normal_neighbour) ** 2<br><br>    # np.argsort sorts the elements smalles to largest but stores their order in values of their indices.<br>    sorted_feature_indices = np.argsort(squared_differences)[::-1]<br><br>    print(f&quot;--- ANOMALY EXPLANATION REPORT ---&quot;)<br>    print(f&quot;Test Transaction Index: {test_index}&quot;)<br>    print(f&quot;Total Anomaly Score: {distances[0][k-1]:.2f}\n&quot;)<br>    print(&quot;Top 5 Feature Culprits (What made this look like fraud):&quot;)<br><br>    top_features = []<br>    top_contributions = []<br>    <br>    for i in range(5):<br>        feature_index = sorted_feature_indices[i]<br>        feature_name = feature_names[i]<br><br>        score = squared_differences[feature_index]<br>        contribution = (score / np.sum(squared_differences)) * 100<br><br>        print(f&quot;{i+1}. {feature_name}: Contributed {score:.2f} ({contribution:.1f}% of anomaly)&quot;)<br><br>        top_features.append(feature_name)<br>        top_contributions.append(contribution)<br><br>    plt.figure(figsize=(8, 4))<br>    plt.barh(top_features[::-1], top_contributions[::-1], color=&#39;crimson&#39;)<br>    plt.xlabel(&#39;Squared Distance Contribution&#39;)<br>    plt.title(&#39;Top 5 Anomaly Drivers&#39;)<br>    plt.tight_layout()<br>    plt.savefig(&#39;top_5_anomaly_drivers.png&#39;)<br>    plt.show()</pre><pre>fraud_indices = np.where(y_test == 1)[0]<br>one_sample_fraud = fraud_indices[0]<br>one_sample_fraud<br><br>explain_anomaly(one_sample_fraud, k=100)</pre><p>Oh! I would also highlight that I implemented the following project as a live application so you can check it out. Deployed the frontend on Vercel and the ML brain and FastAPI backend on Render. You can randomly generate a 29 value vector according to the available buttons. As mentioned about SHAP, it would list which features have driven the generated test point to be flagged as fraud. It might not be the best in place application out there but the idea is to get exposure to production grade ML application deployments. I hope you like it ; ) .</p><p><a href="https://fraud-detector-puce.vercel.app/">fraud-dashboard</a></p><h3>Part V: When Exact Math Loses to Approximation</h3><p>Also as an extension, I also thought how can make our euclidean approach to also match the baseline of our HNSW implementation. Then I found out that faiss also has “<em>IndexFlatL2” </em>that is similar to computing Euclidean distance with a catch: We can send more than 10K rows to be indexed. In our case, we sent 75% of our train set (213605 rows) leaving behind 25% to our test set (71202).</p><blockquote><strong>Note</strong>: Here as well we go for semi-supervised approach by allowing our model to train on normal behaviour alone so that an abnormal transaction(a data point lying far from trained normal–behaviour-data-points) can be flagged as fraud.</blockquote><pre>import faiss<br>from sklearn.model_selection import train_test_split<br><br>X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42, stratify=y)<br>X_train.shape, X_test.shape, y_train.shape, y_test.shape<br><br>train_mean = np.mean(X_train, axis=0)<br>train_std = np.std(X_train, axis=0)<br><br>train_std[train_std == 0] = 1e-10<br><br>X_train_scaled = (X_train - train_mean) / train_std<br>X_test_scaled = (X_test - train_mean) / train_std<br><br>X_train_scaled = np.ascontiguousarray(X_train_scaled, dtype=np.float32)<br>X_test_scaled = np.ascontiguousarray(X_test_scaled, dtype=np.float32)<br><br>normal_indices = (y_train == 0)<br>X_train_scaled_normal = X_train_scaled[normal_indices]<br><br>n_features = X_train_scaled_normal.shape[1]<br>index = faiss.IndexFlatL2(n_features)<br><br># train on full dataset<br>index.add(X_train_scaled_normal)<br><br>distances, indices = index.search(X_test_scaled, k=100)<br><br>distances</pre><p>and the following are the results for AUPRC curve.</p><pre>k = 100<br><br>anomaly_scores = distances[:, k-1]<br>anomaly_scores<br><br>from sklearn.metrics import precision_recall_curve, auc<br>import matplotlib.pyplot as plt<br><br>precision, recall, thresholds = precision_recall_curve(y_test, anomaly_scores)<br>auprc_score = auc(recall, precision)<br><br># 8. Plot AUPRC<br>plt.figure(figsize=(8, 6))<br>plt.plot(recall, precision, color=&#39;#3b82f6&#39;, linewidth=2.5, <br>         label=f&#39;FAISS Exact Euclidean (AUPRC = {auprc_score:.4f})&#39;)<br><br># Aesthetics<br>plt.fill_between(recall, precision, alpha=0.1, color=&#39;#3b82f6&#39;)<br>plt.title(&#39;Precision-Recall Curve (Semi-Supervised Anomaly Detection)&#39;, fontsize=14, pad=15)<br>plt.xlabel(&#39;Recall (True Positive Rate)&#39;, fontsize=12)<br>plt.ylabel(&#39;Precision (Positive Predictive Value)&#39;, fontsize=12)<br>plt.legend(loc=&#39;lower left&#39;, fontsize=12)<br>plt.grid(True, linestyle=&#39;--&#39;, alpha=0.6)<br>plt.xlim([0.0, 1.05])<br>plt.ylim([0.0, 1.05])<br>plt.savefig(&#39;flat_l2_index.png&#39;)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Xb-1ETbOXRfg2_6J-JvtzQ.png" /></figure><p>Surprisingly, not much improvement on the baseline 0.2370. Still less than half of what HNSW baseline is.</p><p>How did exact math perform worse than an approximation?</p><p>The answer lies in how high-dimensional space reacts to anomalies and noise. Exact Euclidean distance calculates the absolute straight-line distance across all 29 dimensions. Because of this, it is hyper-sensitive to outliers. If a perfectly safe, normal transaction has just one highly unusual feature, the L2 formula squares that difference, artificially inflating the total spatial distance and instantly triggering a false positive alarm.</p><p>HNSW, on the other hand, is a graph-based topological structure. By navigating through layers of connected neighbors, it inherently smooths out these extreme distances. The “approximation” isn’t actually a flaw — it acts as a powerful form of geometric regularization. It relies on the dense gravity of the local neighborhood rather than a rigid, absolute distance calculation.</p><p>We hadn’t written bad code; we had just mathematically proven that in a 29-dimensional space, topological graph traversal is significantly more robust against noise than brute-force Euclidean distance.</p><h3>Conclusion</h3><p>Nevertheless its worth mentioning tearing down these magic algorithms apart and seeing what’s underneath, understanding the logic and building them on our own gives utmost satisfaction. Because now its no more a magic trick but a cooking recipe that was unfolded out to us. But math alone doesn’t solve real-world problems. True engineering challenge lies in translating that logic into a business context, like tuning high-dimensional distance thresholds to perfectly balance fraud prevention.</p><p>Taking that raw algorithmic brain and wrapping it in a production-ready FastAPI and Vite architecture bridges the massive gap between an isolated Jupyter notebook and a live cloud application. It transforms an abstract model into a tangible system, mirroring the exact engineering standards and end-to-end architectures used by top-tier product companies to solve real problems at scale.</p><p><a href="https://github.com/Sathwik-007/knn-fraud-detection.git">GitHub - Sathwik-007/knn-fraud-detection: This project is built to detect a transaction to be either fraudulent or legit. It has two implementations. Standard implementation with from-scratch KNN logic (Euclidean distance)and another one with FAISS.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3ea61fde180" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Softmax regression: Building a Clinical Triage Classifier from scratch]]></title>
            <link>https://medium.com/@sathwikpinninti/softmax-regression-building-a-clinical-triage-classifier-from-scratch-c2ce967c3c0b?source=rss-b2f0a7b28c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2ce967c3c0b</guid>
            <category><![CDATA[softmax-regression]]></category>
            <category><![CDATA[categorical-cross-entropy]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[clinical-trials]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <dc:creator><![CDATA[Sathwik Pinninti]]></dc:creator>
            <pubDate>Wed, 24 Jun 2026 17:43:27 GMT</pubDate>
            <atom:updated>2026-07-08T18:57:37.864Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://github.com/Sathwik-007/clinical-triage-classifier">GitHub - Sathwik-007/clinical-triage-classifier: This project outlines from scratch implementation of softmax regression which utilises softmax function to even out probabilities across k classes and cross entropy loss as loss function which were purely coded on NumPy and some Pandas</a></p><p>Building on my previous deep dives, I’m unboxing the next algorithm my list: <strong>Softmax Regression</strong>.</p><p>The biggest <strong><em>epiphany</em></strong> for me on this journey was realizing that the math we rely on like the loss functions and sigmoid curves from previous discussions isn’t just pulled from thin air. These formulas are elegant expressions that naturally evolve from a single, atomic piece of intuition.</p><p>So lets talk about what does Softmax regression aims to do: <strong>Softmax Regression</strong> (often called Multinomial Logistic Regression) is a supervised machine learning algorithm used for <strong>multi-class classification</strong>. It is a generalization of standard logistic regression that allows a model to predict the probability of an input belonging to one of three or more mutually exclusive classes.</p><p>Instead of returning a single probability for a binary outcome (e.g., “Is this a cat or not?”), Softmax Regression returns a probability distribution across all possible classes (e.g., “Is this a cat, a dog, or a bird?”).</p><blockquote>It aims to predict how likely it is, given a logit that it belongs to a particular class <strong>j </strong>given <strong>K </strong>classes.</blockquote><p>With that in mind, let’s dive in and build that core intuition from the ground up.</p><h3>Demystifying the Math: Maximum Likelihood and Cross Entropy Loss</h3><p>Imagine you have a dataset of N images. You want your model to look at Image 1 and assign a high probability to the correct answer, look at Image 2 and do the same, and so on.</p><p>In statistics, the <strong>Likelihood</strong> is the total probability that your model guesses the entire dataset correctly. If you flip a coin and want Heads AND Heads AND Heads, you multiply the probabilities together (0.5 * 0.5 * 0.5).</p><p>Similarly, the Likelihood of our model being completely right across N samples is the product of its correct predictions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dqwsBP08BbKfh5kTsAx_kQ.png" /></figure><p>Before we can multiply everything together, we need a mathematical way to extract the probability of just the <em>correct</em>class for a single image, ignoring the wrong classes.</p><p>Remember that our true labels (<em>y</em>) are one-hot encoded. If an image is a Dog (Class 2 out of 3), the true label is <em>y</em>=[0,1,0]. Suppose our model predicts probabilities <em>y^</em>​=[0.1,0.8,0.1].</p><p>How do we mathematically filter <em>y^</em>​ to only extract the 0.8? We use exponents. If we take our predictions (<em>y^</em>​) and raise them to the power of our true labels (<em>y</em>), something clever happens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*chlK9gZEgqORlHvpqDV4fw.png" /></figure><p>Because anything to the power of 0 is exactly 1, the wrong classes vanish into 1s, and only the correct class survives!</p><p>P(correct)=1⋅0.8⋅1=0.8</p><p>So, the general formula for the probability of being correct on a single sample with K classes is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*7O8G5z0zpbPkKYq2fFsm-g.png" /></figure><p>Now we want to multiply our probabilities accross <em>n </em>samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*VNbCMizG51DmBpymqBA7TQ.png" /></figure><p>This equation perfectly describes what we want our model to maximize. But we have a massive problem: <strong>Computer Underflow</strong>. If you multiply thousands of probabilities together (e.g., 0.8⋅0.9⋅0.4⋅0.8…), the number gets so small, so fast, that a computer’s memory rounds it down to exactly 0.0. The math breaks.</p><p>To fix the computer underflow problem, we apply the natural logarithm (log) to both sides of the equation.</p><p>By taking the log, our giant multiplication symbols (<strong>∏</strong>) turn into friendly, stable summation symbols (<strong>∑</strong>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*ghkoe8XJ2YnuahRKrPvvRg.png" /></figure><p>Simplifying the equation further it looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*4iRvOv6N8uG3Pp_QWYe-cQ.png" /></figure><p>In machine learning, algorithms like Gradient Descent are built to <strong>minimise</strong> things. They are designed to walk downhill to find the lowest possible error.</p><p>How do you turn a “Maximization” problem into a “Minimization” problem? You simply multiply it by −1.</p><p>Maximizing profit is the exact same math as minimizing loss. By adding a negative sign, our Log-Likelihood becomes the <strong>Negative Log-Likelihood</strong>, which is exactly what we call <strong>Categorical Cross-Entropy Loss</strong>.</p><p>Finally, because we want our loss to be independent of the size of our dataset, we divide by <strong><em>N</em></strong> to get the average loss per sample.</p><p>Now our final equation transforms into:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*L9d89rAz6AKa7xQDPgAAbw.png" /></figure><pre>def _cross_entropy_loss(self, y_true, y_pred):<br>        samples = y_true.shape[0]<br>        # Clip predictions to avoid log(0) which results in infinity<br>        y_pred = np.clip(y_pred, 1e-15, 1. - 1e-15)<br>        loss = -np.sum(y_true * np.log(y_pred)) / samples<br>        return loss</pre><p>That is how the <strong>Categorical Cross-Entropy Loss equation </strong>arrived.</p><h3>Demystifying the Math: From Raw Scores to Probabilities</h3><p>Our starting point is the standard linear equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*OUIo7Q3xDHbQKQhbAOmX0Q.png" /></figure><p>This equation generates a raw score (a logit). The problem is that <em>z</em> can be literally any number from −∞ to +∞. We need our model to output a <strong>probability</strong> (<em>p</em>), which is strictly locked between 0 and 1.</p><p>How do we mathematically bridge the gap between [−∞,∞] and [0,1]?</p><p>Statisticians realized they couldn’t predict probability directly with a linear line. So, they changed the target. Instead of predicting the probability of winning, they decided to predict the <strong>Odds</strong> of winning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*ZD5igTnliRlwNIB5YAUl0g.png" /></figure><p>If you have an 80% chance of winning (p=0.8), your odds are 0 * 20 * 8​=4. (You are 4 times more likely to win than lose). Odds fix half of our problem: they can go from 0 to +∞. But they still can’t be negative, and our linear equation z can definitely output negative numbers.</p><p>To fix the negative number problem, statisticians applied the natural logarithm (log or ln) to the Odds.</p><p>If you take the log of a number between 0 and 1, you get a negative number. If you take the log of a number greater than 1, you get a positive number.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*hYFYtos_YjTovDP7fe1Qsw.png" /></figure><p>This is the holy grail. The Log-Odds function maps a probability strictly between 0 and 1 to a range of −∞ to +∞. Because the ranges finally match, we can set our linear equation (z) equal to the Log-Odds:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*ruyjvZZUk7o3k1KCOQcLVQ.png" /></figure><p>The equation above is what the model <em>computes</em>, but we want to know what p is. We need to solve for p. This requires some basic algebra.</p><p>Get rid of the log by exponentiating both sides:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*tXkRMq8XEsjj2QAVFrU9QQ.png" /></figure><p>Multiply both sides by (1−p):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*QS99keDMAz1YiBu4mmRUqA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Wg9IEQsweE2ui0pEIhuCA.png" /></figure><p>This resulting equation is the famous <strong>Sigmoid Function</strong>. It perfectly converts a raw linear score (z) back into a probability (p) for <strong>two</strong> classes (Binary Classification).</p><p>How do we get from Sigmoid (2 classes) to Softmax (K classes)? We just have to look at the Sigmoid denominator a little differently.</p><p>Remember that any number to the power of 0 is 1 (<em>e^0=1</em>). So, we can rewrite the Sigmoid equation like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*hvBGOQr8PnCBtk7YU7uH4A.png" /></figure><p>In binary classification, the math naturally assumes a “baseline” class with a locked score of 0 (e0), and a “target” class with a calculated score of <em>z</em> (<em>e^z</em>).</p><p>But what if we have 3, 5, or 100 classes? We don’t have a single baseline anymore. Every single class gets its own independent score (<em>z1​,z2​,z3​…zK</em>​).</p><p>Therefore, instead of adding e0 and ez in the denominator, we must add up the exponentiated scores of <strong>all</strong> K classes to find our total denominator.</p><p>If we want the probability of a specific class i, we put its specific exponentiated score on top, and the sum of all exponentiated scores on the bottom:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-AG2rJCSymFFIAIQjSwLKQ.png" /></figure><p>Which is exactly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*6k-2wbE8jJatLtVREXPSuw.png" /></figure><pre>def _softmax(self, z):<br>        # Subtract max(z) for numerical stability to prevent overflow<br>        shift_z = z - np.max(z, axis=1, keepdims=True)<br>        exp_z = np.exp(shift_z)<br>        return exp_z / np.sum(exp_z, axis=1, keepdims=True)</pre><p>It necessarily says:</p><blockquote>To find the probability of a specific class, take the exponent of its raw score, and divide it by the sum of the exponents of all the raw scores.</blockquote><h3>Calculating the Gradients (Loss):</h3><p>To understand where dw and db come from, we have to use the <strong>Chain Rule</strong> from calculus. We want to find out how much a tiny change in a weight (W) or bias (b) affects the final Loss (L).</p><p>To get from L back to W, we step backward through our three variables:</p><ol><li><strong>L</strong>: The Cross-Entropy Loss</li><li><strong>y^​</strong>: The Softmax Probabilities</li><li><strong>Z</strong>: The raw Logits (XW+b)</li></ol><p>The Chain Rule (just the same from our previous blogs as well) tells us:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*b3sckY7Tjt5fDscgqSaYQw.png" /></figure><p>Before we get all the way back to the weights, we first need to find the derivative of the Loss with respect to the raw logits (Z). This requires combining the first two parts of our chain</p><h4>1. Derivative of Cross Entropy:</h4><p>The loss function for a single sample across <strong><em>K</em></strong> classes is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*gos5UEMGU5RZmjfJM6i1GQ.png" /></figure><p>Its derivative with respect to a specific predicted probability <em>y^​​</em> is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*fuawXl6BKbAz0BoOV3wHeQ.png" /></figure><h4>2. Derivative of Softmax function</h4><p>The Softmax function for a specific class <em>i</em> is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*LYVcaqaE96jiU1NB9PUOMg.png" /></figure><p>We want to know how a specific predicted probability (<em>y^​</em>) changes when we tweak a specific raw logit (<em>zj​</em>). To do this, we need the <strong>Quotient Rule</strong> from calculus, which states that for a function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*flkGhLStmkP_fFWwc2w5TA.png" /></figure><p>Here is where the two scenarios come in. Because the denominator g contains <em>all</em> the logits, tweaking <em>zj​</em> affects the probability <em>y^</em>differently depending on whether <em>i</em> and <em>j</em> are the same class.</p><h4>Scenario 1: The classes are the same (i=j)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IBg7iMiErqw5_G7_rCnEYg.png" /></figure><p>simplifying the fraction,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*6ZQlAF_ZJqX_C_4Ocb-0gA.png" /></figure><p>Notice that the first part is exactly our Softmax definition. We can rewrite the second part using the same logic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*GcS22R_0UzpuM6EwitZ8hQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kXvnbbdiSo61syHdM_326g.png" /></figure><h4>Scenario 2: The classes are different (i !=j)</h4><p>We are tweaking the logit for a <em>different</em> class, but we want to see how it affects class <em>i’s</em> probability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xQnkYCWFg_xOlxJ-bxvf_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rEOxeWDN6erYq59u0tZ1Tg.png" /></figure><p>Again, recognize the Softmax definitions hidden inside:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*XWR_niP1l_51fBgjo1K_GQ.png" /></figure><h3>The Grand Cancellation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oDdcgPqkCMEvhLmkpTYOnw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSVmyhS8nnM4X5XUBGxuSA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Uol63sPbrJyhMU8JcLz1A.png" /></figure><h4>The Gradient of Logits with respect to Weights</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nozd44lRjjEKNNAJ4JyF4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GdTRrskUnn3ly4ZRJLEl6Q.png" /></figure><p>However, in Python, we don’t calculate this one weight at a time using for loops. We calculate it for the entire matrix of weights and all samples in our batch simultaneously using NumPy dot products.</p><p>Let’s look at the matrix shapes:</p><ul><li><strong><em>X (Inputs)</em>:</strong> Shape is (N, F) — N samples, F features.</li><li><strong><em>y^​−y (Errors)</em>:</strong> Shape is (N, K) — N samples, K classes.</li><li><strong><em>W (Weights)</em>:</strong> Shape is (F, K) — F features, K classes.</li></ul><p>How do we multiply (N, F) and (N, K) to get (F, K)? <strong>We have to transpose the input matrix X so its shape becomes </strong><strong>(F, N).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*jeI1ErvJD7fPXtyfCj_h_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*GCkfGmfAM9FPoJ99ndZpSA.png" /></figure><pre>dw = (1 / n_samples) * np.dot(X.T, (y_pred - y_encoded))</pre><h3>Calculating the Gradients (bias):</h3><p>The bias is even easier. In the equation <em>Z=XW+b</em>, the derivative of <em>Z</em> with respect to <em>b</em> is just 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/1*2bAIethceEV_sOuQkMe6Mg.png" /></figure><p>Therefore, the chain rule for the bias is just the error signal multiplied by 1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*FSIJ8Ximd339kbCegkN4pw.png" /></figure><pre>db = (1 / n_samples) * np.sum(y_pred - y_encoded, axis=0, keepdims=True)</pre><h3>Beyond the Black Box: Building a Clinical Triage Classifier from Scratch (And Why I Didn’t Trust My 100% Accuracy)</h3><p>When building this project, my main goal was to move beyond foundational machine learning exercises and tackle a real-world operational challenge. I wanted to experience what it takes to not just train a model, but to integrate business logic, weigh the real-world costs of misclassification, and build a full deployment architecture. To do this, I chose a problem that health-tech platforms face every day: <strong>Clinical Triage</strong>.</p><p>When a patient walks into a hospital or uses a tele-health app and lists their symptoms, they need to be routed to the correct department (Cardiology, Neurology, Dermatology, etc.) as quickly as possible. More importantly, if they are exhibiting signs of a critical emergency — like a stroke or heart attack — the system needs to flag them immediately.</p><p>Here is a breakdown of how I built an AI-powered Clinical Symptom Triage Classifier from absolute scratch, the architectural decisions I made, and the massive lesson I learned about perfect accuracy.</p><h3>🏗️ The Architecture: Doing the Hard Math</h3><p>The problem is a classic multi-class classification task: mapping 131 distinct symptoms to 41 possible disease categories. To solve this, I built a <strong>Softmax Regressor</strong>.</p><p>Instead of relying on black-box libraries, I manually implemented:</p><ol><li><strong>The Forward Pass:</strong> Vectorizing the input features and applying the Softmax activation function (with shifting for numerical stability) to generate a probability distribution across all 41 diseases.</li><li><strong>Cross-Entropy Loss:</strong> Calculating the precise error between the predicted probabilities and the one-hot encoded ground truth.</li><li><strong>The Backward Pass (Gradient Descent):</strong> Calculating the matrix calculus derivatives (<em>dW</em> and <em>db</em>) to update the weights over 2,500 epochs.</li></ol><p>For feature engineering, I didn’t just use binary 1s and 0s. I mapped a clinical severity dictionary to the dataset. A symptom like “high fever” received a heavy mathematical weight of 7, while “itching” received a 1. This domain-specific weighting allowed the custom algorithm to easily identify severe mathematical signals.</p><h3>🚀 Deployment: The Microservice Approach</h3><p>A machine learning model is useless if it’s trapped in a Jupyter Notebook.</p><p>To deploy the model, I avoided the “monolithic” approach and built a decoupled microservice architecture:</p><ul><li><strong>The Backend (FastAPI):</strong> I exported my trained model weights as static matrices and loaded them into a FastAPI server. This server exposes a /predict endpoint that performs the Softmax matrix multiplication (the Forward Pass) in microseconds.</li><li><strong>The Frontend (Streamlit):</strong> I built a lightweight, interactive UI where users can search and select their symptoms. The Streamlit app acts as a dumb client — it simply sends the symptom vector to the API and renders the resulting diagnosis.</li></ul><h4><strong>Adding Business Logic:</strong></h4><p>Machine learning models deal in probabilities, but businesses deal in decisions. I programmed a business-logic layer into the API. If the model predicts a disease in the HIGH_RISK array (e.g., Heart Attack) with greater than 30% confidence, the API returns an is_critical: true flag, triggering a massive red emergency banner on the Streamlit frontend.</p><h3>🚩 The Elephant in the Room: 100% Accuracy</h3><p>When I ran my custom model against the unseen test data, the terminal spit out a terrifying metric: <strong>Accuracy: 100.0%</strong>. You can check the dataset out from here:</p><p><a href="https://www.kaggle.com/datasets/itachi9604/disease-symptom-description-dataset">Disease Symptom Prediction</a></p><p>If you are a Data Scientist, your heart sinks when you see 100%(that happened with me). In the real world, perfect accuracy almost always means data leakage, severe overfitting, or a broken evaluation pipeline. Real clinical data is incredibly noisy patients forget symptoms, exaggerate pain, and have overlapping comorbidities. A real-world triage model usually peaks around 85–90%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hmUrbC_QWTIv2YrPrFhGhw.png" /></figure><p>So, did I break the math? No. I just encountered the reality of <strong>Synthetic Data</strong>.</p><p>The model was trained on the publicly available Disease Symptom Prediction dataset from Kaggle. Because real hospital data is heavily guarded by HIPAA laws, this dataset was artificially generated by a script. The creators took a “master list” of symptoms for each disease and generated perfect, mathematically distinct permutations.</p><p>Because there was no human noise, the dataset was <strong>perfectly linearly separable</strong>. My custom Gradient Descent algorithm easily navigated the 131-dimensional space and found the absolute global minimum without encountering a single contradictory data point.</p><p>In this specific context, the 100% accuracy wasn’t a bug; <strong>it was a mathematical validation.</strong> It proved that the custom matrix calculus and the multi-hot encoded feature pipeline were functioning flawlessly.</p><h3>Implementation</h3><pre>class SoftmaxRegression:<br>    def __init__(self, epochs=1000, lr=0.01):<br>        self.epochs = epochs<br>        self.lr = lr<br>        self.weights = None<br>        self.bias = None<br>        self.losses = []<br><br>    def _softmax(self, Z):<br>        Z_shifted = Z - np.max(Z, axis=1, keepdims=True)<br>        exp_z = np.exp(Z_shifted)<br>        return exp_z / np.sum(exp_z, axis=1, keepdims=True)<br><br>    def _intialise_weights(self, n_features, k_classes):<br>        self.weights = np.random.randn(n_features, k_classes) * 0.01<br>        self.bias = np.zeros((1, k_classes))<br><br>    def _cross_entropy_loss(self, n_samples, y, probs):<br>        y_probs = np.clip(probs, 1e-15, 1 - 1e-15)<br>        return -(1 / n_samples) * np.sum(y * np.log(y_probs))<br><br>    def fit(self, X, y):<br>        self._intialise_weights(X.shape[1], y.shape[1])<br>        n_samples = X.shape[0]<br>        for i in range(self.epochs):<br>            Z = np.dot(X, self.weights) + self.bias<br><br>            probs = self._softmax(Z)<br><br>            error = probs - y<br><br>            # the math of matrix multiplication automatically sums up the values across all your samples. <br>            #Hence no use of np.sum(np.dot())<br>            dw = (1 / n_samples) * np.dot(X.T, error)<br>            # error is a matrix of (N, 41). You have N different errors. You need one single bias adjustment for each of the 41 classes. <br>            # You must sum the errors vertically across all patients.<br>            db = (1 / n_samples) * np.sum(error, axis=0, keepdims=True)<br><br>            self.weights -= self.lr * dw<br>            self.bias -= self.lr * db<br><br>            loss = self._cross_entropy_loss(n_samples, y, probs)<br>            self.losses.append(loss)<br>            if i % 100 == 0:<br>                print(f&#39;Loss at epoch {i} is {loss:.4f}&#39;)<br><br>        return self<br><br>    def _predict(self, X):<br>        &quot;&quot;&quot;<br>        Predicts the disease class for the given symptoms.<br>        &quot;&quot;&quot;<br>        Z = np.dot(X, self.weights) + self.bias<br>        probs = self._softmax(Z)<br>        return np.argmax(probs, axis=1)</pre><h3>Scaling to v2.0: The 99.75% Accuracy</h3><p>When transitioning this clinical triage engine from a pure-NumPy foundation to a hardware-accelerated PyTorch pipeline, the goal was architectural scale. By leveraging Apple Silicon’s Metal Performance Shaders (MPS) and writing custom lazy-loading data structures, the system was suddenly capable of chewing through the <strong>DDXPlus</strong> dataset — a massive, complex repository of over a million synthetic patient records. You can check the dataset out from here (<em>I sincerely admire the researchers who simulated this dataset to make it as much apt to machine learning and also more close to real world</em>):</p><p><a href="https://figshare.com/articles/dataset/DDXPlus_Dataset_English_/22687585">JavaScript is disabled</a></p><p>The pipeline compiled, the <strong>Adam optimizer</strong> went to work, and the results were staggering. The model evaluated over 130,000 unseen test patients and hit an accuracy of <strong>99.71%</strong>, alongside a Micro-Average AUPRC (Area Under the Precision-Recall Curve) of <strong>0.9987</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6rkiCodgA1L7eGJCFLGwbw.png" /></figure><p>If you have spent any time in clinical data science, those numbers should immediately set off alarm bells. Real-world medical data is famously chaotic. Patients misremember symptoms, doctors mistype notes, and biologies overlap in illogical ways. In a real hospital setting, a diagnostic AI usually caps out around 85–90% accuracy. Hitting 99.7% usually means one of two things: catastrophic data leakage, or severe overfitting.</p><p>But in this specific case, it meant something much more profound: <strong>The PyTorch engine successfully reverse-engineered the simulator.</strong></p><p>The secret lies in how the DDXPlus dataset was created. To protect patient privacy while still providing a massive, hierarchically complex dataset for machine learning, researchers generated DDXPlus using a highly advanced <strong>Bayesian Network Simulator</strong>. The data behaves exactly like human biology, but underneath, it is ultimately governed by strict, synthetic mathematical rules and conditional probabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j51Q3NEQnes0mbL6GmLQJg.png" /></figure><p>When my PyTorch model processed those multi-hot encoded symptom tensors, it wasn’t just memorizing patients. Driven by the Cross-Entropy loss function, the neural network systematically peeled back the layers of the data and mathematically deduced the exact probability matrix the researchers used to build the dataset in the first place.</p><p>The 99.7% accuracy isn’t a claim that this AI is ready to replace human doctors tomorrow. Rather, it is mathematical proof that the underlying pipeline i.e the Just-In-Time tensor generation, the one-vs-rest evaluation logic, and the hardware-accelerated matrix calculus is architecturally flawless. The engine is built; now it just waits for real-world data.</p><h3>v2.0 PyTorch Implementation</h3><pre>class DDXPlusDatasetLoader(Dataset):<br>    def __init__(self, csv_filepath):<br>        print(&quot;Loading Massive dataset into Memory&quot;)<br>        # load the dataset into memory<br>        self.df = pd.read_csv(csv_filepath)<br>        print(f&#39;Pandas can see the columns: {self.df.columns.tolist()}&#39;)<br><br>        # changing the EVIDENCES column to column of lists instead of str.<br>        # print(f&#39;Formatting each EVIDENCE in EVIDENCES column to a list instead of str using ast.literal_eval&#39;)<br>        # self.df[&#39;EVIDENCES&#39;] = self.df[&#39;EVIDENCES&#39;].apply(ast.literal_eval)<br><br>        # lets forumlate disease to index dictionary.<br>        print(&quot;Mapping symptoms and diseases to numerical indices...&quot;)<br>        self.all_diseases = self.df[&#39;PATHOLOGY&#39;].unique().tolist()<br>        self.disease_to_idx = {d: i for i, d in enumerate(self.all_diseases)}<br><br>        # extract all unique symptoms<br>        all_symptoms = set()<br>        for evidences in self.df[&#39;EVIDENCES&#39;]:<br>            symptoms_list = ast.literal_eval(evidences)<br>            all_symptoms.update(symptoms_list)<br>        # all_symptoms = set(evidences for evidences in self.df[&#39;EVIDENCES&#39;])<br><br>        # lets formulate symptom to index dictionary<br>        self.all_symptoms = list(all_symptoms)<br>        self.symptom_to_idx = {s: i for i, s in enumerate(self.all_symptoms)}<br><br>        self.num_symptoms = len(self.all_symptoms)<br>        self.num_classes = len(self.all_diseases)<br><br>        print(f&#39;No. of unique symptoms and diseases detected are: {self.num_symptoms} and {self.num_classes}&#39;)<br><br>    def __len__(self):<br>        return len(self.df)<br><br>    # to return one-hot encoded symptom list and its class (disease)<br>    def __getitem__(self, idx):<br>        row = self.df.iloc[idx]<br><br>        # create a vector of tensor type of all zeros. Flip to 1.0 as per evidence index<br>        x_tensor = torch.zeros(self.num_symptoms, dtype=torch.float32)<br>        patient_symptoms = ast.literal_eval(row[&#39;EVIDENCES&#39;])<br>        for symptom in patient_symptoms:<br>            symptom_idx = self.symptom_to_idx[symptom]<br>            # flip to 1.0<br>            x_tensor[symptom_idx] = 1.0<br><br>        y_idx = self.disease_to_idx[row[&#39;PATHOLOGY&#39;]]<br>        y_tensor = torch.tensor(y_idx, dtype=torch.long)<br><br>        return x_tensor, y_tensor<br><br>&#39;&#39;&#39;<br>By inheriting nn.Module class we get the ability to track gradients, <br>move data to M4 GPU and save model weights.<br>&#39;&#39;&#39;<br>class DDXClinicalTriageModel(nn.Module):<br>    def __init__(self, input_size, num_classes):<br>        &#39;&#39;&#39;<br>        Explicitly tell python to run super method, <br>        gives mechanism to track math update gradients.<br>        &#39;&#39;&#39;<br>        super(DDXClinicalTriageModel, self).__init__()<br>        self.linear = nn.Linear(input_size, num_classes)<br><br>    def forward(self, x):<br>        return self.linear(x)</pre><pre>csv_path = &quot;../data/raw/release_train_patients.csv&quot;<br><br>dataset = DDXPlusDatasetLoader(csv_path)<br><br>dataloader = DataLoader(dataset, batch_size=1024, shuffle=True)<br><br>model = DDXClinicalTriageModel(dataset.num_symptoms, dataset.num_classes).to(device)<br><br>criterion = nn.CrossEntropyLoss()<br>optimizer = optim.Adam(model.parameters(), lr=0.01)<br><br>epochs = 20<br><br>print(f&#39;\nStarting Training loop:&#39;)<br><br>for epoch in range(epochs):<br>    epoch_loss = 0.0<br>    for batch_X, batch_y in dataloader:<br>        batch_X = batch_X.to(device)<br>        batch_y = batch_y.to(device)<br><br>        optimizer.zero_grad()<br>        predictions = model(batch_X)<br>        loss = criterion(predictions, batch_y)<br>        loss.backward()<br>        optimizer.step()<br><br>        epoch_loss += loss.item()<br><br>    print(f&#39;Epoch: {epoch+1}/{epochs}, Loss: {epoch_loss/len(dataloader):.4f}&#39;)<br>        <br>print(&quot;✅ Enterprise Training Complete!&quot;)</pre><h3>Conclusion</h3><p>Let’s be honest: writing a Gradient Descent loop in pure NumPy and another with PyTorch are challenging ways to do this. But it is also the most rewarding.</p><p>Throughout this project, I had to confront the realities of matrix dimensionalities, the necessity of numerical stability (<strong><em>subtracting the max value</em></strong><em> before applying Softmax saved my model from crashing</em>!), and the hilarious consequences of minor math bugs turning a Heart Attack diagnosis upside down. Not to mention splitting the .csv to seperate column values from a single column from the DDXPlus dataset was also hard to identify untill pandas has to slam few errors to my face (this is just me, most of you might make your way through this way faster).</p><p>The epiphany that algorithms aren’t magic they are just beautifully structured linear algebra was the most fun part of this entire journey. Pushing this custom math to a live FastAPI microservice and seeing it actually work in a frontend UI made all the debugging worth it and the deep mathematical intuition I gained from this project will follow me into every model I build from here on out.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2ce967c3c0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logistic Regression: Building a Profit-Driven Credit Model from Scratch]]></title>
            <link>https://medium.com/@sathwikpinninti/logistic-regression-building-a-profit-driven-credit-model-from-scratch-aa1ce99b79d4?source=rss-b2f0a7b28c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa1ce99b79d4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <dc:creator><![CDATA[Sathwik Pinninti]]></dc:creator>
            <pubDate>Sun, 31 May 2026 18:18:12 GMT</pubDate>
            <atom:updated>2026-06-03T18:35:46.279Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://github.com/Sathwik-007/Predict-Credit-Card-Default.git">GitHub - Sathwik-007/Predict-Credit-Card-Default: This project aims to predict whether a given customer is a good customer or a defaulter using Logistic Regression machine learning model built from scratch.</a></p><p>With the same amount of healthy skepticism and after a few struggles of being able to invest time over the past few work weeks, I finally am able to unpack the next algorithm in my list. Here is yet another classification model I decided to uncover. But before we get to the business logic, we need to dive into the math first</p><h3>The Math: Squashing Numbers with Sigmoid</h3><p>The most fascinating thing about the Sigmoid function is its mechanical elegance. It can take absolutely any real-valued number no matter how massively positive or negative and perfectly squash it into a clean probability between 0 and 1.</p><p>The formula looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*yPRNa0n3J3uHTJOJ2lpQsg.png" /></figure><p><strong><em>where:</em></strong></p><ul><li><strong><em>z: </em></strong><em>The raw, unbounded numerical output from our linear weights (the wTx+b).</em></li><li><strong><em>σ(z): </em></strong><em>The final predicted probability (a value between 0 and 1).</em></li></ul><h3>Calculating loss: Binary Cross Entropy</h3><p>Just as we use Mean Squared Error (MSE) in Linear Regression to calculate how far our predictions stray from actual values, Logistic Regression requires a different approach. Because we are dealing with probabilities, we calculate this error using <strong>Binary Cross-Entropy</strong> (often called Log Loss), combined with <strong>L2 Regularization</strong>.</p><p>Before looking at the final equation, let’s break down the regularization part.</p><h4>What are L1 and L2 Regularization, and why L2?</h4><p>Machine learning models are greedy. If you give them a dataset, they will happily assign massive weights to a few specific features just to memorize the training data, leading to severe overfitting. Regularization introduces a penalty term (λ) that actively discourages this behavior.</p><p><strong>L1 Regularization (Lasso):</strong> Adds the absolute sum of the weights to the loss function. It encourages sparsity, meaning it can forcefully push the weights of less important features down to exactly zero. It effectively acts as an automatic feature-deletion tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*DBU2c7O_30Ixp87THrya-g.png" /></figure><p><strong>L2 Regularization (Ridge)</strong>: Adds the squared sum of the weights to the loss function. It shrinks all weights proportionally but rarely reduces them to exactly zero. This results in a model where all features are used, but weights are kept small and balanced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*fy1MtcHbFJBExHxJo6H1Uw.png" /></figure><p><strong><em>Note:</em></strong><em> We go with L2 (also a most commonly used general technique) because we don’t want the model to completely ignore the features with weights that sparsed to 0 (which happens with L1). All of them features a significant value to contribute to a most accurate result while predicting the final output.</em></p><h4>The Final Loss Equation</h4><p>When we combine Binary Cross-Entropy with our L2 penalty, the complete mathematical objective for our Logistic Regression model is born:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FU3wpky01havRLeGGCw_5Q.png" /></figure><p><strong><em>where</em></strong><em>:</em></p><ul><li><strong><em>n</em></strong><em>: Number of samples (rows in the dataset)</em></li><li><strong><em>yi</em></strong><em>​: The actual value (0 or 1)</em></li><li><strong><em>y^​i</em></strong><em>​: The predicted probability</em></li><li><strong><em>λ</em></strong><em>: The regularization penalty</em></li></ul><p>When combined with regularisation techniques, the model is penalised when attempting to assign massive weights to features thereby preventing overfitting.</p><p>With the core math laid out, you might notice that Logistic Regression is surprisingly similar to Linear Regression. The process of calculating slope, computing loss, and finding gradients is practically identical under the hood. Speaking of gradients, let me lay out exactly how we calculate them to make the model learn.</p><h3>The Chain Rule for the “Weights”</h3><h3>1. The Setup: The Three Layers</h3><p>To find out how a tiny change in our weights (w) affects the final Loss (L), we have to use the <strong>Chain Rule</strong> from calculus. We have to work backward through the three steps the model took to make a prediction:</p><ol><li><strong>The Linear Equation:</strong> z=wx+b</li><li><strong>The Sigmoid Function:</strong> y^​=σ(z)</li><li><strong>The Loss Function (BCE):</strong> L=−(ylog(y^​)+(1−y)log(1−y^​))</li></ol><p>To find the gradient for the weights (∂w∂L​), the Chain Rule says we multiply the derivatives of these three steps together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*ml9Qgeo5Be-me8XB7P11mg.png" /></figure><h3>Step 1: Derivative of the Loss with respect to the Prediction (∂y^​/∂L​)</h3><p>Lets look at first part of the equation. We differentiate L w.r.to y^. Then the equation turns into …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*neS80-_xUxb1AEt9MCd3uA.png" /></figure><p>Further simplification and the final term at the end looks like …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*aWmAcuy4s8RKrWadf8Nhiw.png" /></figure><h3>Step 2: Derivative of the Sigmoid with respect to z (∂z/∂y^​​)</h3><p>Our target function is the Sigmoid function, which takes an input z:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*54DXXwqC4k01A3dH1XQnUA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*yaNvhVaQuMu_RoEFe0_GGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*MlzdktC9OQ4mL8DKNfaU4g.png" /></figure><p>By norms of differentiation, chain rule demands us to differentiate internal terms as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*YXCcyMUCTG7xL0x6XFDt6Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*9NjwhGvZV6Xp0Qaf4cZIvA.png" /></figure><p>Now, we rewrite it back as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*v0ortZWMdC7xOIRE6dvmOw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P0OdFVF0TUEvf5HM9zg5ZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UEw3T_G-r5LewBPF9B3Vw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2rrGdFHKFUf_bKlfYwqM-A.png" /></figure><h3>Step 3: Derivative of the Linear Equation with respect to w (∂w/∂z​)</h3><p>Since z=wx+b, taking the derivative with respect to the weight w leaves us with just the input feature:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*h-dOPTGeWzpTxg8LOmeCgw.png" /></figure><h3>Step 4: The Magic Cancellation (Putting it together)</h3><p>Now, we multiply all three parts together according to our Chain Rule equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJjn0bNib4CbQweFnxcUhg.png" /></figure><h3>Step 5: Adding the Summation and L2 Regularization Back In</h3><p>Now we bring back the full scope of your image. We need to average this error over all n samples in the dataset, and we need to add the derivative of the L2 penalty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*0edUyucMQok8bIr9HFVMjw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQ5xQ50xCGRMN1PQFJqQCQ.png" /></figure><p>The final Loss equation is now given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*aHLD3b4KnVI_d6iRF-ojTw.png" /></figure><h3>The Chain Rule for the “Bias”</h3><p>Just like we did for the weights, we have to look at the three steps the model takes to make a prediction, and apply the Chain Rule to work backward.</p><ol><li><strong>The Linear Equation:</strong> z=wx+b</li><li><strong>The Sigmoid Function:</strong> y^​=σ(z)</li><li><strong>The Loss Function (BCE):</strong> L=BCE(y^​,y)</li></ol><p>To find out how a tiny change in the bias (b) affects the final Loss (L), the Chain Rule dictates we multiply the derivatives of those three steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*yJW3IbhfV2gWZK6d5lRePQ.png" /></figure><p>Notice that the first two parts of this Chain Rule (∂y^​/∂L​⋅∂z/∂y^​​) are <strong>exactly the same</strong> as they were for the weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*xzPLXQ2-lYMRpn9ZxU93vQ.png" /></figure><p>Now we just need the final piece. For which we consider the equation:</p><ul><li><strong><em>z = wx+b.</em></strong></li></ul><p>Derivating (z) w.r.to b gives 1. The rest are treated as constants and are evaluated to 0. Hence it can be given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*haUFcRR2ShPe9ujLt7EvKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Rh5QoJS8kfIwNN7mwcIAA.png" /></figure><p>The final equation results into:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*Sdk9VA1kinc_U5jjzD8JeA.png" /></figure><h3>Now, into the project building paradigm …</h3><p>Well well well, I know. The math have sounded a bit just whelming (I believe it can get more serious ahead) but it was the <strong>“epiphany”</strong> moment of this entire journey.</p><p>To wrap this learning into a practical implementation, I decided to build a custom Logistic Regression from scratch to predict credit card defaults. I didn’t use standard ML libraries because I needed understand the actual underlying implementation and have absolute control over the mathematical architecture to solve a specific business problem.</p><p>What started as an exercise in calculus quickly evolved into an engineering lesson on risk, greed, and the danger of building models in a vacuum. Here is what happens when you stop optimizing for standard metrics and start optimizing for actual business profit.</p><h3>Stop Optimizing for Accuracy: How I Built a Profit-Driven Credit Default Model from Scratch</h3><p>Most machine learning tutorials end the exact same way: you split your data, call .fit(), and celebrate when your accuracy hits 95%. But when you deploy that code into a live financial system, that &quot;95% accuracy&quot; can quietly drain millions of dollars from the vault.</p><p>Recently, I built a custom Logistic Regression model from scratch to predict credit card defaults. I didn’t use standard ML libraries because I needed absolute control over the mathematical architecture to solve a specific business problem.</p><p>What started as an exercise in calculus quickly evolved into an engineering lesson on risk, greed, and the danger of building models in a vacuum. Here is what happens when you stop optimizing for standard metrics and start optimizing for actual business profit.</p><h3>The Business Problem: The Cost of Being Wrong</h3><p>In the world of credit lending, not all mistakes are created equal.</p><p>Imagine a bank evaluating applications. If the model makes a <strong>False Positive</strong> (falsely rejecting a good customer), the bank loses the opportunity to make money a frustrating, but survivable, <strong>$2,000</strong> loss in potential lifetime interest.</p><p>But if the model makes a <strong>False Negative</strong> (falsely approving a customer who ultimately defaults), the bank loses the original principal. That is a <strong>$50,000</strong> direct hit to the vault.</p><p>The baseline risk matrix is highly asymmetric:</p><p>Total Cost = (False Negatives * 50,000) + (False Positives * 2,000)</p><p>Because the dataset is heavily imbalanced (the vast majority of people pay their bills), standard ML metrics fail completely. If a dataset has 95% good customers and 5% defaulters, a “dumb” model that simply approves <em>everyone</em> will achieve <strong>95% accuracy </strong>and immediately bankrupt the company. Accuracy is a lie metric in risk management.</p><h3>The Architecture Trap: “Double Dipping”</h3><p>Once the mathematical engine was running, I ran into a massive architectural trap. My model was outputting probabilities, but I needed to convert those into a hard business decision: <strong>Approve (0) or Reject (1)</strong>.</p><p>Initially, I tried to protect the bank by using two safety nets simultaneously:</p><ol><li><strong>Class Weights:</strong> During training, I mathematically forced the model to fear defaulters. This artificially inflated the probabilities (e.g., a true 10% risk was output as a 40% risk). Because the numbers were artificially high, I could leave the final decision threshold at the standard <strong>0.50</strong>.</li><li><strong>Threshold Tuning:</strong> I then ran a loop testing hundreds of thresholds against the Total Cost equation to find the absolute safest cutoff line.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JqvDKdvfuJJp7BWI0fYX_A.png" /></figure><p>This resulted in a disaster: <strong>a 0% approval rate.</strong></p><p>By using Class Weights, I pushed the minimum possible probability the model could output up to around 20%. But by testing against the strict $50,000 penalty, my threshold loop demanded a cutoff of <strong>11%</strong>. The threshold line was drawn <em>below</em> the floor of the model. The lines crossed, and the bank rejected every single applicant.</p><p>Then I realised: <em>If I were to use Class Weights, I’m intentionally spiking the probabilities. To build a transparent, calibrated system, I must drop the weights, let the model output the brutal, honest truth, and use a strict final threshold as my only safety net.</em></p><h3>From Fear to Greed: Finding the True Threshold</h3><p>After dropping the Class Weights, my model was perfectly calibrated. A 10% prediction meant exactly a 10% risk.</p><p>When I ran my threshold optimization loop using purely the cost equation trying to avoid the $50,000 penalty at all costs — the algorithm settled on an incredibly strict optimal threshold of <strong>9% (0.09)</strong>. The vault was perfectly secure, but the bank was barely approving any customers. The model was operating in a state of pure fear.</p><p>To fix this, I had to completely change the mathematical objective. I stopped calculating <em>Cost</em> and started calculating <em>Net Profit</em>.</p><p>I introduced <strong>Opportunity Cost</strong>: Every time the model correctly approves a good customer (True Negative), the bank generates <strong>$5,000</strong> in interest over the life of the card. The new equation became:</p><p>Total Profit = (True Negatives * 5000) - (False Negatives * 50000)</p><p>I fed this new, profit-driven reality into the optimization loop, and the math did something beautiful. The optimal threshold shifted from <strong>9% to 11%</strong>.</p><p>The algorithm realized that by relaxing the threshold just 2%, it might occasionally let a defaulter slip through and lose $50,000. But that exact same 2% shift would approve hundreds of new, excellent customers, generating millions in interest. The risk was mathematically justified by the reward.</p><h3>The Results: Proving the Math in Production</h3><p>Writing the calculus in Python is fun, but a model is only as good as its behavior in the real world. To see if the custom engine actually worked, I plotted the final results across three different metrics. This is where the gap between theoretical math and actual business value gets bridged.</p><h4>1. Is the model honest? (Calibration Curve)</h4><p>Before we can even think about setting business rules, we have to answer one question: <em>When the model says there is a 40% chance of default, is there actually a 40% chance of default?</em></p><p>To prove this, I plotted a Reliability Diagram (Calibration Curve).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PSSqMUbsLm-aayO5v0tSRg.png" /></figure><p>The dotted diagonal line represents perfect reality. If the model is perfectly calibrated, its predictions will sit exactly on that line.</p><p>Looking at the blue line (our custom Logistic Regression), it hugs that dotted line remarkably well. This is the ultimate payoff of dropping the Class Weights I mentioned earlier. Because we didn’t artificially warp the training data, the model outputs raw, honest truth. A 20% predicted risk maps beautifully to a real-world 20% default rate. It proves that the numpy matrix math we wrote from scratch is fundamentally sound.</p><h4>2. The Trade-off: Precision vs. Recall</h4><p>Because our dataset is heavily imbalanced (only about 22% of the dataset actually defaults), looking at standard accuracy doesn’t help. Instead, we look at the Precision-Recall curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SKShG3212LPcBUMkVZTY-Q.png" /></figure><p>This curve is the visual representation of the fundamental tug-of-war in risk management:</p><ul><li><strong>Recall (X-axis):</strong> What percentage of the actual defaulters are we catching?</li><li><strong>Precision (Y-axis):</strong> When our model flags someone as a defaulter, how often is it actually right?</li></ul><p>The graph slopes downward, which is exactly what you expect with imbalanced data. If we want to catch absolutely <em>every</em>single defaulter (pushing Recall to 1.0 on the far right), the model has to become so incredibly strict that it flags almost everyone. As a result, its Precision plummets down to around 22% — meaning 78% of the people it rejected were actually perfectly good customers.</p><p>This graph visually proves why we couldn’t just use a default 0.50 threshold. The model needs a custom cutoff point to balance this slide.</p><h4>3. The Grand Finale: Optimizing for Profit</h4><p>This is the single most important chart in the entire project. It takes the probabilities from the Calibration Curve, the trade-offs from the Precision-Recall curve, and translates them entirely into dollars.</p><p>I ran the model through the simulated financial environment, calculating the net profit at every possible threshold from 0.01 to 0.99.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mg5K4UNK_65_QnxvMkcp-g.png" /></figure><p>This graph is just awesome to look at. It perfectly visualizes the “Fear vs. Greed” dynamic.</p><ul><li>On the far left (Thresholds near 0.00), the model operates in pure fear. It rejects everyone, so the bank loses no money, but it also makes zero profit.</li><li>On the far right (Thresholds near 1.0), the model operates with reckless greed. It approves everyone, letting every single defaulter into the vault, resulting in a catastrophic plunge into negative $40,000,000.</li></ul><p>But right there on the left, marked by the red dashed line, is the mathematical peak. At exactly a <strong>0.11 threshold</strong>, the model finds the absolute perfect balance. It approves enough good customers to generate massive interest, while staying just strict enough to block the most expensive defaults.</p><p>Seeing that peak emerge from the raw calculations was one of those “mind-blown” moments. It is tangible proof that with a bit of calculus and some clean Python code, you aren’t just predicting numbers — you are engineering a system that mathematically guarantees the smartest possible business decision.</p><h3>More feature engineering (Adding polynomial features and interaction columns)</h3><p>To bring the most out of the model, I decided to add polynomial features and interaction columns. So what polynomial featuers have I exactly added?</p><h4>Polynomial features:</h4><ol><li><strong><em>Age_squared</em></strong>: A person with age 18 is of high risk, where as a person around mid 30’s or 40’s are low risk and a person with age of 65 is again a high risk. The risk assessment is no longer a straight line(with increase in age, risk is also greater that they would default) now it’s a U-shaped curve (U-shaped in conjunction with original “AGE” column.</li><li><strong><em>utilisation_squared</em></strong>: A low utilisation_rate would have smaller value where as a high utilisation trend would have exponentially larger value. This is to help the model understand that the risk is exponentially large with increase in utilisation_rate.</li><li><strong><em>delinquency_squared</em></strong>: To show case similar relationship behaviour as utilisation_squared.</li></ol><h4>Interaction columns:</h4><p>Introduce interaction columns to capture the following trends:</p><ol><li><strong><em>age_x_utilisation</em></strong>: To expose how utilisation trends vary with age.</li><li><strong><em>utilisation_x_delinquency</em></strong>: To highlight to what rate of utilisation, delinquency plays along.</li></ol><h3>The Pivot: Risk-Based Pricing</h3><p>A static $50,000 credit limit forces the model to be terrified of risk. Because the penalty for a mistake is so massive ($50000), it rejects a huge portion of the customer base just to protect the vault.</p><p>But what if we rewrote the business rules to match the algorithm’s strengths?</p><p>The solution is <strong>Risk-Based Pricing</strong>: dynamically changing the credit limit based on the model’s exact probability output. By breaking the approval process into three distinct tiers, we completely change the baseline cost equation:</p><ul><li><strong>Tier 1: Low Risk (&lt; 11%)</strong> The math proves these customers are safe. We reward them with the maximum <strong>$50,000 credit limit</strong> to rake in maximum interest.</li><li><strong>Tier 2: The Grey Area (11% to 25%)</strong> This is where the magic happens. Instead of rejecting this massive group, we approve them — but restrict their limit to <strong>$3,000</strong>. By dropping the cost of a mistake from fifty grand down to three grand, the risk becomes completely survivable. We can safely approve thousands of extra customers and unlock a massive new revenue stream.</li><li><strong>Tier 3: High Risk (&gt; 25%)</strong> The model’s polynomial features indicate these customers are a mathematical liability. We issue a hard rejection (<strong>$0 limit</strong>) to protect the vault.</li></ul><p>We didn’t change the underlying calculus or add a new library. We just introduced a simple if/elif/else block that aligned the algorithm&#39;s honest probabilities with smart financial boundaries. Your approval rate skyrockets, your risk stays capped, and the code acts as an automated revenue engine.</p><blockquote><strong>Cheat-code 😁🙈</strong><br>To see all 3 outputs on frontend(Approved, Approved with less limit, Reject), simply increase credit limit in ranges of $50000, $100000, $1000000</blockquote><h3>The Takeaway</h3><p>By taking the time to build the math from scratch, abandoning standard metrics like accuracy, and directly tying the algorithm’s output to a corporate profit/loss ledger, the code stops being just a script. It becomes an automated financial engine.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa1ce99b79d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linear Regression: Building a Delivery Time estimation model from Scratch]]></title>
            <link>https://medium.com/@sathwikpinninti/unboxing-the-black-box-building-machine-learning-from-scratch-b2c54fbf8e21?source=rss-b2f0a7b28c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2c54fbf8e21</guid>
            <category><![CDATA[ridge-regression]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sathwik Pinninti]]></dc:creator>
            <pubDate>Sat, 09 May 2026 13:35:51 GMT</pubDate>
            <atom:updated>2026-07-28T06:31:03.843Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://github.com/Sathwik-007/Food-Delivery-Time-Estimation">GitHub - Sathwik-007/Food-Delivery-Time-Estimation: Predicting food delivery times using raw linear algebra. A deep dive into building Custom Linear Regression models (Normal Eq, Ridge, BGD) from scratch, handling categorical anomalies, and using interaction terms to model real-world logistics</a></p><p>While I have some development experience in my past, it’s not my strong suit today. Honestly, I approach most tech trends with a heavy dose of skepticism. So, when I decided to dive into Machine Learning to build a Food Delivery Time Estimator, the idea of just importing scikit-learn and calling model.fit() didn&#39;t sit right with me. I didn&#39;t want to rely on &#39;magic&#39; algorithms; I wanted to tear them apart and understand the raw linear algebra underneath.</p><p>So, I built three foundational regression algorithms entirely from scratch using pure NumPy: The <strong>Normal Equation</strong>, <strong>Ridge Regression</strong>, and <strong>Batch Gradient Descent</strong>.</p><p>What followed was a masterclass in matrix math, frustrating silent bugs, and a stark lesson in how machine learning models can be perfectly accurate while being entirely wrong about reality.</p><h3>The Math behind</h3><p>Honestly, deep diving into algorithms offcourse involves “math”. As tiring as it sounds to me (jst me and myself), it is also interesting how numbers can actually reveal information about real world. The more I dive into, the more I realise, “<strong>Math is the language of universe</strong>”. I’m impressed on how matrix operations like dot product, transposing, finding inverse and etc can precisely unwrap meaningful insights.</p><h3>Batch Gradient Descent</h3><p>Lets begin by talking about the typical Slope equation :</p><pre>y = mx + c</pre><p>where,</p><h4>1. y: independent variable</h4><h4>2. x: dependent variable</h4><h4>3. m: gradient(steepness of the line)</h4><h4>4. c: intercept. (the exact point where line crosses the y-axis).</h4><p>Now, a typical Gradient descent would follow similar formula as is. By performing same operation over and over again by calculating slope, finding the error (how far the model is away from actual values) and updating its individual <strong><em>Gradients(m)</em></strong> and <strong><em>Intercept (c)</em></strong>. Repeat the same loop till the model can train to least possible amount of error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ApVeaDo9IjsTxPHiNyzUhQ.png" /></figure><p>Initially, our <strong><em>Gradients</em></strong> (<strong><em>W</em></strong> which we used to refer to as “m” till now) is nothing but a matrix of 0’s of size (no. of features, 1) and <strong><em>bias</em></strong> (<strong><em>b</em></strong> which we referred to as “c” till now) is matrix of size (1, 1).</p><ol><li>So we capture the resultant , then we simply subtract it from the actual value to know how far we are away from it: pred and err respectively.</li><li>Then we calculate loss using the well known <strong>MSE </strong>(<strong>Mean Squared Error</strong> - squaring and calculating the mean of the loss). We optionally append it to the array to see how loss is reducing and closing into 0 as we tweak learning-rateand epochs.</li><li>Most importantly, we update the <em>Gradients</em> <em>(W)</em> and</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*LDlzx62vzgkr326_jE4emw.png" /></figure><p>and <em>bias (b) </em>as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*H8PjBzut4MlJG7SlkHfOKA.png" /></figure><p>These are nothing but the derivatives of the squared error. This is where the actual learning happens. Gradient descent takes tiny steps on the line and iteratively calculates which next big step to take so that we reach the bottom of the error valley where slope is almost zero.</p><p><strong>2/len(y)</strong>: Derived from the derivative of the squared error (x2→2x) averaged over all samples (n).</p><p>I would like to highlight my best understanding of Gradient descent as Gemini helped me understand.</p><p>Instead of calculating the perfect answer in one heavy shot, it makes a <strong>bad guess</strong>, measures how wrong it is, and takes a small step in the right direction. It repeats this until it stops improving.</p><blockquote>Think of a marble left to roll around in a bowl. The marble can be considered by taking small steps at a time to find the centre where slope is almost 0. That exact point is called the <strong>global minimum<em> </em></strong>that Linear regression attempts to find where error is almost negligible and close to 0.</blockquote><h3>Normal Equation (Closed-form solution)</h3><p>Best if independent variable depends on more than one dependent variable. For Ex: We cannot predict delivery time solely depending on the distance. It also involves other features like <em>delivery person’s age(x1), rating(x2), order type(x3), type of vehicle(x4) </em>etc.</p><p>To expand that in terms of gradient descent, the slope equation would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WcES-6pI6Pv8fedGP_vQbw.png" /></figure><p>Finding optimal slopes <em>(m1), (m2), etc. </em>is impossible as the features interact. For instance, older drivers might drive slower (Age), but perhaps they take shorter routes (Distance). You can’t calculate the slope for Age without simultaneously adjusting the slope for Distance.</p><p>This is where matrix math and the <strong>Closed-form solution</strong> step in.</p><p>It is like using a GPS to teleport directly to your exact destination. On side note: With the Closed-form Solution, you need to calculate the inverse of a matrix. In real life, if you have millions of rows and thousands of features, calculating that inverse (X.T @ X)^−1 requires a massive amount of memory and computational power viz. sometimes literally impossible for a computer to do.</p><p>The Closed-form solution calculates all those slopes (m) and the intercept simultaneously in a single, elegant mathematical step, without any iterative guessing (like Gradient Descent does).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*QOtW0t_xLKwCZxj3OvfFgg.png" /></figure><ul><li><strong>Where θ (Theta):</strong> A single vector containing your intercept © and all your slopes (m1​,m2​,m3​…).</li><li><strong>X (The Design Matrix):</strong> Your entire dataset of features. We also add a column of pure 1s to the start of this matrix. That column of 1s is what mathematically creates your intercept (c).</li><li><strong>y:</strong> Your target column (Delivery Time).</li></ul><p><strong>Step 1: Calculate X.T @ X (The Covariance Matrix)</strong> When you multiply a matrix by its transpose (XT), you get a square matrix that captures the relationships between every feature. It calculates how Distance varies with Distance, how Rating varies with Rating, and crucially, how Distance varies with Rating. <em>Think of XTX as a mathematical summary of how all your inputs overlap.</em></p><p><strong>Step 2: Invert it (X.T @ X)−1</strong> In matrix math, inversion is like division. By inverting the covariance matrix, we are essentially “untangling” the overlapping relationships. If Distance and Rating are correlated, the inversion separates their distinct impacts.</p><p><strong>Step 3: Multiply by X.T @ y</strong> calculates the raw correlation between your features and the target (Time).</p><p>To understand how the information unwinds, lets look at a sample example:</p><ul><li><strong>Delivery 1:</strong> Distance 5km, Rating 4.0, Time 40 mins</li><li><strong>Delivery 2:</strong> Distance 10km, Rating 4.5, Time 52.5 mins</li><li><strong>Delivery 3:</strong> Distance 6km, Rating 3.0, Time 37 mins</li></ul><p>First, we organize this into our Feature Matrix (X) and Target Vector (y). Notice the column of 1s on the far left of X—this is a mathematical trick that creates our Intercept (c).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*OrRSkuOWqR99D0mtPWDYUQ.png" /></figure><h4>Step 1: Generating the “Covariance” Matrix (XTX)</h4><p>We multiply the transposed matrix (XT) by the original matrix (X).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*oWBMY6X3_dcrTVWRCv2fLA.png" /></figure><p><strong>The Insight:</strong> X.T @ X is a mathematical summary of how your features behave and overlap.</p><ul><li><strong>The Top-Left (3):</strong> This is just 1⋅1+1⋅1+1⋅1. It counts your total data points.</li><li><strong>The Center (161):</strong> This is how Distance varies with Distance (52+102+62). It represents the raw variance or “spread” of your distance data.</li><li><strong>The Bottom-Right (45.25):</strong> This is how Rating varies with Rating (4.02+4.52+3.02).</li><li><strong>The Cross-Sections (83):</strong> This is the magic number. It is the sum of (Distance × Rating) across all rows (5⋅4.0+10⋅4.5+6⋅3.0=83). This number tells the algorithm exactly how Distance and Rating vary <em>together</em> in the real world.</li></ul><h4>Step 2: Untangling the Overlap (X.T @ X)^−1</h4><p>Because our features overlap (represented by that 83), we can’t just calculate their weights in isolation. In matrix algebra, finding the inverse of a matrix is equivalent to division.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*3-qTDeAVxwbpzfiGLLi3sg.png" /></figure><p><strong>The Insight:</strong> By inverting XTX, the algorithm mathematically “divides out” the overlap. It isolates the <em>pure</em> independent effect of Distance and the <em>pure</em> independent effect of Rating, completely untangling them from each other.</p><h4>Step 3: Finding the Raw Target Correlation (X.T @ y)</h4><p>Next, we multiply the transposed features by the actual delivery times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*yv7nWEfzNpezGdY54_czhg.png" /></figure><p><strong>The Insight:</strong> This matrix represents the direct, brute-force correlation between your inputs and the target. The top is the sum of all times, the middle is the sum of (Distance × Time), and the bottom is the sum of (Rating × Time).</p><h4>Step 4: The Final Result (θ)</h4><p>Finally, we multiply the untangled features from Step 2 by the raw target correlations from Step 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BjE_yIz2wSwZyt5EVaAzoA.png" /></figure><p><strong>The Final Insight:</strong> The formula outputs our absolute optimal weights instantly.</p><ul><li><strong>Intercept:</strong> 10 minutes base time.</li><li><strong>Distance:</strong> +2 minutes per kilometer.</li><li><strong>Rating:</strong> +5 minutes per unit of rating (in this synthetic example).</li></ul><h3>Ridge Regression</h3><p>If the Closed-form Solution is a perfect mathematical GPS, and Gradient Descent is walking down a mountain blindfolded, <strong>Ridge Regression</strong> is like adding shock absorbers to your GPS so it doesn’t spin out of control when the roads get messy.</p><p>In <strong>Ridge Regression</strong>, we make one tiny but incredibly powerful modification. We add a penalty term (λ) multiplied by an Identity matrix (I) to the covariance matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*pATud5K2LJ7IJDn5UYfcUQ.png" /></figure><p>If we wind back to same example from Normal equation’s step 2 where we calculated covariance matrix, the a</p><p>If featues are highly corelated, then matrix inverse would become unstable and could cause “<strong><em>divide by zero error</em></strong>”. To fix it, Ridge Regression creates an Identity matrix (I), which is just 1s on the diagonal and 0s everywhere else. We multiply it by λ (Lambda), which is our “penalty strength.” Let’s use a heavy penalty of <strong>λ=10</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*M5Cq6ZLatEONSWWYm_Naug.png" /></figure><p><strong>The Insight:</strong> Why did I change the top-left 1 to a <strong>0</strong>? Because that represents the Intercept (the base delivery time). In Machine Learning, we <strong>never</strong> penalize the intercept. We only want to shrink the slopes (the influence of the features), not the base starting point.</p><h4>Dropping the Mathematical Anchor</h4><p>Now, we add our Penalty Matrix to our Covariance Matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ExMljdF_DeBouHChHpat0g.png" /></figure><p><strong>The Insight:</strong> Look at the diagonal! The numbers 161 and 45.25 were artificially inflated to 171 and 55.25. By heavily beefing up the diagonal numbers, we are mathematically forcing the matrix to be stable and easily invertible. It is like widening the base of a tall tower so it won’t tip over.</p><p>Ridge Regression looked at our penalty (λ=10) and said: <em>“These feature weights are getting too cocky. I am going to shrink them down so the model doesn’t rely too heavily on any single feature.”</em></p><h3>Now the challenges on to building the models went like …</h3><h3>Act I: The Three Flavours of Linear Regression</h3><p>Before building the model, I needed to understand what my code was actually trying to do. It turns out, finding the “line of best fit” can be done in three completely different ways:</p><ol><li><strong>The Normal Equation (The Mathematical GPS):</strong> This is a closed-form solution. It uses matrix inversion (X.T @ X)^-1 @ X.T @ y to teleport directly to the exact mathematical center of the error valley. It’s incredibly fast for small datasets but crashes if features overlap perfectly.</li><li><strong>Batch Gradient Descent (The Blindfolded Hiker):</strong> Instead of calculating the answer instantly, BGD takes tiny, iterative steps down the error mountain using a learning rate. It’s essential for massive datasets where matrix inversion would blow up your computer’s RAM.</li><li><strong>Ridge Regression (The Shock Absorbers):</strong> Real-world data is messy and features often overlap. Ridge adds a mathematical penalty (L2 Regularization) ((X.T @ X) + (λI))^-1 @ X.T @ y that gently squishes the model’s weights to prevent it from overreacting to noisy data.</li></ol><p><em>The takeaway:</em> In production, standard Linear Regression is rarely used. You almost always use Ridge to keep your weights stable.</p><h3>Act II: The Bugs That Didn’t Throw Errors (Until They Did)</h3><p>Translating pure math into Python code is a minefield. Here are the most painful lessons I learned about data engineering:</p><h4>1. The NumPy Broadcasting Trap</h4><p>My Batch Gradient Descent kept throwing a bizarre ValueError: operands could not be broadcast together with shapes (10,) (10,45162) (10,).</p><p>The math was right, so what gives?</p><p>It turned out my target variable y was a 2D column vector (45162, 1). When NumPy tried to subtract it from a flat 1D array, its &quot;broadcasting&quot; feature panicked and silently created a massive 45,162 x 45,162 grid in memory.</p><p><strong>The Fix:</strong> Always append .flatten() to your y array before feeding it into an iterative ML loop.</p><h4>2. The Dummy Variable Trap</h4><p>When handling categorical data (like Type_of_vehicle), I used One-Hot Encoding to turn &quot;Scooter&quot; and &quot;Motorcycle&quot; into 1s and 0s. But when I fed this into the Normal Equation, the math violently exploded with a &quot;Singular Matrix&quot; divide-by-zero error.</p><p>Why? Because I had added a Bias column of 1s to act as my Intercept. If you add up the One-Hot Encoded vehicle columns, they also equal 1. The matrix realized it had duplicate data and couldn’t be inverted.</p><p><strong>The Fix:</strong> Always use pd.get_dummies(drop_first=True). Dropping one category forces the algorithm to use it as the baseline (Intercept), saving the math.</p><h4>3. The “True/False” RAM Killer</h4><p>By default, modern Pandas outputs One-Hot Encoded columns as True/False booleans. While this is memory efficient, passing booleans into NumPy&#39;s strict linear algebra engine causes massive slowdowns and type-casting crashes.</p><p><strong>The Fix:</strong> Pass dtype=&#39;uint8&#39; into get_dummies. Unsigned 8-bit integers use the exact same tiny amount of RAM (1 byte) as a boolean, but they keep the math engine perfectly happy.</p><h3>Act III: When the Math is Right, but the World is Wrong</h3><p>Once the pipeline was structurally sound, I ran a sample prediction. The result blew my mind: <strong>The model predicted that a bicycle would deliver food faster than a motorcycle on a 5km cross-town trip.</strong></p><p>Was my math broken? No. The math was flawless. The model had learned two parallel lines, and the motorcycle had a strict +3.99 minute penalty universally applied. I had hit the limits of Linear Regression: <em>it cannot bend.</em></p><h4>The Solution: Interaction Terms</h4><p>To allow a motorcycle to travel at a different speed (slope) than a bicycle, I had to multiply the Distance column by the is_Motorcycle column. This created an <strong>Interaction Term</strong>.</p><p>Suddenly, the math snapped into focus. The motorcycle still had a heavy baseline penalty (time spent parking and gearing up), but its new Interaction Weight was negative. It ate up distance faster. The lines crossed! At 1km, the bike won. But past a certain distance, the motorcycle overtook it. (more conclusion at the bottom)</p><h4>The Golden Rule of Scaling Interactions</h4><p>When adding interaction terms, you must standardize the continuous variable <em>first</em>, and then multiply it by the 0/1 dummy. If you standardize the final interaction column, you turn all your 0s into negative numbers, completely contaminating the baseline math.</p><h4>The Final Twist: Omitted Variable Bias</h4><p>Even with the interaction terms, the model still predicted the bicycle was faster up to 19 kilometers. Why? Because algorithms lack common sense. They only know the data they are given.</p><p>My dataset did not contain a Traffic_Density column. In the real world, dispatchers only assign bicycles in gridlocked downtown areas where cars can&#39;t move. Because the algorithm couldn&#39;t &quot;see&quot; the traffic, it assumed bicycles were just magically faster vehicles. This is known as <strong>Omitted Variable Bias</strong>—the model is only as smart as the context you feed it.</p><h3>Act IV: The Wrap</h3><p>Building the math in a .ipynb notebook is only step one. To actually use this, I decoupled the architecture:</p><ol><li><strong>The Engine:</strong> A FastAPI backend that loads a model_artifacts.pkl file containing the weights, means, standard deviations, and the exact column order.</li><li><strong>The UI:</strong> A Streamlit frontend where users adjust sliders.</li><li><strong>The Link:</strong> The UI sends JSON payloads to the API. The API dynamically calculates the distance buckets, standardizes the inputs on the fly using the <em>training set’s</em> metrics, performs the dot product (Theta @ Input), and returns the time.</li></ol><h3>Act IV: The X-Ray (Analysing the Residuals)</h3><p>To truly see where my model was failing, I plotted the <strong>Residuals</strong> (Actual Time - Predicted Time). If a model is perfect, the errors sit at 0. My plot revealed a harsh reality:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bcBW8lIh3GuVkBmi31x_4w.png" /></figure><ol><li><strong>The Massive Rectangle of Guessing:</strong> My errors stretched from -30 to +30 minutes. If an app quotes “30 minutes,” the food could arrive in 0 minutes or 60 minutes. The model was effectively guessing.</li><li><strong>The Diagonal Floor:</strong> The bottom of my error cloud had a subtle, unnatural diagonal cutoff. This is the physical limit of time. A model can occasionally predict a 40-minute delivery that arrives in 15 mins (residual -25), but it can never predict a 15-minute delivery that arrives in -10 mins.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6uvtB9YoV9y3sw-MMdnpdw.png" /></figure><ol><li><strong>Feature Importance:</strong> Looking at my standardised Ridge weights, I realised what actually mattered. Motorcycle penalties, Distance, and Driver Rating heavily drove the time. But “Order Type” (Meal vs. Snack) had weights close to 0.0. The math proved that <em>what</em> the customer ordered had almost zero impact on the delivery time.</li></ol><h3>Act V: The Business Psychology of Errors</h3><p>In the real world of logistics apps, not all mathematical errors are created equal.</p><p>If my model <strong>underestimates</strong> by 10 minutes (quotes 20, takes 30), it’s a business disaster. The customer stares at the map, feels lied to, leaves a 1-star review, and the driver gets a reduced tip. If my model <strong>overestimates</strong> by 10 minutes (quotes 40, takes 30), the customer is thrilled the food is “early” and the driver looks like a hero.</p><p>Data Science isn’t about finding Error = 0. It&#39;s about designing <strong>Asymmetric Loss</strong>. In production, we intentionally bias models to slightly overpredict to win the &quot;Under-promise, Over-deliver&quot; psychological game.</p><h3>Act VI: Moving to Production</h3><p>Linear Regression ultimately hit its ceiling. It assumes the world is a straight line, but adding 1km in a flooded downtown street during rush hour is not the same as adding 1km on an empty highway. To fix the massive 60-minute error spread, I would need to switch to <strong>Tree-Based Models</strong> (like XGBoost or Random Forest) which use “If/Then” splits to isolate localized chaos.</p><p>But to deploy my V1 model today, I decoupled the architecture:</p><ol><li><strong>The Engine:</strong> A FastAPI backend that loads a .pkl artifact containing my mathematically pure weights and standard deviation metrics.</li><li><strong>The UI:</strong> A Streamlit frontend.</li></ol><p>Building models from scratch takes ten times longer than importing a library. But the intuition you gain — understanding exactly how L2 regularization squishes a weight, or how a missing parenthesis in standardization can make your model think a driver is 200 years old — is invaluable. Machine learning isn’t just about calling APIs; it’s about translating the messy, illogical real world into pure, strictly-typed math.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2c54fbf8e21" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>