<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aryan Bisht on Medium]]></title>
        <description><![CDATA[Stories by Aryan Bisht on Medium]]></description>
        <link>https://medium.com/@AryanBeast?source=rss-1db4e17cc56d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*uB2mV7LaDAmdq_srqMC61Q.jpeg</url>
            <title>Stories by Aryan Bisht on Medium</title>
            <link>https://medium.com/@AryanBeast?source=rss-1db4e17cc56d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:52:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@AryanBeast/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Imbalanced Data & Use of Optuna Explained]]></title>
            <link>https://medium.com/@AryanBeast/imbalanced-data-use-of-optuna-explained-c51ef7975a59?source=rss-1db4e17cc56d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c51ef7975a59</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[smote]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[imbalanced-dataset]]></category>
            <dc:creator><![CDATA[Aryan Bisht]]></dc:creator>
            <pubDate>Sun, 10 May 2026 04:01:00 GMT</pubDate>
            <atom:updated>2026-05-10T04:01:00.831Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to another post in my ongoing machine learning adventure. This blog is part of a series where I’m diving into the world of ML — learning, experimenting, and (hopefully) not getting completely lost in the algorithms! 😅 If you’re interested in learning along with me, check out the whole series <a href="https://docs.google.com/spreadsheets/d/1tyC8Wg1jMQnMdGMlhbfG29GuN1DRxggTNWpKlYCJsRU/edit?usp=sharing"><strong>HERE 📚</strong></a>. Or, if you’re just here for this particular topic, no worries — you’re in the right spot too..!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*pupWgQsGlGkaqUIrhrEtSg.png" /></figure><p>So this article will provides a comprehensive guide to one of the most common yet overlooked challenges in machine learning: <strong>imbalanced data</strong>. A dataset is said to be imbalanced when one class (the majority class) significantly outnumbers the other class(es) (the minority class). If not handled carefully, this imbalance can lead to biased models and misleading performance evaluations.</p><p>Imbalanced datasets are not rare edge cases — they are the norm in many high-impact, real-world applications:</p><ul><li><strong>Finance:</strong> Fraud detection and credit risk assessment, where fraudulent transactions are extremely rare compared to legitimate ones.</li><li><strong>Healthcare:</strong> Prediction of rare diseases, where missing a positive case can have serious consequences.</li><li><strong>Business:</strong> Customer churn prediction, where only a small percentage of users typically leave a service.</li><li><strong>Earth Sciences:</strong> Predicting events like earthquakes or volcanic eruptions, which occur infrequently but are critically important.</li></ul><p>In all these domains, failing to correctly identify the minority class can be far more costly than making mistakes on the majority class.</p><h3>Key Problems Caused by Class Imbalance</h3><p><strong>1. Model Bias</strong></p><p>Most machine learning algorithms are designed to minimize overall error. When the data is imbalanced, the model naturally focuses on the majority class because it dominates the training data, often learning very little about the minority class.</p><p><strong>2. Misleading Evaluation Metrics</strong></p><p>Accuracy becomes unreliable in imbalanced scenarios. For example, if 90% of the data belongs to the class <em>“No Placement”</em>, a model that always predicts <em>“No Placement”</em> will achieve 90% accuracy — yet it completely fails to identify students who actually get placed. This makes accuracy a poor metric for judging real performance in such cases.</p><h3>Five Techniques to Handle Imbalanced Data</h3><p>The video demonstrates five practical strategies using the <strong>Imbalanced-learn</strong> library:</p><p><strong>1. Undersampling</strong></p><p>This technique reduces the number of samples in the majority class to match the minority class. While it helps balance the dataset and speeds up training, it risks losing valuable information by discarding potentially important data points.</p><p><strong>2. Oversampling (Random Oversampling)</strong></p><p>Here, minority class samples are randomly duplicated until both classes are balanced. Although this preserves all original data, it can lead to overfitting — especially with models like Decision Trees that can memorize repeated samples.</p><p><strong>3. SMOTE (Synthetic Minority Over-sampling Technique)</strong></p><p>SMOTE is one of the most popular approaches for handling imbalance. Instead of duplicating samples, it creates new synthetic minority instances by interpolating between existing ones. This generally reduces overfitting compared to random oversampling, but it can be sensitive to outliers in the data.</p><p><strong>4. Ensemble Methods</strong></p><p>Ensemble techniques such as <strong>Balanced Random Forest</strong> modify traditional algorithms by ensuring that each bootstrap sample used to train a tree is balanced. This leads to more robust performance but increases model complexity and training time.</p><p><strong>5. Cost-Sensitive Learning</strong></p><p>Rather than changing the data distribution, this approach modifies the learning process itself. Higher misclassification costs are assigned to the minority class using class weights or custom loss functions. While easy to implement, it is not supported by all algorithms and requires careful tuning.</p><p>So, imbalanced data is not a niche problem — it’s the default in real-world machine learning. Whether you’re detecting fraud, predicting rare diseases, or identifying customer churn, blindly trusting accuracy can quietly sabotage your model’s usefulness. The techniques discussed — sampling strategies, synthetic data generation, ensemble methods, and cost-sensitive learning — aren’t silver bullets, but practical tools. The key takeaway is simple: <strong>handle imbalance deliberately, evaluate with the right metrics, and choose techniques based on data size, noise, and model behavior</strong>. A model that finds the minority class is often far more valuable than one that merely looks accurate.</p><h3><strong>Optuna</strong></h3><p><strong>Optuna</strong>, a modern framework used for hyperparameter tuning in Machine Learning and Deep Learning. Optuna is introduced as an open-source tool built for speed and efficiency, aiming to remove the manual effort and guesswork involved in selecting optimal hyperparameters.</p><p>The key motivation behind using Optuna is its reliance on Bayesian Optimization rather than brute-force techniques. Traditional methods like Grid Search are computationally expensive because they try all possible combinations, while Random Search can waste trials on unimportant regions of the search space. Optuna, on the other hand, intelligently learns from previous trials and gradually shifts its focus toward more promising hyperparameter values, making the optimization process faster and more effective.</p><p>To understand how Optuna works internally, the video explains a few core concepts. A <strong>study</strong> represents the overall optimization task and manages all experiments. Each <strong>trial</strong> is a single run of the objective function with a specific set of hyperparameters. These values are referred to as <strong>trial parameters</strong>. The <strong>objective function</strong> defines what needs to be optimized, such as maximizing accuracy or minimizing loss, and the <strong>sampler</strong> is the strategy that decides which hyperparameters should be tried next based on past results, commonly using the Tree-structured Parzen Estimator (TPE).</p><p>One of Optuna’s strongest advantages is its flexibility. While Bayesian Optimization is the default approach, Optuna also supports Random Search and Grid Search when required. It additionally provides powerful visualization tools such as optimization history plots, parallel coordinate plots, and contour plots, which help in analyzing how different hyperparameters influence model performance and in identifying optimal regions of the search space.</p><p>A standout feature discussed in the video is Optuna’s <strong>define-by-run</strong> capability, which allows dynamic search spaces. This means that even the choice of machine learning algorithm can be treated as a hyperparameter. Within a single study, Optuna can decide whether to use models like SVM, Random Forest, or XGBoost, and then tune the corresponding hyperparameters of the selected model. This makes Optuna highly adaptable for real-world experimentation.</p><p>Finally, the video demonstrates a simple workflow for using Optuna in practice: installing it via pip, defining an objective function that receives a trial object, creating a study, and running the optimization for a fixed number of trials. The conclusion emphasizes that Optuna’s intelligent search strategy, combined with its dynamic nature and strong framework integrations, makes it one of the most powerful and practical hyperparameter tuning tools available today.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>You can buy me coffee at :- <a href="https://www.buymeacoffee.com/aryanbisht">https://www.buymeacoffee.com/aryanbisht</a></p><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 If you find this helpful Clap for the story and follow the author</li><li>🔔 Follow us: <a href="https://www.linkedin.com/in/spaa-club-50b302216/">LinkedIn</a>, <a href="https://twitter.com/bisht_aryan_ji"><em>Twitter</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c51ef7975a59" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DBSCAN Explained: Clustering Data by Density, Not Distance]]></title>
            <link>https://medium.com/@AryanBeast/dbscan-explained-clustering-data-by-density-not-distance-907d1f607a31?source=rss-1db4e17cc56d------2</link>
            <guid isPermaLink="false">https://medium.com/p/907d1f607a31</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[k-means-clustering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[clustering-algorithm]]></category>
            <category><![CDATA[dbscan-algorithm]]></category>
            <dc:creator><![CDATA[Aryan Bisht]]></dc:creator>
            <pubDate>Wed, 06 May 2026 04:01:04 GMT</pubDate>
            <atom:updated>2026-05-06T04:01:04.291Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to another post in my ongoing machine learning adventure. This blog is part of a series where I’m diving into the world of ML — learning, experimenting, and (hopefully) not getting completely lost in the algorithms! 😅 If you’re interested in learning along with me, check out the whole series <a href="https://docs.google.com/spreadsheets/d/1tyC8Wg1jMQnMdGMlhbfG29GuN1DRxggTNWpKlYCJsRU/edit?usp=sharing"><strong>HERE 📚</strong></a>. Or, if you’re just here for this particular topic, no worries — you’re in the right spot too..!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*x7Ztu-GdtNdFzy_RmcTT7g.png" /></figure><h3>Introduction to DBSCAN</h3><p>DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a density-based clustering algorithm that groups data points based on how closely they are packed together. Unlike centroid-based algorithms such as K-Means, DBSCAN does not rely on distances from cluster centers. Instead, it identifies regions of high point density and treats sparsely populated regions as noise. This makes DBSCAN especially effective for real-world datasets where clusters may have irregular shapes and where outliers are present.</p><p><strong>Why Use DBSCAN Instead of K-Means?</strong></p><p>One of the biggest limitations of K-Means is that it requires the number of clusters to be specified in advance. In many practical scenarios, this information is not known beforehand. DBSCAN overcomes this limitation by automatically determining the number of clusters based on data density. Another major drawback of K-Means is its sensitivity to outliers. Since K-Means assigns every point to a cluster, extreme values can significantly shift centroids and distort clusters. DBSCAN, on the other hand, explicitly identifies such points as noise and excludes them from clusters. Additionally, K-Means assumes clusters are spherical in shape, which causes it to fail on datasets with complex structures such as concentric circles or curved patterns. DBSCAN does not make any assumptions about cluster shape and can successfully identify clusters of arbitrary forms.</p><p><strong>Hyperparameters of DBSCAN</strong></p><p>DBSCAN relies on two key hyperparameters: epsilon (ε) and MinPoints. Epsilon defines the radius of the neighborhood around a data point and determines how close points must be to be considered neighbors. MinPoints specifies the minimum number of points that must exist within a point’s ε-neighborhood for it to be considered a dense region. The choice of these parameters is critical: a very small ε may label most points as noise, while a very large ε can merge distinct clusters into one.</p><p><strong>Types of Data Points in DBSCAN</strong></p><p>Based on ε and MinPoints, DBSCAN classifies each data point into one of three categories. A <strong>Core Point</strong> is a point that has at least MinPoints within its ε-neighborhood and serves as the foundation for forming clusters. A <strong>Border Point</strong> has fewer than MinPoints in its neighborhood but lies within the ε-neighborhood of a Core Point; such points belong to a cluster but cannot expand it further. A <strong>Noise Point</strong> is a point that is neither a Core Point nor a Border Point and is treated as an outlier. These noise points are ignored during cluster formation.</p><p><strong>Working of the DBSCAN Algorithm</strong></p><p>The DBSCAN algorithm starts by examining each data point and identifying its ε-neighborhood to determine whether it is a Core, Border, or Noise point. When an unvisited Core Point is found, a new cluster is created, and all points that are density-connected to it are added to the cluster. This expansion continues recursively as long as new Core Points are discovered within the neighborhood. Border Points encountered during this process are assigned to the cluster but do not contribute to further expansion. Points that are not reachable from any Core Point remain unassigned and are labeled as noise, often represented by -1 in implementations.</p><p>DBSCAN performs exceptionally well on datasets with complex patterns and noise, such as moon-shaped or circular clusters, where K-Means fails. It is robust to outliers and does not require prior knowledge of the number of clusters. However, DBSCAN struggles when clusters have varying densities, as a single ε value cannot appropriately capture all dense regions. This limitation often motivates the use of advanced variants like HDBSCAN.</p><p>Below is a simple Python implementation of DBSCAN using Scikit-Learn on a non-spherical dataset:</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn.cluster import DBSCAN<br>from sklearn.datasets import make_moons<br><br># Create dataset<br>X, _ = make_moons(n_samples=300, noise=0.05, random_state=42)<br><br># Apply DBSCAN<br>dbscan = DBSCAN(eps=0.3, min_samples=5)<br>labels = dbscan.fit_predict(X)<br><br># Visualize clusters<br>plt.scatter(X[:, 0], X[:, 1], c=labels, cmap=&#39;viridis&#39;, s=30)<br>plt.title(&quot;DBSCAN Clustering&quot;)<br>plt.xlabel(&quot;Feature 1&quot;)<br>plt.ylabel(&quot;Feature 2&quot;)<br>plt.show()</pre><p>In this implementation, points labeled -1 represent noise, while non-negative integers represent different clusters. DBSCAN successfully separates the curved clusters without being affected by outliers.</p><p><strong>Choosing the Right Value of Epsilon</strong></p><p>Selecting an appropriate ε value is crucial for DBSCAN’s performance. A common approach is the <strong>k-distance graph</strong>, where distances to the k-th nearest neighbor (usually k = MinPoints) are plotted in ascending order. The point where the graph shows a sharp bend, known as the elbow, is often a good estimate for ε.</p><p>DBSCAN is a powerful clustering algorithm that excels in discovering clusters based on data density rather than distance. Its ability to handle arbitrary cluster shapes and ignore noise makes it highly suitable for real-world datasets. However, its sensitivity to parameter selection and difficulty in handling varying densities should be carefully considered before use.</p><p>DBSCAN offers a fundamentally different way of thinking about clustering by focusing on data density rather than predefined structure. Its ability to automatically discover clusters, handle arbitrary shapes, and explicitly identify noise makes it a powerful tool for real-world datasets where assumptions often fail. While careful parameter tuning is required, DBSCAN remains an essential algorithm to understand when working with unsupervised learning and complex data distributions.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>You can buy me coffee at :- <a href="https://www.buymeacoffee.com/aryanbisht">https://www.buymeacoffee.com/aryanbisht</a></p><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 If you find this helpful Clap for the story and follow the author</li><li>🔔 Follow us: <a href="https://www.linkedin.com/in/spaa-club-50b302216/">LinkedIn</a>, <a href="https://twitter.com/bisht_aryan_ji"><em>Twitter</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=907d1f607a31" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[XGBoost Explained from the Inside: Regression, Classification, and Tree Construction]]></title>
            <link>https://medium.com/@AryanBeast/xgboost-explained-from-the-inside-regression-classification-and-tree-construction-e7657f882578?source=rss-1db4e17cc56d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7657f882578</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gradient-boosting]]></category>
            <dc:creator><![CDATA[Aryan Bisht]]></dc:creator>
            <pubDate>Thu, 30 Apr 2026 04:01:01 GMT</pubDate>
            <atom:updated>2026-04-30T04:01:01.372Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to another post in my ongoing machine learning adventure. This blog is part of a series where I’m diving into the world of ML — learning, experimenting, and (hopefully) not getting completely lost in the algorithms! 😅 If you’re interested in learning along with me, check out the whole series <a href="https://docs.google.com/spreadsheets/d/1tyC8Wg1jMQnMdGMlhbfG29GuN1DRxggTNWpKlYCJsRU/edit?usp=sharing"><strong>HERE 📚</strong></a>. Or, if you’re just here for this particular topic, no worries — you’re in the right spot too..!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*MJHZ3dFQ8YpDkU2MpzESEg.png" /></figure><p>Now we will be studying about one of the most used &amp; important concept in today’s ML world, XGBoost. XGBoost (Extreme Gradient Boosting) is <strong>not a new algorithm</strong>, but a <strong>highly optimized implementation of Gradient Boosting</strong>. Its success comes from combining <strong>solid ML theory</strong> with <strong>serious systems-level engineering</strong>. Because of this combination, it has become a dominant choice in machine learning competitions and real-world industry applications, especially where performance, scalability, and accuracy are critical.</p><p>The term “Extreme” in XGBoost refers to the aggressive optimizations applied at both the algorithmic and system levels. XGBoost is designed to extract maximum performance from available hardware by reducing computation time, improving memory usage, and enabling scalability across CPUs, GPUs, and distributed systems. These extreme optimizations allow XGBoost to train models significantly faster than standard Gradient Boosting while maintaining or even improving predictive accuracy.</p><h3>Computational Speed and Software Optimizations</h3><p>XGBoost achieves exceptional speed through multiple software-level optimizations. It supports parallel processing during tree construction by evaluating feature splits simultaneously across multiple CPU cores, even though trees themselves are built sequentially. The library uses a column block data structure that stores data in a column-wise format, enabling efficient memory access and parallel computation. Additionally, XGBoost is cache-aware, meaning it is designed to utilize CPU cache effectively and minimize slow memory accesses. For datasets larger than system memory, XGBoost supports out-of-core computation by processing data in chunks and storing compressed representations on disk. It also enables distributed training across multiple machines and supports GPU acceleration for highly parallel tasks such as histogram generation, resulting in substantial reductions in training time.</p><h3>Machine Learning Improvements Over Gradient Boosting</h3><p>Beyond speed, XGBoost introduces important improvements to the learning process itself. Unlike traditional Gradient Boosting, XGBoost includes L1 and L2 regularization terms directly in its objective function, which helps control model complexity and reduce overfitting. It also handles missing values automatically using sparsity-aware split finding, where the algorithm learns the optimal direction to assign missing values during training. Furthermore, XGBoost employs approximate tree learning using weighted quantile sketches, which reduces the cost of split evaluation by grouping feature values into bins while preserving most of the model’s accuracy.</p><h3>Flexibility and Practical Applicability</h3><p>XGBoost is highly flexible and practical for real-world use. It supports multiple programming languages such as Python, R, Java, Scala, and C++, allowing models to be trained in one environment and deployed in another. The library can be applied to a wide range of supervised learning problems, including regression, binary and multiclass classification, ranking tasks, and time-series forecasting, making it suitable for diverse machine learning workflows.</p><h3>Comparison with Other Gradient Boosting Libraries</h3><p>While XGBoost is widely regarded as a gold standard, other gradient boosting libraries also offer competitive performance. LightGBM focuses on faster training and lower memory consumption, particularly for large datasets, whereas CatBoost is optimized for handling categorical features without extensive preprocessing. Despite these alternatives, XGBoost remains a popular and reliable choice due to its balanced performance, robustness, and extensive ecosystem.</p><p><strong>Core Workflow of XGBoost Regression</strong></p><p>XGBoost follows a stage-wise additive modeling approach similar to traditional Gradient Boosting. In the first stage, the model starts with a base prediction, which is typically the mean of the target values. This constant prediction minimizes the initial squared error loss. Next, residuals are calculated by subtracting the predicted values from the actual target values. These residuals represent the error that the current model has failed to explain. From the second stage onward, XGBoost trains decision trees to predict these residuals rather than the original target. After building a tree, its output is multiplied by a learning rate (η) and added to the previous prediction. This iterative process continues for a fixed number of estimators, with each new tree attempting to reduce the remaining residuals toward zero.</p><p><strong>What Makes XGBoost Trees “Special”</strong></p><p>The key difference between XGBoost and standard Gradient Boosting lies in how decision trees are constructed. Instead of relying on impurity measures such as Gini index or entropy, XGBoost builds trees using optimization-based metrics derived from the loss function. These metrics allow the algorithm to evaluate splits and leaf nodes while explicitly incorporating regularization, making the resulting trees more robust and less prone to overfitting.</p><p><strong>Similarity Score in XGBoost</strong></p><p>To evaluate the quality of a leaf node, XGBoost uses a metric called the Similarity Score. This score measures how well the residuals grouped within a leaf node align with each other. It is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*z4adsifX4zwdC-pWbTyDbw.png" /></figure><p>Here, λ (lambda) is a regularization parameter that penalizes overly complex trees by discouraging leaves with very few data points. In the video’s example, λ is set to zero for simplicity, which reduces the similarity score to a function of only the residual values.</p><p><strong>Gain and Split Selection</strong></p><p>To determine the best split at a node, XGBoost computes the Gain for each possible split. Gain measures how much improvement a split provides over keeping the node unsplit. It is calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*4kS3qF4WJZXYBiMr_nqdpw.png" /></figure><p>The split that produces the highest gain is selected. This ensures that every split meaningfully reduces the overall loss while accounting for model complexity. By using gain-based split selection, XGBoost constructs trees that are both efficient and well-regularized.</p><p><strong>Output Value of Leaf Nodes</strong></p><p>Once the tree structure is finalized, XGBoost assigns an output value to each leaf node. This value determines how much the prediction should be adjusted for data points that fall into that leaf. The output value is computed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*_irPQjt_31SA77eSTjqG_g.png" /></figure><p>If λ is set to zero, this formula simplifies to the average of the residuals in that leaf. These output values are then scaled by the learning rate and added to the existing predictions during model updates.</p><p><strong>Tree Construction Algorithms</strong></p><p>XGBoost supports multiple algorithms for finding optimal splits in decision trees. For small datasets, it uses the Exact Greedy Algorithm, which evaluates every possible split point to find the one with the maximum gain. This approach is computationally expensive but highly accurate and is the method demonstrated in the video. For large datasets, XGBoost uses an Approximate Algorithm that relies on quantiles and bucket-based splitting. This significantly reduces computation time while maintaining near-optimal accuracy.</p><h3>XGBoost for Classification</h3><p>XGBoost for classification follows the stage-wise additive modeling approach, similar to regression, but with important differences in how predictions are represented. Instead of starting with the mean of the target variable, the model begins by computing the Log(Odds) of the positive class. This initial prediction represents the baseline confidence of belonging to the positive class and is calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*xQpTs2xdwYdbRqktlPl50g.png" /></figure><p>where P is the probability of the positive class. This Log(Odds) value is then converted into a probability using the sigmoid function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*JFHLv7IVlwNrRTx140giPw.png" /></figure><p>This probability becomes the model’s initial prediction for all data points.</p><p><strong>Residual Calculation in Classification</strong></p><p>Once the initial probability is obtained, XGBoost computes residuals, often called pseudo-residuals in classification. These residuals are calculated by subtracting the predicted probability from the actual class label, which is either 0 or 1. The residual therefore represents how much the model underestimates or overestimates the probability for each data point. These residuals serve as the target values for the next stage of the model.</p><p><strong>Building Trees in XGBoost Classification</strong></p><p>From the second stage onward, XGBoost trains decision trees to predict the residuals rather than the original class labels. However, unlike standard decision trees, XGBoost classification trees are constructed using metrics derived from the second-order Taylor expansion of the loss function. These metrics allow XGBoost to directly optimize classification loss while incorporating regularization.</p><p><strong>Similarity Score in Classification</strong></p><p>To evaluate how good a leaf node is during tree construction, XGBoost uses a classification-specific Similarity Score. This score measures how well the residuals in a leaf node align while accounting for prediction uncertainty. It is calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*dMJlpV7fsvSV5ViJLgFR1Q.png" /></figure><p>Here, the term Previous Prob×(1−Previous Prob) represents the variance of the Bernoulli distribution, reflecting uncertainty in probability predictions. The regularization parameter λ penalizes overly complex trees and is set to zero in the video for simplicity.</p><p><strong>Gain and Split Selection</strong></p><p>To decide the best split at any node, XGBoost computes the Gain for each possible split. Gain measures how much improvement a split provides compared to keeping the node intact. It is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*9msXGbw4xyBMVDn14irZJA.png" /></figure><p>The split that maximizes the gain is selected, ensuring that each split significantly reduces classification loss while respecting regularization constraints.</p><h3>Output Value of Leaf Nodes</h3><p>After the tree structure is finalized, XGBoost assigns an output value to each leaf node. This value determines how much the Log(Odds) should be adjusted for data points that fall into that leaf. The output value is calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*EhUNBzf87oldofiG5UI8lw.png" /></figure><p>This output is then multiplied by the learning rate and added to the previous Log(Odds) during prediction updates.</p><p><strong>Iteration and Final Prediction</strong></p><p>The boosting process continues iteratively, with each new tree attempting to push the residuals closer to zero. The final model prediction for a new data point is obtained by summing the initial Log(Odds) and the contributions of all trees, each scaled by the learning rate. This final Log(Odds) value is then converted back into a probability using the sigmoid function, and the probability is used to determine the final class label.</p><p>So at the end I will say XGBoost excels by combining gradient boosting principles with strong mathematical foundations and efficient software design. Its consistent stage-wise framework for both regression and classification, along with regularized tree construction and gain-based optimization, makes it a powerful and reliable model. Understanding these internal mechanisms helps practitioners move beyond treating XGBoost as a black box and apply it more effectively to real-world machine learning problems. If want to learn math behind each formula you can check this video <a href="https://www.youtube.com/watch?v=0Eo-_5bfers&amp;list=PLKnIA16_Rmvbr7zKYQuBfsVkjoLcJgxHH&amp;index=130&amp;pp=iAQB0gcJCZEKAYcqIYzv">LINK</a>.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>You can buy me coffee at :- <a href="https://www.buymeacoffee.com/aryanbisht">https://www.buymeacoffee.com/aryanbisht</a></p><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 If you find this helpful Clap for the story and follow the author</li><li>🔔 Follow us: <a href="https://www.linkedin.com/in/spaa-club-50b302216/">LinkedIn</a>, <a href="https://twitter.com/bisht_aryan_ji"><em>Twitter</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7657f882578" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Probability Basics to Naive Bayes: Intuition, Math, and Real-World Examples]]></title>
            <link>https://medium.com/@AryanBeast/from-probability-basics-to-naive-bayes-intuition-math-and-real-world-examples-c3281152023f?source=rss-1db4e17cc56d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3281152023f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[bayes-theorem]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[naive-bayes]]></category>
            <dc:creator><![CDATA[Aryan Bisht]]></dc:creator>
            <pubDate>Sat, 25 Apr 2026 04:01:01 GMT</pubDate>
            <atom:updated>2026-04-25T04:01:01.398Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to another post in my ongoing machine learning adventure. This blog is part of a series where I’m diving into the world of ML — learning, experimenting, and (hopefully) not getting completely lost in the algorithms! 😅 If you’re interested in learning along with me, check out the whole series <a href="https://docs.google.com/spreadsheets/d/1tyC8Wg1jMQnMdGMlhbfG29GuN1DRxggTNWpKlYCJsRU/edit?usp=sharing"><strong>HERE 📚</strong></a>. Or, if you’re just here for this particular topic, no worries — you’re in the right spot too..!</p><p>Now we will be studying little bit of probability as it is required for our future algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*RsgtddBwEf8FSA2czlV-YA.png" /></figure><h3>Conditional Probability</h3><p>Conditional probability answers a simple but powerful question: <em>what is the probability of an event A, given that another event B has already occurred?</em> Instead of considering all possible outcomes, conditional probability forces us to rethink the situation by focusing only on outcomes where the given condition is true. This idea is mathematically captured by the formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/1*Ejvh5ldAIjCUBUQc6CuOgA.png" /></figure><p>Here, P(B) represents the probability of the condition, while P(A∩B) represents the probability that both the event of interest and the condition occur together. Conceptually, this formula tells us that conditional probability is about <strong>restricting the sample space first and then measuring probability within that restricted space</strong>.</p><p>The most important takeaway from this example is not the numerical result, but the way the reasoning changes. Conditional probability is not about applying a formula mechanically; it is about <strong>changing the universe of possible outcomes</strong> based on the given condition. Once this new universe is defined, probability calculations often become straightforward and intuitive.</p><h3>Independent Events</h3><p>Independent events are events where the occurrence of one does not affect the probability of the other. Knowing that one event has happened gives no extra information about whether the second event will occur. This idea is fundamental in probability theory and plays a key role in understanding how the Naive Bayes classifier works. Two events A and B are said to be independent if the probability of both occurring together is equal to the product of their individual probabilities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*QsxxtTBt955toZUQcZ8D4Q.png" /></figure><p>Independence has a direct connection with conditional probability. If events A and B are independent, then conditioning on one does not change the probability of the other:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/229/1*Om0M-UxeX4V9LGoRI12N_w.png" /></figure><p>This happens because event B does not provide any useful information about event A.</p><h3>Mutually Exclusive</h3><p>Mutually exclusive events are events that <strong>cannot occur at the same time</strong>. If one event happens, the other becomes impossible. In probability terms, this means the intersection of two mutually exclusive events is zero:</p><p>P(A∩B)=0</p><p>This concept is simple but important, especially when distinguishing between different types of probability relationships. Mutual exclusivity has a direct implication for conditional probability. If two events are mutually exclusive, then:</p><p>P(A∣B)=0</p><p>This is because once event B has occurred, there is no possible outcome left in which event A could also occur. The condition completely eliminates the event of interest.</p><h3>Mutually Exclusive vs. Independent Events</h3><p>Mutually exclusive events are often confused with independent events, but they are fundamentally different. Independent events can occur together and do not influence each other’s probabilities. Mutually exclusive events, on the other hand, <strong>cannot occur together at all</strong>. In fact, two non-zero probability events cannot be both independent and mutually exclusive at the same time.</p><h3>Bayes’ Theorem: The Foundation of Naive Bayes</h3><p>Bayes’ Theorem, named after <strong>Thomas Bayes</strong>, is a fundamental result in probability theory and the backbone of Bayesian statistics and the Naive Bayes classifier. The theorem provides a systematic way to update our beliefs about an event when new information becomes available. It is expressed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*ki9rK8NKLgIXI_uVyFKY0g.png" /></figure><p>This formula allows us to compute the probability of event A given that event B has occurred, by reversing the conditional probability from P(B∣A). Each term in Bayes’ Theorem has a specific interpretation. The probability we want to compute, P(A∣B), is called the <strong>posterior probability</strong>. The term P(B∣A) is known as the <strong>likelihood</strong>, representing how likely the evidence is given the event. The probability P(A)is the <strong>prior</strong>, which captures our belief about event A before seeing any evidence. Finally, P(B) is the <strong>evidence</strong>, acting as a normalization factor to ensure the probabilities remain valid.</p><p>Let B be an event whose probability we want to calculate, but assume that B can happen due to <strong>multiple distinct situations</strong>. Let those situations be events A1,A2,A3. These events are <strong>mutually exclusive and exhaustive</strong>, meaning only one of them can occur at a time, and together they cover all possible cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*y8BpO9HHlPXW8OzuANIedw.png" /></figure><h3>Naive Bayes Classifier</h3><p>The Naive Bayes classifier is designed to predict a <strong>categorical outcome</strong> by combining multiple input features using Bayes’ Theorem. Instead of looking at one factor in isolation, it asks: <em>given all the observed features, which class is most likely?</em> The power of Naive Bayes lies not in complex modeling, but in how it breaks a difficult probability question into smaller, manageable pieces. The algorithm is called “naive” because it assumes that all input features are independent of each other given the class. This assumption allows the joint probability of all features to be written as a simple product of individual conditional probabilities. While this assumption is rarely true in real life, it dramatically simplifies computation and works surprisingly well in practice. To make a final prediction, Naive Bayes computes this probability score for each possible class, such as <strong>Win</strong> and <strong>Lose</strong>, and compares them. The class with the higher score is chosen as the prediction. Importantly, the denominator of Bayes’ Theorem (the probability of the evidence) is the same for all classes, so it can be ignored during comparison. This makes classification faster without affecting correctness.By decomposing a complex conditional probability into smaller independent components, Naive Bayes turns an otherwise difficult problem into a computationally efficient one. This same intuition scales to high-dimensional problems like spam detection and document classification, where dozens or hundreds of features must be handled simultaneously.</p><p><strong>Real-World Example: Predicting a Match Result</strong></p><p>Suppose we want to predict whether <strong>Chennai Super Kings</strong> will <strong>Win (W)</strong> or <strong>Lose (L)</strong> a match based on three observed conditions: the team <strong>lost the toss</strong>, the match is played in <strong>Mumbai</strong>, and the weather is <strong>Sunny</strong>. Our task is to decide which outcome — Win or Lose — is more likely given these conditions. This prediction problem can be written using conditional probability as and similarly for losing :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*DlUNjUMu-wHJljmuIyvjuA.png" /></figure><p>The Naive Bayes classifier computes both probabilities and compares them. The class with the higher probability becomes the prediction.</p><p>Now applying Bayes’ Theorem, the probability of a win given the observed conditions is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*_Ip3moBSCY6zNfmw_JinNw.png" /></figure><p>At this stage, the denominator represents the probability of observing those match conditions overall. Since this denominator is <strong>the same for both Win and Lose</strong>, it does not affect the comparison and can be ignored. This gives:</p><p>P(W∣Loss,Mumbai,Sunny) ∝ P(Loss,Mumbai,Sunny∣W)×P(W)</p><p>Naive Bayes now makes its key assumption: the features are conditionally independent given the class. This allows the joint probability to be broken into simpler parts:</p><p>P(Loss,Mumbai,Sunny∣W) = P(Loss∣W)×P(Mumbai∣W)×P(Sunny∣W)</p><p>Substituting this back:</p><p>P(W∣Loss,Mumbai,Sunny) ∝ P(W)×P(Loss∣W)×P(Mumbai∣W)×P(Sunny∣W)</p><p>The same calculation is performed for the losing class:</p><p>P(L∣Loss,Mumbai,Sunny) ∝ P(L)×P(Loss∣L)×P(Mumbai∣L)×P(Sunny∣L)</p><h3>Final Decision Rule</h3><p>The Naive Bayes classifier compares these two values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*iKxpvk3kPuS3C1buhgrK2g.png" /></figure><p>Whichever side is larger determines the prediction. No exact probability is needed — only the <strong>relative magnitude</strong> matters.</p><blockquote>Naive Bayes predicts the class that maximizes P(Class)×∏P(Feature∣Class)P(Class)</blockquote><p>Many of you are now might be wondering why we <em>Can’t</em> Directly Compute P(W∣Loss,Mumbai,Sunny). In theory, you <em>could</em> compute</p><p>P(W∣Loss,Mumbai,Sunny) , <strong>directly</strong> — but only if you had <strong>enough data</strong> to do so reliably.</p><p>By definition:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*xIAWTVBMPjgejLNnG9pG8A.png" /></figure><p>In principle, it may seem that we should be able to directly compute P(W∣Loss,Mumbai,Sunny) from data, just like we compute P(Loss,Mumbai,Sunny∣W). However, the difficulty arises from how real-world data is distributed. Directly estimating P(W∣Loss,Mumbai,Sunny) requires us to count how many times this <em>exact combination</em> of features appears in the dataset and how often it leads to a win. In practice, such exact combinations occur very rarely, especially as the number of features increases.</p><p>As more features are added, the dataset becomes sparse: most feature combinations either appear only once or do not appear at all. This makes the direct conditional probability unstable or even zero, which is useless for prediction. Therefore, although the formula for P(W∣Loss,Mumbai,Sunny) is mathematically valid, it is not practically reliable when working with limited data.</p><p>On the other hand, probabilities like P(Loss∣W)P, P(Mumbai∣W), and P(Sunny∣W) are much easier to estimate. These probabilities only require us to look at individual features within all winning matches, which provides many more data points. By assuming conditional independence, Naive Bayes combines these simpler probabilities to approximate the joint probability P(Loss,Mumbai,Sunny∣W).</p><p>Bayes’ Theorem then allows us to reverse this probability and obtain P(W∣Loss,Mumbai,Sunny) without ever computing it directly. In this way, Naive Bayes avoids the data sparsity problem by transforming a hard-to-estimate probability into a set of simpler, more reliable estimates.</p><h3>Handling Numerical Data in Naive Bayes</h3><p>So far, Naive Bayes has worked with categorical features, where probabilities can be estimated by simple counting. However, this approach breaks down when features are <strong>continuous numerical values</strong>, such as height, weight, or salary. For numerical data, the probability of observing an exact value like 175.5 is usually zero or extremely small, which makes direct probability estimation unreliable. To handle numerical features, Naive Bayes makes an additional assumption: the data for each feature, within each class, follows a known probability distribution. The most commonly used choice is the <strong>Gaussian (Normal) distribution</strong>, because many real-world measurements naturally cluster around a mean and form a bell-shaped curve. Instead of counting frequencies, we model how values are spread using this distribution.</p><p>For every numerical feature and for every class, Naive Bayes computes two parameters from the training data: the <strong>mean (μ)</strong>, which represents the average value, and the <strong>standard deviation (σ)</strong>, which captures how much the values vary. These parameters summarize the entire distribution of the feature for that class. When a new data point arrives, Naive Bayes does not ask “how often did this exact value appear?” Instead, it asks “how likely is this value under the class’s distribution?” This likelihood is computed using the Gaussian probability density function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*g_RtWgwQn1mbTMLeKql5ig.png" /></figure><p>By plugging the feature value into this formula using the class-specific mean and standard deviation, the model obtains a probability-like score that reflects how well the value fits that class. While Gaussian Naive Bayes is the most popular, it is not the only option. If the data does not resemble a bell curve, other distributions can be used. Multinomial Naive Bayes is common for text data such as word counts, and Bernoulli Naive Bayes is suitable for binary features. Choosing the right distribution depends on the nature of the data and can often be guided by visualizing it.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>You can buy me coffee at :- <a href="https://www.buymeacoffee.com/aryanbisht">https://www.buymeacoffee.com/aryanbisht</a></p><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 If you find this helpful Clap for the story and follow the author</li><li>🔔 Follow us: <a href="https://www.linkedin.com/in/spaa-club-50b302216/">LinkedIn</a>, <a href="https://twitter.com/bisht_aryan_ji"><em>Twitter</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3281152023f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Support Vector Machines Explained with Geometric Intuition]]></title>
            <link>https://medium.com/@AryanBeast/support-vector-machines-explained-with-geometric-intuition-3f0ca52e075a?source=rss-1db4e17cc56d------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f0ca52e075a</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[svm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[support-vector-machine]]></category>
            <dc:creator><![CDATA[Aryan Bisht]]></dc:creator>
            <pubDate>Mon, 20 Apr 2026 04:01:01 GMT</pubDate>
            <atom:updated>2026-04-20T04:01:01.363Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to another post in my ongoing machine learning adventure. This blog is part of a series where I’m diving into the world of ML — learning, experimenting, and (hopefully) not getting completely lost in the algorithms! 😅 If you’re interested in learning along with me, check out the whole series <a href="https://docs.google.com/spreadsheets/d/1tyC8Wg1jMQnMdGMlhbfG29GuN1DRxggTNWpKlYCJsRU/edit?usp=sharing"><strong>HERE 📚</strong></a>. Or, if you’re just here for this particular topic, no worries — you’re in the right spot too..!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*cIxK9GdR3-M1fQbY8RsXYg.png" /></figure><p>Support Vector Machines (SVM) are supervised learning algorithms designed to find the <em>best possible boundary</em> between different classes of data. Unlike models such as Logistic Regression, which aim to find <strong>any boundary</strong> that separates the classes, SVM is much more selective. Its primary objective is to find the <strong>Maximum Margin Hyperplane</strong> — a decision boundary that maximizes the distance between itself and the closest data points from each class. This idea of maximizing the margin is crucial because a wider margin generally leads to better generalization and more reliable predictions on unseen data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*py-5hn0-cZdktJbd6-5PtQ.png" /></figure><p>Geometrically, imagine two groups of points in a 2D space. Many lines can separate them without errors, but SVM chooses the one that maximizes the <strong>margin</strong>, which is the distance between the boundary and the nearest data points. These nearest points are called <strong>support vectors</strong>, and they are the only points that actually define the model. Moving a support vector changes the boundary, while moving other distant points has no effect.</p><p>One major advantage of SVM is its <strong>robustness</strong>. By focusing on margin maximization, it becomes less sensitive to noise and small variations near the decision boundary. Although the basic idea is explained using linear separation, SVM can also handle <strong>non-linear data</strong> using the <strong>kernel trick</strong>, which allows it to find complex boundaries in higher-dimensional spaces.</p><p>We have two type of SVM, hard Margin &amp; Soft Margin.</p><p>Hard Margin Support Vector Machines aim to find an <strong>optimal separating hyperplane</strong> that divides two classes while maximizing the margin between them. The decision boundary is defined as<br> <strong>W · X + b = 0</strong>, where <strong>W</strong> is the weight vector perpendicular to the hyperplane and <strong>b</strong> is the bias. The key objective is not just separation, but achieving the <strong>maximum possible distance</strong> from the closest data points of each class, known as support vectors.</p><p>To determine classification, SVM relies on <strong>vector projection</strong>. Any data point is projected onto the normal vector <strong>W</strong>, and the sign of <strong>W · X + b</strong> decides which side of the hyperplane the point lies on. For mathematical convenience and margin normalization, two parallel supporting hyperplanes are defined as<br> <strong>W · X + b = 1</strong> and <strong>W · X + b = −1</strong>. Using class labels <strong>yᵢ ∈ {+1, −1}</strong>, this leads to a single constraint for all correctly classified points:<br> <strong>yᵢ (W · Xᵢ + b) ≥ 1</strong>.</p><p>The distance between these two supporting hyperplanes represents the <strong>margin width</strong>, which is mathematically derived as, <strong>2 / ‖W‖</strong>. Maximizing the margin therefore becomes equivalent to <strong>minimizing ‖W‖</strong>, or more conveniently, minimizing</p><p>Hard Margin SVM assumes that the data is <strong>perfectly linearly separable</strong>. No points are allowed inside the margin or on the wrong side of the boundary. While this leads to a clean mathematical formulation, it is also a major limitation — if even a single outlier exists, the model fails to find a solution. This limitation motivates <strong>Soft Margin SVM</strong>, which relaxes these constraints using slack variables to handle real-world, noisy data.</p><p>Soft Margin Support Vector Machines extend <strong>Hard Margin SVM</strong> to handle <strong>real-world data</strong>, where perfect linear separation is rare. Hard Margin SVM strictly requires all data points to lie outside the margin, but even a single outlier can make this impossible or force the model to learn an extremely narrow margin that generalizes poorly.</p><p>To overcome this limitation, <strong>slack variables</strong> ζi are introduced for each data point. These variables measure how much a point violates the margin constraints. If ζi=0, the point is correctly classified and lies outside or exactly on the margin. If 0&lt;ζi≤1, the point is correctly classified but lies inside the margin. If ζi&gt;1, the point is misclassified and lies on the wrong side of the decision boundary.</p><p>With slack variables, the optimization objective now balances <strong>two competing goals</strong>: maximizing the margin and minimizing classification errors. This leads to the Soft Margin optimization problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*4eXUtTxvvY-Y3uT12qoLmg.png" /></figure><p>The hyperparameter <strong>C</strong> controls this trade-off. A <strong>high C</strong> heavily penalizes errors, pushing the model to classify training points correctly at the cost of a smaller margin — behaving similarly to Hard Margin SVM. A <strong>low C</strong> allows more violations, resulting in a wider margin that usually generalizes better on unseen data.</p><p>The loss function underlying Soft Margin SVM is <strong>hinge loss</strong>, which penalizes points that lie inside the margin or are misclassified. This formulation closely resembles regularized optimization in Logistic Regression, where C is inversely related to the regularization strength.</p><p>In essence, Soft Margin SVM makes SVM practical for noisy, overlapping datasets by allowing controlled violations while still preserving the maximum-margin principle.</p><p><strong>Kernel Trick</strong> enables Support Vector Machines (SVMs) to handle data that is <strong>not linearly separable</strong> in its original feature space. In many real-world datasets, classes overlap in complex patterns — such as one class forming a ring around another — making it impossible to separate them using a straight line or flat hyperplane.</p><p>The key idea behind the Kernel Trick is <strong>dimensionality transformation</strong>. If data cannot be separated in its current dimension, SVM maps it into a <strong>higher-dimensional space</strong> where separation becomes possible. For example, non-separable 1D data can be transformed into 2D, and non-separable 2D data into 3D. In this higher-dimensional space, a <strong>simple linear hyperplane</strong> can cleanly divide the classes.</p><p>The video illustrates this using a 1D example where points of one class lie in the center and the other class lies at the ends. In 1D, no single point can separate them. However, after applying a transformation (such as a quadratic or Gaussian function), the middle points are lifted into a second dimension. Once transformed, a straight line can easily separate the classes in 2D.</p><p>A <strong>kernel</strong> is the mathematical function that performs this transformation. Instead of explicitly computing new coordinates in higher dimensions — which would be computationally expensive — the Kernel Trick allows SVM to compute similarities between points <strong>as if</strong> they were in that higher-dimensional space. Common kernels include the <strong>Polynomial Kernel</strong> and the <strong>RBF (Gaussian) Kernel</strong>, which creates flexible, distance-based decision boundaries.</p><p>In essence, the Kernel Trick allows SVMs to model <strong>complex non-linear boundaries</strong> in the original space by solving a <strong>linear problem in a higher dimension</strong>, making SVM both powerful and efficient.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>You can buy me coffee at :- <a href="https://www.buymeacoffee.com/aryanbisht">https://www.buymeacoffee.com/aryanbisht</a></p><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 If you find this helpful Clap for the story and follow the author</li><li>🔔 Follow us: <a href="https://www.linkedin.com/in/spaa-club-50b302216/">LinkedIn</a>, <a href="https://twitter.com/bisht_aryan_ji"><em>Twitter</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f0ca52e075a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Complete Guide to KNN: From Intuition to Decision Boundaries]]></title>
            <link>https://medium.com/@AryanBeast/a-complete-guide-to-knn-from-intuition-to-decision-boundaries-ccc73a895e05?source=rss-1db4e17cc56d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccc73a895e05</guid>
            <category><![CDATA[knn-algorithm]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Aryan Bisht]]></dc:creator>
            <pubDate>Tue, 14 Apr 2026 04:01:01 GMT</pubDate>
            <atom:updated>2026-04-14T04:01:01.475Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to another post in my ongoing machine learning adventure. This blog is part of a series where I’m diving into the world of ML — learning, experimenting, and (hopefully) not getting completely lost in the algorithms! 😅 If you’re interested in learning along with me, check out the whole series <a href="https://docs.google.com/spreadsheets/d/1tyC8Wg1jMQnMdGMlhbfG29GuN1DRxggTNWpKlYCJsRU/edit?usp=sharing"><strong>HERE 📚</strong></a>. Or, if you’re just here for this particular topic, no worries — you’re in the right spot too..!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yvZY1yx4y_5_Y3iuxTXf9A.png" /></figure><p>Let’s start with KNN now. K-Nearest Neighbors (KNN) is a <strong>supervised learning algorithm</strong> used for both classification and regression. Unlike most machine learning algorithms, KNN does not learn parameters or build a model during training. Instead, it stores the entire training dataset and delays computation until a prediction is required. This makes KNN intuitive, flexible, and easy to understand — but also computationally expensive.</p><p>At its heart, KNN follows a simple idea: <strong>similar data points tend to belong to the same class</strong>. Similarity is measured using distance in the feature space. The intuition behind KNN is rooted in real life. If you want to predict something about a person, you often look at people most similar to them. KNN applies this idea mathematically. When a new data point appears, the algorithm looks around it in the feature space and asks: <em>what labels do nearby points have?</em></p><p>The assumption here is that points that are close together share similar properties. KNN therefore makes <strong>local decisions</strong>, not global ones. This locality is both a strength (captures fine details) and a weakness (sensitive to noise).</p><h3>Steps for KNN</h3><p><strong>Step 1: Choosing the Value of K</strong></p><p>The first explicit decision in KNN is selecting <strong>K</strong>, the number of neighbors to consider. This choice controls how local or global the algorithm’s behavior is.</p><p>If K is small, the prediction depends on very few nearby points. This makes the model sensitive to noise and outliers. A single mislabeled or noisy point can strongly influence the result. If K is large, the algorithm considers a broader neighborhood, smoothing out noise but potentially ignoring meaningful local patterns.</p><p>A common heuristic is choosing K ≈ √n, where n is the number of training samples. However, this is only a starting point. In practice, the best K is found using <strong>cross-validation</strong>, where multiple K values are tested and the one with the highest validation accuracy is selected. This balances bias and variance effectively.</p><p><strong>Step 2: Representing Data in Feature Space</strong></p><p>Before distance can be computed, each data point must be represented as a vector in feature space. For example, if a dataset has two features — say height and weight — each data point becomes a point in 2D space. With more features, the space becomes higher dimensional.</p><p>At this stage, <strong>feature scaling is critical</strong>. Since distance calculations are sensitive to magnitude, features with larger numeric ranges can dominate smaller ones. For example, if age ranges from 0–100 and salary ranges from 0–1,000,000, salary will overpower age unless scaling is applied. Standardization or normalization ensures that all features contribute fairly.</p><p><strong>Step 3: Computing Distances</strong></p><p>Once a query point arrives, KNN computes the distance between this point and <strong>every point in the training dataset</strong>. The most commonly used metric is <strong>Euclidean distance</strong>, which measures straight-line distance between points.</p><p>This step is computationally expensive because it requires calculating distances to all stored samples. For a dataset with n points and d dimensions, this step takes O(n·d) time per prediction. This is why KNN becomes slow for large datasets.</p><p>Different distance metrics (Manhattan, Minkowski, cosine similarity) can be used depending on the problem, but the underlying idea remains the same: quantify similarity numerically.</p><p><strong>Step 4: Finding the K Nearest Neighbors</strong></p><p>After computing distances, the algorithm sorts them in ascending order. The K data points with the smallest distances are selected as the <strong>nearest neighbors</strong>.</p><p>This step defines the neighborhood that will influence the prediction. These neighbors form a local region around the query point. The composition of this region — how many points from each class it contains — directly determines the final output.</p><p>Efficient implementations may use data structures like KD-trees or Ball trees to speed this step, but the conceptual process remains unchanged.</p><p><strong>Step 5: Making the Prediction (Voting or Averaging)</strong></p><p>For <strong>classification</strong>, KNN uses <strong>majority voting</strong>. Each of the K neighbors votes for its class label, and the class with the most votes is assigned to the query point. Some variants use <strong>distance-weighted voting</strong>, where closer neighbors have more influence than farther ones.</p><p>For <strong>regression</strong>, KNN computes the <strong>average (or weighted average)</strong> of the target values of the K neighbors. This makes predictions smooth and locally adaptive.</p><p>This prediction mechanism is simple, transparent, and directly tied to the data — there are no learned coefficients or parameters.</p><p>The choice of K strongly affects the <strong>decision boundary</strong>, which separates different predicted regions in the feature space. With K=1, the decision boundary becomes extremely irregular, tightly wrapping around training points. This results in overfitting, as the model memorizes noise.</p><p>As K increases, the decision boundary becomes smoother and more stable. However, if K becomes too large, the model loses sensitivity to local structure and may classify most points as the majority class. This leads to underfitting. Thus, K controls the bias-variance tradeoff in KNN.</p><p>KNN performs almost no computation during training — it simply stores the data. All the heavy computation happens during prediction. This is why it is referred to as a <strong>lazy learning algorithm</strong>, in contrast to eager learners like linear regression or decision trees that build a model upfront.</p><p>This laziness makes KNN flexible and easy to update with new data, but it also makes prediction slow and memory-intensive.</p><h3>Limitation of KNN</h3><ul><li><strong>Large Datasets:</strong> KNN is a “lazy learner,” meaning it does no work during training but performs heavy calculations during prediction, making it very slow for large data.</li><li><strong>Curse of Dimensionality:</strong> In very high-dimensional spaces, distance metrics become less reliable.</li><li><strong>Outliers:</strong> If K is too small, outliers can easily misguide the classification.</li><li><strong>Imbalanced Data:</strong> The model tends to favor the majority class</li><li><strong>Interpretability:</strong> KNN is a “black box” model; it can predict a class but cannot easily explain which features were most influential in that decision</li></ul><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>You can buy me coffee at :- <a href="https://www.buymeacoffee.com/aryanbisht">https://www.buymeacoffee.com/aryanbisht</a></p><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 If you find this helpful Clap for the story and follow the author</li><li>🔔 Follow us: <a href="https://www.linkedin.com/in/spaa-club-50b302216/">LinkedIn</a>, <a href="https://twitter.com/bisht_aryan_ji"><em>Twitter</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccc73a895e05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hierarchical Clustering Explained : Agglomerative & Divisive]]></title>
            <link>https://medium.com/@AryanBeast/hierarchical-clustering-explained-agglomerative-divisive-75b8b2831a6b?source=rss-1db4e17cc56d------2</link>
            <guid isPermaLink="false">https://medium.com/p/75b8b2831a6b</guid>
            <category><![CDATA[k-means-clustering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[divisive-clustering]]></category>
            <category><![CDATA[agglomerative-clustering]]></category>
            <category><![CDATA[hierarchical-clustering]]></category>
            <dc:creator><![CDATA[Aryan Bisht]]></dc:creator>
            <pubDate>Fri, 10 Apr 2026 04:01:03 GMT</pubDate>
            <atom:updated>2026-04-10T04:01:03.152Z</atom:updated>
            <content:encoded><![CDATA[<h3>Hierarchical Clustering Explained : Agglomerative &amp; <strong>Divisive</strong></h3><p>Welcome to another post in my ongoing machine learning adventure. This blog is part of a series where I’m diving into the world of ML — learning, experimenting, and (hopefully) not getting completely lost in the algorithms! 😅 If you’re interested in learning along with me, check out the whole series <a href="https://docs.google.com/spreadsheets/d/1tyC8Wg1jMQnMdGMlhbfG29GuN1DRxggTNWpKlYCJsRU/edit?usp=sharing"><strong>HERE 📚</strong></a>. Or, if you’re just here for this particular topic, no worries — you’re in the right spot too..!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ygtd81shVluLuO_UA74gPw.png" /></figure><p>Let’s start with some new type of learning algorithm, Hierarchical Clustering. <strong>Hierarchical Clustering </strong>is an <strong>unsupervised learning algorithm</strong> that groups data points based on similarity, but unlike K-Means, it does <strong>not</strong> require you to decide the number of clusters beforehand. Instead, it builds a <strong>hierarchy of clusters</strong>, giving you a complete picture of how data points relate to each other at different levels of granularity.</p><p>To understand why Hierarchical Clustering is powerful, we first need to clearly understand the <strong>limitations of K-Means</strong>.</p><h3>Why K-Means Is Not Always Enough</h3><p><strong>a) Geometric Constraints of K-Means</strong></p><p>K-Means works by computing distances between data points and <strong>cluster centroids</strong>. Each cluster is represented by a centroid (mean of points), and points are assigned based on the shortest distance to these centroids.</p><p>This approach assumes:</p><ul><li>Clusters are <strong>convex</strong></li><li>Clusters are <strong>spherical</strong></li><li>Clusters are roughly <strong>equal in size</strong></li></ul><p>These assumptions hold for simple datasets but completely break down for <strong>complex shapes</strong>. For example:</p><ul><li>Concentric circles</li><li>Moon-shaped or crescent-shaped clusters</li><li>Elongated or curved clusters</li></ul><p>In such cases, the centroid is not a good representation of the cluster’s structure, leading to incorrect grouping.</p><p><strong>b) Sensitivity to Noise and Outliers</strong></p><p>K-Means <strong>forces every data point into a cluster</strong>, even if a point is very far from all others. This means:</p><ul><li>Outliers can distort centroids</li><li>Cluster boundaries become unstable</li><li>Noise points get treated as meaningful data</li></ul><p>Hierarchical Clustering, especially with certain linkage methods, is <strong>less sensitive</strong> to this problem and provides better structural insight into the data.</p><p>Hierarchical Clustering creates clusters in a <strong>tree-like structure</strong>, known as a <strong>hierarchy</strong>. Instead of producing a single flat partition, it records:</p><ul><li>Which points merged</li><li>At what distance they merged</li><li>In what order the merges occurred</li></ul><p>This hierarchy allows you to explore clustering at <strong>multiple resolutions</strong>.</p><h3>Types of Hierarchical Clustering</h3><p><strong>a) Agglomerative Clustering (Bottom-Up)</strong></p><p>In agglomerative hierarchical clustering, the algorithm begins with each data point treated as an individual cluster. At every step, the two most similar clusters are merged together. As this merging process continues, small clusters gradually combine to form larger ones until all data points belong to a single cluster. This bottom-up approach is widely used because it is conceptually simple and relatively efficient compared to alternatives.</p><p><strong>b) Divisive Clustering (Top-Down)</strong></p><p>Divisive clustering takes the opposite approach. It starts with all data points grouped into a single cluster and repeatedly splits this cluster into smaller ones. While this top-down strategy is intuitive, it is computationally expensive and requires complex decisions at each split, which is why it is rarely used in practical applications.</p><h3>Core Algorithm: Step-by-Step Breakdown</h3><p><strong>Step 1: Compute the Proximity Matrix</strong><br> The algorithm begins by calculating a proximity (or distance) matrix that contains the distances between every pair of data points in the dataset. This matrix has a size of n×n, where n is the number of data points. Each entry represents how similar or dissimilar two points are, based on a chosen distance metric such as Euclidean distance. This matrix forms the foundation of hierarchical clustering, as all future merging decisions are based on these distance values.</p><p><strong>Step 2: Treat Each Data Point as an Individual Cluster</strong><br> At the start of the clustering process, every data point is considered its own cluster. At this stage, there is no grouping or structure — only isolated points. This represents the finest possible level of clustering, where no assumptions about relationships between points have yet been made.</p><p><strong>Step 3: Identify the Closest Pair of Clusters</strong><br> Using the proximity matrix, the algorithm searches for the two clusters that are closest to each other. Since clusters initially contain only one point, this step first finds the closest pair of points. As clustering progresses, “closeness” is determined based on the selected linkage method, which defines how distances between clusters are calculated.</p><p><strong>Step 4: Merge the Closest Clusters</strong><br> Once the closest pair of clusters is identified, they are merged into a single new cluster. This merge is permanent and represents a key characteristic of hierarchical clustering: once two clusters are joined, the algorithm never splits them apart or reconsiders earlier decisions.</p><p><strong>Step 5: Update the Proximity Matrix</strong><br> After merging two clusters, the proximity matrix must be updated. The distances between the newly formed cluster and all remaining clusters are recalculated according to the chosen linkage strategy. This update ensures that future merge decisions correctly reflect the current cluster structure.</p><p><strong>Step 6: Repeat the Merge-and-Update Process</strong><br> The algorithm repeatedly identifies the closest clusters, merges them, and updates the proximity matrix. With each iteration, the total number of clusters decreases. This process continues step by step, gradually building larger and larger clusters.</p><p><strong>Step 7: Continue Until a Single Cluster Remains</strong><br> The algorithm terminates when all data points have been merged into one single cluster. At this point, the complete hierarchical structure has been formed. Although the algorithm technically ends here, meaningful clusters are obtained by cutting this hierarchy at an appropriate level using a dendrogram.</p><h3>The Role of Linkage in Shaping Clusters</h3><p>The notion of “closeness” between clusters is defined by the linkage method, and this choice has a profound impact on the final clustering structure.</p><p><strong>Single linkage </strong>measures the distance between the closest points in two clusters. This allows clusters to grow by forming chains of nearby points, which can capture complex shapes but also makes the algorithm sensitive to noise.</p><p><strong>Complete linkage</strong>, in contrast, considers the distance between the farthest points in two clusters. This produces tighter and more compact clusters, as merges only occur when all points are relatively close to each other. While this reduces sensitivity to outliers, it can sometimes split natural clusters into smaller pieces.</p><p><strong>Average linkage</strong> strikes a balance by computing the average distance between all pairs of points across two clusters. This approach tends to be more stable and less extreme than single or complete linkage, making it a good general-purpose choice.</p><p><strong>Ward’s method</strong> takes a different perspective by focusing on variance. Instead of directly measuring distances, it merges clusters in a way that minimizes the increase in within-cluster variance. This often results in well-balanced, compact clusters and is the default choice in many machine learning libraries.</p><h3>Understanding Dendrograms and Cluster Selection</h3><p>The entire merging process of hierarchical clustering is visualized using a dendrogram. This tree-like diagram records each merge along with the distance at which it occurred. The height of a merge in the dendrogram represents how dissimilar the clusters were when they were combined.</p><p>Choosing the number of clusters becomes a visual task rather than a guessing game. By identifying the largest vertical gap in the dendrogram and drawing a horizontal cut across it, one can determine a natural number of clusters. This makes hierarchical clustering especially appealing when the optimal number of clusters is unknown.</p><h3>Strengths and Limitations</h3><p>Hierarchical clustering is highly flexible and does not impose strict geometric assumptions on the data. It provides a complete and interpretable view of cluster relationships, which is invaluable for understanding complex datasets. However, this power comes at a cost. The need to store and update an n×n distance matrix leads to high memory usage, and the repeated merging process results in high computational complexity. As a result, hierarchical clustering is not suitable for very large datasets.</p><p>Hierarchical Clustering excels as an exploratory technique that reveals the hidden structure of data. While it may not scale to millions of points, its interpretability and flexibility make it a strong choice when understanding relationships matters more than raw performance. In scenarios where K-Means oversimplifies the data, hierarchical clustering often provides the clarity that simpler algorithms cannot.</p><h3>Hyperparameter</h3><p>The n_clusters parameter defines how many clusters you want in the final output. The algorithm first builds the hierarchical tree and then cuts it at a level that produces the specified number of clusters. This is useful when you already know how many groups you expect in the data.</p><p>The metric parameter controls how distances between data points are calculated. Euclidean distance is the default and works well for most numerical data, but other metrics can be used depending on the nature of the features. When using Ward linkage, the metric must be Euclidean because Ward’s method is based on variance minimization. Can be “euclidean”, “l1”, “l2”, “manhattan”, “cosine”, or “precomputed”</p><p>The linkage parameter determines how distances between clusters are computed during merging. Single linkage focuses on the closest points, complete linkage uses the farthest points, average linkage takes the mean distance, and Ward linkage minimizes the increase in within-cluster variance. This choice strongly influences the shape and compactness of the resulting clusters. <strong><em>{‘ward’, ‘complete’, ‘average’, ‘single’}, default=’ward’</em></strong></p><p>The distance_threshold parameter provides an alternative way to stop clustering. Instead of fixing the number of clusters, the algorithm stops merging when clusters are farther apart than a specified distance. When this parameter is used, n_clusters must be set to None.</p><p>The connectivity parameter restricts which points can be merged by defining neighborhood relationships. This is especially useful for spatial or image data, as it enforces local structure and can also improve computational efficiency. <strong><em>array-like, sparse matrix, or callable, default=None.</em></strong></p><p>This is all about hierarchical clustering from my side.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>You can buy me coffee at :- <a href="https://www.buymeacoffee.com/aryanbisht">https://www.buymeacoffee.com/aryanbisht</a></p><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 If you find this helpful Clap for the story and follow the author</li><li>🔔 Follow us: <a href="https://www.linkedin.com/in/spaa-club-50b302216/">LinkedIn</a>, <a href="https://twitter.com/bisht_aryan_ji"><em>Twitter</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75b8b2831a6b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced Ensemble Learning: Stacking and Blending]]></title>
            <link>https://medium.com/@AryanBeast/advanced-ensemble-learning-stacking-and-blending-18e69c0eb8bd?source=rss-1db4e17cc56d------2</link>
            <guid isPermaLink="false">https://medium.com/p/18e69c0eb8bd</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[stacking]]></category>
            <category><![CDATA[blending]]></category>
            <category><![CDATA[ensemble-learning]]></category>
            <dc:creator><![CDATA[Aryan Bisht]]></dc:creator>
            <pubDate>Sun, 05 Apr 2026 04:01:03 GMT</pubDate>
            <atom:updated>2026-04-05T04:01:03.214Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to another post in my ongoing machine learning adventure. This blog is part of a series where I’m diving into the world of ML — learning, experimenting, and (hopefully) not getting completely lost in the algorithms! 😅 If you’re interested in learning along with me, check out the whole series <a href="https://docs.google.com/spreadsheets/d/1tyC8Wg1jMQnMdGMlhbfG29GuN1DRxggTNWpKlYCJsRU/edit?usp=sharing"><strong>HERE 📚</strong></a>. Or, if you’re just here for this particular topic, no worries — you’re in the right spot too..!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*891Yr4P1DhdhQtECJ_nEdA.png" /></figure><p>We will studying one more ensemble technique called stacking. Stacking can be seen as an advanced version of Voting Ensembles. Instead of simply taking a majority vote (classification) or average (regression), stacking <strong>learns how to combine models</strong>. Multiple <strong>base models</strong> are first trained on the original dataset. Their predictions are then used as <strong>input features</strong> to train another model called the <strong>Meta Model</strong>, which produces the final prediction. This allows the ensemble to learn which base models to trust more in different situations.</p><p>The <strong>base models</strong> are usually diverse algorithms such as Linear Regression, Decision Trees, KNN, or SVM. Each of these models captures different patterns in the data. The <strong>Meta Model</strong> (often Logistic Regression or Random Forest) learns from the predictions of these base models and figures out the best way to combine them. Instead of manually assigning weights, the Meta Model automatically learns the optimal combination.</p><p>If base models are trained on the full dataset and their predictions are used directly to train the Meta Model, it can lead to <strong>overfitting</strong>. This happens because the Meta Model would see overly optimistic predictions. To solve this, the video introduces two safe training strategies: <strong>Blending</strong> and <strong>Stacking (K-Fold)</strong>.</p><h3>Blending Ensemble (Hold-Out Method)</h3><p>Blending is one of the simplest ways to combine multiple models using a Meta Model while avoiding overfitting. The process begins by splitting the available dataset into two distinct parts. The first part is used exclusively to train all the base models, while the second part is kept aside as a validation set. This separation is crucial because the Meta Model must only learn from predictions generated on data that the base models have never seen during training.</p><p>Once the data is split, several diverse base models such as Linear Regression, Decision Trees, or KNN are trained on the first subset of the data. Each base model independently learns its own representation of the problem. After training, these models are used to generate predictions on the validation dataset that was kept aside earlier. These predictions are not final outputs; instead, they act as new features, often referred to as meta-features.</p><p>In the next step, the Meta Model is trained using these meta-features as input and the true labels of the validation dataset as the target. The Meta Model learns how to optimally combine the predictions coming from different base models. Because the base models never predicted on the data they were trained on, the Meta Model receives unbiased inputs, which helps reduce overfitting. However, since a portion of the dataset is reserved only for meta-training, blending does not fully utilize the available data, making it less effective when the dataset is small.</p><h3>Stacking Ensemble (K-Fold Method)</h3><p>Stacking improves upon blending by ensuring that all available data is used efficiently while still preventing data leakage. Instead of a single hold-out split, stacking relies on K-Fold cross-validation to generate safe predictions for meta-training. The dataset is first divided into K equal folds. Each fold gets an opportunity to act as a validation set while the remaining folds are used for training.</p><p>For each fold, the base models are trained on K-1 folds and then used to predict on the remaining fold. This process is repeated K times so that every data point in the dataset receives exactly one prediction from a model that was not trained on it. These predictions are known as out-of-fold predictions and serve as reliable meta-features for training the Meta Model.</p><p>After completing the K-Fold process, the Meta Model is trained on the entire dataset using the out-of-fold predictions as input and the true labels as output. Since every prediction used for meta-training is generated without data leakage, the Meta Model learns a robust and generalized combination strategy. Although stacking is computationally more expensive because each base model is trained multiple times, it typically produces more stable and accurate results than blending, especially on small and medium-sized datasets.</p><h3>How Stacking Differs from Bagging and Boosting</h3><p>Bagging, Boosting, and Stacking are all ensemble techniques, but they differ in how models are trained and combined. <strong>Bagging</strong> trains multiple models of the same type independently on different bootstrap samples of the dataset and combines their predictions using simple averaging or majority voting. Its main goal is to reduce variance, and it does not try to learn which model is better than others.</p><p><strong>Boosting</strong> also uses models of the same type, but trains them sequentially. Each new model focuses more on the data points that previous models predicted incorrectly. The final prediction is obtained through a weighted combination of all models, where the weights are determined by training errors. Boosting mainly reduces bias and improves performance on difficult patterns.</p><p><strong>Stacking</strong> takes a different approach by learning how to combine models instead of using fixed rules. It allows different types of base models and uses their predictions as input features for a Meta Model, which learns the optimal way to combine them. Unlike bagging and boosting, stacking does not rely on averaging or predefined weights, making it more flexible and powerful.</p><h3>Advanced Multi-Layer Stacking</h3><p>Multi-layer stacking extends standard stacking by using <strong>more than one level of Meta Models</strong>. In this approach, predictions generated by base models in the first layer are used as input features for models in the next layer, instead of going directly to a single final Meta Model. Each layer learns how to combine the outputs of the previous layer, gradually improving the overall prediction.</p><p>The intuition is that higher layers can learn relationships <strong>between models</strong>, not just between features. For example, one layer may learn when tree-based models perform better, while another learns when linear models are more reliable. To prevent overfitting, cross-validation is used at every layer so that predictions are always made on unseen data.</p><p>Although multi-layer stacking can boost performance, it is computationally expensive and requires careful design. As a result, it is mainly used in competitive machine learning scenarios like Kaggle, where small accuracy improvements are crucial.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>You can buy me coffee at :- <a href="https://www.buymeacoffee.com/aryanbisht">https://www.buymeacoffee.com/aryanbisht</a></p><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 If you find this helpful Clap for the story and follow the author</li><li>🔔 Follow us: <a href="https://www.linkedin.com/in/spaa-club-50b302216/">LinkedIn</a>, <a href="https://twitter.com/bisht_aryan_ji"><em>Twitter</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18e69c0eb8bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gradient Boosting Made Intuitive: Regression and Classification Explained]]></title>
            <link>https://medium.com/@AryanBeast/gradient-boosting-made-intuitive-regression-and-classification-explained-8e08f44178c3?source=rss-1db4e17cc56d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e08f44178c3</guid>
            <category><![CDATA[gradient-boosting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ensemble-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Aryan Bisht]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 04:01:01 GMT</pubDate>
            <atom:updated>2026-03-31T04:01:01.929Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to another post in my ongoing machine learning adventure. This blog is part of a series where I’m diving into the world of ML — learning, experimenting, and (hopefully) not getting completely lost in the algorithms! 😅 If you’re interested in learning along with me, check out the whole series <a href="https://docs.google.com/spreadsheets/d/1tyC8Wg1jMQnMdGMlhbfG29GuN1DRxggTNWpKlYCJsRU/edit?usp=sharing"><strong>HERE 📚</strong></a>. Or, if you’re just here for this particular topic, no worries — you’re in the right spot too..!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*edv_sG4xH5h9KBOyGduUuw.png" /></figure><p>So we are starting Gradient Boosting now. Gradient Boosting is an ensemble learning technique that builds a strong regression model by <strong>adding weak models one after another</strong>. Unlike methods where models are trained independently, Gradient Boosting follows a strictly sequential approach — each new model is trained to fix the mistakes made by the current ensemble. Gradient Boosting is based on <strong>additive modeling</strong>, where a complex relationship between input features X and the target y is approximated by <strong>adding multiple simple models together</strong>. Instead of learning the entire function at once, the algorithm builds it gradually. Each new model is added to the existing ensemble with the goal of improving its accuracy by correcting previous errors. Mathematically, the final model is expressed as a sum of simpler functions, typically decision trees.</p><p>To guide this learning process, Gradient Boosting relies on a <strong>loss function</strong>, which quantifies how far the model’s predictions are from the true values. For regression problems, the most commonly used loss is the <strong>Least Squares loss</strong>, defined as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/333/1*kDbiRyv17NEaOHEdH3Hg5w.png" /></figure><p>The factor 1/2​ is included purely for mathematical convenience, as it simplifies derivatives during optimization without affecting the solution.</p><h3>Steps for finding Gradient Boosting</h3><h3>Step 1: Initialization</h3><p>The algorithm begins by finding a <strong>constant prediction</strong> that minimizes the loss function over the entire training dataset. When using the Least Squares loss, this optimal constant turns out to be the <strong>mean of the target variable</strong> y. This initial prediction is denoted as F0(x). At this stage, the model ignores all input features and provides the same output for every data point.</p><h3>Step 2: Iterative Boosting (for m=1 to M)</h3><p>Gradient Boosting then improves the model through a sequence of iterations, where each iteration adds one new decision tree.</p><p>a. First, the algorithm computes the <strong>pseudo-residuals</strong> for every training example. These residuals are defined as the <strong>negative gradient of the loss function</strong> with respect to the current model’s predictions. For Least Squares loss, this simplifies to a very intuitive quantity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*etkhDjqgCpaUVk1lVGsVYg.png" /></figure><p>These residuals represent exactly what the current model is failing to predict.</p><p>b. Next, a <strong>regression decision tree</strong> is trained to predict these residuals using the input features. The tree partitions the feature space into several terminal regions (leaf nodes), each representing a subset of the data where the errors behave similarly.</p><p>d. Once the tree structure is fixed, the algorithm computes an optimal <strong>leaf value</strong> for each terminal node. This value, often denoted as γ, is chosen to minimize the loss function for the samples that fall into that leaf. Under Least Squares loss, this optimal value is simply the <strong>mean of the residuals</strong> within that leaf.</p><p>e. After computing these leaf values, the model is updated by <strong>adding the new tree’s predictions</strong> (optionally scaled by a learning rate) to the existing model. This produces the updated function Fm(x), which is now a better approximation of the true target.</p><h3>Step 3: Final Model</h3><p>After repeating this process for M iterations, the final Gradient Boosting model is obtained. It consists of the initial mean prediction plus the contributions of all the decision trees added sequentially. Each tree corrects the mistakes of the previous ensemble, resulting in a powerful and flexible predictive model.</p><h3>Key Differences: AdaBoost vs. Gradient Boosting</h3><p><strong>1. Tree Depth: Stumps vs. Deeper Trees</strong></p><p>AdaBoost typically uses <strong>decision stumps</strong>, which are trees with a depth of just one. Each stump makes a decision using only a single feature and a single split, meaning it is extremely weak on its own. AdaBoost compensates for this weakness by combining many such stumps, where each stump focuses on correcting mistakes made by the previous ones. The simplicity of stumps ensures that AdaBoost improves mainly by <strong>reweighting difficult data points</strong>, rather than by increasing model complexity.</p><p>Gradient Boosting, on the other hand, usually employs <strong>deeper trees</strong>, often with depths ranging from 8 to 32. These trees are much more expressive and can capture feature interactions and non-linear patterns within a single tree. Instead of relying on many extremely simple models, Gradient Boosting allows each tree to learn richer structures, but carefully controls their influence through gradual updates.</p><p><strong>2. Weights vs. Learning Rate: How Corrections Are Controlled</strong></p><p>In AdaBoost, every weak learner is assigned a <strong>model-specific weight</strong> based on how well it performs. If a stump correctly classifies many samples, it receives a higher weight and has a stronger influence on the final prediction. Poor-performing models receive lower weights. This dynamic weighting mechanism allows AdaBoost to emphasize stronger learners while still keeping weaker ones in the ensemble.</p><p>In Gradient Boosting, all trees are treated more uniformly. Instead of assigning different weights to different trees, Gradient Boosting uses a <strong>constant learning rate</strong> that scales the contribution of every new tree equally. This learning rate acts as a global control knob, ensuring that each tree makes only a small correction to the existing model. As a result, learning becomes more stable and less prone to overfitting.</p><h3>Gradient Boosting for Classification</h3><p>Gradient Boosting for classification follows the principle of <strong>additive modeling</strong>, where a strong classifier is built by adding multiple weak learners sequentially. Each learner is usually a <strong>shallow decision tree</strong>, and instead of learning the full decision boundary at once, the model improves gradually. Every new tree is trained to correct the errors made by the combined model so far. By stacking many high-bias models in a controlled way, Gradient Boosting achieves low bias while keeping variance in check.</p><p><strong>Step 1: Initial Model Using Log-Odds</strong></p><p>Unlike regression, classification uses <strong>Log Loss</strong> as the loss function. The algorithm starts with a constant model that predicts the <strong>log-odds</strong> of the positive class. This value is computed as the logarithm of the ratio of positive to negative samples in the dataset. This initial log-odds value is then passed through the <strong>logistic (sigmoid) function</strong> to obtain an initial probability estimate for every data point.</p><p><strong>Step 2: Compute Pseudo-Residuals</strong></p><p>Next, the model measures how wrong it is by computing <strong>pseudo-residuals</strong>. For binary classification, these residuals are simply the difference between the actual label and the predicted probability, written as y−p^​. These values indicate both the direction and magnitude of error and represent what the current model has failed to learn.</p><p><strong>Step 3: Train a Tree on Residuals</strong></p><p>A <strong>regression tree</strong> is then trained using the original input features, but with the residuals as the target. Even though this is a classification problem, regression trees are used because residuals are continuous values. The tree divides the data into leaf nodes where errors behave similarly.</p><p><strong>Step 4: Compute Leaf Values (γ)</strong></p><p>A crucial difference from regression appears at this stage. Since predictions are made in <strong>log-odds space</strong>, residuals cannot be added directly. For each leaf, a special value γ is computed using a formula that accounts for the predicted probabilities and their variance. This step ensures that updates are mathematically compatible with Log Loss.</p><p><strong>Step 5: Update the Model</strong></p><p>The model is updated by adding the scaled leaf values to the previous log-odds prediction. A <strong>learning rate</strong> is applied to control how much influence each tree has. This updated log-odds value is then converted back into a probability using the logistic function. The same process is repeated for multiple iterations, with each stage refining the classifier further.</p><p>Key difference between classification &amp; regression : While Gradient Boosting for regression directly adds residual predictions, <strong>classification requires transforming leaf outputs to work in log-odds space</strong>, even though the underlying trees are still regression trees.</p><p>If you want to learn in further detail you can check in this article <a href="https://towardsdatascience.com/all-you-need-to-know-about-gradient-boosting-algorithm-part-2-classification-d3ed8f56541e/"><em>LINK</em></a><em>.</em></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>You can buy me coffee at :- <a href="https://www.buymeacoffee.com/aryanbisht">https://www.buymeacoffee.com/aryanbisht</a></p><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 If you find this helpful Clap for the story and follow the author</li><li>🔔 Follow us: <a href="https://www.linkedin.com/in/spaa-club-50b302216/">LinkedIn</a>, <a href="https://twitter.com/bisht_aryan_ji"><em>Twitter</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e08f44178c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding K-Means Clustering Through Geometry]]></title>
            <link>https://medium.com/@AryanBeast/understanding-k-means-clustering-through-geometry-566857571150?source=rss-1db4e17cc56d------2</link>
            <guid isPermaLink="false">https://medium.com/p/566857571150</guid>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[k-means-clustering]]></category>
            <dc:creator><![CDATA[Aryan Bisht]]></dc:creator>
            <pubDate>Thu, 26 Mar 2026 04:01:00 GMT</pubDate>
            <atom:updated>2026-03-26T04:01:00.687Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to another post in my ongoing machine learning adventure. This blog is part of a series where I’m diving into the world of ML — learning, experimenting, and (hopefully) not getting completely lost in the algorithms! 😅 If you’re interested in learning along with me, check out the whole series <a href="https://docs.google.com/spreadsheets/d/1tyC8Wg1jMQnMdGMlhbfG29GuN1DRxggTNWpKlYCJsRU/edit?usp=sharing"><strong>HERE 📚</strong></a>. Or, if you’re just here for this particular topic, no worries — you’re in the right spot too..!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*CaAHd5Pz5u044ZfisA59EQ.png" /></figure><p>So know we will learn about k-means clustering. K-Means is one of the most popular <strong>unsupervised learning algorithms</strong>, designed to discover hidden patterns in <strong>unlabeled data</strong>. Unlike supervised learning, there is no correct answer provided beforehand. Instead, the algorithm groups data points into clusters purely based on similarity, which is typically measured using distance. This makes K-Means especially useful in real-world scenarios such as customer segmentation, image grouping, and educational or behavioral data analysis.</p><p>Steps-by-Step Working :</p><p><strong>Step 1: Choosing the number of clusters (K).</strong><br> The algorithm begins by selecting a value for <strong>K</strong>, which represents the number of clusters to be formed. Since K-Means has no prior knowledge of how many natural groups exist in the data, this value must be provided by the user.</p><p><strong>Step 2: Initializing centroids.</strong><br> Next, K data points are randomly chosen as <strong>initial centroids</strong>. These centroids act as temporary centers of the clusters. Because this step is random, different initializations can lead to slightly different clustering results.</p><p><strong>Step 3: Assigning data points to clusters.</strong><br> Each data point is then assigned to the nearest centroid using <strong>Euclidean distance</strong>. Geometrically, this divides the feature space into regions, where every point belongs to the cluster of the closest centroid.</p><p><strong>Step 4: Updating the centroids.</strong><br> After the assignment step, the centroid of each cluster is recalculated by taking the <strong>mean of all points assigned to that cluster</strong>. The centroid moves toward the region where data points are most dense, improving cluster quality.</p><p><strong>Step 5: Repeating until convergence.</strong><br> The assignment and update steps are repeated multiple times. This process continues until the centroids stop moving significantly or a maximum number of iterations is reached. At this point, the clusters become stable, and the algorithm converges.</p><p><strong>Finding the optimal value of K (Elbow Method)</strong><br>To choose a good K, the <strong>Elbow Method</strong> is commonly used. It relies on <strong>Within-Cluster Sum of Squares (WCSS)</strong>, which measures how compact the clusters are. It is distance between each point in a cluster and its centroid. As K increases, WCSS decreases, but after a certain point the improvement slows down. The point where this change becomes less significant forms an “elbow” in the graph and is chosen as the optimal K.</p><p>The <strong>Elbow Method</strong> is a commonly used technique to determine the <strong>optimal number of clusters (K)</strong> in K-Means. Since K-Means requires K to be specified beforehand, the elbow method helps us make a data-driven decision instead of guessing.</p><p><strong>Step 1: Run K-Means for different values of K.</strong><br> First, the K-Means algorithm is executed multiple times using different values of K, usually starting from K = 1 and increasing gradually (for example, up to 10 or 15). For each value of K, the clustering result is recorded.</p><p><strong>Step 2: Compute WCSS for each K.</strong><br> For every run, we calculate <strong>WCSS (Within-Cluster Sum of Squares)</strong>. WCSS measures how close data points are to their respective cluster centroids. Smaller WCSS values indicate tighter and more compact clusters.</p><p><strong>Step 3: Plot the Elbow Curve.</strong><br> Next, a graph is plotted with the <strong>number of clusters (K)</strong> on the x-axis and <strong>WCSS</strong> on the y-axis. As K increases, WCSS always decreases because adding more clusters reduces the distance between points and centroids.</p><p><strong>Step 4: Observe the rate of improvement.</strong><br> Initially, increasing K leads to a <strong>sharp drop in WCSS</strong>, meaning clustering quality improves significantly. However, after a certain value of K, the reduction in WCSS becomes much smaller and more gradual.</p><p><strong>Step 5: Identify the “elbow point.”</strong><br> The point where this sharp decrease starts to slow down forms a visible <strong>bend or “elbow”</strong> in the curve. This elbow represents a balance between cluster compactness and model simplicity and is usually chosen as the <strong>optimal value of K</strong>.</p><p>Implementation:</p><pre>from sklearn.cluster import KMeans<br>import numpy as np<br><br><br># Create K-Means model<br>kmeans = KMeans(n_clusters=2, random_state=42)<br><br># Fit the model<br>kmeans.fit(X)<br><br># Cluster labels for each point<br>labels = kmeans.labels_<br><br># Centroid coordinates<br>centroids = kmeans.cluster_centers_<br><br>print(&quot;Cluster Labels:&quot;, labels)<br>print(&quot;Centroids:\n&quot;, centroids)</pre><p>This was all about K-means clustering algorithm. It’s simple algorithm which can be implemented easily.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>You can buy me coffee at :- <a href="https://www.buymeacoffee.com/aryanbisht">https://www.buymeacoffee.com/aryanbisht</a></p><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 If you find this helpful Clap for the story and follow the author</li><li>🔔 Follow us: <a href="https://www.linkedin.com/in/spaa-club-50b302216/">LinkedIn</a>, <a href="https://twitter.com/bisht_aryan_ji"><em>Twitter</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=566857571150" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>