<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohith on Medium]]></title>
        <description><![CDATA[Stories by Mohith on Medium]]></description>
        <link>https://medium.com/@mohith-g?source=rss-6575ac4820f4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5usig-bkFlGzO3j3AdWZrw.jpeg</url>
            <title>Stories by Mohith on Medium</title>
            <link>https://medium.com/@mohith-g?source=rss-6575ac4820f4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 05:01:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mohith-g/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stuff I Learned While Building Models — The Notes I Wish I Had Earlier]]></title>
            <link>https://mohith-g.medium.com/stuff-i-learned-while-building-models-the-notes-i-wish-i-had-earlier-982156d69168?source=rss-6575ac4820f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/982156d69168</guid>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Mohith]]></dc:creator>
            <pubDate>Thu, 01 Jan 2026 13:49:26 GMT</pubDate>
            <atom:updated>2026-01-01T13:49:26.025Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stuff I Learned While Building Models — The Notes I Wish I Had Earlier</h3><p>When I try to implement ideas from my readings, sometimes my models don’t learn, and sometimes they learn but inefficiently. Over time I’ve noticed a few common mistakes I made, figured out why they happen, and picked up small changes that make training work better. This is just a list of things I wish I had known earlier.</p><ol><li><strong>Never use clamping when there is a gradient flow</strong></li></ol><p>When I wanted to find out whether I can model the convolutional network’s output based on how uncertain the model is on each feature in feature-space before sending it to a classifier. I used clamp inside the training loop, this ended up preventing gradients from flowing through parts of the uncertainty head, effectively stopping those parameters from receiving meaningful learning signals.</p><p>2. <strong>Batch Normalization is non-negotiable, it makes the theory work</strong></p><p>Using non-linear activation functions does gives us the advantages of non-linearity but how shallow our gradients are depends on how saturated our activation is. I don’t wanna say Using BatchNorm ‘avoids vanishing gradients problem’ — but — limiting it because vanishing gradients became habitual as we developed deeper networks. For example,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/215/1*RuELxo6ZsAINtz70E9H5Mw.png" /></figure><p>For a two layer network,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*lwCDNi2KbA6NNT82VPQDlA.png" /></figure><p>Since,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*NQa7g9ziAWHS3CLYhpgifw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/1*uxsh6fbi-YRGTvy-6CrYFA.png" /></figure><p>For k layers,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/122/1*vIp-yodASAVuTX9PutiH9w.png" /></figure><p><strong>The pre-activation(<em>z)</em> contribution to vanishing gradients:</strong></p><p>The absolute scale of the <em>z</em> values doesn’t really matter — what matters is the signal they carry. So instead of letting the network deal with wildly scaled activations, BatchNorm explicitly models that scale. It standardizes the <em>z</em> values and then starts with γ=1 (std) and β=0 (mean), so initially the layer is seeing the same distribution as before — just stabilized.</p><p>This keeps both the <strong>inputs flowing into the layer</strong> and the <strong>weights inside the layer</strong> in a moderate range. Because if either the weights or the incoming activations explode, the <em>z</em> values explode too — and once <em>z</em> becomes large, the gradients don’t flow cleanly anymore. That’s when you start seeing vanishing or unstable updates. So BatchNorm isn’t about changing what the layer learns — it’s about keeping the signal clean enough that the layer can actually learn.</p><p><strong>The activation (a or x) contribution to unstable gradients:</strong></p><p>The absolute value of the activation itself doesn’t carry meaning — what matters is how it <strong>scales the learning signal</strong>. In back-prop,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/215/1*RuELxo6ZsAINtz70E9H5Mw.png" /></figure><p>So the activation isn’t just data flowing forward — it directly multiplies the gradient flowing backward. When the input coming into a layer is large, this term blows up the weight gradients even if the loss signal is perfectly reasonable. That’s how a harmless-looking activation can destabilize learning: the model isn’t wrong, the signal is just being amplified at every update. So controlling the scale of activations isn’t about changing representation — it’s about preventing the learning signal from exploding before it ever reaches the weights that actually need to adapt.</p><p><strong>The weights(w) contribution to both vanishing and exploding gradients:</strong></p><p>If the weights become large, the back-prop terms like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/1*uxsh6fbi-YRGTvy-6CrYFA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/122/1*vIp-yodASAVuTX9PutiH9w.png" /></figure><p>start to <strong>explode</strong>. And if the weights are too small, the same chain keeps shrinking — that’s <strong>vanishing gradient</strong>.</p><p>So the deeper you go, the smaller the update becomes when weights are tiny. Lower layers end up learning almost nothing, not because the model is wrong, but because the signal just can’t survive the journey backward.</p><p>We can avoid all this problems just by using BatchNorm of activation values.</p><p>3. <strong>Accuracy and Efficiency are orthogonal problems and must be treated separately.</strong></p><p>I made my training workflow into two core stages:</p><ul><li>Designing the model architecture to match the statistical structure of the task, and</li><li>System-level optimization using PyTorch’s profiler to identify compute, memory, and data-pipeline bottlenecks before tuning hyper-parameters.</li></ul><p>I don’t treat hyper-parameter tuning as blind search — I profile some training runs using torch.profiler to understand GPU utilization, memory behavior, and what kernels my GPU is using, and only then adjust learning rate schedules, batch size, precision, and data-loading strategy.</p><p>4.<strong> Try to fully saturate GPU’s silicon:</strong></p><p>For a long time, these are the few misconceptions I had and didn’t put much effort into using my GPU efficiently, partly because this aspect was not something I was exposed to during college.</p><ul><li>After reading tutorials from 2016, I thought there should be CUDA tool-kit installed and ready to use in my pc, so that PyTorch can use it for my use-cases.</li><li>Using torch-gpu .</li><li>The only code I used that is closest to GPU is torch.cuda.is_available() .</li></ul><p>Later I realized the only thing that I need to make sure when I’m installing PyTorch from CUDA wheel is that</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*bZsJZn8GnDPByXC4oDa44w.png" /></figure><p>So I don’t get into any CUDA errors.</p><p>I did used GPU, particularly most of the time I used CUDA cores which are better for parallel computation compared to CPU. Since CUDA cores execute mainly scalar operations per clock cycle, they are inefficient for the large-scale matrix multiplications that dominate deep-learning workloads.</p><p>To improve FP16 Tensor Core utilization, I now choose batch size, hidden dimensions, channel counts, and intermediate sizes (such as FFN expansion widths) as multiples of 8 or 16. This aligns FP16 GEMM dimensions with Tensor Core tile shapes, avoids inefficient remainder kernels, and gives much better throughput during mixed-precision training.</p><p>I use autocast with GradScaler so that unsafe operations stay in FP32 while the heavy matrix multiplications run in FP16 without underflow issues. I also enable torch.backends.cuda.matmul.allow_tf32 = True, so any FP32 matrix multiplications that slip through are executed using TF32 Tensor Cores instead of slow CUDA-core paths.</p><p>For convolution-heavy models, I also send both the model and inputs to channels_last memory format to improve memory access patterns and keep Tensor Core kernels better fed.</p><p>By following this setup, I’ve gone from effectively using <strong>only CUDA cores</strong> to actually <strong>using FP16 Tensor Cores for matrix multiplications</strong> in both forward and backward passes, storing weights, activations, and gradients in FP32, <strong>using TF32 Tensor Cores for any remaining FP32 GEMMs</strong>, and leaving<strong> CUDA cores mainly for activation functions, LayerNorm, and residual adds. </strong>Overall trying to keep most parts of the GPU busy instead of leaving large portions of the silicon idle.</p><p>5.<strong> Mixed-Precision Training Needs a Full Setup — Not Just </strong><strong>autocast()</strong></p><p>When I first started using FP16, I thought enabling autocast() alone was enough. It wasn’t. My models would sometimes diverge, sometimes silently stop improving. Over time I realized that FP16 training only works smoothly when I treat it as a <strong>system</strong>, not a toggle. For a stable and predictable gradient flow, I now always combine:</p><ul><li>Proper <strong>weight initialization</strong></li><li><strong>Input scaling</strong></li><li><strong>BatchNorm or LayerNorm</strong> to control activation scale</li><li>amp.autocast(&quot;cuda&quot;) so heavy ops run in FP16</li><li>amp.GradScaler() to prevent gradient underflow</li><li><strong>Residual connections</strong> to preserve learning signals in deeper networks</li></ul><p>This setup keeps activations in a reasonable range, prevents tiny gradients from disappearing, and makes mixed-precision training behave like FP32 training — just much faster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=982156d69168" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bayesian Neural Networks]]></title>
            <link>https://mohith-g.medium.com/bayesian-neural-networks-203e2e4d56f0?source=rss-6575ac4820f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/203e2e4d56f0</guid>
            <category><![CDATA[bayesian-networks]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[uncertainty]]></category>
            <dc:creator><![CDATA[Mohith]]></dc:creator>
            <pubDate>Tue, 30 Dec 2025 16:22:51 GMT</pubDate>
            <atom:updated>2025-12-30T16:22:51.986Z</atom:updated>
            <content:encoded><![CDATA[<h4>How can we get epistemic and aleatoric uncertainties just by converting from point-estimates to random-variables ?</h4><p>The way I’d like to jog my memory if I have to implement or model uncertainty in neural net predictions is to think there are three kinds of modeling techniques:</p><ol><li><strong>Traditional Neural Networks (No Explicit Uncertainty) : </strong>This nets are our usual go-to universal function approximators, we train this models through negative log-likelihood as loss function in a update rule. A prior we should agree to use these models is that the data is homoskedastic (For instance, in the California Housing dataset, assuming homoskedastic noise is questionable. House price variance is not constant and likely depends on covariates such as house size, with larger houses exhibiting different variability than smaller ones).</li><li><strong>Heteroskedastic Neural Networks (Aleatoric Uncertainty) </strong>: When homoskedastic assumption fails, we can take advantage of free variance parameter(For instance, instead of fixing variance as 1 in gaussian negative log-likelihood, we let the model output variance too). This gives us aleatoric uncertainty (uncertainty in prediction due to data noise), we can later use it to perform inference like confidence intervals, etc.</li><li><strong>Bayesian Neural Networks (Epistemic + Aleatoric Uncertainty) : </strong>Instead of performing regular point-estimation in neural nets we consider the model’s parameters as random variables, this let’s us find epistemic uncertainty (uncertainty in prediction due to uncertainty in model parameters). In this blog I want to talk about how the idea of BNN is introduced and how to implement it from scratch.</li></ol><p>From bayes theorem,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/199/1*D5L7VTfUZG1kaisTtAxFdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*EkxBAFYxybSY5ZauRf1S4A.png" /></figure><p>Since, evidence doesn’t depend on weights:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*DilH0nMlJ_7Yibur0OPemw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*AoUQ6rA6OmVjuN_z-gWx2Q.png" /></figure><p>Note this above term is similar to regularized negative log-likelihood (Regularized loss) in point-estimation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*MCLmVHJ5-LEknzjNQBqhdQ.png" /></figure><p>If the prior is from Laplace or Gaussian distribution with 0 mean, the regularization would be L1 or L2 respectively.</p><blockquote><strong>Introducing Posterior Weights Distribution :</strong></blockquote><p>Now that prior is a random-variable but not a point-estimate, if we compute posterior weight distribution with the help of the likelihood of observing data with prior weights</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*Ajo4tj0rTWa1reOWRl-Jyg.png" /></figure><p>Where alpha, betta are prior weight’s precision(controls epistemic uncertainty) and noise precision(aleatoric uncertainty) respectively. To make it more clear with gaussian assumption :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/229/1*f5f5TZptpioyEohmx4KQcQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*5dDXhkJaQ_aaBdUh2eB7kA.png" /></figure><p>Now we can use this posterior to make predictions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*n35QxpPcSQCREIMo1RiQOQ.png" /></figure><p>Where,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*dHkG6oc1JdrAivPaz1q3Zw.png" /></figure><p>Practically using Monte-Carlo approximation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*KNVv0rUcRDfcG_7rrgfvHg.png" /></figure><p>Now we can perform</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*AzoZQXPw_wSCqXRFV2P9jg.png" /></figure><p>to get a mean prediction and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*y3eDCqGpdn86OzYYTXGO1g.png" /></figure><p>to get total uncertainty on the prediction.</p><blockquote><strong>Computing Posterior Weights Distribution:</strong></blockquote><p>To get the mean and uncertainty values we need to compute posterior weights distribution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/105/1*3Q6kb0wloOwskiJujmx9UQ.png" /></figure><p>These are some of the ways to compute the posterior:</p><ul><li><strong>MAP estimation</strong></li><li><strong>Laplace approximation</strong></li><li><strong>Variational Inference (VI)</strong></li><li><strong>Mean-field Variational Inference</strong></li><li><strong>Markov Chain Monte Carlo (MCMC)</strong></li><li><strong>Hamiltonian Monte Carlo (HMC)</strong></li><li><strong>Stochastic Gradient Langevin Dynamics (SGLD)</strong></li><li><strong>Expectation Propagation (EP)</strong></li><li><strong>Monte Carlo Dropout</strong></li></ul><p>I’ll be using Variational Inference(VI) here.</p><blockquote><strong>BNN Using</strong> <strong>Variational Inference (VI):</strong></blockquote><p>In VI, we introduce a new distribution Q(w) with pre-defined initial values and try to make Q(w) as identical as possible to posterior weight distribution P(w | D).</p><p>We can use KL divergence to find out how similar two distributions are</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*7da4HVf-BKei2dmAGz6WVA.png" /></figure><p>Which can also be written as,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/1*_uHGrnH3r9OdcJmXQc8d4g.png" /></figure><p>Solving,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*Pe3eJK9mlBpQ-ehbU3QBaQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*OP2TbPi6wFrddwXxSQ0vXw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*lhzTbJfGJXP4cvZDanNOJg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*XK77lNtRHkKmOvkMLYcgug.png" /></figure><p>Now,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*XXVJnvUm73c1g-Ig4rBu5A.png" /></figure><p>Since posterior KL is always greater than 0, the term</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*6x2a7zxYfiQSm-43mEiOfw.png" /></figure><p>acts as the lower bound of log-evidence. Let’s call it <strong>Evidence Lower Bound (ELBO)</strong>.</p><p>From Equation 1, log-evidence is constant w.r.t to Q(w), so</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*BNlQHaNvvdAl7YzX6ZmMfA.png" /></figure><p>Then</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*ndcQWGe3tyw5oZfPOeg5YQ.png" /></figure><p>If we consider negative ELBO as loss,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*P3Wi3f429Z2PJgd0GO8pBA.png" /></figure><p>Now in optimization, if the loss decreases then the difference between Q(w) and posterior P(w|D) decreases.</p><p>Let’s code the loss functions:</p><pre>import torch<br><br>def gaussian_nll(y_mean, y_std, y_true):<br>    var = y_std**2 + 1e-6<br>    return torch.mean(<br>        0.5 * torch.log(2 * torch.pi * var) +<br>        0.5 * (y_true - y_mean)**2 / var<br>    )<br><br>def kld_gaussian(mu, log_var):<br>    var = torch.exp(log_var)<br>    return 0.5 * torch.sum(mu**2 + var - log_var - 1)</pre><p>gaussian_nll is the log-likelihood part of the ELBO and kld_gaussian is KL divergence part of the ELBO. This two will update the weights so that the likelihood of observing data increases while also respecting the prior so that they don’t trust the data completely.</p><p>The model:</p><pre>from torch import nn<br>from typing import List, Tuple<br><br>class BayesianNN(nn.Module):<br>    def __init__(<br>        self,<br>        input_dim: int,<br>        hidden_layers: List[int],<br>        output_dim: int,<br>        activation: nn.Module = nn.ReLU()<br>    ):<br>        super().__init__()<br><br>        # Posterior parameters q(w) = N(μ, σ²)<br>        self.weight_mu: nn.ParameterList = nn.ParameterList()<br>        self.weight_log_var: nn.ParameterList = nn.ParameterList()<br>        self.bias_mu: nn.ParameterList = nn.ParameterList()<br>        self.bias_log_var: nn.ParameterList = nn.ParameterList()<br><br>        self.activation = activation<br><br>        layers: List[int] = [input_dim] + hidden_layers + [output_dim * 2]<br><br>        for in_f, out_f in zip(layers[:-1], layers[1:]):<br>            self.weight_mu.append(nn.Parameter(torch.randn(out_f, in_f) * 0.05))<br>            self.weight_log_var.append(nn.Parameter(torch.full((out_f, in_f), -3.0)))<br>            self.bias_mu.append(nn.Parameter(torch.randn(out_f) * 0.05))<br>            self.bias_log_var.append(nn.Parameter(torch.full((out_f,), -3.0)))<br><br>    def forward(self, X, sample=True) -&gt; Tuple[torch.Tensor, torch.Tensor]:<br>        for i in range(len(self.weight_mu)):<br>            if sample:<br>                w = self.weight_mu[i] + torch.exp(0.5 * self.weight_log_var[i]) * torch.randn_like(self.weight_mu[i])<br>                b = self.bias_mu[i] + torch.exp(0.5 * self.bias_log_var[i]) * torch.randn_like(self.bias_mu[i])<br>            else:<br>                w, b = self.weight_mu[i], self.bias_mu[i]<br><br>            X = X @ w.T + b<br>            if i &lt; len(self.weight_mu) - 1:<br>                X = self.activation(X)<br><br>        mean, log_var = torch.chunk(X, 2, dim=1)<br>        std = torch.exp(0.5 * log_var)<br>        return mean, std</pre><p>hidden_layers is a type of list[int] having length of how many hidden layers we want in the neural network where each integer is the number of neurons in each layer.</p><p>The training loop:</p><pre>for epoch in range(epochs):<br>    model.train()<br>    train_loss = 0<br>    train_samples = 0<br><br>    for X_batch, y_batch in train_loader:<br>        X_batch = X_batch.to(device)<br>        y_batch = y_batch.to(device)<br>        batch_size = X_batch.size(0)<br><br>        # Forward pass (samples stochastic weights)<br>        y_mean, y_std = model(X_batch)<br><br>        # Aleatoric term : Negative Log Likelihood<br>        nll = gaussian_nll(y_mean, y_std, y_batch)<br><br>        # Epistemic term : KL(q(w) || p(w))<br>        kld = 0<br>        for i in range(len(model.weight_mu)):<br>            kld += kld_gaussian(model.weight_mu[i], model.weight_log_var[i])<br>            kld += kld_gaussian(model.bias_mu[i], model.bias_log_var[i])<br><br>        # ELBO<br>        loss = nll + kld / num_train_samples<br><br>        # Backprop<br>        optimizer.zero_grad()<br>        loss.backward()<br>        optimizer.step()<br><br>        train_loss += loss.item() * batch_size<br>        train_samples += batch_size<br><br>    print(f&quot;Epoch {epoch+1} - Train ELBO: {train_loss/train_samples:.4f}&quot;)</pre><p><strong>Note:</strong> <em>I summarised the code by removing all data preprocessing, temperature calibration, Monte-Carlo validation logic, metric tracking, PICP–MPIW evaluation, dynamic architecture handling, saving/loading utilities, and all packaging scaffolding. The full implementation additionally performs posterior sampling, ELBO scaling by dataset size, predictive uncertainty decomposition, probabilistic calibration via temperature scaling, and confidence-interval evaluation, all of which are available in the original </em><em>BnnRegressor class of the package </em><a href="https://pypi.org/project/dynamodelx/"><em>dynamodelx</em></a><em>.</em></p><h3>Conclusion</h3><p>The key idea behind Bayesian Neural Networks is simple: instead of learning a single best set of weights, we learn a <em>distribution</em> over weights. Once weights are treated as random variables, epistemic uncertainty naturally appears through posterior sampling, while aleatoric uncertainty is captured by explicitly modeling the output variance.</p><p>From an implementation point of view, nothing fundamentally changes in how we train neural networks. We still define a loss, call backward(), and update parameters using gradient-based optimization. The only difference is that the loss is now the negative ELBO, which balances data fit (log-likelihood) with regularization toward the prior (KL divergence).</p><p>This framework gives us a principled way to quantify uncertainty in predictions, making Bayesian Neural Networks particularly useful in settings where knowing <em>how uncertain</em> a model is matters as much as the prediction itself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=203e2e4d56f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Perceptrons to Deep Learning]]></title>
            <link>https://mohith-g.medium.com/from-perceptrons-to-deep-learning-241cd1994a33?source=rss-6575ac4820f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/241cd1994a33</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[backpropagation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[perceptron]]></category>
            <dc:creator><![CDATA[Mohith]]></dc:creator>
            <pubDate>Thu, 16 Oct 2025 06:10:34 GMT</pubDate>
            <atom:updated>2025-10-16T07:03:03.417Z</atom:updated>
            <content:encoded><![CDATA[<h4>The History, Math, and Mechanics of Neural Networks</h4><p>It’s 1958, an American psychologist <strong>Frank Rosenblatt</strong> invented perceptron, a model that mimics how brain neuron works. Basically, he mimicked a neuron by using weighted sum and thresholding. For instance, <em>x</em> is a <em>d</em> dimensional vector such that</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/68/1*te0RHIcFp_YW30Vy_wwuMg.png" /></figure><p>And there exists a non-linear, discontinuous and not differentiable step-function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/206/1*RJOu39qKb922qSLCITuL6g.png" /></figure><p>If <em>w</em> be the <em>d</em> dimensional weight vector of that neuron, the working process of perceptron is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/143/1*cHkYZAftHV9L2r2x33-rKw.png" /></figure><p>The real breakthrough of this invention is it’s learning(weight update) rule. Without any statistics assumptions or MLE, Rosenblatt heuristically designed the update rule as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/1*b6F_VZj4Y8llppA5gdKSrQ.png" /></figure><p>This moves the weights in the direction of loss reduction. Since, the step-function is non-differentiable we can’t compute gradients. That’s why <strong>backpropagation can’t be used</strong> with the original perceptron.</p><p>He also introduced a threshold(θ), the neuron fires only if the weighted sum crosses this threshold, it is often assumed 0 to make the formulas simpler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/181/1*SJ5bsC-pEDb_UrwqZf-HJw.png" /></figure><p>Later we called it bias, by considering</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/65/1*5OuuRbT1tmgpMpTf9qdPNQ.png" /></figure><p>And made it a trainable parameter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/179/1*oRy6gSe12OEIpjKlY6Qurw.png" /></figure><pre>from sklearn.datasets import load_iris<br>from sklearn.model_selection import train_test_split<br>from sklearn.decomposition import PCA as pca<br>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt</pre><pre>data = load_iris()<br>X , y = data.data, data.target<br>df = pd.DataFrame(np.column_stack((X, y)), columns=data.feature_names + [&#39;target&#39;])<br>df = df[df[&#39;target&#39;] != 2] # Only sertosa and versicolor<br>X = df.drop(&#39;target&#39;, axis=1).values<br>y = df[&#39;target&#39;].values<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BlibYASn6DUxx--nsb_xoQ.png" /></figure><pre>def step_function(z):<br>    return 1 if z &gt; 0 else 0<br><br>w = np.random.randn(X_train.shape[1])<br>learning_rate = 0.001<br><br>for epoch in range(10):<br>    errors = 0<br>    for xi, target in zip(X_train, y_train):<br>        z = np.dot(xi, w)<br>        y_pred = step_function(z)<br>        update = learning_rate * (target - y_pred)<br>        w += update * xi<br>        errors += int(update != 0)<br>    print(f&#39;Epoch {epoch}, Misclassifications: {errors}&#39;)</pre><pre>Epoch 0, Misclassifications: 22<br>Epoch 1, Misclassifications: 2<br>Epoch 2, Misclassifications: 2<br>Epoch 3, Misclassifications: 2<br>Epoch 4, Misclassifications: 1<br>Epoch 5, Misclassifications: 1<br>Epoch 6, Misclassifications: 1<br>Epoch 7, Misclassifications: 1<br>Epoch 8, Misclassifications: 0<br>Epoch 9, Misclassifications: 0</pre><pre>X_test_pred = [step_function(np.dot(xi, w)) for xi in X_test]<br>print(f&#39;Misclassifications in test data: {np.sum(X_test_pred != y_test)}&#39;)</pre><pre>Misclassifications in test data: 0</pre><p>Rosenblatt claimed it could learn to recognize complex patterns — even said things like <em>“The perceptron will eventually learn to walk, talk, see, and understand” . </em>The U.S. Navy even funded the <strong>Mark I Perceptron</strong> hardware computer in 1958. Media called it the <strong>first machine that could think</strong>.</p><p>Then Marvin Minsky and Seymour Papert wrote <strong>“Perceptrons” (1969), </strong>they mathematically proved perceptron can’t even solve problems like XOR. The linear combination create a line/plane/hyperplane, the step-function turns the linear function into a binary decision. So, perceptron can only handle linearly separable data. This book crushed the hype of perceptron, they introduced multi-layer perceptron(input, hidden and output layers) and said in theory it could solve this problem, but no one knew how to train them. This started the first <strong>AI Winter</strong> from 1970’s to early 1980’s. Neural Networks considered dead end, funding dropped sharply.</p><p>In 1986 Rumelhart, Hinton, and Williams formalized the <strong>backpropagation algorithm</strong>. Finding gradient of the loss function w.r.t all the weights through chain rule made the multi-layer perceptron to learn non-linear decision-boundaries effectively.</p><p>After multi-layer perceptron with differentiable activations became trainable (thanks to backprop), researchers proved that <strong>a sufficiently large neural network can approximate any continuous function</strong> to arbitrary accuracy. This is known as the <strong>Universal Approximation Theorem (1989–1991).</strong></p><blockquote>Training a Neural Network from Scratch :</blockquote><p><strong>Feed-forward Network Functions:</strong></p><p>A feed-forward phase can be described as a series of functional transformations. For 2 layer neural network, first we construct <em>M </em>perceptron models of input variable <em>x1, …, xd</em> . <em>j’th</em> model’s output in M models is computed in the form</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/1*licYOe0YC2E8CWXs1cAOdA.png" /></figure><p>The subscript (1) indicates that the corresponding parameters are in first layer of the network. <em>a_j </em>is the pre-activation output of model <em>j</em>. Using a non-linear activation function helps us model complex patterns, while making sure it is differentiable helps in gradient-based backpropagation. Let <em>h(.)</em> be a differentiable, non-linear activation function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/119/1*N0k7RqbhZ7kVTz_s4RJ23A.png" /></figure><p><em>z_j</em> is the activated or hidden unit of model <em>j</em>. These values again linearly combined to give output unit activations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/217/1*_2CUWjqK0u-9ut-xeSyWVA.png" /></figure><p>The output layer’s dimension or activation function depends on what type of problem we are solving if it’s a regression, the activation function is identity. So for a given n’th sample</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/79/1*VBCXce19HahGkbnd7MRm6A.png" /></figure><p>Note if the output is a scalar prediction, k = 1.</p><p>In regression, sum of squared error(SSE) is the loss we often use</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/114/1*kigXqgQLLjAi-6ezI8mcnw.png" /></figure><p>This was invented in 19th century by Gauss. There are other variants for SSE, like if you want simple gradients in optimization</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/135/1*uN-ZFAmMT47jLAsj4u6JFg.png" /></figure><p>The factor 1/2 cancels 2 in derivative.</p><p>Mean Squared Error (MSE)— makes the loss scale-independent of dataset size, useful for smoother updates during training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/137/1*x6MQeqn06IsLwlZ1rxAwLg.png" /></figure><p>If there are<em> k</em> outputs per sample (e.g., multi-output regression), this averages over both samples and outputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/1*1FCnmPqEA0nnbjuWx0-1Kw.png" /></figure><p>These variants are used accordingly. Using different factors just effects the scale of the gradients, it doesn’t change where the minimum of the negative log likelihood is after performing MLE.</p><p>Consider performing MLE for the model’s prediction assuming gaussian error, for a unit variance we get an error of</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/131/1*H0Oqe7fkrcKP3Va_9fJC3g.png" /></figure><p>Instead of factor 1/2 if I choose to use 1/n, I would get smoother updates but with twice the magnitude.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/1*t-5JzNo9cy-lFQI9JX3d7g.png" /></figure><p>The goal of backpropagation is to find out how much the error changes w.r.t each layer’s weights, so we can update the weights and bias by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/1*AMJ8qL5CbhdX8jMtWAVrRQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/167/1*1HrG4Csdr-DqkLhQhWVzVA.png" /></figure><p><strong>Backpropagation :</strong></p><p>By applying chain rule, we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/169/1*4Snjlfjl2bwjJxTdNW3VHw.png" /></figure><p>Let</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/101/1*J7eghE82MqsXPMU530HzXw.png" /></figure><p>We know that,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*m-KLxfUwebpu8ktZdRxhpA.png" /></figure><p>Then,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/123/1*vUrmG7hQGGG1_X_U2zB9eg.png" /></figure><p>Substituting back in the chain rule,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/151/1*qu_6v5Q610sXdaQcY0ZXWg.png" /></figure><p>This tells us that the required derivatives is obtained simply by multiplying the value of δ for the unit at the output end of the weight by the value of <em>z</em> for the unit at the input end of the weight. In order to evaluate the derivatives, we only need to calculate the value of δ.</p><p>For our error function, if there are <em>L</em> layers in feed-forward, for the last layer’s</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/175/1*WKSfZ7VWwZ_vm4u0dbVIzw.png" /></figure><p>so in our 2-layers case, the last layer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/217/1*_2CUWjqK0u-9ut-xeSyWVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/109/1*cRsUVmr8x1RC2X06ML2IAQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/1*_K--OpAFc3rx4Ls_2oGEpQ.png" /></figure><p>For remaining layers,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*bcz4S2CYp8-K9Nv15pw4Bg.png" /></figure><p>we are making use of the fact that variations in <em>a_j</em> give rise to the variations in error function only through variations in variable <em>a_k</em>.</p><p>Solving this, we know that</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/137/1*r7FZ1crntLtrQmV5NABBjQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/1*hv5dDH-RGlRkgDaemQowNw.png" /></figure><p>Substituting</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*akQci5iPODSEEYsZXQ720w.png" /></figure><p>For bias <em>b, </em>in layer <em>l</em>, applying the chain rule gives:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/175/1*j1SCHV8Taej_rX2ecRYflg.png" /></figure><p>solving this would result in,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/140/1*c58BiN0-OAGcxH4dB4bnvQ.png" /></figure><p>Notice that during feed-forward, the computation proceeds over the second index of the weight matrix — because each neuron in layer <em>l+1</em> receives inputs from all neurons in the previous layer (summing over that index).<br> In contrast, during backpropagation, the computation proceeds over the first index — since we already know the errors of the output neurons in the next layer and need to propagate them backward through their incoming connections.</p><p>Training an MLP with 3 hidden-layers and 1 output layer</p><pre>from sklearn.datasets import fetch_california_housing<br>from dataclasses import dataclass, field<br>import torch<br><br>device = &#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39;<br>print(&quot;Using device:&quot;, device)</pre><pre>Using device: cuda</pre><pre>data = fetch_california_housing()<br>X, y = data.data, data.target<br>features = data.feature_names<br>target = data.target_names<br>print(&#39;Input features : &#39;,features)<br>print(&#39;Target feature : &#39;,target)<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)<br><br>X_train = torch.tensor(X_train, dtype=torch.float32, device=device)<br>y_train = torch.tensor(y_train, dtype=torch.float32, device=device)<br><br>X_test = torch.tensor(X_test, dtype=torch.float32, device=device)<br>y_test = torch.tensor(y_test, dtype=torch.float32, device=device)<br><br>print(&#39;Training samples : &#39;, X_train.shape[0])<br>print(&#39;Testing samples : &#39;, X_test.shape[0])</pre><pre>Input features :  [&#39;MedInc&#39;, &#39;HouseAge&#39;, &#39;AveRooms&#39;, &#39;AveBedrms&#39;, &#39;Population&#39;, &#39;AveOccup&#39;, &#39;Latitude&#39;, &#39;Longitude&#39;]<br>Target feature :  [&#39;MedHouseVal&#39;]<br>Training samples :  14448<br>Testing samples :  6192</pre><pre>@dataclass<br>class Params:<br>    input_dim: int<br>    hidden_layer1 : int = 16<br>    hidden_layer2 : int = 8<br>    hidden_layer3 : int = 4<br>    lr : float = 0.01<br>    epoch : int = 0<br>    loss : float = float(&#39;inf&#39;)<br>    r_square : float = 0<br>    seed : int = 42<br><br>    W1: torch.Tensor = field(init=False, repr=False)<br>    b1: torch.Tensor = field(init=False, repr=False)<br>    W2: torch.Tensor = field(init=False, repr=False)<br>    b2: torch.Tensor = field(init=False, repr=False)<br>    W3: torch.Tensor = field(init=False, repr=False)<br>    b3: torch.Tensor = field(init=False, repr=False)<br>    W4: torch.Tensor = field(init=False, repr=False)<br>    b4: torch.Tensor = field(init=False, repr=False)<br><br>    def __post_init__(self):<br>        torch.manual_seed(self.seed)<br>        self.W1 = torch.randn(self.hidden_layer1, self.input_dim, device=device) * torch.sqrt(torch.tensor(2/self.input_dim, device=device))<br>        self.b1 = torch.zeros(self.hidden_layer1, 1, device=device)<br><br>        self.W2 = torch.randn(self.hidden_layer2, self.hidden_layer1, device=device) * torch.sqrt(torch.tensor(2/self.hidden_layer1, device=device))<br>        self.b2 = torch.zeros(self.hidden_layer2, 1, device=device)<br><br>        self.W3 = torch.randn(self.hidden_layer3, self.hidden_layer2, device=device) * torch.sqrt(torch.tensor(2/self.hidden_layer2, device=device))<br>        self.b3 = torch.zeros(self.hidden_layer3, 1, device=device)<br><br>        self.W4 = torch.randn(1, self.hidden_layer3, device=device) * torch.sqrt(torch.tensor(2/self.hidden_layer3, device=device))<br>        self.b4 = torch.zeros(1, 1, device=device)<br><br>p = Params(input_dim = X_train.shape[1])</pre><pre>def relu(a):<br>    return torch.maximum(torch.zeros_like(a), a)<br><br>def relu_grad(a):<br>    return (a &gt; 0).float()<br><br>def feed_forward(X, p):<br>    A1 = p.W1 @ X.T + p.b1<br>    Z1 = relu(A1)<br><br>    A2 = p.W2 @ Z1 + p.b2<br>    Z2 = relu(A2)<br><br>    A3 = p.W3 @ Z2 + p.b3<br>    Z3 = relu(A3)<br><br>    A4 = p.W4 @ Z3 + p.b4<br>    cache = (A1, A2, A3, Z1, Z2, Z3, A4)<br>    return A4, cache<br><br><br>def back_prop(cache, X, y, p):<br>    A1, A2, A3, Z1, Z2, Z3, A4 = cache<br>    m = X.shape[0]<br><br>    dA4 = (2/m) * (A4 - y.reshape(1, -1))<br>    dW4 = dA4 @ Z3.T<br>    db4 = torch.sum(dA4, dim=1, keepdim=True)<br><br>    dA3 = relu_grad(A3) * (p.W4.T @ dA4)<br>    dW3 = dA3 @ Z2.T<br>    db3 = torch.sum(dA3, dim=1, keepdim=True)<br><br>    dA2 = relu_grad(A2) * (p.W3.T @ dA3)<br>    dW2 = dA2 @ Z1.T<br>    db2 = torch.sum(dA2, dim=1, keepdim=True)<br><br>    dA1 = relu_grad(A1) * (p.W2.T @ dA2)<br>    dW1 = dA1 @ X<br>    db1 = torch.sum(dA1, dim=1, keepdim=True)<br><br>    return dW1, db1, dW2, db2, dW3, db3, dW4, db4</pre><pre>m = X_train.shape[0]<br>epochs = 10000<br><br>X_mean = X_train.mean(axis=0)<br>X_std = X_train.std(axis=0)<br><br>y_mean = y_train.mean()<br>y_std = y_train.std()<br><br># Normalize<br>X_train_norm = (X_train - X_mean) / X_std<br><br>y_train_norm = (y_train - y_mean) / y_std<br>y_train_norm = y_train_norm.reshape(1, -1)<br><br>loss_track = []<br>r_square_track = []<br><br>for i in range(epochs):<br>    <br>    A4, cache = feed_forward(X_train_norm, p)<br><br>    p.epoch += 1<br>    <br>    p.loss = torch.mean((y_train_norm - A4)**2).item()<br>    <br>    loss_track.append(p.loss)<br><br>    rss = torch.sum((y_train_norm - A4)**2)<br>    tss = torch.sum((y_train_norm - torch.mean(y_train_norm))**2)<br><br>    p.r_square = 1 - (rss/tss).item()<br>    r_square_track.append(p.r_square)<br><br>    dW1, db1, dW2, db2, dW3, db3, dW4, db4 = back_prop(cache, X_train_norm, y_train_norm, p)<br>    <br>    p.W1 -= p.lr * dW1<br>    p.b1 -= p.lr * db1<br>    <br>    p.W2 -= p.lr * dW2<br>    p.b2 -= p.lr * db2<br>    <br>    p.W3 -= p.lr * dW3<br>    p.b3 -= p.lr * db3<br>    <br>    p.W4 -= p.lr * dW4<br>    p.b4 -= p.lr * db4<br>    <br>    if i % 100 == 0:<br>        print(f&quot;Epoch {i}: Loss = {p.loss:.4f}, R^2 = {p.r_square:.4f}&quot;)</pre><pre>Epoch 9900: Loss = 0.2452, R^2 = 0.7548</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*56aoqJA67IAly0ip3dCoKQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w0rIHhGg7qBo4dw_OfRzGg.png" /></figure><pre>X_test_mean = X_test.mean(axis=0)<br>X_test_std = X_test.std(axis=0)<br><br>y_test_mean = y_test.mean()<br>y_test_std = y_test.std()<br><br># Normalize<br>X_test_norm = (X_test - X_test_mean) / X_test_std<br><br>y_test_norm = (y_test - y_test_mean) / y_test_std<br><br>l1 = relu(p.W1 @ X_test_norm.T + p.b1)<br>l2 = relu(p.W2 @ l1 + p.b2)<br>l3 = relu(p.W3 @ l2 + p.b3)<br>l4 = p.W4 @ l3 + p.b4<br>print(f&#39;Test Loss: {torch.mean((y_test_norm.reshape(1, -1) - l4)**2).item():.4f}&#39;)</pre><pre>Test Loss: 1.4066</pre><p>ReLU simply passes positive values and clips everything below zero. It fixed the vanishing gradient issue that sigmoid and tanh had, but it introduced a new one, dead neurons. Once ReLU outputs zero, the gradient also becomes zero, and that neuron stops learning.</p><p>He initialization was designed for this. Since ReLU kills roughly half of the activations, He scales the weights by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/52/1*V2MFxzbrNNEQbb3PWJAb-w.png" /></figure><p>so the variance stays balanced across layers. It basically keeps the signal flow stable even when half the neurons are inactive.</p><p>If I use the loss,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/1*jtuO1n4Hnu4cjk0_jQx5-w.png" /></figure><p>Combining both SSE with factor 1/2 and 1/n</p><p>The gradient in the code would be,</p><pre>    dA4 = (1/m) * (A4 - y.reshape(1, -1))</pre><p>Multiplying with 2 before made the gradients scaled twice it’s magnitude. After removing 2, to get the same loss I got for the same number of iterations, I need to double the learning rate to 0.02</p><p><strong>Conclusion :</strong></p><p>What began as Rosenblatt’s single neuron now forms the foundation of deep learning — every neural network today, from GPT to CNNs, still follows the same principles: weighted sums, nonlinear activations and backpropagation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=241cd1994a33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Comprehensive Guide On Classification]]></title>
            <link>https://mohith-g.medium.com/a-comprehensive-guide-on-classification-c27f90558afa?source=rss-6575ac4820f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c27f90558afa</guid>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[support-vector-machine]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[bayes-theorem]]></category>
            <dc:creator><![CDATA[Mohith]]></dc:creator>
            <pubDate>Thu, 18 Sep 2025 05:56:18 GMT</pubDate>
            <atom:updated>2025-09-18T06:02:29.832Z</atom:updated>
            <content:encoded><![CDATA[<h4>Applying and exploring the foundations of classification algorithms with derivations</h4><p>In regression, we try to find a function r(x) that takes an input x and predicts the output y as accurately as possible. A simple example of such a regression function is :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/217/1*QQIrff3DF3lGnWK1OGKtyg.png" /></figure><p>In classification, instead of predicting a continuous output, we need a decision boundary that splits the space. The goal is to separate data points such that those on one side of the boundary belong to one class, and those on the other side belong to another. In two dimensions, this boundary is a line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/0*n8CNxOqVqAByFQde" /></figure><p>For classification, we assign the point to a class based on which side of the boundary it is located, positive or negative. Points on one side of the boundary satisfies :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*D8AWaEcOYNKqGxSyEpwhKQ.png" /></figure><p>And the other satisfies :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/0*MMd5IEO-AF-Aqk1H" /></figure><p>The points that are exactly on the boundary are either used to define the boundary (as support vectors in some algorithms) or assigned to a class according to the rules of the algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3FtT-P5rz159iDJvkxqyw.png" /></figure><p>The main goal of a classification problem is, for a given data point x, predict which side of the decision boundary the point belongs to i.e. to find the posterior :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/246/1*7vW5DXznf3-dZ2MnfzSXig.png" /></figure><p>This is a bayes problem.</p><blockquote>Bayes Classifier :</blockquote><p>According to bayes rule,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*gdSnTil0dhLOcfC8agDOsQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*WusRiWnv6hwL2q6KHWkmRg.png" /></figure><ul><li><strong>f1(x)</strong> is the density function of the data points that belongs to class 1</li><li><strong>f0(x)</strong> is the density function of the data points that belongs to class 0</li><li><strong>π</strong> is the P(Y = 1)</li><li><strong>1 - π</strong> is the P(Y = 0)</li></ul><p>The model would be,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*zW9WBu4b7k8akVXQofESdQ.png" /></figure><p>And the points that satisfies,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/1*0ZXlx1eG9aianv-sNwLAnQ.png" /></figure><p>lies on the boundary.</p><p>The <strong>true error rate</strong> of a classifier <em>h</em> is :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/212/1*YF91vdlTX9Wi2CQxXTIrLQ.png" /></figure><p>However we don’t know the population data, so we find<strong> empirical error rate</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*TFcEMbp9M0SVVQyH97Fk4w.png" /></figure><p>The bayes rule is optimal, that is, if <em>h</em> is any other classification rule then</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/133/1*_GBQlZUqUIjzcUuKuzQV1Q.png" /></figure><p>The bayes rule depends on unknown quantities so we need to use the data to find some approximation to the bayes rule. There are three main approaches:</p><ol><li><strong>Empirical Risk Minimization :</strong></li></ol><p>The goal of a bayes classifier is to find P(Y = 1 | x) as accurately as possible, this is also mean to reduce the empirical error rate of the model.</p><p>The Empirical Risk Minimization states that, <em>Choose a set of classifiers H and find a classifier h </em>∈ <em>H, such that it minimizes the empirical error rate.</em></p><p>The goal of bayes classifier and <strong>ERM’s </strong>ended up being the same thing. Not only for model selection but <strong>ERM </strong>can be extended to improve model’s performance by selecting weights or parameters such that, they reduce loss function.</p><p><strong>2. Density Estimation </strong>:</p><p>Using the bayes classifier as a frame, we can go all data driven and find individual components that give us the posterior. Estimate f1 from X for which Yi = 1, estimate f0 from X for which Yi = 0 and let prior be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/1*UBcsU4tdZ78UjVDZYDxNZg.png" /></figure><p>Then,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*wPNhS3b6zRnGtuEk8UvS8A.png" /></figure><p>Suppose that Y ∈ {1, 2, …, k}, the optimal rule is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/1*GYoIAn2UBceDR4l9hgjt5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*zWiUfsP0M5vcHcOUahk24g.png" /></figure><p>Since the denominator is same for all classes, ignoring it won’t effect our comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*IjroGBb1tFLz2pwYuGraWw.png" /></figure><p>You may have heard this type of classification as<strong> Generative models</strong>.</p><p><strong>3. Regression :</strong></p><p>This method might sound like cheating but it actually works. Instead of estimating all the pieces like density functions and priors and stick them together to get posterior, we can find a regression that directly estimates the posterior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/246/1*7vW5DXznf3-dZ2MnfzSXig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*zW9WBu4b7k8akVXQofESdQ.png" /></figure><p>And you may have heard these type of models as <strong>Discriminative models.</strong></p><p>Remember that bayes classifier is theoretical which trained using the true underlying population distribution, so it gives optimal decision boundary because it minimizes classification error. But we don’t have access to true population data, hence we estimate. Every model, whether it is generative or discriminative is trying to estimate this optimal bayes classifier in some way. According to law of large numbers, both converges towards the bayes solution as more data is available. In this blog, we’ll have fun by going deep into the classification models that established with this idea and their advantages, disadvantages and use cases.</p><p>The models we’ll be implementing in order:</p><ol><li>Quadratic Discriminant Analysis (QDA)</li><li>Linear Discriminant Analysis (LDA)</li><li>Naive Bayes</li><li>Logistic Regression</li><li>Decision Tree Classifier</li><li>Support Vector Machines (SVM)</li><li>Kernelization</li></ol><p>QDA, LDA, Naive Bayes are generative models and other are discriminative models.</p><p><strong><em>Generative Models :</em></strong></p><p>They’re called <strong>generative</strong> because they don’t just figure out how to separate classes — they actually learn how the data itself is created. They model the full picture, like “what’s the chance this feature appears in this class?” By understanding that, they can classify data or even generate new samples that look real. So, they’re learning how the world works, not just drawing a boundary between classes. That’s why they’re called generative.</p><blockquote>Quadratic Discriminant Analysis (QDA) :</blockquote><p>Perhaps the simplest classification approach is to use density estimation strategy and assume a parametric model for the densities. Suppose that problem is binary classification y ∈ {0, 1} and that f0(x) = f(x | Y = 0) and f1(x) = f1(x| Y = 1) are both multivariate Gaussians:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*lzs34AcmgmjB2tIxb5XpDg.png" /></figure><p>The distributions is centered at the mean of the data points that belongs to class k.</p><p>To use this for modeling, incorporating bayes rule for comparing,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*7Ffqtgl-SG-830Vm7YtyNw.png" /></figure><p>We assign the data point to class 1 if</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*7DIjoHA3S_Q1ExLTXLy4gQ.png" /></figure><p>This is also equivalent to</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/118/1*se4kGFSmY-WWLgtcLQ_5Sg.png" /></figure><p>Taking log for simplicity,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/155/1*uPzpCAujLu3IYEdcFchIVA.png" /></figure><p>Plugging in Gaussian densities and rearranging would give us,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*ljqa3oG9IxbIZOs8OQ1FyQ.png" /></figure><p>where,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*75pEipGDul4xJ03sZoVTXA.png" /></figure><p>This is <strong>Mahalanobis Distance, </strong>the lower the value the closer the data point is to the mean of the distribution, the more likely the data point belongs to that class.</p><p>The term</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/75/1*ymsP9w4c1OLfwxb0hAa9Bg.png" /></figure><p>encodes how prior influence the classification. For example, If x is close to class 1 mean i.e. for a point x, mahalanobis distance to class 1 is lower than class 0 but the class 0 is more common i.e. the prior of class 0 is high, then the term value decreases making the model to prefer class 0.</p><p>The term</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/92/1*v617de11K0Wr6OJdhqc8sw.png" /></figure><p>encodes how the uncertainty influence the classification. If there is high uncertainty in class 1’s data (denominator) increases, making the model to pick class 0.</p><p>Now the model would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*iSfTYOrXuCj9FfC2HsRDZA.png" /></figure><p>Now we know how the decision are being made to classify the data point, unless you wanna go too technical like observing how the prior and uncertainty is effecting our classification you don’t have to use this formula, we can come up with a simpler condition achieving the same performance. Coming from top :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*uxHGxFmgsvCOmQ-681dsYQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*LFFpP7dehpMZBP9W1dDSrQ.png" /></figure><p>Expanding,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*blCn0r129vXER63HqVQeFA.png" /></figure><p>Removing the first term, which is common for all the classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*klyC_LgmiIn5SWESiPz0hA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*Ol9J9oeKBs9iEwpKBkRAqw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*AFHXTkNI8JmLd08DoMgnfw.png" /></figure><p>The decision boundary of the above classifier,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/131/1*F5bQ9L2Cm2lOonvyf2LaIw.png" /></figure><p>It’s called <strong>Quadratic Discriminant Analysis</strong> because the decision boundary between the classes becomes a curve — a quadratic — when the data from the two classes have <strong>different covariances (variances)</strong>.</p><p>To build some intuition, imagine two Gaussian distributions in a one-dimensional space. These distributions represent the data for two different classes. Now, picture that each class has its own spread — meaning their variances are different. One might be narrow and tightly packed, while the other is wider and more spread out.</p><p>Because of this difference in spread, the two bell-shaped curves can cross at more than one point. In fact, depending on how far apart their centers are and how different their spreads are, they could intersect at two points! These crossing points x value’s density is same for both distributions and that’s where the model has to decide which class a new data point belongs to. A straight line wouldn’t be enough to separate the classes in this case — the boundary needs to bend and curve around the regions where the classes overlap in order to correctly classify the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qAeaR-J4vzn2b4WI38fa0Q.png" /></figure><p>Like in our example plot above, the data points less than 1 belong to class 1 as it has higher density, data points in between 1 and 7.5 belongs to class 0 and data points above 7.5 belongs to class 1 again. The curve(black line) might look like this in input space:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_9_viJGVFiKaAeXWCBzUOw.png" /></figure><p>When the variance differs between classes, the equation that defines the boundary includes quadratic terms — where variables are squared — which allows the decision boundary to curve in space. Because of this quadratic nature of the boundary, the method is called <strong>Quadratic Discriminant Analysis</strong>.</p><p>Let’s code this</p><pre>import numpy as np<br>from sklearn.datasets import load_iris<br>from sklearn.model_selection import train_test_split<br><br>iris = load_iris()<br>X = iris.data<br>y = iris.target<br>x_train, x_test, y_train, y_test =  train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)<br><br>print(f&#39;Shape of training data : {x_train.shape}&#39;)<br>print(f&#39;Shape of test data : {x_test.shape}&#39;)<br>print(f&#39;Total number of classes : {len(np.unique(y))}&#39;) # Setosa, Versicolor, Virginica</pre><pre>Shape of training data : (105, 4)<br>Shape of test data : (45, 4)<br>Total number of classes : 3</pre><pre>class_dict = {}<br><br>for i in range(len(np.unique(y))):<br>    <br>    class_data = x_train[y_train == i]<br>    <br>    class_det = np.linalg.det(np.cov(class_data, rowvar=False))<br>    <br>    class_inv = np.linalg.pinv(np.cov(class_data, rowvar=False))<br>    <br>    class_mean = np.mean(class_data, axis=0)<br>    <br>    class_prior = np.sum(y_train == i) / len(y_train)<br>    <br>    class_dict[i]  = {&#39;det&#39;: class_det, &#39;inv&#39;: class_inv, &#39;mean&#39;: class_mean, &#39;prior&#39;: class_prior}<br><br>QDA = lambda i, sample: -0.5 * np.log(class_dict[i][&#39;det&#39;]) - 0.5 * ((sample - class_dict[i][&#39;mean&#39;]).T @ class_dict[i][&#39;inv&#39;] @ (sample - class_dict[i][&#39;mean&#39;])) + np.log(class_dict[i][&#39;prior&#39;])<br><br>h_QDA = lambda x: np.argmax([QDA(i, x) for i in class_dict])<br><br>y_pred = [h_QDA(x) for x in x_train]<br><br>print(f&#39;Total incorrect predictions in train: {np.sum(y_pred != y_train)}&#39;)<br>print(f&#39;Accuracy in train: {np.mean(y_pred == y_train):.2f}&#39;)<br><br>y_test_pred = [h_QDA(x) for x in x_test]<br><br>print(f&#39;Total incorrect predictions in test: {np.sum(y_test_pred != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_test_pred == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in train: 2<br>Accuracy in train: 0.98<br>Total incorrect predictions in test: 1<br>Accuracy in test: 0.98</pre><p>This gave us around 98% of accuracy in Iris dataset. We can use In-built class <strong>QuadraticDiscriminantAnalysis() </strong>to<strong> </strong>avoid implementing from scratch.</p><pre>from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis<br><br>qda = QuadraticDiscriminantAnalysis()<br>qda.fit(x_train, y_train)<br><br>y_pred = qda.predict(x_test)<br>print(f&#39;Total incorrect predictions in test: {np.sum(y_pred != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_pred == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in test: 1<br>Accuracy in test: 0.98</pre><p><strong>Advantages :</strong></p><ul><li>The quadratic decision boundary allows QDA to classify more complex, non-linear patterns.</li><li>Being a parametric model, it provides stable and consistent predictions when the underlying assumptions hold.</li></ul><p><strong>Disadvantages:</strong></p><ul><li>If the data isn’t normally distributed, the model’s accuracy can drop significantly because it relies on that assumption.</li></ul><p><strong>Takeaways from QDA :</strong></p><ul><li>We assumed the distribution the classes came from follows multivariate Gaussian and selected the class which has higher likelihood for a given sample x, since the higher the likelihood the closer the value to the mean of that class.</li><li>We will get the same accuracy even if i use multivariate_normal.pdf(), but the numbers might not be similar because we optimized the likelihood by removing terms that won’t be useful for comparison.</li></ul><blockquote>Linear Discriminant Analysis (LDA) :</blockquote><p>What if data points from both classes have same variance/covariance ?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FOcrusZx3sPv746OXXnV4A.png" /></figure><p>There will only be single contact between two PDF’s. The contact would exactly be at the center of the distance between two classes means, 8.5 in the above graph. Now the model will classify the data points below 8.5 as class 0 and class 1 for above 8.5</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*04BGFM_2dQ9tw7x-Gdlzkw.png" /></figure><p>Mathematically, if we assume same covariance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/93/1*APjS1_CYO3cPlMMHPxFWgA.png" /></figure><p>We know QDA,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*Ol9J9oeKBs9iEwpKBkRAqw.png" /></figure><p>If the covariance is same, we can eliminate first term in the formula</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/1*vKDVLhvQX-u9hQt9ITvJKg.png" /></figure><p>Expanding the likelihood part,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*sf8-aXzD3W1gCiYoFpXvkQ.png" /></figure><p>Notice the first term doesn’t depend of <em>k</em>, it can be ignored for comparison</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*JepAx7ZgQOOmPUd3bgqC-g.png" /></figure><p>The MLE of covariance is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/1*M8w77_ra50RAT46ZxIKcFQ.png" /></figure><p>where,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*xj3yRRISeWLBvRFTuCN6Fg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*biHxU_xcSZvXGa3l6hHjzg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/167/1*TeTFO-t0qIoFszQIYTnepg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/1*6ASz5SdeTokv5h2NqZ8FZQ.png" /></figure><p>Substituting,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*PMwaaenP5XFolyXqKPVS0A.png" /></figure><p>Modeling this with <em>k</em> classes,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/211/1*dKAVdrD91LUgyGNnrtXkpg.png" /></figure><p>This is <strong>Linear Discriminant Analysis (LDA)</strong>.</p><p>Let’s code this,</p><pre>n_samples = x_train[y_train == 0].shape[0] # same no of samples in all classes<br><br>s = lambda j : (x_train[y_train == j] - np.mean(x_train[y_train == j], axis=0)).T @ (x_train[y_train == j] - np.mean(x_train[y_train == j], axis=0)) / n_samples<br><br>S = (n_samples * s(0) + n_samples * s(1) + n_samples * s(2)) / (n_samples * 3)<br><br><br>LDA = lambda i, sample: sample.T @ np.linalg.pinv(S) @ np.mean(x_train[y_train == i], axis=0) \<br>                        - 0.5 * np.mean(x_train[y_train == i], axis=0).T @ np.linalg.pinv(S) @ np.mean(x_train[y_train == i], axis=0) \<br>                        + np.log(np.sum(y_train == i) / len(y_train))<br><br><br>h_LDA = lambda x : np.argmax([LDA(i, x) for i in class_dict])<br><br>y_pred = [h_LDA(x) for x in x_train]<br><br>print(f&#39;Total incorrect predictions in train: {np.sum(y_pred != y_train)}&#39;)<br>print(f&#39;Accuracy in train: {np.mean(y_pred == y_train):.2f}&#39;)<br><br>y_test_pred = [h_LDA(x) for x in x_test]<br><br>print(f&#39;Total incorrect predictions in test: {np.sum(y_test_pred != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_test_pred == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in train: 3<br>Accuracy in train: 0.97<br>Total incorrect predictions in test: 1<br>Accuracy in test: 0.98</pre><p>For quick modeling,</p><pre>from sklearn.discriminant_analysis import LinearDiscriminantAnalysis<br><br>lda = LinearDiscriminantAnalysis()<br>lda.fit(x_train, y_train)<br><br>y_pred = lda.predict(x_test)<br>print(f&#39;Total incorrect predictions in test: {np.sum(y_pred != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_pred == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in test: 1<br>Accuracy in test: 0.98</pre><p><strong>Advantages :</strong></p><ul><li>The linear decision boundary makes it simple and easy to interpret.</li><li>It’s a parametric model, so predictions are stable and computationally efficient.</li><li>Works well when the classes are normally distributed and share the same covariance structure.</li><li>Performs well even with limited data because it estimates fewer parameters compared to more complex models.</li></ul><p><strong>Disadvantages :</strong></p><ul><li>It assumes that all classes have the same covariance matrix, which may not hold in real-world data.</li><li>If the data isn’t normally distributed or the classes overlap a lot, its accuracy can drop.</li><li>The linear boundary can’t capture more complex, non-linear patterns.</li></ul><p><strong>Takeaways from LDA :</strong></p><ul><li>LDA because the decision boundary is a straight line or a hyperplane.</li><li>LDA assumes the classes in the data are linearly separable. Don’t mistake with linear regression which assumes linearity in input and output variables</li><li>Choosing between <strong>QDA and LDA depends on whether the covariance (or variance) across classes is similar or not</strong>. If the covariances are roughly the same, LDA works well. If they’re different, QDA is better because it models each class’s spread separately.</li><li>There’s a test called <strong>Box’s M test</strong> that helps check if the covariances are similar across classes — it’s useful when deciding which model to use.</li></ul><blockquote>Naive Bayes :</blockquote><p>In QDA and LDA we assumed the density function is from known family (multivariate Gaussian) making them parametric models. Another strategy is to estimate density with some non-parametric density estimator, such as <strong>Kernel Density Estimator</strong>. Visit <a href="https://mohith-g.medium.com/moving-past-histograms-smoothing-out-distributions-with-kde-9c5a7700f667"><em>Moving Past Histograms: Smoothing Out Distributions with KDE</em></a> if you want to know more about KDE.</p><p>But the problems that we face with this technique are :</p><ol><li><strong>Kernel Density Estimation (KDE) doesn’t work well in high dimensions</strong> — once you go beyond a few dimensions (say more than 5), the estimation becomes unreliable because data becomes sparse and variance explodes.</li><li><strong>Modeling joint distributions is computationally expensive</strong> — if you want to fully capture dependencies between variables, you need to work with the joint distribution, which grows exponentially with the number of variables, making it infeasible.</li></ol><p>Turns out the solution to both problems is to just assume that each feature is independent. That way, we don’t have to estimate a complex joint distribution or deal with unreliable density estimates in high dimensions (we can compute density function for each variable individually).</p><p>For each group <em>k</em>, compute density estimation for each variable’s data points i.e. if there are <em>k</em> = 3 classes and <em>d</em> = 4 independent variables, then we’ll have total of <em>k*d</em> density functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*AnBTiUZYK-ergcXHihZBaQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*HpeDuDic0qZYroKFxkgdew.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/237/1*563shgnHe2N8H3unOsdTlA.png" /></figure><p>Since we want non-parametric, instead of using multivariate Gaussian to the<em> d</em> dimensional dataset we are using KDE and seeing that it won’t perform well in higher dimension, we assumed independence — resulted in <em>d</em> density functions. We have <em>k</em> classes to work with, so total of <em>k*d</em> density functions.</p><p>It’s called <strong>Naive Bayes</strong> because it <em>naively</em> assumes all features are independent given the class, even though in reality they often aren’t.</p><p>Let’s code this,</p><pre>from statsmodels.nonparametric.kde import KDEUnivariate<br>import pandas as pd<br><br>df_train = pd.DataFrame(np.column_stack((x_train, y_train)), columns=iris.feature_names + [&#39;target&#39;])<br><br>df_class0 = df_train[df_train[&#39;target&#39;] == 0].drop(columns=[&#39;target&#39;])<br>df_class1 = df_train[df_train[&#39;target&#39;] == 1].drop(columns=[&#39;target&#39;])<br>df_class2 = df_train[df_train[&#39;target&#39;] == 2].drop(columns=[&#39;target&#39;])<br><br>kernels = []<br><br>for frame in [df_class0, df_class1, df_class2]:<br>    class_kdes = []<br>    for column in frame.columns:<br>        kde = KDEUnivariate(frame[column])<br>        kde.fit(kernel=&#39;gau&#39;)<br>        class_kdes.append(kde)<br>    kernels.append(class_kdes)<br><br><br>pi = lambda k : np.sum(y_train == k) / len(y_train)<br><br>f = lambda k, x: pi(k) * np.prod([kernels[k][i].evaluate(x[i])[0] for i in range(len(x))])<br><br>classes = np.unique(y_train)<br><br>def h(x):<br>    return max(classes, key=lambda k: f(k, x))<br><br>y_pred = [h(x) for x in x_train]<br><br>print(f&#39;Total incorrect predictions in train: {np.sum(y_pred != y_train)}&#39;)<br>print(f&#39;Accuracy in train: {np.mean(y_pred == y_train):.2f}&#39;)<br><br>y_test_pred = [h(x) for x in x_test]<br><br>print(f&#39;Total incorrect predictions in test: {np.sum(y_test_pred != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_test_pred == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in train: 2<br>Accuracy in train: 0.98<br><br>Total incorrect predictions in test: 3<br>Accuracy in test: 0.93</pre><p>When I first learned about Naive Bayes and saw the problems with assuming all variables are independent or using all of them and ending up with unreliable density estimates, I got an idea. What if we split the variables into smaller sets of highly correlated ones — usually with less than 5 variables each — and then perform KDE within each set? This way, we don’t assume full independence, but we also avoid the density estimation problems in high dimensions. Turns out, this approach has already been explored and is called <strong>Semi-Naive Bayes</strong>.</p><p><strong>Advantages :</strong></p><ul><li>It’s super fast and easy to implement.</li><li>Works surprisingly well even when the independence assumption isn’t fully true.</li><li>Requires less data because it estimates fewer parameters.</li><li>Handles high-dimensional data well since it breaks things down into simpler parts.</li></ul><p><strong>Disadvantages :</strong></p><ul><li>The assumption that features are independent is often unrealistic and can hurt accuracy.</li><li>If the real distribution of the data is far from dependent, the model’s predictions can be off.</li><li>It can struggle when some features are strongly correlated but treated as independent.</li></ul><p><strong>Takeaways from Naive Bayes :</strong></p><ul><li>It works because by assuming independence, we avoid unreliable density estimates in high dimensions and simplify computation.</li><li>Estimating each feature’s density separately makes the problem more manageable while still capturing enough information for classification.</li></ul><p><strong><em>Discriminative Models:</em></strong></p><p>Discriminative models directly estimate the posterior, r(x) = P(Y = 1 | X = x). Logistic Regression is a best example for a discriminative model.</p><blockquote>Logistic Regression :</blockquote><p>We use a <strong>regression function</strong> in logistic regression because we want a way to combine the input features and predict something about the outcome — in this case posterior, the probability of a class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*zwo-NijuPCe4Y10Pt8elag.png" /></figure><p>But, the above equation is invalid. <em>P </em>is probability where 0 ≤ <em>P</em> ≤ 1. The regression function returns values in the range of (-inf, inf). For this specific reason <strong>we assume the relationship between the input variables and the log-odds of the outcome is linear</strong>,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*KjA6wIuJ36J89mFYiU_lzw.png" /></figure><p>log-odds because it spans through all the real numbers and it is mathematically convenient.</p><p>Until now, we assumed whatever the scalar that comes out of a regression function is a log-odds value. However, our goal is to get probability <em>P</em> not the log-odds, since the log-odds is derived from <em>p</em>, we inverse it to get <em>p :</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/143/1*xKQJ-dC9AamHyueXb0kxig.png" /></figure><p>Exponentiate both sides to remove log,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/97/1*O2Nzbpegl4uLyESUcGeU8w.png" /></figure><p>multiply both sides with (<em>1-p)</em> to isolate p,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/126/1*1MKLRjJNZFn2QjppH7-VEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/119/1*EHEzZ-Ilg2lEHXcwSsemhg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/118/1*gC2RqPsRIbAnTdCM50AxMw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/131/1*mVUu_bDQtYEiKiW5t5WBGA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/98/1*F-pNI63-znSZLADw0KwAWw.png" /></figure><p>This is sigmoid function. To make it easy to interpret, divide the numerator and denominator with exponent of <em>z</em> ,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/104/1*6ww-6ytJkPR55zInPrK4aA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/110/1*bJBIf9R4nVYMtQekKidw4A.png" /></figure><p>You might have seen this in textbooks, more like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*t_SxSdtk6NbJV7w7BHBpVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*XeddfnPImqF60cQIWrI0SQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*xcIHQML3odaEMuhfLoTvCw.png" /></figure><p>Now, we mapped the linear combination of input features to probability, it’s time to find weights that best fits the data.</p><p>If the problem is binary classification, we just need a single linear model as there are only 2 classes to classify. Unlike regression, we can’t be using RSS or MSE errors due to categorical outputs. Since y ∈ {0, 1} we assume the random variable follows Bernoulli Distribution i.e.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/1*YYT3pyo8ptOZa_iS3PHV-w.png" /></figure><p>where we compute,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*D3cACe9m5Skvkppejo2M6A.png" /></figure><p>the likelihood of observing accurate binary output depends on getting accurate probability for the data points, getting accurate probabilities depends on optimal weights β0, β values (β is a vector containing β1, β2, …) The likelihood of observing data given β0, β values is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*PwQmW-3-noJ51o470fP0qQ.png" /></figure><p>Applying log for simplicity and to avoid numeric underflow,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*o89kkiHMBvMj2wEAILMMrA.png" /></figure><p>Turning this into negative log-likelihood because it aligns with our update rule and it’s convenient to minimize,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/1*HnWLYyA2xRMN2OBxAP-gFw.png" /></figure><p>In log-likelihood, if the real output yi = 1, then for predicting higher probability</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/246/1*RZ3zh69-NfZhf8YpGNKmiw.png" /></figure><p>And predicting lower probability for being 1 even though it is 1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*kmvC1rUX5ukU4INibvXh2w.png" /></figure><p>The log-likelihood increased when it gave higher probability for correct class and decreased when probability got lower. Turing into negative log-likelihood</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/1*S-yFuvayQSRRzM9MLjnScg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*bNiVkxpmPxf88-uUrEQieQ.png" /></figure><p>This is the reason we assume negative log-likelihood as a loss function, this particular loss function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/1*HnWLYyA2xRMN2OBxAP-gFw.png" /></figure><p>is called Binary cross-entropy used for binary classification.</p><p>Now, taking derivative w.r.t β0, β would result in the slopes of the tangent passing through that loss function at β0, β points. After performing derivative w.r.t β0, β</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/181/1*jp_E-MzlRb44YxysR2NOOw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/169/1*fCelA7-bvoK5sTUupSFZDQ.png" /></figure><p>Now we keep on updating β0, β values until they converge to global minima. And using negative log-likelihood here helps it reach global minima no matter how much it oscillates</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/130/1*XcAlej76w135gm7fYVTVyA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/151/1*QCTPeQSyidswDnpuKX5CTw.png" /></figure><p>The multiplying variable we see is eta, seventh letter of Greek alphabet, we use it to represent learning rate, if the learning rate is too high (like 1), the updates take big steps and the model keeps overshooting, making it harder to settle. If it’s too low (like 0.0001), the steps are tiny and it takes forever to converge. So, it shouldn’t be too high or too low — it should be somewhere in between. A good place to start is 0.01, and since it’s a hyperparameter, we can adjust it depending on what we’re trying to achieve.</p><p>What if the problem is not a binary classification and it has more than 2 classes ? There are two most common approaches for multi-class classification</p><ol><li><strong>One-vs-All / One-vs-Rest (OvA / OvR)</strong></li></ol><p>If the problem has <em>k</em> classes, we build <em>k</em> binary classification models and select a model’s output which predicts with higher confidence. For example, in Iris dataset there are 3 classes Setosa, Versicolor and Virginica. We can generate 3 datasets containing Setosa vs not-Setosa, Versicolor vs not-Versicolor, Virginica vs not-Virginica and train 3 binary classification models.</p><p>One problem with this method is that we could experience some ties as a model won’t adjust its confidence level related to other models.</p><p>2. <strong>Multinomial</strong></p><p>Instead of training k separate models, we can combine them into a single problem by using matrix multiplication. For example, if X has a shape of (n, d), instead of using one weight vector, we can column stack k weight vectors with shape (d, k), which gives us an output of shape (n, k). It’s like asking a data point for its opinion on all k classes at once. After that, we pass this k-dimensional output through the softmax function.</p><p>The softmax function takes these raw scores and turns them into probabilities. It basically highlights which class has the highest score while keeping everything proportional. The higher a score, the higher its probability after softmax, but all the outputs sum to 1. This way, we can interpret the result as the model’s confidence for each class.</p><p>Now, once we have these probabilities, we can compare them with the true labels using a loss function like categorical cross-entropy. This helps the model learn which class it should be more confident about. By combining everything into a single matrix operation, training becomes faster and more efficient, especially when dealing with a large number of classes.</p><p>The optimization and update template is same for the multinomial too, however this is not a binary classification anymore so the loss function becomes,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*xSLgPWD6dxrNHoMb28Bedw.png" /></figure><p>And</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*asethuystgBnQh0Io-ZM-g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/1*CfXqXjVwtkEQ6ZX3d8Rb-A.png" /></figure><p>These two are most commonly used approaches but there are also other approaches like One-vs-One, Error-Correcting Output Codes, and more.</p><p>Let’s code them,</p><pre>n_classes = len(np.unique(y_train))<br>df_train = pd.DataFrame(np.column_stack((x_train, y_train)), columns=iris.feature_names + [&#39;target&#39;])<br># One-vs-Rest<br><br># Model 1 : Setosa vs not-Setosa<br><br>epsilon = 1e-5<br>max_iter = 1000<br>batch_size = 5<br>lr = 0.01<br><br>w = np.zeros((df_train.shape[1] - 1, 1))<br>b = 0.0<br>w_old = w.copy()<br>b_old = b<br><br>for i in range(max_iter):<br>    <br>    df_train = df_train.sample(frac=1, random_state=i).reset_index(drop=True)<br>    batches = np.array_split(df_train, batch_size)<br>    <br>    epoch_loss = 0<br><br>    for batch in batches:<br>        y_train_class0 = (batch[&#39;target&#39;] == 0).astype(float).to_numpy().reshape(-1, 1)<br>        x_batch = batch.drop(columns=[&#39;target&#39;]).to_numpy()<br>        <br>        linear_model = np.dot(x_batch, w) + b<br>        linear_model = np.clip(linear_model, -500, 500) # prevent overflow<br>        y_predicted = 1 / (1 + np.exp(-linear_model)) # probabilities<br><br>        error = y_predicted - y_train_class0<br>        dw = (1 / x_batch.shape[0]) * np.dot(x_batch.T, error) # average gradient for stable updates<br>        db = (1 / x_batch.shape[0]) * np.sum(error) # average gradient for stable updates<br>        <br>        w -= lr * dw<br>        b -= lr * db<br><br>        loss = -np.mean(<br>            y_train_class0 * np.log(y_predicted + 1e-9) + <br>            (1 - y_train_class0) * np.log(1 - y_predicted + 1e-9)<br>        )<br>        epoch_loss += loss<br> <br>    epoch_loss /= len(batches)<br>    <br>    if i % 10 == 0:<br>        print(f&quot;Iteration {i+1}, Loss: {epoch_loss:.3f}&quot;)<br>    <br>    if np.allclose(w, w_old, rtol=1e-5, atol=1e-8) and np.isclose(b, b_old, rtol=1e-5, atol=1e-8):<br>        print(f&quot;Converged after {i+1} iterations&quot;)<br>        break<br>    <br>    w_old = w.copy()<br>    b_old = b</pre><pre>Iteration 991, Loss: 0.014</pre><pre># Model 2 : Versicolor vs not-Versicolor<br><br>w2 = np.zeros((df_train.shape[1] - 1, 1))<br>b2 = 0.0<br>w2_old = w2.copy()<br>b2_old = b2<br><br>for i in range(max_iter):<br>    <br>    df_train = df_train.sample(frac=1, random_state=i).reset_index(drop=True)<br>    batches = np.array_split(df_train, batch_size)<br>    <br>    epoch_loss = 0<br><br>    for batch in batches:<br>        y_train_class1 = (batch[&#39;target&#39;] == 1).astype(float).to_numpy().reshape(-1, 1)<br>        x_batch = batch.drop(columns=[&#39;target&#39;]).to_numpy()<br>        <br>        linear_model = np.dot(x_batch, w2) + b2<br>        linear_model = np.clip(linear_model, -500, 500) # prevent overflow<br>        y_predicted = 1 / (1 + np.exp(-linear_model)) # probabilities<br><br>        error = y_predicted - y_train_class1<br>        dw = (1 / x_batch.shape[0]) * np.dot(x_batch.T, error)# average gradient for stable updates<br>        db = (1 / x_batch.shape[0]) * np.sum(error)# average gradient for stable updates<br>        <br>        w2 -= lr * dw<br>        b2 -= lr * db<br><br>        loss = -np.mean(<br>            y_train_class1 * np.log(y_predicted + 1e-9) + <br>            (1 - y_train_class1) * np.log(1 - y_predicted + 1e-9)<br>        )<br>        epoch_loss += loss<br> <br>    epoch_loss /= len(batches)<br>    <br>    if i % 10 == 0:<br>        print(f&quot;Iteration {i+1}, Loss: {epoch_loss:.3f}&quot;)<br><br>    if np.allclose(w2, w2_old, rtol=1e-5, atol=1e-8) and np.isclose(b2, b2_old, rtol=1e-5, atol=1e-8):<br>        print(f&quot;Converged after {i+1} iterations&quot;)<br>        break<br>    <br>    w2_old = w2.copy()<br>    b2_old = b2</pre><pre>Iteration 991, Loss: 0.513</pre><pre># Model 3 : Virginica vs not-Virginica<br><br>w3 = np.zeros((df_train.shape[1] - 1, 1))<br>b3 = 0.0<br>w3_old = w3.copy()<br>b3_old = b3<br><br>for i in range(max_iter):<br>    <br>    df_train = df_train.sample(frac=1, random_state=i).reset_index(drop=True)<br>    batches = np.array_split(df_train, batch_size)<br>    <br>    epoch_loss = 0<br><br>    for batch in batches:<br>        y_train_class2 = (batch[&#39;target&#39;] == 2).astype(float).to_numpy().reshape(-1, 1)<br>        x_batch = batch.drop(columns=[&#39;target&#39;]).to_numpy()<br>        <br>        linear_model = np.dot(x_batch, w3) + b3<br>        linear_model = np.clip(linear_model, -500, 500) # prevent overflow<br>        y_predicted = 1 / (1 + np.exp(-linear_model)) # probabilities<br><br>        error = y_predicted - y_train_class2<br>        dw = (1 / x_batch.shape[0]) * np.dot(x_batch.T, error)# average gradient for stable updates<br>        db = (1 / x_batch.shape[0]) * np.sum(error)# average gradient for stable updates<br>        <br>        w3 -= lr * dw<br>        b3 -= lr * db<br><br>        loss = -np.mean(<br>            y_train_class2 * np.log(y_predicted + 1e-9) + <br>            (1 - y_train_class2) * np.log(1 - y_predicted + 1e-9)<br>        )<br>        epoch_loss += loss<br> <br>    epoch_loss /= len(batches)<br>    <br>    if i % 10 == 0:<br>        print(f&quot;Iteration {i+1}, Loss: {epoch_loss:.3f}&quot;)<br><br>    if np.allclose(w3, w3_old, rtol=1e-5, atol=1e-8) and np.isclose(b3, b3_old, rtol=1e-5, atol=1e-8):<br>        print(f&quot;Converged after {i+1} iterations&quot;)<br>        break<br><br>    w3_old = w3.copy()<br>    b3_old = b3</pre><pre>Iteration 991, Loss: 0.160</pre><pre>def OvR(x):<br>    probs = [<br>        1 / (1 + np.exp(-(np.dot(x, w) + b))),<br>        1 / (1 + np.exp(-(np.dot(x, w2) + b2))),<br>        1 / (1 + np.exp(-(np.dot(x, w3) + b3)))<br>    ]<br>    return np.argmax(probs)<br><br><br>y_train_ovr = [OvR(x) for x in x_train]<br><br>print(f&#39;Total incorrect predictions in train: {np.sum(y_train_ovr != y_train)}&#39;)<br>print(f&#39;Accuracy in train: {np.mean(y_train_ovr == y_train):.2f}&#39;)<br><br>y_test_ovr = [OvR(x) for x in x_test]<br><br>print(f&#39;Total incorrect predictions in test: {np.sum(y_test_ovr != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_test_ovr == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in train: 3<br>Accuracy in train: 0.97<br>Total incorrect predictions in test: 3<br>Accuracy in test: 0.93</pre><p>Using built-in class,</p><pre>from sklearn.linear_model import LogisticRegression<br>from sklearn.multiclass import OneVsRestClassifier<br><br>base_lr = LogisticRegression(solver=&#39;liblinear&#39;, random_state=42, max_iter=1000)<br><br>ovr_model = OneVsRestClassifier(base_lr)<br><br>ovr_model.fit(x_train, y_train)<br><br>y_pred = ovr_model.predict(x_train)<br>y_test_pred = ovr_model.predict(x_test)<br><br>print(f&#39;Total incorrect predictions in train: {np.sum(y_pred != y_train)}&#39;)<br>print(f&#39;Accuracy in train: {np.mean(y_pred == y_train):.2f}&#39;)<br>print(f&#39;Total incorrect predictions in test: {np.sum(y_test_pred != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_test_pred == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in train: 4<br>Accuracy in train: 0.96<br>Total incorrect predictions in test: 4<br>Accuracy in test: 0.91</pre><p>The small difference in incorrect predictions and accuracy can happen because of small differences in how we implement things, like how we handle gradients, shuffle data, or check for convergence.</p><pre># Multinomial<br><br>max_iter = 1000<br>batch_size = 5<br>lr = 0.01<br>n_classes = len(np.unique(y_train))<br><br>w_multinomial = np.zeros((df_train.shape[1] - 1, n_classes))<br>b_multinomial = np.zeros((1, n_classes))<br><br>w_old = np.zeros_like(w_multinomial)<br>b_old = np.zeros_like(b_multinomial)<br><br>for i in range(max_iter):<br>    df_train = df_train.sample(frac=1, random_state=i).reset_index(drop=True)<br>    batches = np.array_split(df_train, batch_size)<br>    <br>    epoch_loss = 0<br>    <br>    for batch in batches:<br>        y_train_multiclass = batch[&#39;target&#39;].to_numpy().reshape(-1, 1)<br>        x_batch = batch.drop(columns=[&#39;target&#39;]).to_numpy()<br>        <br>        <br>        y_onehot = np.zeros((x_batch.shape[0], n_classes))<br>        y_onehot[np.arange(x_batch.shape[0]), y_train_multiclass.flatten().astype(int)] = 1<br>        <br>        linear_model = np.dot(x_batch, w_multinomial) + b_multinomial<br>        linear_model = np.clip(linear_model, -500, 500)<br>        probs = np.exp(linear_model) / np.sum(np.exp(linear_model), axis=1, keepdims=True)<br>        <br>       <br>        dw = (1 / x_batch.shape[0]) * np.dot(x_batch.T, (probs - y_onehot))<br>        db = (1 / x_batch.shape[0]) * np.sum(probs - y_onehot, axis=0, keepdims=True)<br>        <br>        w_multinomial -= lr * dw<br>        b_multinomial -= lr * db<br>        <br>        <br>        loss = -np.mean(np.sum(y_onehot * np.log(probs + 1e-9), axis=1))<br>        epoch_loss += loss<br>    <br>    epoch_loss /= len(batches)<br>    <br>    if i % 10 == 0:<br>        print(f&quot;Iteration {i+1}, Loss: {epoch_loss:.3f}&quot;)<br>    <br>    if np.allclose(w_multinomial, w_old, rtol=1e-5, atol=1e-8) and np.allclose(b_multinomial, b_old, rtol=1e-5, atol=1e-8):<br>        print(f&quot;Converged after {i+1} iterations&quot;)<br>        break<br>    <br>    w_old = w_multinomial.copy()<br>    b_old = b_multinomial.copy()</pre><pre>Iteration 991, Loss: 0.170</pre><pre>linear_model = np.dot(x_train, w_multinomial) + b_multinomial<br>linear_model = linear_model - np.max(linear_model, axis=1, keepdims=True)  # stability trick<br>probs = np.exp(linear_model) / np.sum(np.exp(linear_model), axis=1, keepdims=True)<br><br>y_pred_multinomial = np.argmax(probs, axis=1)<br>print(f&#39;Total incorrect predictions in train: {np.sum(y_pred_multinomial != y_train)}&#39;)<br>print(f&#39;Accuracy in train: {np.mean(y_pred_multinomial == y_train):.2f}&#39;)<br><br>linear_model = np.dot(x_test, w_multinomial) + b_multinomial<br>linear_model = linear_model - np.max(linear_model, axis=1, keepdims=True)<br>probs = np.exp(linear_model) / np.sum(np.exp(linear_model), axis=1, keepdims=True)<br><br>y_pred_multinomial = np.argmax(probs, axis=1)<br>print(f&#39;Total incorrect predictions in test: {np.sum(y_pred_multinomial != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_pred_multinomial == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in train: 2<br>Accuracy in train: 0.98<br>Total incorrect predictions in test: 1<br>Accuracy in test: 0.98</pre><pre>lr_multinomial = LogisticRegression(<br>    multi_class=&#39;multinomial&#39;,<br>    solver=&#39;lbfgs&#39;,    <br>    max_iter=1000,<br>    random_state=42<br>)<br><br>lr_multinomial.fit(x_train, y_train)<br><br>y_train_pred = lr_multinomial.predict(x_train)<br>train_acc = np.mean(y_train_pred == y_train)<br>print(f&#39;Total incorrect predictions in train: {np.sum(y_train_pred != y_train)}&#39;)<br>print(f&quot;Training Accuracy: {train_acc:.2f}&quot;)<br><br>y_test_pred = lr_multinomial.predict(x_test)<br>test_acc = np.mean(y_test_pred == y_test)<br>print(f&#39;Total incorrect predictions in test: {np.sum(y_test_pred != y_test)}&#39;)<br>print(f&quot;Test Accuracy: {test_acc:.2f}&quot;)</pre><pre>Total incorrect predictions in train: 3<br>Training Accuracy: 0.97<br>Total incorrect predictions in test: 3<br>Test Accuracy: 0.93</pre><p><strong>Advantages :</strong></p><ul><li>Gives us probabilities, so we know how confident the model is.</li><li>Works well when the relationship is roughly linear.</li><li>Fast to train and doesn’t need much computation.</li></ul><p><strong>Disadvantages</strong>:</p><ul><li>Assumes a linear relationship between features and the output, which might not always be true.</li><li>Doesn’t work well with complex or non-linear boundaries unless we manually create new features. The reason behind high loss with Model 2 in OvR</li></ul><p><strong>Takeaway from Logistic Regression :</strong></p><ul><li>We assumed that the relationship between the features and the target’s log-odds is linear. Because that assumption holds well in this dataset, the model is able to separate the classes effectively and gives good accuracy.</li><li>We can add polynomial features to the dataset and send the log-odds to sigmoid or softmax. This can capture more complex relations. Watch out for overfitting, we can use regularization forms like L1, L2 penalties and tuning hyperparameters to avoid over fitting.</li><li>So, it’s not just about the algorithm — it’s about how well the data fits the assumptions behind it.</li></ul><blockquote>Decision Tree Classifier:</blockquote><p>Decision Trees partition the input space into disjoint pieces and classify observations according to which partition element they fall in.</p><p>Suppose we have two variables, x1 = Age and x2 = Blood Pressure, and a binary classification problem where y∈{0,1}. The algorithm iterates through the input variables to find a split condition <em>t</em>, such as x1 &lt; 50 or x2​ &lt; 110, that divides the data into two subsets: A1=(−∞,t] and A2=(t,∞). Then the proportion of data samples, for split <em>s</em> and class <em>j</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*01C4xEr0xiIiQixrVY427g.png" /></figure><p>We use these proportions to measure the impurity of the set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/1*bW8DCB8CqTVgMW1rlofXFQ.png" /></figure><p>The total impurity resulting from the split is computed as the weighted sum of the impurities of the two subsets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*9qX9wWOOzcLaTv8vl36wCA.png" /></figure><p>This particular measure of impurity is known as <strong>Gini index. </strong>There is also another technique that measures impurity called <strong>Entropy</strong>, It’s theory grounded in information theory. But Gini and Entropy both measures impurity and they often pick the same or very similar splits.</p><p>Let’s code this,</p><pre>class TreeNode:<br>    def __init__(self, feature=None, threshold=None, left=None, right=None, value=None):<br>        self.feature = feature<br>        self.threshold = threshold<br>        self.left = left<br>        self.right = right<br>        self.value = value<br><br>def compute_impurity(y_left, y_right, classes):<br><br>    def split_impurity(split):<br>        proportions = [np.sum(split == c) / len(split) for c in classes]<br>        return 1 - sum(p ** 2 for p in proportions)<br>    <br>    total_size = len(y_left) + len(y_right)<br>    impurity = (len(y_left) * split_impurity(y_left) + len(y_right) * split_impurity(y_right)) / total_size<br>    return impurity<br><br>def build_tree(X, y, depth=0, max_depth=None, min_samples_split=2):<br>    <br>    if len(y) &lt; min_samples_split or (max_depth is not None and depth &gt;= max_depth):<br>        leaf_value = np.bincount(y).argmax()<br>        return TreeNode(value=leaf_value)<br>    <br>    best_feature = None<br>    best_threshold = None<br>    best_impurity = float(&#39;inf&#39;)<br>    classes = np.unique(y)<br><br>    for j in range(X.shape[1]):<br>        sorted_indices = np.argsort(X[:, j])<br>        X_sorted = X[sorted_indices]<br>        y_sorted = y[sorted_indices]<br><br>        for i in range(1, len(y_sorted)):<br>            threshold = (X_sorted[i-1, j] + X_sorted[i, j]) / 2<br>            left_mask = X_sorted[:, j] &lt;= threshold<br>            right_mask = X_sorted[:, j] &gt; threshold<br><br>            y_left = y_sorted[left_mask]<br>            y_right = y_sorted[right_mask]<br><br>            if len(y_left) == 0 or len(y_right) == 0:<br>                continue<br><br>            impurity = compute_impurity(y_left, y_right, classes)<br><br>            if impurity &lt; best_impurity:<br>                best_impurity = impurity<br>                best_feature = j<br>                best_threshold = threshold<br>    <br>    if best_feature is None:<br>        leaf_value = np.bincount(y).argmax()<br>        return TreeNode(value=leaf_value)<br>    <br>    left_mask = X[:, best_feature] &lt;= best_threshold<br>    right_mask = X[:, best_feature] &gt; best_threshold<br><br>    left_subtree = build_tree(X[left_mask], y[left_mask], depth+1, max_depth, min_samples_split)<br>    right_subtree = build_tree(X[right_mask], y[right_mask], depth+1, max_depth, min_samples_split)<br><br>    return TreeNode(feature=best_feature, threshold=best_threshold, left=left_subtree, right=right_subtree)</pre><pre>tree = build_tree(x_train, y_train, max_depth=4)<br><br>def predict_tree(node, sample):<br>    if node.value is not None:<br>        return node.value<br>    <br>    if sample[node.feature] &lt;= node.threshold:<br>        return predict_tree(node.left, sample)<br>    else:<br>        return predict_tree(node.right, sample)<br><br>def predict(X, tree):<br>    return np.array([predict_tree(tree, sample) for sample in X])<br><br>y_pred = predict(x_train, tree)<br>y_test_pred = predict(x_test, tree)<br><br>print(f&#39;Total incorrect predictions in train: {np.sum(y_pred != y_train)}&#39;)<br>print(f&#39;Accuracy in train: {np.mean(y_pred == y_train):.2f}&#39;)<br>print(&#39;--&#39;*30)<br>print(f&#39;Total incorrect predictions in test: {np.sum(y_test_pred != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_test_pred == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in train: 1<br>Accuracy in train: 0.99<br>------------------------------------------------------------<br>Total incorrect predictions in test: 4<br>Accuracy in test: 0.91</pre><p>We can use built-in class</p><pre>from sklearn.tree import DecisionTreeClassifier<br><br>tree = DecisionTreeClassifier(max_depth=4, min_samples_split=2, criterion=&#39;gini&#39;, random_state=42)<br>tree.fit(x_train, y_train)<br><br>y_pred = tree.predict(x_train)<br>print(f&#39;Total incorrect predictions in train: {np.sum(y_pred != y_train)}&#39;)<br>print(f&#39;Accuracy in train: {np.mean(y_pred == y_train):.2f}&#39;)<br>print(&#39;--&#39;*30)<br>y_pred = tree.predict(x_test)<br>print(f&#39;Total incorrect predictions in test: {np.sum(y_pred != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_pred == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in train: 1<br>Accuracy in train: 0.99<br>------------------------------------------------------------<br>Total incorrect predictions in test: 5<br>Accuracy in test: 0.89</pre><p><strong>Advantages :</strong></p><ul><li>Easy to see what’s happening. You can literally look at the splits and follow the decision.</li><li>Works for both regression and classification.</li><li>Captures non-linear patterns.</li></ul><p><strong>Disadvantages :</strong></p><ul><li>Overfits easily if tree gets too deep</li><li>Even a small change in data may shake the entire split conditions, resulting in completely different tree</li></ul><p><strong>Takeaway from Decision Tree:</strong></p><p>Decision tree works because it splits on features to reduce impurity (like Gini or entropy) at each node. It’s greedy — picks the best local split, repeats recursively. No assumptions about linearity or distribution, so the model structure comes from data itself, and purity-based splits capture patterns efficiently without complex formulas.</p><blockquote>Support Vector Machines (SVM) :</blockquote><p>Support Vector Machines is a class of linear classifier. Assume the problem is binary classification, it is convenient to label outcomes -1 and +1 instead of 0 and 1 in SVM. So, the linear classifier can be written as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/1*ypXAjOSAmoV4MmzVNnMS1w.png" /></figure><p>Where x is a <em>d</em> dimensional vector.</p><p>If the data is linearly separable, there exist many separating hyperplanes. How should we choose one ?</p><p>Intuitively, it seems reasonable to <strong>choose the hyperplane in the middle that is “furthest” from the data on both sides, in the sense that if separates -1&#39;s and +1’s and maximizes the distance to the closest data points. This hyperplane is called Maximum Margin Hyperplane.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CEVIenyMlk6il8VEJOl7bA.png" /></figure><p>The margin is the distance from the hyperplane to the nearest data points. Points on the boundary of the margin are called <strong>support vectors</strong>.</p><p>The SVM finds a decision boundary like this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/121/1*lVDQK2WQlRvbJN_AJ86RRg.png" /></figure><p>The margin is the distance between the two parallel lines where the support vectors lie. These lines are</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*9y0u9uPzWs57lvTCMoB0oQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/137/1*GyDUEY5kJ7IYU8vztIwlAQ.png" /></figure><p>The distance between two parallel lines,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/1*_p-qW2OyBhCv_N7YC0oCgQ.png" /></figure><p>So the margin is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/108/1*R9ZZ75D6frHTfYU7wt2Zeg.png" /></figure><p>Our goal is to find a hyperplane that maximizes this margin, which is also equivalent to minimize</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/131/1*kx5rk0lPjOmOd_mvk_vt2A.png" /></figure><p>It’s just mathematically easier to minimize instead of maximizing. Note that the margin could be any positive number so for simplicity we consider unit value so we can scale without any issue</p><p>Now, we have two constraints that our classifier needs to follow</p><ol><li>The class prediction should be accurate, means data points should be on the side where they belong.</li><li>These data points should be outside or on the margin but not inside</li></ol><p>Mathematically, we can formulate these two conditions into a single expression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/117/1*J9hHCM6Vx8XZcajY2hEzWQ.png" /></figure><p>If Yi = 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/97/1*tR_JPJeIxdCqtQ4uJ-r2dA.png" /></figure><p>Saying that positive class points must be on or outside the margin</p><p>If Yi = -1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/114/1*kmgOtJZSL-3D6G_P_mETlA.png" /></figure><p>Saying the negative class points must be on or outside the margin or opposite side of positive data points</p><p>Normally to get the weights <em>a, a0 </em>we can simply use derivative on the minimization term. But we have a constraint to respect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/117/1*J9hHCM6Vx8XZcajY2hEzWQ.png" /></figure><p>This<strong> means we need to maximize the margin such that all the data points are in there class groups and on or outside the margin.</strong></p><p>We can use Lagrangian method for this situation, unlike simply taking derivatives, <strong>it’s a constrained optimization technique</strong></p><p>Lagrangian gives you a single equation to work with by embedding both the goal and the constraint</p><p>Lagrangian multiplier α ≥ 0 , α tells us which points matter and how strongly they affect the final model. Since, there are n data points we have n constraints (one for each data point). So, αi ≥ 0 for i∈{1,…,n}</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*zzUvONzu0qi12-gn_xTeIw.png" /></figure><p>The first term is what we want to minimize, the second term penalizes constraint violations using lagrangian multiplier α. Combining the objective and constraint</p><p>So now the goal is to minimize the norm of <em>a</em>, which in turn maximizes the margin. We can do this by finding both</p><ul><li>a, controls the orientation and scaling of the hyperplane</li><li>a0, controls the shift or offset of the hyperplane</li></ul><p>We also want to maximize the lagrangian multiplier because we want high penalty to the data points which won’t obey the constraint.</p><p>After performing derivative to lagrangian term w.r.t <em>a</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/238/1*8zuhlxdnuVM4nnWrfz6k1g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/1*F3Zp__y2919QQtQDx6F25g.png" /></figure><p>w.r.t a0</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/198/1*-WqVMa7KboQ61OddUX-png.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/115/1*Ke2oRCnFj12rjjxCoDVvlA.png" /></figure><p>This leads to a balance constraint on the αi​ across classes. It’s something the lagrangian needs to respect so that the multipliers stay symmetric and the solution behaves properly. Intuitively, <strong>once you set it up this way, only the support vectors end up with non-zero αi the rest don’t affect the boundary because they’re already correctly classified and far from the margin.</strong></p><p>For example, say you have 100 data points from class ‘-1’ with 20 support vectors, and 500 from class ‘1’ with 5 support vectors. The lagrangian is designed so that only these support vectors contribute, and the rest have αi=0. Now, the non-zero multipliers must balance each other so their total effect sums to zero. So, the multipliers for class ‘-1’ could be 0.5 each (0.5 × 20 = 10), and for class ‘1’ they could be 2 each (2 × 5 = 10), making the sums equal. This symmetry ensures that the boundary stays centered and doesn’t favor one class over the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/125/1*hXMO1YKBLoK2wpVZ9rUAmw.png" /></figure><p>So basically, this constraint and symmetry happen because only the points on the margin matter, and the lagrangian sets it up so their combined influence is balanced between classes.</p><p>Now, we know how to compute a, a0. Substituting these two equations back into lagrangian term and let’s maximize it w.r.t α to get αi values. So we can make predictions,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*BkQuLiGatc3Zx0xyRpZWHQ.png" /></figure><p>Subject to,</p><p>This is called dual problem because it’s derived from the original optimization problem by focusing on the lagrangian multipliers. So, now this is an optimization problem depending only on α with constraints</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*cTG1cP6BDTL1kACD1BGBIw.png" /></figure><p>It can be solved by numerical solver like cvxopt library. It is used to solve convex optimizations and QP(Quadratic Programming) is one of the main problem types it handles. It provide solvers that handle problems like :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*e1UZl7YkAyLARXDc1dmy3g.png" /></figure><p>with constraints,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*fyXNeoZLUjVBNw6M0xmw7w.png" /></figure><p>Our QP and constraints are</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*BkQuLiGatc3Zx0xyRpZWHQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*cTG1cP6BDTL1kACD1BGBIw.png" /></figure><p>Now converting our QP to the format the cvxopt solves result in,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/1*JdviNMXPSyPkDRj99_3Odg.png" /></figure><p>Assuming,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/54/1*TDc2t4ByGlBbh7P2Wj683w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*UfHATnqCpeD85-r6tunv0g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/65/1*Ud7MzGZniLcC3eNIw9D33g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/132/1*FboCtU13YnruAwZSsX9-yw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/62/1*IlleOsBUJj0Ir8E48yO42g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/1*aHlRAbmXvIu0My7detKXdw.png" /></figure><p>For the constraints,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*cTG1cP6BDTL1kACD1BGBIw.png" /></figure><p>to turn into the format of</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*fyXNeoZLUjVBNw6M0xmw7w.png" /></figure><p>The new values would be,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/73/1*8h6YWKRBTAQHrFshcte2Ow.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/54/1*TDc2t4ByGlBbh7P2Wj683w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/132/1*FboCtU13YnruAwZSsX9-yw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/62/1*IlleOsBUJj0Ir8E48yO42g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/1*aHlRAbmXvIu0My7detKXdw.png" /></figure><p>Substituting this new values and performing cvxopt, will result in α values. Remember only α values of Support Vectors(SV) will be greater than 0.</p><p>We have α values, we know how to compute a, a0. Now, we can make predictions by,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/143/1*v_EEXC6zavFEgp2ids6suA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/1*9qh0NqKoyKsjmX-SmgiJTQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/159/1*Mr88J3Hq6pTADuuJuINZnQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/177/1*sSjvYkiNt2TEVnFlmGvZYQ.png" /></figure><p>Let’s code this,</p><p>Note, since Iris dataset is multi-class we’ll be using One-vs-One model. Since the Iris dataset has 3 classes, the One-vs-One approach will train 3(3−1)/2 = 3 binary SVM classifiers each distinguishing a pair of classes.</p><pre>from cvxopt import matrix, solvers<br><br>df = pd.DataFrame(np.column_stack((x_train, y_train)), columns=iris.feature_names + [&#39;target&#39;])<br><br><br>data1 = df[df[&#39;target&#39;] != 2.0]<br>data1[&#39;target&#39;].replace({0.0: 1, 1.0: -1}, inplace=True) # setosa = 1 vs versicolor = -1<br><br>data2 = df[df[&#39;target&#39;] != 1.0]<br>data2[&#39;target&#39;].replace({0.0: 1, 2.0: -1}, inplace=True) # setosa = 1 vs virginica = -1<br><br>data3 = df[df[&#39;target&#39;] != 0.0]<br>data3[&#39;target&#39;].replace({1.0: 1, 2.0: -1}, inplace=True) # versicolor = 1 vs virginica = -1</pre><pre># model 1 for setosa vs versicolor<br><br>X1 = data1.drop(columns=[&#39;target&#39;]).to_numpy()<br>y1 = data1[&#39;target&#39;].to_numpy()<br>n1, d1 = X1.shape<br><br>C = 1.0<br>K = X1 @ X1.T<br>P = matrix(np.outer(y1, y1) * K)<br>q = matrix(-1 * np.ones(n1))<br>A = matrix(y1, (1, n1), &#39;d&#39;)<br>b = matrix(0.0)<br>G = matrix(np.vstack((-np.eye(n1), np.eye(n1))))<br>h = matrix(np.hstack((np.zeros(n1), np.ones(n1) * C)))<br><br>solvers.options[&#39;show_progress&#39;] = False<br>solvers.options[&#39;abstol&#39;] = 1e-7<br>solvers.options[&#39;reltol&#39;] = 1e-6<br>solvers.options[&#39;feastol&#39;] = 1e-7<br><br>solution = solvers.qp(P, q, G, h, A, b)<br><br>alphas1 = np.array(solution[&#39;x&#39;]).flatten()<br><br>sv1 = (alphas1 &gt; 1e-5) &amp; (alphas1 &lt;= C)<br>ind = np.arange(len(alphas1))[sv1]<br><br>print(&quot;Support vector indices:&quot;, ind)<br>print(&quot;Support vector alphas:&quot;, alphas1[sv1])<br><br>a1 = np.sum(alphas1[:, None] * y1[:, None] * X1, axis=0)<br>print(&quot;Weight vector a:&quot;, a1)<br><br>a0_1 = np.mean(y1[sv1] - np.dot(X1[sv1], a1))<br>print(&quot;Bias a0:&quot;, a0_1)<br><br>model1_predict = lambda x: np.sign(np.dot(x, a1) + a0_1)<br><br>y1_pred = np.array([model1_predict(x) for x in X1])<br><br>print(f&#39;Total incorrect predictions in train: {np.sum(y1_pred != y1)}&#39;)</pre><pre>Support vector indices: [ 0 32 35 52]<br>Support vector alphas: [0.70078918 0.48886359 0.15566161 0.05626382]<br>Weight vector a: [-0.24005609  0.48198802 -0.86426521 -0.60389136]<br>Bias a0: 2.276392123880816<br>Total incorrect predictions in train: 0</pre><pre># model 2 for setosa vs virginica<br><br>X2 = data2.drop(columns=[&#39;target&#39;]).to_numpy()<br>y2 = data2[&#39;target&#39;].to_numpy()<br>n2, d2 = X2.shape<br><br>C = 1.0<br>K = X2 @ X2.T<br>P = matrix(np.outer(y2, y2) * K)<br>q = matrix(-1 * np.ones(n2))<br>A = matrix(y2, (1, n2), &#39;d&#39;)<br>b = matrix(0.0)<br>G = matrix(np.vstack((-np.eye(n2), np.eye(n2))))<br>h = matrix(np.hstack((np.zeros(n2), np.ones(n2) * C)))<br><br>solution = solvers.qp(P, q, G, h, A, b)<br><br>alphas2 = np.array(solution[&#39;x&#39;]).flatten()<br><br>sv2 = (alphas2 &gt; 1e-5) &amp; (alphas2 &lt;= C)<br>ind = np.arange(len(alphas2))[sv2]<br><br>print(&quot;Support vector indices:&quot;, ind)<br>print(&quot;Support vector alphas:&quot;, alphas2[sv2])<br><br>a2 = np.sum(alphas2[:, None] * y2[:, None] * X2, axis=0)<br>print(&quot;Weight vector a:&quot;, a2)<br><br>a0_2 = np.mean(y2[sv2] - np.dot(X2[sv2], a2))<br><br>print(&quot;Bias a0:&quot;, a0_2)<br><br>model2_predict = lambda x: np.sign(np.dot(x, a2) + a0_2)<br><br>y2_pred = np.array([model2_predict(x) for x in X2])<br><br>print(f&#39;Total incorrect predictions in train: {np.sum(y2_pred != y2)}&#39;)</pre><pre>Support vector indices: [43 53 55]<br>Support vector alphas: [0.0349535  0.15923256 0.12427907]<br>Weight vector a: [-0.15418374  0.15923257 -0.46526983 -0.22991631]<br>Bias a0: 2.1572325014369014<br>Total incorrect predictions in train: 0</pre><pre># model 3 for versicolor vs virginica<br><br>X3 = data3.drop(columns=[&#39;target&#39;]).to_numpy()<br>y3 = data3[&#39;target&#39;].to_numpy()<br>n3, d3 = X3.shape<br><br>C = 1.0<br>K = X3 @ X3.T<br>P = matrix(np.outer(y3, y3) * K)<br>q = matrix(-1 * np.ones(n3))<br>A = matrix(y3, (1, n3), &#39;d&#39;)<br>b = matrix(0.0)<br>G = matrix(np.vstack((-np.eye(n3), np.eye(n3))))<br>h = matrix(np.hstack((np.zeros(n3), np.ones(n3) * C)))<br><br>solution = solvers.qp(P, q, G, h, A, b)<br><br>alphas3 = np.array(solution[&#39;x&#39;]).flatten()<br><br>sv3 = (alphas3 &gt; 1e-5) &amp; (alphas3 &lt;= C)<br>ind = np.arange(len(alphas3))[sv3]<br><br>print(&quot;Support vector indices:&quot;, ind)<br>print(&quot;Support vector alphas:&quot;, alphas3[sv3])<br><br>a3 = np.sum(alphas3[:, None] * y3[:, None] * X3, axis=0)<br>print(&quot;Weight vector a:&quot;, a3)<br><br>a0_3 = np.mean(y3[sv3] - np.dot(X3[sv3], a3))<br>print(&quot;Bias a0:&quot;, a0_3)<br><br>model3_predict = lambda x: np.sign(np.dot(x, a3) + a0_3)<br><br>y3_pred = np.array([model3_predict(x) for x in X3])<br><br>print(f&#39;Total incorrect predictions in train: {np.sum(y3_pred != y3)}&#39;)</pre><pre>Support vector indices: [ 1  6  7 11 12 16 21 24 33 36 37 40 43 49 53 54 63]<br>Support vector alphas: [9.99998473e-01 9.99999875e-01 9.99999291e-01 9.99999973e-01<br> 9.99999956e-01 9.99999871e-01 9.99999881e-01 9.99999961e-01<br> 3.07122193e-05 3.37438483e-01 8.87326117e-01 9.99999979e-01<br> 6.26004939e-01 9.99999865e-01 9.99999963e-01 5.98788175e-01<br> 9.99999944e-01]<br>Weight vector a: [ 0.57792639  0.18148076 -1.89513938 -1.86161793]<br>Bias a0: 8.289948474987956<br>Total incorrect predictions in train: 2</pre><pre>models = [<br>    (model1_predict, 0, 1),  # setosa vs versicolor<br>    (model2_predict, 0, 2),  # setosa vs virginica<br>    (model3_predict, 1, 2),  # versicolor vs virginica<br>]<br><br>def one_vs_one_predict(x):<br>    votes = {0: 0, 1: 0, 2: 0}<br>    for model, class_a, class_b in models:<br>        pred = model(x)<br>        if pred == 1:<br>            votes[class_a] += 1<br>        else:<br>            votes[class_b] += 1<br>    max_votes = max(votes.values())<br>    top_classes = [cls for cls, v in votes.items() if v == max_votes]<br>    return min(top_classes)<br><br>y_pred = np.array([one_vs_one_predict(x) for x in x_test])<br><br>print(f&#39;Total incorrect predictions in test: {np.sum(y_pred != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_pred == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in test: 0<br>Accuracy in test: 1.00</pre><p>Using built-in class,</p><pre>from sklearn.svm import SVC<br><br>svc_model = SVC(kernel=&#39;linear&#39;)<br><br>svc_model.fit(x_train, y_train)<br><br>y_svc_pred = svc_model.predict(x_test)<br><br>print(f&#39;Total incorrect predictions in test: {np.sum(y_svc_pred != y_test)}&#39;)<br>print(f&#39;Accuracy in test: {np.mean(y_svc_pred == y_test):.2f}&#39;)</pre><pre>Total incorrect predictions in test: 0<br>Accuracy in test: 1.00</pre><p><strong>Advantages :</strong></p><ul><li>Effective even if number of features &gt; number of samples</li><li>Finds the optimal separating hyperplane, so usually robust</li><li>Can handle non-linear boundaries using kernels</li></ul><p><strong>Disadvantages :</strong></p><ul><li>Not great with very large datasets (training can be slow)</li><li>Choosing the right kernel and tuning parameters can be tricky</li><li>Doesn’t directly give probabilities</li></ul><p><strong>Takeaway from SVM :</strong></p><p>SVM is all about <strong>maximizing the margin</strong> using only the critical points (support vectors). Works really well when classes are separable, but can get tricky with huge or messy data.</p><blockquote>Kernelization :</blockquote><p>There is a trick called kernelization for improving the computationally simple classifier <em>h</em>. The goal is to map the input variables X to higher dimensional space Z and apply classifier in the bigger space Z. This can yield a more flexible classifier while maintaining computational simplicity.</p><p>A mapping function Φ takes input in X dimension and returns output in Z dimension. In higher dimension the Yi’s are separable by linear boundary, in other words : <strong>a linear classifier in a higher dimensional space corresponds to the non-linear classifier in the original space.</strong></p><p>The point is, to get richer set of classifiers we do not need to give up the convenience of linear classifiers. We simply can map the input space to higher dimension space. This is similar to making the linear regression more flexible by introduction polynomial features</p><p>There is a potential drawback. If we significantly increase the dimension of the problem we might increase the computational burden. For example, if x is in 256 input dimensional space and we wanted to use all the fourth-order terms, then</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/80/1*Xw45yofOAOBWWY9DXxw7sw.png" /></figure><p>has 183,181,376 dimensions. After converting to this high-dimensional data, we burn more time, power and space to compute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*szIQe1SM9gGqFkbf7T9Dzg.png" /></figure><p>If we observe how we are using the input dataset in modeling, we are always computing the inner product, like in SVM we are performing inner product in 2 places. first, is in computing lagrangian and second is in making predictions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*BkQuLiGatc3Zx0xyRpZWHQ.png" /></figure><p>and,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/143/1*v_EEXC6zavFEgp2ids6suA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/159/1*Mr88J3Hq6pTADuuJuINZnQ.png" /></figure><p>In both cases we are performing inner product. So our goal is to get the inner product of the transformed data, not to get the transformed data itself. What if we have a function that we call Kernel, that takes data in input space, apply some transformation and returns their inner product and what if this inner product happens to be similar to taking the inner product of transformed data using Φ.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*r4aHlApnxevXW3V9dXJLUA.png" /></figure><p>In fact, we never need to construct the Φ at all. We only need to specify a kernel <em>K</em> that corresponds to the inner product of transformed data for some mapping function Φ. This raises an interesting question, given a function of two variables <em>K(X, Y)</em>, does there exist a mapping function Φ such that</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*r5Vef8SgViWU4r9gHwEnXg.png" /></figure><p>In other words, we can only be sure the output of the kernel function is valid if and only if there exists some Φ, such that performing inner product with this Φ results in the same output as the kernel’s.</p><p>According to mercer’s theorem, a kernel will have a Φ only if the kernel is valid. A kernel is valid if it satisfies symmetry</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/206/1*zp3t3tKuJaanU7ih-LhEjQ.png" /></figure><p>and positive semidefiniteness(PSD) i.e. for any set of points X1, ..,Xn and any real coefficients C1, ..,Cn</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*uLpvPj_IWsAcjxufFwCmig.png" /></figure><p>Examples of commonly used kernels are;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*fVrGEL3CaA_jjugNy7bOYQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*NiA09FsaKS2nQcFKWEHnvA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*-cujg1XXegedu_UCmi_8yg.png" /></figure><p>Using kernel trick in SVM lets the model take advantage of higher dimensional representation without explicitly converting the data into higher dimensional space. After deciding on which kernel suits our problem lagrangian term and prediction terms changes to</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*0qYYowVGjkjUP4-TrbKxgA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*c1QE0H53nr-c1TTvAqNAJA.png" /></figure><p><strong>Conclusion :</strong></p><p>In this blog, we started with the main idea — classification is about finding a decision boundary, and Bayes classifier gives the optimal one. Since we don’t know the true distribution, every model we looked at is just an approximation of that.</p><ul><li><strong>Generative models (QDA, LDA, Naive Bayes)</strong> try to model how the data is generated and then use Bayes rule to classify.</li><li><strong>Discriminative models (Logistic Regression, Decision Trees, SVMs, Kernel methods)</strong> don’t bother modeling the full distribution — they directly learn the boundary that separates the classes.</li></ul><p>Each one comes with its own pros and cons. Some are simple and interpretable, others are more flexible but can overfit. But the core idea is the same: get as close as possible to the Bayes solution.</p><p>There are also a few other important methods we didn’t go into here</p><ul><li><strong>K-Nearest Neighbors (KNN):</strong> a non-parametric method that classifies points based on their neighbors.</li><li><strong>Bagging and Boosting:</strong> ensemble methods that combine multiple weak models (usually trees) to make a stronger one.</li><li><strong>Neural Networks:</strong> flexible models that can learn very complex boundaries by stacking many layers.</li></ul><p>All of these are just extensions of the same principle — different ways to approximate the optimal classifier depending on the problem and the data. There isn’t a universal “best” model, model selection varies with problem and it always depends on the trade-off between interpretability, flexibility, and computation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c27f90558afa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Moving Past Histograms: Smoothing Out Distributions with KDE]]></title>
            <link>https://mohith-g.medium.com/moving-past-histograms-smoothing-out-distributions-with-kde-9c5a7700f667?source=rss-6575ac4820f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c5a7700f667</guid>
            <category><![CDATA[kernel-density-estimation]]></category>
            <category><![CDATA[gaussian-kernel]]></category>
            <dc:creator><![CDATA[Mohith]]></dc:creator>
            <pubDate>Fri, 22 Aug 2025 08:07:06 GMT</pubDate>
            <atom:updated>2025-08-22T08:07:06.370Z</atom:updated>
            <content:encoded><![CDATA[<h4>The role of kernels and bandwidth in smoothing data</h4><p>The histogram density function we discussed in <a href="https://mohith-g.medium.com/density-function-estimation-39383466e969">Density Function Estimation</a> is <strong>discontinuous, </strong>means the density abruptly changes between ending of a bin and starting of another bin. To estimate more continuous density function of a random variable we can use Kernel Density Estimation(KDE).</p><p>A kernel is essentially a PDF used for smoothing in density estimation, a kernel is chosen to be symmetric(they don’t have to be). The other properties of a kernel are non-negative,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/71/0*gPABVRpbJA-JSAMV" /></figure><p>area under the curve is 1,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/110/0*Zz357iVcfbn3TbNe" /></figure><p>centered around 0,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/121/0*TYrVnaU_Qk5FSogx" /></figure><p>has a non-zero variance,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/175/0*J7MIsbjKsYndP-S6" /></figure><p>The main idea behind using kernel is:</p><p>In a histogram with bins like [1, 3) and [3, 5), we lose detail about points between 2 and 3, so the estimate looks blocky and discontinuous. With kernels, instead of bins, we place a smooth curve (say <em>K(u)</em> ~ Gaussian) on each datapoint — its peak sits on the point, and its spread covers the gaps. To estimate the density at a point x, we check how far x is from each datapoint in units of the bandwidth <em>h</em>. Then we plug those distances into the kernel function — closer points give higher kernel values, while points farther away contribute less. Mathematically, the formula for KDE is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/203/0*_-5h24l_DpJ4sfK0" /></figure><p>Let’s see how <em>K(x — Xi / h) </em>works by looking at single datapoint example,</p><p>For the sake of this example let <em>K(u)</em> is a gaussian kernel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/0*EKiXwD99VGxBv6L_" /></figure><p>For the data we used in Histogram Density Estimation waiting_times , the kernel on a single datapoint looks like</p><pre>waiting_times = np.sort(waiting_times)<br><br>K = lambda u: (2 * np.pi) ** (-0.5) * np.exp(-0.5 * u ** 2)<br><br>plt.figure(figsize=(10, 6))<br>plt.ylim(0, 0.001)<br>plt.scatter(waiting_times, np.zeros_like(waiting_times), color=&#39;red&#39;, zorder=15)<br><br>for X in np.random.choice(waiting_times, size=1, replace=False):<br>    kernel = [(1/h) * K((x - X) / h) / len(waiting_times) for x in waiting_times]<br>    # we will talk about h in a bit<br>    plt.plot(waiting_times, kernel, color=&quot;#2C5B03&quot;)<br>    plt.axvline(x=X, ymin=0, color=&#39;#D08770&#39;, linestyle=&#39;-&#39;, alpha=1, linewidth=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*RlMg5pwlLiKClmnvm4R9bg.png" /></figure><p>Here the X is 75, if we want to find the density of x, the X = 75 contribution would be lower if x is far (x = 50). It won’t be 0(since it’s gaussian) but very small that the contribution is negligible. So, if <em>h</em> is small in</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/89/0*7bkenRFX3iG2mof3" /></figure><p><em>u</em> increases, then when we send <em>u</em> into <em>K(u)</em> where <em>Xi</em> = 75 , for x = 50 the contribution from 75 gaussian kernel is very small.<strong><em> If h is too small, each kernel is very narrow, so the density looks too wiggly and noisy — capturing random fluctuations instead of the real shape. If h is too large, the kernels are too wide, and everything gets oversmoothed — you lose the important details in the data. Choosing the right h is about finding the sweet spot between too much noise and too much smoothing.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/110/0*vwRPk7JNDEGGmQLP" /></figure><p>This is known as <strong>Silverman’s rule of thumb</strong>. It uses the sample standard deviation σ and sample size <strong>n</strong> to give a quick, reasonable starting value for the bandwidth. It’s not always the optimal <em>h</em> for every dataset, but it balances bias and variance fairly well in many cases, so it’s often used as the default choice before fine-tuning.</p><pre>waiting_times = np.sort(waiting_times)<br><br>K = lambda u: (2 * np.pi) ** (-0.5) * np.exp(-0.5 * u ** 2)<br><br>sigma = np.std(waiting_times)<br>n = len(waiting_times)<br>h = 1.06 * sigma * n**(-1/5)<br><br>plt.figure(figsize=(10, 6))<br>plt.ylim(0, 0.001)<br>plt.scatter(waiting_times, np.zeros_like(waiting_times), color=&#39;red&#39;, zorder=15)<br><br>for X in waiting_times:<br>    kernel = [(1/h) * K((x - X) / h) / len(waiting_times) for x in waiting_times]<br>    plt.plot(waiting_times, kernel, color=&quot;#2C5B03&quot;)<br>    plt.axvline(x=X, ymin=0, color=&#39;#D08770&#39;, linestyle=&#39;-&#39;, alpha=1, linewidth=1)<br><br>plt.savefig(&quot;kernel_densities.png&quot;, dpi=300)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WkuSosgES7vL666c5AamkQ.png" /></figure><p>From this, we can say that the mean of sample data x̄ has higher density value(because we add high kernel densities) and the value that is far from these datapoints like 10 or something, will have lower density (because we add lower kernel densities)</p><p>In KDE formula, when we changed the distance scale into <em>h</em> the area in each kernel function changed from unit area to <em>h</em>. To get back into unit area we divide each kernel function with <em>h</em>. Then we sum those <strong>n</strong> unit areas giving us area of n, so we normalize by dividing it with <strong>n</strong>. Giving us</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/211/0*v228nT4vGZ4nDlcH" /></figure><pre>waiting_times = np.sort(waiting_times)<br><br>K = lambda u: (2 * np.pi) ** (-0.5) * np.exp(-0.5 * u ** 2)<br><br>sigma = np.std(waiting_times)<br>n = len(waiting_times)<br>h = 1.06 * sigma * n**(-1/5)<br><br>def f(x, h):<br>    kernel_values = [(1/h) * K((x - X) / h) for X in waiting_times]<br>    return np.sum(kernel_values) / len(waiting_times)<br><br>pdf = [f(x, h) for x in waiting_times]<br><br>plt.figure(figsize=(10, 6))<br>plt.plot(waiting_times, pdf, color=&#39;#E15759&#39;, linewidth=2, label=&#39;Kernel Density Estimate&#39;)<br>plt.xlabel(&quot;Waiting Time (minutes)&quot;, fontsize=14)<br>plt.ylabel(&quot;Density&quot;, fontsize=14)<br>plt.title(&quot;Kernel Density Estimation of Waiting Times&quot;, fontsize=16, pad=15)<br>plt.legend()<br>plt.tight_layout()<br>plt.savefig(&quot;kde_plot.png&quot;, dpi=300)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K8bFtm2K35tkr7QfPy2NSw.png" /></figure><p>While Gaussian kernels are the most common choice, they’re not the only option. For example, the <strong>Epanechnikov kernel</strong> gives higher weight to points close to x, but it completely ignores points that are too far away. In other words, its contribution becomes zero outside a certain range. This makes it computationally efficient since distant datapoints don’t affect the density at x. The kernel function for epanechnikov would be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/0*2xIfFRzO20pGYZz4" /></figure><p>See how the contribution got to 0 after a unit distance from <em>Xi</em></p><p>Another example is the <strong>Uniform kernel</strong>. Here, every point within the bandwidth window contributes equally, and points outside contribute nothing. In fact, if you use the uniform kernel, Kernel Density Estimation (KDE) reduces to the <strong>Histogram Density Estimator (HDE)</strong> — just a smoothed-out version of the histogram.</p><p>But in practice, we don’t need to write KDE by hand — libraries already implement it for us. <strong>Statsmodels</strong> does exactly what we coded, but the density curve looks a lot smoother because it evaluates the function over a fine grid of points rather than only at the sample values. Similarly, <strong>Seaborn’s</strong> kdeplot also uses a Gaussian kernel by default and automatically chooses a bandwidth, giving you a clean and smooth curve with just one line of code.</p><pre>from statsmodels.nonparametric.kde import KDEUnivariate<br>import seaborn as sns<br><br>kde = KDEUnivariate(waiting_times)<br>kde.fit(kernel=&quot;gau&quot;) # gau = gaussian kernel, epa = epanechnikov kernel, uni = uniform kernel<br><br>plt.figure(figsize=(10, 6))<br>plt.plot(kde.support, kde.density, color=&#39;#E15759&#39;, label=&#39;Gaussian Kernel (Statsmodels)&#39;)<br>sns.kdeplot(waiting_times, color=&#39;#4C78A8&#39;, label=&#39;Gaussian Kernel (Seaborn)&#39;)<br>plt.xlabel(&quot;Waiting Time (minutes)&quot;, fontsize=14)<br>plt.ylabel(&quot;Density&quot;, fontsize=14)<br>plt.title(&quot;Different Kernels for KDE&quot;, fontsize=16, pad=15)<br>plt.legend()<br>plt.tight_layout()<br>plt.savefig(&quot;kde.png&quot;, dpi=300)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iGL2fCv1zB9QhkPd71dWQQ.png" /></figure><p><strong>Conclusion </strong>:</p><p>Histograms gave us a rough idea, but they were jumpy and blocky. KDE fixes that by placing smooth curves on data points and adding them up, giving us a continuous view of the density. The kernel shape doesn’t change much in practice, but the bandwidth <em>h</em> is everything — it decides whether the curve is too noisy or too smooth. And since libraries like Statsmodels or Seaborn can do this in one line, we don’t really need to code it ourselves. Still, now we know exactly what’s going on under the hood.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c5a7700f667" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Density Function Estimation]]></title>
            <link>https://mohith-g.medium.com/density-function-estimation-39383466e969?source=rss-6575ac4820f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/39383466e969</guid>
            <category><![CDATA[histograms]]></category>
            <category><![CDATA[statistical-inference]]></category>
            <category><![CDATA[confidence-interval]]></category>
            <category><![CDATA[density-estimation]]></category>
            <category><![CDATA[probability-density]]></category>
            <dc:creator><![CDATA[Mohith]]></dc:creator>
            <pubDate>Thu, 21 Aug 2025 08:01:06 GMT</pubDate>
            <atom:updated>2025-08-21T08:01:06.786Z</atom:updated>
            <content:encoded><![CDATA[<h4>Histogram to estimate density function</h4><p>Whether the goal is prediction or statistical inference, density function helps us in either case. To understand how the mass of a random variable is distributed across the datapoints — estimating an accurate density function of the random variable would be a real help. Before seeing how we can estimate a density function, let’s take look at how density would help us with simple example.</p><p>Say you have a data of general and malnourished children heights in normal distribution:</p><pre>import matplotlib.pyplot as plt<br>import numpy as np<br>from scipy.stats import norm, expon, binom<br>import seaborn as sns<br><br># General population heights<br>pop_heights = np.sort(np.random.normal(loc=170, scale=10, size=500))<br><br># Malnourished children heights<br>mal_heights = np.sort(np.random.normal(loc=150, scale=12, size=500))</pre><p>To get density function(PDF) of these random variables:</p><pre>pdf_pop = norm.pdf(pop_heights, loc=np.mean(pop_heights), scale=np.std(pop_heights))<br>pdf_mal = norm.pdf(mal_heights, loc=np.mean(mal_heights), scale=np.std(mal_heights))</pre><p>norm.pdf() takes a datapoint and return it’s density in Standard Normal Distribution. We can change the SND to our data’s scale by giving the mean loc and standard deviation scale</p><pre>plt.figure(figsize=(10,5))<br>plt.plot(pop_heights, pdf_pop, color=&#39;#4C78A8&#39;, lw=2.5, label=&#39;General Children PDF&#39;)<br>plt.plot(mal_heights, pdf_mal, color=&#39;#E15759&#39;, lw=2.5, label=&#39;Malnourished Children PDF&#39;)<br>plt.fill_between(pop_heights, pdf_pop, color=&#39;#4C78A8&#39;, alpha=0.1)<br>plt.fill_between(mal_heights, pdf_mal, color=&#39;#E15759&#39;, alpha=0.1)<br>plt.title(&quot;Estimating Children Heights&quot;)<br>plt.xlabel(&quot;Height (cm)&quot;)<br>plt.ylabel(&quot;Density&quot;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="Density functions of general vs malnourished children" src="https://cdn-images-1.medium.com/max/899/1*9JvWGWZ82GNvsjgzhjQSsQ.png" /></figure><p><strong>Observations from the Graph:</strong></p><p><strong>Shift in Mean Heights:</strong></p><ul><li>The peak of the general children is centered around ~170 cm.</li><li>The peak of the malnourished children is centered around ~150 cm.</li><li>This shows that malnourished children are, on average, shorter than the general children.</li></ul><p><strong>Overlap Between Distributions:</strong></p><ul><li>There is some overlap between the two distributions in the 150–165 cm range.</li><li>This means that within this height range, it is harder to distinguish whether a child belongs to the malnourished group or the general group based on height alone.</li></ul><p><strong>Spread (Variance) Difference:</strong></p><ul><li>The red curve is wider, showing <strong>greater variability</strong> in heights.</li><li>The blue curve is narrower, showing <strong>less variability</strong>.</li></ul><p><strong>Tail Behavior:</strong></p><ul><li>The general children distribution extends more towards taller heights ~200 cm.</li><li>The malnourished distribution is more spread but with very few children beyond ~185 cm, showing a ceiling effect in growth.</li></ul><p><strong>Practical Implication:</strong></p><ul><li>Height can be used as a proxy indicator for nutritional status, but due to overlap, <strong>statistical inference</strong> is needed to make reliable group distinctions rather than relying on raw comparison.</li></ul><p>From these density functions with predictions we can infer probabilities like :</p><p><em>What fraction of children are taller than 180 cm?</em></p><p><em>What’s the most common height range?</em></p><p>also performing hypothesis tests, finding confidence intervals, doing interpretation, etc. This helps in applications like public health or clothing industry sizing.</p><blockquote>Some other examples like:</blockquote><p><em>Time until next customer arrives in queue management, call centers, and traffic modeling.</em></p><figure><img alt="PDF of customer arrival time" src="https://cdn-images-1.medium.com/max/898/1*UC62tszZP2zTkEaaZXIOtQ.png" /></figure><p><em>Probability of heads in coin flips, the foundation of hypothesis testing and fairness checks in experiments.</em></p><figure><img alt="PMF of heads in coinflips" src="https://cdn-images-1.medium.com/max/892/1*fUMeO_IHxlRi6Y0VM2iy1Q.png" /></figure><p>For all these above examples we created the data on specific distribution in mind. We can plot histogram for the data to observe how the mass is distributed just like I did for the above two plots. The real-world data may won’t follow these text book distributions. We can see some weird shapes like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X8r-EXlZtqYfJlN1qSRrFQ.png" /></figure><p>Plots help intuition, but density functions give numbers we can compute, compare, and infer from.</p><p>These density functions allow us to move beyond pictures and actually quantify aspects of the distribution, like where it tends to center, how spread out it is, or whether it has heavy tails. This is where <em>density estimation</em> comes in — instead of just plotting data, we aim to mathematically estimate the underlying density function that generated it. That way, we can use the data not only to visualize but also to reason, compare, and make inferences. A simple method to estimate the density of a random variable is <strong>Histogram density estimation.</strong></p><p><strong>Histogram Density Estimation :</strong></p><p>Let <em>X = { x1, x2, x3, …, xn}</em> be a random variable with <strong><em>n</em></strong> observations sampled from some population density function <em>f</em> .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/196/0*89FuBDDnBNKXJRyL" /></figure><p>Now divide the interval <em>[a, b]</em> into <strong><em>m </em></strong>equal bins</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/0*Zq9jNB3o58BXFlS_" /></figure><p>each bin has an equal width of:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/73/0*AI9e-fRr5eoYd0Iu" /></figure><p><strong><em>Our goal is to estimate the density at each data point xi​. Instead of calculating the density for every single point, we can group nearby points into bins and use the density of each bin as a good approximation for the density of the points it contains</em>.</strong> Now assign each data point <em>xi</em> to a bin according to the value it contains. Let <em>vj </em>be the number of observations in bin <em>j</em>, then the probability of an observation getting into bin <em>j </em>would be :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/57/0*wpxBp5LqNDQA8s5Y" /></figure><p>This is only an estimate of the true probability of an observation<br>falling into bin <em>j</em>. If the underlying population has density <em>f</em>, then the actual probability of landing inside bin <em>j</em> is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/129/0*hFZBP-YyD5l5c7hJ" /></figure><p>By the law of large numbers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/83/0*3UTLw3hfqnLsSD22" /></figure><p>Now we know each bin(rectangle) has a width of <em>h</em> and we know the probability(area) of getting into that bin. Our goal is to find the density(height) of the bin, we know</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*nfnLBJOkEE901X8R" /></figure><p>that would make our bin’s density</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/0*pPe_LBpuyLx4vyGH" /></figure><p>The <strong>histogram estimator</strong> is defined by :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/179/0*W96QHGBKoD_USLQU" /></figure><p>which we can write more clearly as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/0*b4wrfdo44sEFtMZ1" /></figure><p>Note that this density function is an estimation, that means</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/0*a1evuAVa4YNOHgXe" /></figure><p>The bin-width <em>h</em> is smoothing parameter,</p><p><strong><em>As the bin-width h increases, each bin averages over more data points, so individual points have less influence on the histogram. (high-bias, over-smoothing)</em></strong></p><p><strong><em>As the bin-width h decreases, each bin contains fewer data points, so individual points have more influence on the histogram. (high-variance, under-smoothing)</em></strong></p><p>This is where the trade-off between bias and variance comes in, high-bias results in overly smooth, inaccurate density estimates, while high-variance makes the histogram overly sensitive to individual points, leading to a noisy estimate.</p><p><strong>Optimal Smoothing Parameter(<em>h</em>):</strong></p><p>As a loss function we will use the <strong>Integrated Squared Error(ISE)</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/0*3Bnxa_D6NYBf3fg4" /></figure><p>The <strong>Risk </strong>or <strong>Mean Integrated Squared Error(MISE) </strong>for a bin-width <em>h</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/0*oz6jgRooBLgySOXQ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/0*W24plsWbXHVfMIXW" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/0*LmtWD6VyjUq5NsGy" /></figure><p>The <strong>Risk</strong> can be written as,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/0*E67SOI-doOSdJXle" /></figure><p>Minimizing risk corresponds to balancing bias and variance. In histogram estimation we do that with smoothing parameter <em>h</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JnPanoZZ7eegeq26nPFpgQ.png" /></figure><p>To find out optimal <em>h</em> value, we can use cross-validation to compute the risk of the density function with bin-width of <em>h </em>and compare the risk with other <em>h</em> values risks.</p><p>Normally, cross-validation for histograms would mean doing leave-one-out CV. That is — for every single data point, we throw it out, rebuild the histogram with the remaining n−1 samples, then check how well the left-out point is predicted. Doing this <strong>n</strong> times for each candidate bin-width <em>h</em> quickly becomes painful.</p><p>The nice thing about the formula</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/0*VRwDrWKwT_0n1i4i" /></figure><p>is that it’s a <strong>shortcut</strong>. Instead of refitting the histogram <strong>n</strong> times, you only need to compute it once per <em>h</em>. So the shortcut doesn’t eliminate the need to try different bin widths — it just saves you from the heavy lifting of recomputing the histogram <strong>n</strong> times for each candidate <em>h</em>.</p><p><strong>Expectation and Variance of Histogram Density Function :</strong></p><p>Consider fixed <em>x</em> and fixed <em>m</em>, and let <em>Bj </em>be the bin containing <em>x</em>. Then,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/107/0*YnM67VAMGzQrnXab" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/169/0*iA-cyJScSuwxPUWR" /></figure><p><strong>Enough of theory :</strong></p><p>let’s see the <strong>HDE</strong> working on <em>Old Faithful Dataset</em>, particularly the column named <em>waiting , </em>which represent the waiting times for the next volcanic eruption.</p><pre>df = pd.read_csv(&#39;data.tsv&#39;, sep=&#39;\t&#39;)<br>waiting_times = df[&#39;waiting&#39;].to_numpy()<br><br>m = 50 # Initial no of bins<br>min_value = waiting_times.min()<br>max_value = waiting_times.max()<br>bins = np.linspace(min_value, max_value, m+1) # m+1 for edges<br>h = bins[1] - bins[0]  # Bin width<br><br># Counting the values with in each bin<br>lengths = np.zeros(m)<br><br>for i in range(m): <br><br>    if i &lt; m-1:<br>        lengths[i] = len(waiting_times[(waiting_times &gt;= bins[i]) &amp; (waiting_times &lt; bins[i+1])])<br>    else:<br>        lengths[i] = len(waiting_times[(waiting_times &gt;= bins[i]) &amp; (waiting_times &lt;= bins[i+1])])<br>        <br>areas = lengths / len(waiting_times)<br><br>densities = areas / h<br><br>plt.figure(figsize=(10, 6))<br>plt.plot(bins[:-1], densities, color=&#39;#E15759&#39;, linewidth=2, label=&#39;Density Estimate&#39;)<br>plt.xlabel(&quot;Waiting Time (minutes)&quot;, fontsize=14)<br>plt.ylabel(&quot;Density&quot;, fontsize=14)<br>plt.title(&quot;Density Estimate of Waiting Times&quot;, fontsize=16, pad=14)<br>plt.legend()<br>plt.tight_layout()<br>plt.savefig(&quot;density_estimate.png&quot;, dpi=250)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d_HeacE-DyB9MfGoRQQcWA.png" /></figure><p>Let’s make a bar plot and see how each bins density looks like:</p><pre>plt.figure(figsize=(10, 6))<br>plt.bar(bins[:-1], densities, width=h, align=&#39;edge&#39;, color=&#39;#4C78A8&#39;, alpha=0.7, edgecolor=&#39;black&#39;)<br>plt.xlabel(&quot;Waiting Time (minutes)&quot;, fontsize=14)<br>plt.ylabel(&quot;Estimated PDF&quot;, fontsize=14)<br>plt.title(&quot;Histogram-Based PDF Estimation&quot;, fontsize=16, pad=14)<br>plt.tight_layout()<br>plt.savefig(&quot;histogram_pdf.png&quot;, dpi=250)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xb3CQb8mML1YZtDWudfiOg.png" /></figure><p>We got this histogram plot from the idea of Histogram Density Function, then it should be similar to matplotlib’s histogram plot given the waiting_times ,bins = mand density = True</p><pre>plt.figure(figsize=(10, 6))<br>plt.hist(waiting_times, bins=m, density=True, color=&#39;#4C78A8&#39;, alpha=0.7, edgecolor=&#39;black&#39;)<br>plt.xlabel(&quot;Waiting Time (minutes)&quot;, fontsize=14)<br>plt.ylabel(&quot;Density&quot;, fontsize=14)<br>plt.title(&quot;Normalized Histogram of Waiting Times&quot;, fontsize=16, pad=14)<br>plt.tight_layout()<br>plt.savefig(&quot;normalized_histogram.png&quot;, dpi=250)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hH2HFQ52bVif4-R-ziIPdw.png" /></figure><p>Let’s take a look at how over-smoothed and under-smoothed density functions looks like</p><p><strong>Over-Smoothing :</strong></p><pre>m = 80<br>min_value = waiting_times.min()<br>max_value = waiting_times.max()<br>h = (max_value - min_value) / m<br><br>print(&quot;h :&quot;,h) # 0.6625<br><br>bins = np.linspace(min_value, max_value, m+1)<br><br>lengths, _ = np.histogram(waiting_times, bins=bins) #np.histogram does the same thing as our counting loop is doing<br><br>areas = lengths / len(waiting_times)<br>density = areas / h<br><br>plt.figure(figsize=(10, 6))<br>plt.plot(bins[:-1], density, color=&#39;#E15759&#39;, linewidth=2, label=&#39;Density Estimate (Overfitting)&#39;)<br>plt.xlabel(&quot;Waiting Time (minutes)&quot;, fontsize=14)<br>plt.ylabel(&quot;Density&quot;, fontsize=14)<br>plt.title(&quot;Overfitting Density Estimate (m=80)&quot;, fontsize=16, pad=14)<br>plt.legend()<br>plt.tight_layout()<br>plt.savefig(&quot;overfitting_density.png&quot;, dpi=250)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lTuAKwyW48xn3FM21U_Kjg.png" /></figure><p>See the oscillations and how the data is too much focused on the sample data.</p><p><strong>Under-Smoothing :</strong></p><pre>m = 10<br>min_value = waiting_times.min()<br>max_value = waiting_times.max()<br>h = (max_value - min_value) / m<br><br>print(&#39;h :&#39;, h) #5.3<br><br>bins = np.linspace(min_value, max_value, m+1)<br><br>lengths, _ = np.histogram(waiting_times, bins=bins)<br><br>areas = lengths / len(waiting_times)<br>density = areas / h<br><br>plt.figure(figsize=(10, 6))<br>plt.plot(bins[:-1], density, color=&#39;#E15759&#39;, linewidth=2, label=&#39;Density Estimate (Underfitting)&#39;)<br>plt.xlabel(&quot;Waiting Time (minutes)&quot;, fontsize=14)<br>plt.ylabel(&quot;Density&quot;, fontsize=14)<br>plt.title(&quot;Underfitting Density Estimate (m=10)&quot;, fontsize=16, pad=14)<br>plt.legend()<br>plt.tight_layout()<br>plt.savefig(&quot;underfitting_density.png&quot;, dpi=250)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xyxqGmnJKzGv06eFK5vBkA.png" /></figure><p>Each bin of width 5.3 is averaging more data, reducing the influence of the individual datapoints.</p><p><strong>Confidence Band for density function :</strong></p><p>Remember that our density function is a random variable here ? Since we estimated the density function for the waiting_times (m = 50 PDF)and we also agree with the law of large numbers, we can estimate where the population density function <em>f(x)</em> is using Confidence Bands. To know more about Confidence Intervals you can see <a href="https://medium.com/@mohith-g/confidence-is-the-new-truth-76e9b880ff0a">Confidence Is The New Truth</a></p><p>The <strong>1-α </strong>confidence band for a histogram density function looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/0*FjVzX_qKkXOaGXF8" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/0*H7Htb_L4eYJEiMv7" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/203/0*fTD2x1jBW5XqPE5p" /></figure><pre>alpha = 1 - 0.95 # 95% confidence level<br><br>z = norm.ppf(1 - alpha/2)<br><br>n = len(waiting_times)<br><br>SE = np.sqrt(densities * (1 - densities * h) / n)<br><br>lower = np.maximum(densities - z * SE, 0)<br><br>upper = densities + z * SE<br><br>plt.figure(figsize=(10, 6))<br>plt.plot(bins[:-1], densities, color=&#39;#E15759&#39;, linewidth=2, label=&#39;Density Estimate&#39;)<br>plt.plot(bins[:-1], upper, color=&#39;#4C78A8&#39;, linestyle=&#39;--&#39;, label=&#39;Upper Bound&#39;)<br>plt.plot(bins[:-1], lower, color=&#39;#4C78A8&#39;, linestyle=&#39;--&#39;, label=&#39;Lower Bound&#39;)<br>plt.fill_between(bins[:-1], lower, upper, color=&#39;gray&#39;, alpha=0.3, label=&#39;95% Confidence Interval&#39;)<br>plt.xlabel(&quot;Waiting Time (minutes)&quot;, fontsize=14)<br>plt.ylabel(&quot;Density&quot;, fontsize=14)<br>plt.title(&quot;Density Estimate with Confidence Intervals&quot;, fontsize=16, pad=14)<br>plt.legend()<br>plt.tight_layout()<br>plt.savefig(&quot;confidence_intervals.png&quot;, dpi=250)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n4gYCqNOdNXde5J9--PUDw.png" /></figure><p><strong>Conclusion :</strong><br> We started with the idea that density functions aren’t just pretty plots — they give us a mathematical handle on how data is spread out. Histograms gave us the first step into density estimation, where the choice of bin-width <em>h</em> controls everything: too small and the estimate becomes noisy, too large and it gets oversmoothed. We also saw how to attach confidence bands to remind ourselves that the estimated density is still a random variable.</p><p>Histograms are simple and intuitive, but they’re also <strong>discontinuous</strong>, with abrupt jumps at bin edges. In the next blog, we’ll look at a really cool Kernel Density Estimation (KDE), which smooths out those jumps and gives a more flexible way of estimating densities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39383466e969" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multiple Linear Regression: How Assumptions, Multicollinearity, and Data Shape Your Model’s…]]></title>
            <link>https://mohith-g.medium.com/multiple-linear-regression-how-assumptions-multicollinearity-and-data-shape-your-models-cf2618a81cc4?source=rss-6575ac4820f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf2618a81cc4</guid>
            <category><![CDATA[sklearn]]></category>
            <category><![CDATA[statistical-inference]]></category>
            <category><![CDATA[multiple-linearregression]]></category>
            <category><![CDATA[multicollinearity]]></category>
            <dc:creator><![CDATA[Mohith]]></dc:creator>
            <pubDate>Sun, 10 Aug 2025 13:56:46 GMT</pubDate>
            <atom:updated>2025-09-01T07:08:46.285Z</atom:updated>
            <content:encoded><![CDATA[<h3>Multiple Linear Regression: How Assumptions, Multicollinearity, and Data Shape Your Model’s Performance</h3><h4>Exploring why predictors relationships, dataset size, and transformations matter for trustworthy inference and prediction</h4><p>This will be the continuation for <a href="https://medium.com/@mohith-g/linear-regression-explained-from-theory-to-real-world-implementation-45b43faed743"><em>Linear Regression Explained: From Theory to Real-World Implementation</em></a><em> , </em>where i discussed about how simple linear regression works and what are the assumptions that make it work better. Now multiple linear regression is where we can predict dependent variable Y with two or more independent variables X.</p><p>The idea of linearity between dependent and independent variables is same in multiple linear regression too. In fact all the assumption are still there in multiple linear regression such as linearity, error normality, homoskedasticity, independence of errors. A new assumption is that now in Multiple Linear Regression we assume the variables in X are not highly correlated with each other, we will discuss why that’s a problem in a bit.</p><p>The <strong>concepts and formulas stay the same in spirit,</strong> the only difference is that with multiple predictors you use matrix notation and inversion instead of simple scalar sums and means. To get an estimate of β, performing OLS with RSS will result in</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/143/0*mm31juMIiJJe8-i4" /></figure><p>And</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/0*-1GycQ0sezHHIVLS" /></figure><p>First, we’ll apply multiple linear regression to a dataset to see how poorly it can perform, explore why that happens, understand the underlying reasons, and then apply it to a dataset that better fits the model’s assumptions.</p><pre>from sklearn.datasets import load_linnerud<br>import seaborn as sns<br><br>data = load_linnerud(as_frame=True)<br>df = data.frame<br><br>plt.figure(figsize=(8, 6))<br><br>ax = sns.heatmap( df.corr(), cmap=&quot;RdBu_r&quot;, annot=True, fmt=&quot;.2f&quot;, vmin=-1, vmax=1, linewidths=0.5, cbar_kws={&quot;shrink&quot;: 0.8})<br><br>ax.set_title(&quot;Correlation Heatmap&quot;, fontsize=15, pad=12, weight=&#39;bold&#39;)<br>plt.xticks(rotation=45, ha=&quot;right&quot;)<br>plt.yticks(rotation=0)<br><br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*KnPcTI5KvbzMpZ22uRBeXQ.png" /></figure><p>We can see the variables are highly correlated. As we discussed before, linear regression is a parametric model, good at inference. If we want a reliable inference values we need to address the correlation between the variables.</p><p>The reason why we avoid correlation is that the parameters can become <strong>unstable and unreliable</strong> when predictors are highly correlated — a problem known as <strong>multicollinearity</strong>.</p><p>When this happens:</p><ul><li>The standard errors of the coefficients inflate</li><li>Small changes in the data can cause large swings in estimated parameter values (even flipping their signs)</li><li>It becomes difficult to interpret the effect of each predictor independently, because they carry overlapping information about the target</li><li>Hypothesis tests for coefficients may fail to detect significance even when predictors are genuinely related to the outcome</li></ul><p>In short, <strong>high correlation between predictors doesn’t break the model mathematically you might get prediction score, but it can break its interpretability and trustworthiness</strong>.</p><p>Model doesn’t care about the numbers (parameters) it assigns to each variable until it finds the least squares estimates for the data. If two variables are highly correlated, one of them is not providing any new information for the model to help explain the variability in Y. The model might give (5,0), (0, 5), (2.5,2.5) or worse, (10, -5), (50, -25). If we try to interpret these values like they reflect some precise, meaningful relationship between each predictor and Y, our inference doesn’t make sense — they’re just numbers that happen to fit the data, and in the presence of high correlation, they could’ve easily been something completely different while giving the exact same predictions. This is the reason why we <strong>assume no</strong> <strong>multicollinearity.</strong></p><pre>Y = df.pop(&#39;Pulse&#39;)<br><br>for i in df.columns:<br>    df[i] = (df[i] - df[i].mean()) / df[i].std() # Standardizing raw data<br>    <br>X = df.to_numpy()<br>X = np.column_stack((np.ones(X.shape[0]), X))<br><br>Y = Y.to_numpy()<br><br>print(&quot;X:&quot;,X.shape,&#39;\nY:&#39;, Y.shape)<br><br>beta = np.linalg.pinv(X.T @ X) @ (X.T @ Y)<br><br>Y_pred = X @ beta<br><br>RSS = np.sum((Y - Y_pred)**2)<br>TSS = np.sum((Y - Y.mean())**2)<br><br>R2 = 1 - (RSS/TSS)<br><br>print(f&quot;RSS: {RSS:.2f}&quot;)<br>print(f&quot;TSS: {TSS:.2f}&quot;)<br>print(f&quot;R square: {R2:.2f}&quot;)<br><br>error_var = RSS / (X.shape[0] - X.shape[1])<br>print(f&quot;error variance: {error_var:.2f}&quot;)</pre><pre>X: (20, 6) <br>Y: (20,)<br>RSS: 840.56<br>TSS: 987.80<br>R square: 0.15<br>error variance: 60.04</pre><p>Note that we column stack intercept to X, it makes math easy for us. The model is only explaining 15% of variability in Y compared to a base model.</p><pre>residuals = Y - Y_pred<br><br>plt.figure(figsize=(8, 6))<br><br>plt.scatter(Y_pred, residuals, alpha=0.7, edgecolor=&#39;k&#39;)<br><br>plt.axhline(0, color=&#39;red&#39;, linestyle=&#39;--&#39;, linewidth=1.5, label=&quot;Zero Residual Line&quot;)<br><br>plt.xlabel(&quot;Predicted Values&quot;, fontsize=14)<br>plt.ylabel(&quot;Residuals&quot;, fontsize=14)<br>plt.title(&quot;Residuals vs Predicted Values&quot;, fontsize=16, weight=&#39;bold&#39;)<br><br>plt.legend()<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*MwlvLGkCvMBn7OlYjWZeEw.png" /></figure><p>The predictions are heteroskedastic — the error increases as the magnitude of the prediction increases. This suggests our model struggles to predict pulse accurately when pulse values are high. One possible reason is that higher pulse values come with greater variability in the data, which the model isn’t capturing.</p><p>One solution is we could log transform the Y values</p><pre>Y_log = np.log(Y)<br><br>X = df.to_numpy() # df is already standardized<br>X = np.column_stack((np.ones(X.shape[0]), X))<br><br>print(&quot;X:&quot;,X.shape,&#39;\nY:&#39;, Y_log.shape)<br><br>beta = np.linalg.pinv(X.T @ X) @ (X.T @ Y_log)<br>Y_pred = X @ beta<br><br>RSS = np.sum((Y_log - Y_pred)**2)<br>TSS = np.sum((Y_log - Y_log.mean())**2)<br><br>R2 = 1 - (RSS/TSS)<br><br>print(f&quot;RSS: {RSS:.2f}&quot;)<br>print(f&quot;TSS: {TSS:.2f}&quot;)<br>print(f&quot;R square: {R2:.2f}&quot;)<br><br>error_var = RSS / (X.shape[0] - X.shape[1])<br>print(f&quot;error variance: {error_var:.2f}&quot;)</pre><pre>X: (20, 6) <br>Y: (20,)<br>RSS: 0.25<br>TSS: 0.29<br>R square: 0.15<br>error variance: 0.02</pre><p>Remember, a log transformation doesn’t inherently improve prediction accuracy — a lower RSS here only means less error in the transformed scale, not in the original units. The model is still mapping X to Y based on whatever patterns it can find in X. Changing the scale of Y doesn’t magically reveal new patterns in X, it just changes how errors are measured. This is why the <em>R square</em> score remains the same for both logged and unlogged data. But a noticeable difference after using log is we can make inference with some confidence, let us see the same plot we observed heteroskedasticity with Y_log</p><pre>residuals = Y_log - Y_pred<br><br>plt.figure(figsize=(8, 5))<br>sns.scatterplot(Y_pred, residuals, alpha=0.7, edgecolor=None)<br>plt.axhline(0, color=&#39;red&#39;, linestyle=&#39;--&#39;, linewidth=1.2)<br><br>plt.title(&quot;Residuals vs Predicted (Log-Transformed Y)&quot;, fontsize=14, weight=&#39;bold&#39;)<br>plt.xlabel(&quot;Predicted Values&quot;, fontsize=12)<br>plt.ylabel(&quot;Residuals&quot;, fontsize=12)<br>plt.grid(True, linestyle=&quot;--&quot;, alpha=0.3)<br>sns.despine()<br>plt.tight_layout()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*WBJfqeocW96qRKzM94QdzQ.png" /></figure><p>The residuals are more evenly distributed, with a much smaller spread overall. This suggests that the transformation helped stabilize the variance, reducing heteroskedasticity. While the residuals aren’t perfectly uniform yet, the improvement is clear — especially in how the model handles larger predictions. It’s not perfectly homoskedastic because the data itself is inherently heteroskedastic — higher pulse values naturally come with greater variability.</p><p>The insights from fitting linear regression to load_linnerud dataset:</p><ul><li><strong>Low R Square score:</strong> The R square is very low, meaning the model only explains about 15% of the variance in the data compared to simply predicting the average of the <em>Y</em> values.</li><li><strong>Why the R square is low:</strong> The residual sum of squares (RSS — variance the model didn’t explain) is almost as large as the variance from a baseline model that just returns E[Y]. This suggests that, although our model tried to infer patterns, there’s no strong linear relationship between X and Y. In this case, the model’s “best” fit is still no better than predicting the mean.</li><li><strong>High error variance:</strong> The error variance is around 60. Even though the regression line fits as best it can, such a high error variance indicates that a large amount of randomness or noise remains uncaptured. However, the raw error variance alone doesn’t indicate “good” or “bad” — we need a meaningful comparison point.</li><li><strong>Signs of heteroskedasticity:</strong> The residuals plot shows that as predicted values increase, the variance of the errors also increases. This suggests possible heteroskedasticity, likely because data at extreme values (e.g., very high pulse rates) is sparse and more variable. This also reinforces that a simple linear model may not be the best choice for this dataset.</li></ul><p>By training our model on the log-transformed dependent variable (Y), we observed an improvement toward homoskedasticity. An added benefit of this approach is that the model’s parameters can now be interpreted in terms of percentage changes i.e. we can do coefficient interpretations.</p><pre>np.set_printoptions(suppress=True)<br>percent_interpretation = (np.exp(beta) - 1) * 100 # Converts coefficients to % change in Y for a 1-unit change in X<br><br>for i, col in enumerate(df.columns):<br>    change = percent_interpretation[i + 1]<br>    direction = &#39;increasing&#39; if change &gt; 0 else &#39;decreasing&#39;<br>    print(f&quot;For 1 unit increase in {col}, the pulse is {direction} by {abs(change):.2f}%&quot;)</pre><pre>For 1 unit increase in Chins, the pulse is decreasing by 0.39%<br>For 1 unit increase in Situps, the pulse is increasing by 2.60%<br>For 1 unit increase in Jumps, the pulse is decreasing by 1.67%<br>For 1 unit increase in Weight, the pulse is decreasing by 4.83%<br>For 1 unit increase in Waist, the pulse is increasing by 1.20%</pre><p>Note : these unit changes happens when other variables kept constant.</p><p>Our parameters percent interpretations are mathematically valid, but statistically weak and we can’t trust the coefficient interpretations because the model itself isn’t explaining much(low R square), and the predictors are multicollinear, bad idea to make real-world decisions yet.</p><p>We’ve now seen what happens when the assumptions of multiple linear regression are violated — high multicollinearity, small sample size, heteroskedasticity and weak relationships lead to unstable coefficients, low R square, and unreliable inference.</p><p>But that raises an important question: <strong>What does a “good” case look like?</strong></p><p>To answer that, let’s train our model on a dataset with:</p><ul><li><strong>Lower correlation between predictors</strong> (less multicollinearity)</li><li><strong>A much larger sample size</strong> (more statistical power)</li><li><strong>A stronger linear relationship</strong> between predictors and the target</li></ul><p>The California Housing dataset is a good choice for this. It’s got thousands of rows and a set of predictors that — while not perfectly independent — aren’t tangled up with each other like in the linnerud example. This should give us a chance to see how multiple linear regression performs when the conditions are more in its favor, and check if we actually get more stable coefficients, a better R square, and cleaner residual patterns. To keep it simple let’s use models from scikit-learn.</p><pre>from sklearn.datasets import fetch_california_housing<br>from sklearn.linear_model import LinearRegression<br>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.metrics import r2_score<br><br>data = fetch_california_housing(as_frame=True)<br>df = data.frame<br><br>plt.figure(figsize=(8, 6))<br>ax = sns.heatmap( df.corr(), cmap=&quot;RdBu_r&quot;, annot=True, fmt=&quot;.2f&quot;, vmin=-1, vmax=1, linewidths=0.5, cbar_kws={&quot;shrink&quot;: 0.8})<br>ax.set_title(&quot;Correlation Heatmap&quot;, fontsize=15, pad=12, weight=&#39;bold&#39;)<br>plt.xticks(rotation=45, ha=&quot;right&quot;)<br>plt.yticks(rotation=0)<br>plt.tight_layout()<br>plt.show()<br><br>X = df[df.columns[:-1]].to_numpy() # All columns except MedHouseVal<br>Y = np.log(df[df.columns[-1]].to_numpy())<br><br><br>X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=42)<br><br>scalar = StandardScaler()<br><br>X_train_scaled = scalar.fit_transform(X_train)<br>X_test_scaled = scalar.transform(X_test)<br><br>model = LinearRegression()<br><br>model.fit(X_train_scaled, Y_train)<br><br>Y_pred = model.predict(X_train_scaled)<br><br>RSS = np.sum((Y_train - Y_pred) ** 2)<br><br>R2 = r2_score(Y_train, Y_pred)<br><br>error_var = RSS/(X_train_scaled.shape[0] - (X_train_scaled.shape[1] + 1))<br><br>corr_matrix = np.linalg.pinv(X_train_scaled.T @ X_train_scaled) * error_var<br><br>beta_variances = np.diag(corr_matrix)<br><br>print(f&quot;No of trained samples : {X_train.shape[0]}\nNo of Variables : {X_train.shape[1]}&quot;)<br>print(f&quot;RSS : {RSS:.2f}&quot;)<br>print(f&quot;R Square : {R2:.2f}&quot;)<br>print(f&quot;beta : {model.coef_}&quot;)<br>print(f&quot;beta se : {beta_variances ** 0.5}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*hbeKuxH0PX9IlEEDgtNeMQ.png" /></figure><pre>No of trained samples : 14448<br>No of Variables : 8<br>RSS : 1791.52<br>R Square : 0.62<br>beta : [ 0.36322621  0.0311335  -0.09716159  0.12842756  0.01028357 -0.01818082 -0.60122561 -0.56687028]<br>beta se : [0.00462961 0.00325881 0.00839779 0.0077161  0.00312287 0.00294613 0.00883119 0.00867131]</pre><p>The model is explaining around 62% of the variability in Y_train compared to base model. Things we can interpret:</p><ul><li>The AveRooms and AveBedrms are positively correlated but the coefficients of those variables are high and opposite in sign. This is the textbook example of multicollinearity. We can’t use these for inference, there high SE’s is backing up our interpretation.</li><li>The latitude and longitude are almost perfectly correlated, that resulted in higher coefficients but signs are similar to the direction they are correlated. The model is using the variables by giving unstable coefficient estimates but they are still random values, this means if your focus is on prediction score you can keep them but not accurate for inference.</li></ul><pre>residuals = Y_train - Y_pred<br><br>plt.figure(figsize=(8, 6))<br>sns.scatterplot(Y_pred, residuals, alpha=0.6, edgecolor=None)<br><br>plt.axhline(0, color=&#39;red&#39;, linestyle=&#39;--&#39;, linewidth=1)<br>plt.xlabel(&quot;Predicted Values&quot;, fontsize=12)<br>plt.ylabel(&quot;Residuals&quot;, fontsize=12)<br>plt.title(&quot;Residuals vs. Predicted Values&quot;, fontsize=14, fontweight=&#39;bold&#39;)<br>sns.despine()<br>plt.grid(True, linestyle=&#39;--&#39;, alpha=0.5)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*k4BxxJNoVGd8TGiEixnr3A.png" /></figure><p>We can see heteroskedasticity in data even after applying log, we can use robust standard errors, perform weighted least square to handle heteroskedasticity. I think heteroskedasticity we are seeing came from the fact that we have more options and variations of houses with high prices i.e. in real world higher MedHouseVal has high variability.</p><p>Even though we saw some collinearity in variables, let’s do some coefficient interpretation</p><pre>percent_interpretation = (np.exp(model.coef_) - 1) * 100<br><br>for i, col in enumerate(df.columns[:-1]):<br>    change = percent_interpretation[i]<br>    direction = &#39;increasing&#39; if change &gt; 0 else &#39;decreasing&#39;<br>    print(f&quot;For 1 unit increase in {col}, the MedHouseVal is {direction} by {abs(change):.2f}%&quot;)</pre><pre>For 1 unit increase in MedInc, the MedHouseVal is increasing by 43.80%<br>For 1 unit increase in HouseAge, the MedHouseVal is increasing by 3.16%<br>For 1 unit increase in AveRooms, the MedHouseVal is decreasing by 9.26%<br>For 1 unit increase in AveBedrms, the MedHouseVal is increasing by 13.70%<br>For 1 unit increase in Population, the MedHouseVal is increasing by 1.03%<br>For 1 unit increase in AveOccup, the MedHouseVal is decreasing by 1.80%<br>For 1 unit increase in Latitude, the MedHouseVal is decreasing by 45.19%<br>For 1 unit increase in Longitude, the MedHouseVal is decreasing by 43.27%</pre><p>“For 1 unit increase in AveRooms, the MedHouseVal is decreasing by 9.26%” This is multicollinearity talking. That’s the reason why it doesn’t make sense. From the coefficient interpretation of Latitude and Longitude, it seems houses in southwest California tend to be more expensive — which makes sense given it’s a coastal region. However, since these two variables are highly correlated, including both may cause multicollinearity and increase the risk of overfitting or unstable coefficient estimates.</p><p><strong>Conclusion</strong>:</p><p>Multiple linear regression is powerful but only if its assumptions hold up — especially the one about predictors not being too correlated. When multicollinearity sneaks in, your coefficients become unstable and hard to trust, even if the model still spits out decent predictions.</p><p>Small datasets with highly correlated features leads to shaky inference and low R square. Transforming the target variable, like using a log, can help with heteroskedasticity and make residuals behave better, but it won’t magically fix a weak underlying relationship or multicollinearity.</p><p>On the other side, bigger datasets with more independent predictors, like the California Housing data, give us more stable coefficients, better fit, and clearer insights — though some multicollinearity may still effect interpretation.</p><p>In short: If your goal is solid prediction, sometimes you can live with correlated variables. But if you want trustworthy inference and meaningful interpretation, watch out for multicollinearity and always check your assumptions. Keep an eye on residuals and variance, and don’t be afraid to transform or try other methods when the basic model struggles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf2618a81cc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linear Regression Explained: From Theory to Real-World Implementation]]></title>
            <link>https://mohith-g.medium.com/linear-regression-explained-from-theory-to-real-world-implementation-45b43faed743?source=rss-6575ac4820f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/45b43faed743</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistical-inference]]></category>
            <category><![CDATA[ordinary-least-square]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[maximum-likelihood]]></category>
            <dc:creator><![CDATA[Mohith]]></dc:creator>
            <pubDate>Sat, 09 Aug 2025 14:21:34 GMT</pubDate>
            <atom:updated>2025-08-09T14:21:34.142Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding the math, assumptions, and practical steps to predict continuous outcomes with confidence</h4><p>Predicting the values of dependent continues random variable Y from independent continues random variable X, with the help of a regression function r(x) which takes a value from X and returns a prediction ŷ. Our goal is to find a function r(x) that, for any given x, predicts y as accurate as possible ( ŷ ≈ y ). In statistics, the function that gives the minimum possible bias is :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/63/0*lx4j7zlLmS-enhpI" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/154/0*fK0BnX2Ef0Nz-ARw" /></figure><p>This is the <strong>conditional expectation </strong>— the average value of Y when X is fixed at x. It doesn’t mean we predict every single Y exactly(due to noise and randomness it’s hard) but it means we predict the average outcome given X, which is the best we can do without extra information. A simple example,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/0*ZOqK9M3PXqWLymAM" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*jQIR17JZckzezvQv" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/0*oU9O0rr5jUuhAZGb" /></figure><p>Yes, there is difference in our prediction values with actual values. In cases where ŷ should be 5,7,9,11,13,15 we gave a prediction 1 unit above or below the value which gave us the <strong>Residual Sum of Squares</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/177/0*lbCQgN6ruBZ4cV9H" /></figure><p>of 6.0 which is the sum of units our error predictions deviates from true values. So, our statistical function E[Y | X] which says, <strong>if there are multiple y values for a single fixed x the best prediction we can make with less error is predicting the average of those y values</strong>, which make sense. So, now we are confident that E[Y | X] gives us optimal prediction of random variable Y.</p><p>We don’t know what the real shape of E[Y | X] looks like. In theory, it can take any shape with population data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7xgrGXkS2eqs3n2pW6sUw.png" /></figure><p>In practice, we only get finite and noisy data points. So we need to approximate E[Y | X] curve.</p><p>If you ever used LinearRegression() from scikit-learn package:</p><pre>from sklearn.linear_model import LinearRegression<br><br>model = LinearRegression()<br>model.fit()<br>model.predict()</pre><p>You were agreed with these assumptions:</p><ol><li><strong>Linearity</strong> — Relationship between predictors and response is linear.</li><li><strong>Independence</strong> — Errors are independent from each other.</li><li><strong>Normality of errors</strong> — Errors are normally distributed (mainly for inference).</li><li><strong>Homoskedasticity</strong> — Constant variance of errors.</li><li><strong>No multicollinearity</strong> — Predictors aren’t highly correlated with each other.</li></ol><p>Let’s start from scratch and see how this assumptions are helping linear regression to make better predictions and inference.</p><p>Linearity because, we assume the variables X and Y have linear relationship i.e. We assume the E[Y | X] is a straight line. The straight line formula,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/98/0*ckFfms00cZE06YlL" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/163/0*LTy5Xe2fZ_zuuhHX" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/121/0*EujiSxern4aqgH6s" /></figure><ul><li>β1 is slope</li><li>β0 is intercept</li></ul><p>Just like any other parametric model where we assume our data follows some distribution (like normal, binomial, etc) and make predictions on the distribution, here we assume our E[Y | X] follows a straight line and make prediction using that straight line. This is the reason for calling <strong>linear regression a parametric model</strong>.</p><p>The parametric idea is also same, just like we estimate mean and variance in normal distribution, we can estimate our<em> β0</em> and <em>β1</em> values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/121/0*i2WbTdDmfWyo05z_" /></figure><p>we can use either <strong>MLE </strong>or <strong>Ordinary Least Square(OLS)</strong> method to find our point estimates.</p><p>In OLS :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*kMmulPVHAhDVMpJE" /></figure><p>set the above formula to 0 <em>w.r.t</em> <em>β0</em> and <em>β1</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/0*1Noz_MaCtTAEQgnO" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/0*TthOqeTV4xxiWqbq" /></figure><p>solving this will result in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/0*dsmCSn3F3cKYeFJU" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/98/0*qw2Lf7WJKudHuxSz" /></figure><p>In MLE:</p><p>we can say,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/123/0*eSya7fYUpGqZska_" /></figure><p>Assume error forms a normal distribution with mean 0 and an unbiased error variance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/98/0*qejT5dp_SMapOaUc" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/163/0*-A6LdOqvBjUVBCiO" /></figure><p>Therefore:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/92/0*YfCzlbOTH1CNfXt3" /></figure><p>The likelihood function would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/0*P1vNZZqmzIfdsqWj" /></figure><p>The log likelihood would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/0*JWisES4S7mccZSr0" /></figure><p>The log likelihood increases if the term:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/166/0*OHKQd3PVIdl6NtQ5" /></figure><p>Decreases. This term is the RSS, the thing we shrink in Least Squares. Under the normality assumption, boosting the log-likelihood just means cutting down this RSS — a neat case where MLE and Least Squares end up doing the same thing. So, taking the derivative of log likelihood <em>w.r.t </em>β0 and β1 would give us the same formulas.</p><p>Let’s apply this estimated β0 and β1 to our E[Y | X] example and see if the RSS is 6.0 as it will be if the least square parameters (β0, β1) finds out the best fitting line that gives minimum error:</p><pre>import numpy as np<br><br>x = np.array([1, 1, 1, 2, 2, 2, 3, 3, 3])<br>y = np.array([5, 6, 7, 9, 10, 11, 13, 14, 15])<br><br># Least Squares Estimates<br>beta1 = np.sum((x - np.mean(x)) * (y - np.mean(y))) / np.sum((x - np.mean(x))**2)<br>beta0 = np.mean(y) - beta1 * np.mean(x)<br><br># Fitting A Line<br>y_pred = beta0 + beta1 * x<br><br>RSS = np.sum((y - y_pred)**2)<br><br>print(&#39;Statistical optimal predictions: [6., 6., 6., 10., 10., 10., 14., 14., 14.]&#39;)<br>print(&#39;Model Prediction:&#39;, y_pred)<br>print(&#39;RSS: &#39;, RSS)</pre><pre>Statistical optimal predictions: [6., 6., 6., 10., 10., 10., 14., 14., 14.]<br>Model Prediction: [ 6.  6.  6. 10. 10. 10. 14. 14. 14.]<br>RSS:  6.0</pre><pre>import matplotlib.pyplot as plt<br><br>plt.scatter(x, y, color=&#39;royalblue&#39;, edgecolor=&#39;black&#39;, s=50, alpha=0.8, label=&#39;Data Points&#39;)<br>plt.plot(x, y_pred, color=&#39;darkred&#39;, linewidth=2.5, label=&#39;Fitted Line&#39;)<br><br>plt.xlabel(&quot;X&quot;, fontsize=12)<br>plt.ylabel(&quot;Y&quot;, fontsize=12)<br>plt.title(&quot;Linear Regression Fit&quot;, fontsize=14)<br>plt.legend()<br>plt.grid(True, linestyle=&#39;-.&#39;, alpha=0.6)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*kwwYaxsw_zv0w4nrAlly7w.png" /></figure><pre>residuals = y - y_pred<br>plt.figure(figsize=(7, 5))<br>plt.scatter(y_pred, residuals, alpha=0.9, s=50, edgecolor=&#39;black&#39;, color=&#39;royalblue&#39;)<br>plt.axhline(0, color=&#39;darkred&#39;, linestyle=&#39;--&#39;, linewidth=1.5, label=&#39;Zero Residual Line&#39;)<br><br>plt.xlabel(&quot;Predicted Values&quot;, fontsize=12)<br>plt.ylabel(&quot;Residuals&quot;, fontsize=12)<br>plt.title(&quot;Residuals vs Predicted Values&quot;, fontsize=14)<br>plt.grid(True, linestyle=&#39;-.&#39;, linewidth=0.5, alpha=0.7)<br>plt.xticks(fontsize=10)<br>plt.yticks(fontsize=10)<br>plt.legend()<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*7bAYZfLOZQh8AP7PrrztdA.png" /></figure><p>The way the least square estimates predictions is exactly similar to E[Y | X] is because <strong>least square parameters are designed in a way to reduce the squared error (RSS) , statistically the way to reduce the error when you have multiple y’s for a single x is predicting the mean of those values.</strong></p><p>I like using <em>R square</em> score to judge the regression model performance, you can skip the following explanation if you already know how <em>R square</em> works</p><p><strong>R square :</strong></p><p>We know that,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/72/0*BtTZfyl0-CV_GIYp" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/0*8q7jqDkD3kQaCH3V" /></figure><p>And Total Sum of Squares(TSS) be the total variability in the dependent variable y. If we have a base model that always predicts mean of y for every data point x, then TSS can also be described as “The total variability <em>explained</em> by that base model compared to having no model at all”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/0*P20WmisiM6qSCZLP" /></figure><p>If we perform,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/0*hbLKJLBYP5iaK383" /></figure><p>Since, RSS is the variability our model didn’t explained.</p><ul><li>The resulted scalar will be 1 if our model performed same as base model (RSS = 1, TSS = 1) this means every ŷ our model predicted, is the mean of the dependent variable ȳ (ŷ = ȳ), which also means our model didn’t learned (or) didn’t found any patterns from X to understand variability in Y and just returning the mean of Y.</li><li>Scalar can be greater than 1 if model performs even worse than predicting just ȳ.</li><li>Scalar goes near 0 when numerator RSS goes down, means model explaining most of the variability in Y.</li></ul><p>We can invert the scale by,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/109/0*U2py3EgOWbgg6FRR" /></figure><p>Now,</p><ul><li><em>R square</em> &lt; 0 performing worse than base model</li><li><em>R square</em> = 0, our model isn’t explaining any variability in Y values compared to base model</li><li><em>R square</em> = 0.5, our model is explaining 50% more variability compared to base model</li><li>R square = 1, our model is explaining all the variability in Y. This is not common, especially in regression models where there is noise</li></ul><p>Finding<em> R square</em> of our straight line:</p><pre>RSS = np.sum((y - y_pred)**2)<br>TSS = np.sum((y - np.mean(y))**2)<br><br>R_square = 1 - (RSS/TSS)<br><br>print(f&quot;R square : {R_square:.4f}&quot;)</pre><pre>R square : 0.9412</pre><p>This means the model is explaining around 94% of the variability in Y compared to base model which predicts 10 all-the-time in our case.</p><p>Linear regression does give good prediction score but using “accuracy” as the metric for linear regression doesn’t really make sense — it’s a parametric model, built more for inference and interpretability than for raw prediction scores.</p><p>We can perform techniques like finding Confidence Intervals for parameters and predictions, Hypothesis testing, Interpreting our parametric results, p-values. For our example data, to see the 95% confidence interval for our slope, we can use a direct formula for variance of β1 without performing any bootstrap:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/145/0*W1l_yDWNh8PqeaBo" /></figure><pre>var = (1/(len(y) - 2)) * np.sum((y - y_pred)**2) # error variance<br><br>var_beta1 = var/np.sum((x - np.mean(x))**2)<br><br>se_beta1 = var_beta1 ** 0.5<br><br>alpha = 1 - 0.95<br><br>beta1_interval = [beta1 - norm.ppf(1 - alpha/2) * se_beta1, beta1 + norm.ppf(1 - alpha/2) * se_beta1]<br><br>print(f&quot;Standard Error of β1 : {se_beta1:.2f}&quot;)<br>print(f&quot;Estimated slope : {beta1}&quot;)<br>print(f&quot;With 95% confidence this interval contains the true slope : [{beta1_interval[0]:.2f}, {beta1_interval[1]:.2f}]&quot;)</pre><pre>Standard Error of β1 : 0.38<br>Estimated slope : 4.0<br>With 95% confidence this interval contains the true slope : [3.26, 4.74]</pre><p>From <a href="https://medium.com/@mohith-g/confidence-is-the-new-truth-76e9b880ff0a"><em>Confidence Is The New Truth</em></a> we saw that for the normal interval to work, the sampling distribution needs to be approximately normal. That means the sampling distribution of our slope should be normal — which it is, since we <strong>assumed Normality of Error</strong>. And because <em>β1</em>​ is just a linear combination of those errors, that normality carries over directly to the slope. In other words, if the errors are normal, our slope is normal too — making the whole confidence interval story click.</p><p>We’ve got the slope and intercept, we can now make predictions on new data. That’s one big reason we don’t stop at just writing E[Y∣X] for our sample — we actually fit a model so it can work beyond the data we’ve already seen.</p><pre>x_new = 5<br><br>y_pred_new = beta0 + beta1 * x_new<br><br>print(f&quot;For x:{x_new} y is {y_pred_new}&quot;)</pre><pre>For x:5 y is 22.0</pre><p>The example we took is very convenient for our least squares estimates to predict near true values since the growth or slope of the means is constant 4. This is easy for a straight line to handle. What if i nudge the values a bit, breaking the constant slope? Like</p><pre>x=np.array([1,1,1,2,2,2,3,3,3])<br>y=np.array([5,6,7,9,10,11,17,18,19])</pre><p>Fitting a straight line through 6, 10, 18 is not possible with just 3 units distance in x-axis. The RSS increases, The Error Variance Increases, The Standard Error Increases, The Uncertainty Increases, The width of CI increases. Let’s see if it happens:</p><pre>x=np.array([1,1,1,2,2,2,3,3,3])<br>y=np.array([5,6,7,9,10,11,17,18,19])<br><br><br>beta1_m2 = np.sum((x - np.mean(x))*(y - np.mean(y))) / np.sum((x - np.mean(x))**2)<br>beta0_m2 = np.mean(y) - beta1_m2 * np.mean(x)<br><br>y_pred = beta0_m2 + beta1_m2 * x<br><br>RSS = np.sum((y - y_pred)**2)<br><br>print(&#39;Model Prediction:&#39;, y_pred)<br>print(f&quot;RSS : {RSS:.2f}&quot;)<br>print(f&#39;Slope: {beta1_m2}&#39;)</pre><pre>Model Prediction: [ 5.33333333  5.33333333  5.33333333 11.33333333 11.33333333 11.33333333 17.33333333 17.33333333 17.33333333]<br>RSS : 14.00<br>Slope: 6.0</pre><pre>plt.scatter(x, y, color=&#39;royalblue&#39;, edgecolor=&#39;black&#39;, s=50, alpha=0.8, label=&#39;Data Points&#39;)<br>plt.plot(x, y_pred, color=&#39;darkred&#39;, linewidth=2.5, label=&#39;Fitted Line&#39;)<br>plt.xlabel(&quot;X&quot;, fontsize=12)<br>plt.ylabel(&quot;Y&quot;, fontsize=12)<br>plt.title(&quot;Linear Regression Fit&quot;, fontsize=14)<br>plt.legend()<br>plt.grid(True, linestyle=&#39;-.&#39;, alpha=0.6)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*rGPIKiPZV6Li0tjJaGCb-w.png" /></figure><p>The RSS is increased and we can see the line tried to fit the updated data points with getting as minimum error as possible.</p><pre>residuals = y - y_pred<br>plt.figure(figsize=(7, 5))<br>plt.scatter(y_pred, residuals, alpha=0.9, s=50, edgecolor=&#39;black&#39;, color=&#39;royalblue&#39;)<br>plt.axhline(0, color=&#39;darkred&#39;, linestyle=&#39;--&#39;, linewidth=1.5, label=&#39;Zero Residual Line&#39;)<br><br>plt.xlabel(&quot;Predicted Values&quot;, fontsize=12)<br>plt.ylabel(&quot;Residuals&quot;, fontsize=12)<br>plt.title(&quot;Residuals vs Predicted Values&quot;, fontsize=14)<br>plt.grid(True, linestyle=&#39;-.&#39;, linewidth=0.5, alpha=0.7)<br>plt.xticks(fontsize=10)<br>plt.yticks(fontsize=10)<br>plt.legend()<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*MGY4nUaMgjC5Z9cW9A-xfQ.png" /></figure><p>The straight line is not doing a bad job, it’s doing great with its possible physical traits as we can see in the above plot, none of the predicted values have 0 error but it’s the minimum error it could get, satisfying all the data points.</p><pre>RSS = np.sum((y - y_pred)**2)<br>TSS = np.sum((y - np.mean(y))**2)<br><br>R_square = 1 - (RSS/TSS)<br><br>print(f&quot;R square : {R_square:.4f}&quot;)</pre><pre>R square : 0.9391</pre><p>The <em>R square </em>dropped only a little even though the RSS is higher than before, because it is still explaining a lot of variability in the Y values compared to just predicting ȳ.</p><pre>var = (1/(len(y) - 2)) * np.sum((y - y_pred)**2)<br><br>var_beta1_m2 = var/np.sum((x - np.mean(x))**2)<br><br>se_beta1_m2 = var_beta1_m2 ** 0.5<br><br>alpha = 1 - 0.95<br><br>beta1_interval = [beta1_m2 - norm.ppf(1 - alpha/2) * se_beta1_m2, beta1_m2 + norm.ppf(1 - alpha/2) * se_beta1_m2]<br><br>print(f&quot;Standard Error of β1 : {se_beta1_m2:.2f}&quot;)<br>print(f&quot;Estimated slope : {beta1_m2}&quot;)<br>print(f&quot;With 95% confidence this interval contains the true slope : [{beta1_interval[0]:.2f}, {beta1_interval[1]:.2f}]&quot;)</pre><pre>Standard Error of β1 : 0.58<br>Estimated slope : 6.0<br>With 95% confidence this interval contains the true slope : [4.87, 7.13]</pre><p>Model B has extra residuals compared to model A, and if those residuals increase the spread of errors, they raise the variance. Higher variance means a higher standard Error (as it did for 0.58), which indicates that model B’s β estimates leave more unexplained variation than model A. Higher SE can also increases the width of CI with same confidence.</p><p>Take aways until now:</p><ul><li>In linear regression, we assume normality of errors so that the slope’s sampling distribution is normal, enabling valid normal-based confidence intervals.</li><li>The fitted slope and intercept let us predict on new data, unlike simply stating E[Y∣X] for our sample.</li><li>The residual sum of squares (RSS) is what we minimize in least squares — and under normal errors, this is equivalent to maximizing the likelihood.</li><li>Extra residuals can increase the spread of errors, raising variance and standard deviation, which means the model’s β estimates are leaving more variation unexplained.</li></ul><p>Let’s see how a straight line is performing compared to E[Y|X] on real world data:</p><pre>#  LINEAR REGRESSION<br><br>import pandas as pd<br><br>df = pd.read_csv(&#39;weight-height.csv&#39;, usecols=[&#39;Height&#39;, &#39;Weight&#39;])<br><br>train_samples = int(len(df) * 0.7)<br><br>samples_idx = np.random.choice(len(df), size=train_samples, replace=False)<br><br>df_train = df.iloc[samples_idx]<br>df_test = df.drop(df.index[samples_idx])<br><br>X_train = df_train[&#39;Height&#39;].to_numpy()<br>Y_train = df_train[&#39;Weight&#39;].to_numpy()<br><br>X_test = df_test[&#39;Height&#39;].to_numpy()<br>Y_test = df_test[&#39;Weight&#39;].to_numpy()<br><br>X_mean = np.mean(X_train)<br>Y_mean = np.mean(Y_train)<br><br>beta1 = np.sum((X_train - X_mean) * (Y_train - Y_mean)) / np.sum((X_train - X_mean)**2)<br>beta0 = Y_mean - beta1 * X_mean<br><br>Y_pred = beta0 + beta1 * X_train</pre><pre># Model with E[Y|X] near performance<br>from scipy.stats import binned_statistic<br><br># Bin the heights and compute average weight in each bin -&gt; E[Y|X]<br>bin_means, bin_edges, _ = binned_statistic(X_train, Y_train, statistic=&#39;mean&#39;, bins=50)<br># Get the centers of the bin for plotting accuracy<br>bin_centers = (bin_edges[:-1] + bin_edges[1:]) / 2</pre><p>Binned statistics group nearby x values into bins, since in regression data we rarely get identical x values repeating. This is about as close as we can get to replicating E[Y|X].</p><pre>plt.figure(figsize=(8, 5))<br>plt.scatter(X_train, Y_train, alpha=0.1, label=&#39;Training Data&#39;)<br>plt.plot(bin_centers, bin_means, color=&#39;black&#39;, linewidth=2, label=&#39;Estimated E[Y|X] (bin average)&#39;)<br>plt.plot(np.sort(X_train), np.sort(Y_pred), color=&#39;red&#39;, alpha=0.7,linewidth=2, label=&#39;Linear Regression&#39;)<br>plt.xlabel(&#39;Height&#39;)<br>plt.ylabel(&#39;Weight&#39;)<br>plt.legend()<br>plt.grid(True)<br>plt.title(&#39;Estimated E[Y|X] vs Linear Regression&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*yDymASmXxB1TFOlWZdkLeA.png" /></figure><p>The linear regression model (red line) closely follows the binned estimate of E[Y|X](black line) across the full range of heights, indicating that the linearity assumption holds strongly for this dataset. The small deviations of the binned curve from the straight line are minor and mostly due to local fluctuations in the data rather than systematic bias. The tight clustering of points around both lines suggests low variance and high predictive accuracy, with no visible pattern in residual spread. The model captures the underlying relationship between height and weight very effectively.</p><pre>RSS = np.sum((Y_train - Y_pred)**2)<br>TSS = np.sum((Y_train - Y_mean)**2)<br><br>R_square = 1 - (RSS/TSS)<br><br>print(f&quot;R square : {R_square:.4f}&quot;)</pre><pre>R square : 0.8562</pre><p>The linear regression model capturing around 85% of the variability in the data compared to base model, that’s not bad at all. Let’s see if our assumption of normality in error holds for this data:</p><pre>import seaborn as sns<br><br>residuals = Y_train - Y_pred<br><br>plt.figure(figsize=(8, 5))<br>sns.histplot(residuals, bins=35, kde=True, color=&quot;skyblue&quot;, edgecolor=&quot;black&quot;, alpha=0.7)<br><br>plt.title(&quot;Residuals Distribution&quot;, fontsize=16, fontweight=&#39;bold&#39;)<br>plt.xlabel(&quot;Residual&quot;, fontsize=14)<br>plt.ylabel(&quot;Frequency&quot;, fontsize=14)<br>plt.axvline(0, color=&#39;red&#39;, linestyle=&#39;--&#39;, linewidth=1.5, label=&quot;Zero Residual Line&quot;)<br>plt.legend()<br>plt.grid()<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*cFMfhJkGlCkyv36ZYdUmOw.png" /></figure><p>The residuals appear approximately normally distributed, which supports our normality of errors assumption. This means we can confidently apply statistical inference — such as confidence intervals and hypothesis tests — on our model parameters. While normality of errors ensures our parameter estimates follow the distributions we rely on for inference, there’s another equally important assumption: <strong>homoskedasticity</strong>.</p><p><strong>Homoskedasticity:</strong></p><p>Homoskedasticity means the variance of the errors should be constant across all values of X. If the spread of residuals changes with X (a “fan” or “cone” shape in a residual plot), we have <strong>heteroskedasticity</strong>, which can lead to unreliable standard errors and misleading confidence intervals — even if the errors are normally distributed. Which also means error variance (σ²) is only valid for inference(we used error variance in past to get variance of β1) if and only if the variance of the errors at each data point x is constant. Below plots show the text book definition of homoskedasticity and heteroskedasticity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h-cy8SnMVsWlzBA0z9gR3w.png" /></figure><p>For example, take two x values from the heteroskedasticity plot: x1 = 2 and x2 = 8. After prediction, suppose we get ŷ1 = 3 and ŷ2 = 6, while the actual values are y1 = 2.5 and y2 = 4.</p><p>If we construct confidence intervals for these predictions using the error variance from heteroskedastic data, the results may be misleading. Because the variance of errors changes with x, the CI for ŷ1 might be unnecessarily wide (overestimating uncertainty), while the CI for ŷ​2​ might be too narrow (underestimating uncertainty).</p><p>In contrast, with homoskedasticity, the error variance is constant and stable across all x values, so doing inference is more reliable. This is the reason why we <strong>assume homoskedasticity</strong> in linear regression</p><p>Note: there are statistical techniques to address heteroskedasticity, but some real world data — such as housing prices, where higher-priced homes have more variability — naturally exhibit it.</p><p>We can visualize doing a plot of either ( X vs Residuals) or (Y predictions vs residuals) since ŷ looks similar to x and in multiple regression(regression with more than 1 variable) ŷ can be our go to.</p><pre>plt.figure(figsize=(8, 5))<br>sns.scatterplot(Y_pred, residuals, alpha=0.6, edgecolor=None)<br>plt.axhline(0, color=&#39;red&#39;, linestyle=&#39;--&#39;, linewidth=1.5)<br>plt.xlabel(&quot;Predicted Values&quot;, fontsize=12)<br>plt.ylabel(&quot;Residuals&quot;, fontsize=12)<br>plt.title(&quot;Residuals vs Predicted Values&quot;, fontsize=14, fontweight=&#39;bold&#39;)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*mpKtDABMh3l4NxrPLGZcMQ.png" /></figure><p>I don’t see any patterns or errors expanding with predicted values all the errors are scattered around the 0 errored line, so we don’t have heteroskedastic data.</p><pre>VAR = (1/(len(Y_train) - 2)) * np.sum((Y_train - Y_pred)**2)<br><br>VAR_BETA1 = VAR / np.sum((X_train - X_mean)**2)<br><br>SE_BETA1 = VAR_BETA1 ** 0.5<br><br>print(f&#39;A valid variance due to homoskedasticity : {VAR:.4f}&#39;)<br>print(f&quot;Standard Error of parameter β1: {SE_BETA1 :.4f}&quot;)</pre><pre>A valid variance due to homoskedasticity : 148.0742<br>Standard Error of parameter β1: 0.0376</pre><p>Finding the true β1 (true slope) on true height and weights data would look like:</p><pre>alpha = 1 - 0.95<br><br>beta1_interval = beta1 - norm.ppf(1 - alpha/2)* SE_BETA1, beta1 + norm.ppf(1 - alpha/2) * SE_BETA1<br><br>print(f&quot;Our Estimate of true slope : {beta1:.2f}&quot;)<br>print(f&quot;With 95% confidence this interval contains the true slope : [{beta1_interval[0]:.2f}, {beta1_interval[1]:.2f}]&quot;)</pre><pre>Our Estimate of true slope : 7.71<br>With 95% confidence this interval contains the true slope : [7.63, 7.78]</pre><p>Now we got slope and intercept we can predict weight given height:</p><pre>X_new = 73.5<br>Y_new = beta0 + beta1 * X_new<br><br>SE_Y_new = np.sqrt(<br>    VAR * (1 + 1/len(X_train) + (X_new - X_mean)**2 / np.sum((X_train - X_mean)**2))<br>)<br><br>alpha = 1 - 0.95<br>z = norm.ppf(1 - alpha/2)<br><br>ci_lower = Y_new - z * SE_Y_new<br>ci_upper = Y_new + z * SE_Y_new<br><br><br>print(f&#39;For height of {X_new} the predicted weight is: {Y_new:.2f}&#39;)<br>print(f&quot;95% CI for prediction at X={X_new}: [{ci_lower:.2f}, {ci_upper:.2f}]&quot;)</pre><pre>For height of 73.5 the predicted weight is: 216.42<br>95% CI for prediction at X=73.5: [192.56, 240.28]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/0*ZB-ODmpr_E5MIyzA" /></figure><p>The standard error of a new prediction shows how uncertain that prediction is. It combines the natural spread of data points around the regression line, the fact that we estimated the model from limited data, and how far the new input is from the average of what the model has seen before. Predictions for values far from the average come with more uncertainty. We use this standard error to build confidence or prediction intervals — ranges where the actual value is likely to fall.</p><p>Now let’s compare our parameters and <em>R square</em> score with LinearRegression() in scikit-learn library.</p><pre>from sklearn.linear_model import LinearRegression<br><br>model = LinearRegression()<br>model.fit(X_train.reshape(-1, 1), Y_train)<br><br><br>print(f&quot;β1 (sklearn): {model.coef_[0]:.4f}&quot;)<br>print(f&quot;β0 (sklearn): {model.intercept_:.4f}&quot;)<br><br>print(f&quot;β1 (manual): {beta1:.4f}&quot;)<br>print(f&quot;β0 (manual): {beta0:.4f}&quot;)<br><br>Y_test_pred = model.predict(X_test.reshape(-1, 1))<br><br>rss = np.sum((Y_test - Y_test_pred) ** 2)<br>tss = np.sum((Y_test - np.mean(Y_test)) ** 2)<br><br><br>model_r2_score = 1 - (rss / tss)<br><br>print(f&quot;Model&#39;s R square score (sklearn): {model_r2_score:.4f}&quot;)<br><br>Y_test_pred = beta0 + beta1 * X_test<br><br>RSS_test = np.sum((Y_test - Y_test_pred) ** 2)<br>TSS_test = np.sum((Y_test - np.mean(Y_test)) ** 2)<br><br>R_square_test = 1 - (RSS_test / TSS_test)<br><br>print(f&quot;Our R square score (manual): {R_square_test:.4f}&quot;)</pre><pre>β1 (sklearn): 7.7125<br>β0 (sklearn): -350.4167<br>β1 (manual): 7.7125<br>β0 (manual): -350.4167<br>Model&#39;s R square score (sklearn): 0.8611<br>Our R square score (manual): 0.8625</pre><p>Both sklearn’s LinearRegression and our manual calculation gave almost the exact same slope (β1) and intercept (β0). That means our manual method is spot on.</p><p>Looking at how the model does on new data (the test set), the R square scores are super close too — sklearn’s is 0.8611 and ours is 0.8625. The tiny difference is just from small rounding differences.</p><p>This shows both methods predict almost equally well. The model explains about 86% of the variation in weight based on height, which is pretty solid. So, our manual math works great, but sklearn makes things easier and less error-prone when you’re working with bigger data or more features.</p><p>We just wrapped up simple linear regression — where we modeled the relationship between one predictor and one response. But real-world problems rarely rely on just one factor. That’s where multiple linear regression comes in. It lets us bring in many variables, tease apart their effects, and build more powerful models.</p><p>But heads up: when you add more predictors, new challenges pop up. One biggie is <strong>multicollinearity</strong> — when predictors are highly correlated with each other. This can confuse the model, inflate errors, and make your coefficient estimates unstable and unreliable.</p><p>In the next blog, I’ll walk you through how multiple linear regression works, why multicollinearity matters, how to detect it, and what you can do about it. We’ll also dig into interpreting coefficients when predictors aren’t independent and how to keep your model solid.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45b43faed743" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Hands-On Look At Bayesian Inference]]></title>
            <link>https://mohith-g.medium.com/a-hands-on-look-at-bayesian-inference-e169d6353ab1?source=rss-6575ac4820f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/e169d6353ab1</guid>
            <category><![CDATA[bayes-theorem]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[inferential-statistics]]></category>
            <dc:creator><![CDATA[Mohith]]></dc:creator>
            <pubDate>Thu, 31 Jul 2025 12:57:24 GMT</pubDate>
            <atom:updated>2025-07-31T12:57:24.372Z</atom:updated>
            <content:encoded><![CDATA[<h4>What each type of prior actually does when you apply it — with plots and python</h4><p>Point estimation, Confidence Intervals, and Standard Error are all rooted in frequentist probability. This framework relies on the idea that, as the number of data points increases, our estimates converge to the true population parameter. The confidence interval narrows, the standard error approaches zero — all of this gives us a sense of how far we are from the truth, assuming the experiment can be repeated over and over. But what if repetition isn’t an option?</p><p>Not all real-world tasks allow for endless sampling or repeated experimentation. Think about estimating the failure risk of a one-of-a-kind spacecraft, or predicting election results with just a single survey. In such situations, the frequentist tools struggle — because the foundation they rest on (long-run frequency) simply doesn’t apply.</p><p>This is where Bayesian inference steps in. Instead of relying solely on repeatable data, Bayesian methods combine current evidence (likelihood) with prior beliefs (prior distribution) to update our understanding of the parameter (posterior distribution). It treats the unknown parameter itself as a random variable — not fixed but uncertain — and gives us a full distribution over what values that parameter could plausibly take, given what we know.</p><p>In essence, Bayesian inference isn’t just about data — it’s about belief, uncertainty, and updating. It’s particularly powerful when data is scarce, one-shot, or expensive to collect. And instead of only giving a point estimate or a confidence interval, it gives a complete picture of uncertainty — all in one posterior distribution.</p><p><strong>Bayes Theorem :</strong></p><p>Let A and B are two events, we know the conditional probability states:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/0*PMgy_11ZHvTQK9GN" /></figure><p>To see it in action, let’s take a deck of cards:</p><pre>Card = collections.namedtuple(&#39;Card&#39;,[&#39;rank&#39;, &#39;suit&#39;])<br><br>class FrenchDeck:<br>    <br>    ranks = [str(n) for n in range(2, 11)] + list(&#39;JQKA&#39;)<br>    suits = &#39;spades diamonds clubs hearts&#39;.split()<br>    <br>    def __init__(self):<br>        self._cards = [Card(rank, suit) for suit in self.suits for rank in self.ranks]<br>    <br>    def __len__(self):<br>        return len(self._cards)<br>    <br>    def __getitem__(self, position):<br>        return self._cards[position]<br>    <br>    def shuffle(self):<br>        random.shuffle(self._cards)<br>        <br>    def count(self, condition):<br>        return sum(1 for card in self._cards if condition(card))<br>    <br>    def draw(self):<br>        return random.choice(self._cards)<br><br>Deck = FrenchDeck()<br><br>print(f&quot;Total {len(Deck)} cards&quot;)</pre><pre>Total 52 cards</pre><p>And draw a card from the deck:</p><pre>Deck.shuffle()<br>card = Deck.draw()<br>print(card)</pre><pre>Card(rank=&#39;Q&#39;, suit=&#39;hearts&#39;)</pre><p>We got a queen of hearts, now let event A be drawing a card that’s queen and B be the event that the card is from hearts. What is the probability of getting a queen given the suit is hearts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/0*XaFzxTTzdJChxjph" /></figure><pre>is_queen = lambda card: card.rank == &#39;Q&#39;<br>is_heart = lambda card: card.suit == &#39;hearts&#39;<br>is_queen_and_heart = lambda card: card.rank == &#39;Q&#39; and card.suit == &#39;hearts&#39;<br><br>P_A = Deck.count(is_queen) / len(Deck)<br>P_B = Deck.count(is_heart) / len(Deck)<br>P_A_B = Deck.count(is_queen_and_heart) / len(Deck)<br>P_A_Given_B = P_A_B / P_B<br><br>print(f&quot;P(A): {P_A:.3f}&quot;)<br>print(f&quot;P(B): {P_B:.3f}&quot;)<br>print(f&quot;P(AB) : {P_A_B:.3f}&quot;)<br>print(f&quot;P(A|B) = P(AB) / P(B): {P_A_Given_B:.3f}&quot;)<br><br>hearts = [card for card in Deck if is_heart(card)]<br>P_A_given_B_empirical = sum(1 for card in hearts if is_queen(card)) / len(hearts)<br><br>print(f&quot;From data P(A|B): {P_A_given_B_empirical:.3f}&quot;)</pre><pre>P(A): 0.077<br>P(B): 0.250<br>P(AB) : 0.019<br>P(A|B) = P(AB) / P(B): 0.077<br>From data P(A|B): 0.077</pre><p>As we can see, both the mathematical calculation and the empirical check from the filtered subset give us the same result for P(A | B).</p><p>Now let’s connect this with a general identity in probability because not every time we get the data we want from the real world : the symmetry of joint probability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/0*XQfg2cUI5wxZgf9o" /></figure><p>From this, and using the definition of conditional probability, we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/0*QnQvYns3Afncj9AV" /></figure><p>Then we can write:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/325/0*F19UBotS2H5-EleA" /></figure><p>This is <strong>bayes theorem. </strong>Applying this to our card example:</p><pre>P_B_given_A = 1 / 4 # Only 1 out of 4 queens is from hearts<br>P_B_A = P_B_given_A * P_A<br><br>print(f&quot;To prove, P(BA): {P_B_A:.3f} and P(AB): {P_A_B:.3f}&quot;)<br><br>print(f&quot;P(A|B) = (P(B|A)*P(A)) / P(B) : {P_B_A / P_B:.3f} same as P(A|B) = P(AB) / P(B): {P_A_Given_B:.3f}&quot;)</pre><pre>To prove, P(BA): 0.019 and P(AB): 0.019<br>P(A|B) = (P(B|A)*P(A)) / P(B) : 0.077 same as P(A|B) = P(AB) / P(B): 0.077</pre><p><strong>Role of Bayes Theorem in Inference :</strong></p><p>The goal of inference is to estimate the true parameter θ based on the observed data. In Bayesian inference, this is expressed as the <strong>posterior distribution</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/83/0*WTiMXeFH0ToTaioj" /></figure><p>Now we know bayes theorem,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/223/0*FvrifISAYvCHZjJr" /></figure><p>Applying to our inference,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/0*_3AF7R-SjNbESi3q" /></figure><ul><li><em>P</em>(<em>data </em>| <em>θ</em>) is <strong>Likelihood</strong>, which tells us how probable the observed data is <strong>assuming</strong> a particular value of the parameter θ.</li><li><em>P(θ)</em> is <strong>Prior</strong>, which encodes our beliefs or assumptions about the parameter θ <strong>before seeing the data</strong>.</li><li><em>P(data)</em> is Marginal Likelihood, it acts as a <strong>normalizing constant</strong>.</li></ul><p><strong>Likelihood :</strong></p><p>Depending on the goal of inference, <em>data </em>can be a data point or a list of data points. The well know likelihood formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/0*yvnn3jsrhRYI_LfF" /></figure><p><em>P(xi | θ) </em>is the PDF of assumed distribution (like normal, binomial, etc..). Bayesian inference becomes <strong>parametric</strong> because, to compute the <strong>likelihood</strong> — the joint probability of the observed data given a parameter — we must <strong>assume a distributional form</strong> for the data.</p><p>Likelihood is used in both MLE and Bayesian inference : In MLE, we maximize <em>L(</em>𝜃<em>)</em>. In Bayesian inference, we multiply it by a prior to get the posterior</p><p><strong>Prior :</strong></p><p>The prior reflects our belief about the parameter before seeing any data. We can shape the prior based on domain knowledge or assumptions about how the data behaves. Depending on how confident we are, the prior can either:</p><ul><li>Trust the data completely<strong> </strong>by being vague or flat, letting the likelihood dominate</li><li>Or make it hard to convince by being sharply concentrated, requiring strong evidence from the data to shift our belief</li></ul><p>The prior controls how flexible or stubborn our model is in the face of new data.</p><p><strong><em>Flat Prior :</em></strong></p><p>A flat prior assumes that all values of the parameter are equally likely before seeing any data. It’s basically saying: <em>“I don’t want to favor any value of θ, so just treat them all the same.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/67/0*ewpTGxd1Q9IF6eFK" /></figure><p>This means the prior doesn’t assign higher or lower weight to any region of θ, it’s just a constant across the entire range. But here’s the issue — this kind of prior is <strong>improper</strong>. That means it doesn’t integrate to 1 over the parameter space, so technically it’s not a valid probability distribution.</p><p>Still, we can use flat priors as long as the likelihood is strong enough to “save” the posterior — meaning the product of likelihood and prior is still normalizable (integrates to something finite). But the moment our likelihood is weak or spreads out too much (like with small or noisy data), this improper prior can mess up the posterior, making it improper too — and that ruins the whole inference.</p><p><strong><em>Invariant Prior :</em></strong></p><p>One of the problems with a flat prior is that it depends on how we express the parameter. It might look flat when we write the parameter as θ, but if we reparametrize it (say using log⁡θ or 1/θ), that same prior isn’t flat anymore. That means the prior depends on the coordinate system we are using — which is weird, because our prior belief shouldn’t change just because we chose a different way to write the parameter.</p><p>To fix that, we want a prior that doesn’t change when we reparametrize, a prior that’s invariant under transformations. That’s where Jeffreys Prior comes in.</p><p><strong><em>Jeffreys Prior :</em></strong></p><p>Jeffreys prior is designed to be invariant under reparametrization. It’s based on the fisher Information, which tells us how sensitive the likelihood is to changes in the parameter. The more sensitive the likelihood is, the more information our data gives you about the parameter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/109/0*8e_zSTMlj1k-5T-G" /></figure><p>Where I(θ) is the fisher Information:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*HXPf17M2uTXzMcOV" /></figure><p>So instead of being flat, jeffreys prior adapts to how informative the likelihood is. If the data tells us more about certain regions of θ, the prior puts more density there. That way, it doesn’t favor any specific value arbitrarily, but still respects the structure of the model.</p><p><strong>Inference in Practice :</strong></p><p>Before diving into Bayesian inference, let’s first load a list of body temperatures from the data file:</p><pre>df = pd.read_csv(&#39;thermometry.csv&#39;, usecols=[&#39;body.temp&#39;])<br>temp = df.to_numpy()<br>print(temp[:5])</pre><pre>[[96.3]<br> [96.7]<br> [96.9]<br> [97. ]<br> [97.1]]</pre><p>The distribution of the body temperatures follows an almost normal distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*xr4f2pqcXdftvYnmlM6OCg.png" /></figure><p>If our goal is to estimate the true population mean from this sample, Bayesian inference let’s us go beyond just a single point estimate — it gives us a full probability distribution of what the true mean could be, based on both the data and our prior beliefs.</p><p>Before designing the model, i want to address that the data has some outliers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*CFkMhECgZZ3wCino2Wjq3A.png" /></figure><p>Because the mean is influenced by outliers, we’ll have to use a prior that adds resistance against overfitting to unusual data points.</p><ol><li><strong><em>Flat Prior:</em></strong></li></ol><p>As we discussed, flat prior assumes all the values of the parameter are equally likely. So, let mu_vals be the list of values that our parameter might take on.</p><pre>import numpy as np<br><br>mu_vals = np.linspace(95, 100, 500)<br><br>flat_prior = np.ones_like(mu_vals)<br><br>sigma = np.std(temp)<br><br>log_likelihoods = np.array([np.sum(norm.logpdf(temp, loc=m, scale=sigma)) for m in mu_vals])<br><br>likelihoods = np.exp(log_likelihoods)<br><br># posterior = likelihood * prior<br><br>flat_posterior = likelihoods * flat_prior<br><br># To turn the data into probabilities we normalize the data<br>flat_posterior /= np.sum(flat_posterior)<br><br><br>plt.figure(figsize=(8, 5))<br>sns.lineplot(x=mu_vals, y=flat_posterior, color=&quot;green&quot;, linewidth=1)<br>plt.fill_between(mu_vals, flat_posterior, color=&quot;green&quot;, alpha=0.3)<br><br>plt.xlabel(&quot;Mean Body Temperature (°F)&quot;)<br>plt.ylabel(&quot;Posterior Density&quot;)<br>plt.title(&quot;Posterior Distribution with Flat Prior&quot;)<br>plt.grid(True)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*bd7_6v5RDLnc0btfHaa-ag.png" /></figure><p>A narrow posterior suggests the model is highly confident in the estimated mean. <strong>But in the presence of outliers, certainty may reflect overconfidence rather than accuracy. </strong>To see how we got that posterior,</p><pre>flat_prior_norm = flat_prior / np.sum(flat_prior)<br>likelihoods_norm = likelihoods / np.sum(likelihoods)<br><br>plt.figure(figsize=(10, 6))<br><br>sns.lineplot(x=mu_vals, y=flat_prior_norm, label=&quot;Flat Prior&quot;, color=&quot;green&quot;)<br>sns.lineplot(x=mu_vals, y=likelihoods_norm, label=&quot;Likelihood&quot;, color=&quot;blue&quot;)<br><br>plt.fill_between(mu_vals, flat_prior_norm, color=&quot;green&quot;, alpha=0.15)<br>plt.fill_between(mu_vals, likelihoods_norm, color=&quot;blue&quot;, alpha=0.15)<br><br>plt.xlabel(&quot;Mean Body Temperature (°F)&quot;)<br>plt.ylabel(&quot;Normalized Density&quot;)<br>plt.title(&quot;Flat Prior, Likelihood&quot;)<br>plt.grid(True)<br>plt.legend()<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x74f-3-oy_Geu76j4aY9pg.png" /></figure><p>Since, we multiplied the likelihood with our prior belief (which is constant over all value the parameter could take) the <strong>model blindly trusts the data it sees. </strong>Resulted in fully data driven posterior. To prove that the model blindly trusts the data :</p><pre>flat_mu = mu_vals[np.argmax(flat_posterior)]<br>likelihood_mu = mu_vals[np.argmax(likelihoods_norm)]<br><br>print(f&quot;Posterior mean (flat prior): {flat_mu:.3f}&quot;)<br>print(f&quot;Maximum Likelihood Estimate: {likelihood_mu:.3f}&quot;)<br>print(f&quot;Are they equal? {flat_mu == likelihood_mu}&quot;)</pre><pre>Posterior mean (flat prior): 98.246<br>Maximum Likelihood Estimate: 98.246<br>Are they equal? True</pre><p>This shows that both the posterior (with flat prior) and the likelihood peak at a mean of 98.246 — a value heavily influenced by outliers in the data. Therefore, it’s unlikely to reflect the true underlying population mean.</p><p>Let’s take middle 50% of the data to get rid of outliers from the temp data and see how the posterior is changed from what we got above.</p><pre>sigma_all = np.std(temp)<br>flat_prior = np.ones_like(mu_vals)<br><br>log_lik_all = np.array([np.sum(norm.logpdf(temp, loc=m, scale=sigma_all)) for m in mu_vals])<br>likelihoods_all = np.exp(log_lik_all)<br>posterior_all = likelihoods_all * flat_prior<br>posterior_all /= np.sum(posterior_all)<br><br># posterior_all is our distribution of the mean values containing outliers in the data<br><br>Q1 = np.percentile(temp, 25)<br>Q3 = np.percentile(temp, 75)<br>IQR = Q3 - Q1<br>lower = Q1 - 1.5 * IQR<br>upper = Q3 + 1.5 * IQR<br>temp_clean = temp[(temp &gt;= lower) &amp; (temp &lt;= upper)]<br><br><br>sigma_clean = np.std(temp_clean)<br>log_lik_clean = np.array([np.sum(norm.logpdf(temp_clean, loc=m, scale=sigma_clean)) for m in mu_vals])<br>likelihoods_clean = np.exp(log_lik_clean)<br>posterior_clean = likelihoods_clean * flat_prior<br>posterior_clean /= np.sum(posterior_clean)<br><br># posterior_clean is our distribution of the mean values containing middle 50% of data<br><br>plt.figure(figsize=(9, 5))<br>sns.lineplot(x=mu_vals, y=posterior_all, label=&quot;With Outliers&quot;, color=&quot;red&quot;, linewidth=1.5)<br>sns.lineplot(x=mu_vals, y=posterior_clean, label=&quot;Outliers Removed&quot;, color=&quot;green&quot;, linewidth=1.5)<br><br>plt.fill_between(mu_vals, posterior_all, color=&quot;red&quot;, alpha=0.2)<br>plt.fill_between(mu_vals, posterior_clean, color=&quot;green&quot;, alpha=0.2)<br><br>plt.xlabel(&quot;Mean Body Temperature (°F)&quot;)<br>plt.ylabel(&quot;Posterior Density&quot;)<br>plt.title(&quot;Effect of Outliers on Posterior (Flat Prior)&quot;)<br>plt.legend()<br>plt.grid(True)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*uBF2-NciHo0H6bZb8s0_og.png" /></figure><p>The posterior with outliers has <strong>lower density and higher standard deviation.</strong> This makes sense because, the likelihood of a parameter value is the product of each data points PDF values <em>w.r.t </em>the parameter value. Outliers have a small PDF value because they’re far from the mean, and yeah — it’s small because the normal distribution scales everything using the standard deviation and the standard deviation for the data with outliers is high. Because of this reason we multiply the PDF values with values that are very small (near 0) making the entire product smaller. This happens to every parametric value. So the density effects in the presence of outliers.</p><p>Since, the mean is sensitive to outliers. The peak(MLE) might also change.</p><pre>mle_with_outliers = mu_vals[np.argmax(posterior_all)]<br>mle_without_outliers = mu_vals[np.argmax(posterior_clean)]<br><br>print(f&quot;MLE with outliers: {mle_with_outliers} °F&quot;)<br>print(f&quot;MLE without outliers: {mle_without_outliers} °F&quot;)</pre><pre>MLE with outliers: 98.24649298597194 °F<br>MLE without outliers: 98.2565130260521 °F</pre><p>There’s just a small shift to the right — that’s because most of the outliers were on the left, as we saw in the box plot. When we trimmed the data to just the middle 50%, it reduced the weight of those left-side outliers. So the right-side data had more influence now. But since we’re using fewer data points now, even small shifts or imbalances in what’s left can nudge the mean a bit. To prove the standard deviation will be higher for the data with outliers:</p><pre>print(f&quot;{sigma_all:.3f} &gt; {sigma_clean:.3f}&quot;)</pre><pre>0.730 &gt; 0.662</pre><p><strong><em>Take Away from Flat Prior:</em></strong></p><p>Using a flat prior is like telling the model: “I have no clue about the parameter, so just trust the data.” It doesn’t push back at all. That’s cool when your data is clean, but risky when it’s noisy or has outliers — because the model blindly believes whatever the data says. In my case, it locked onto the outlier-affected mean, proving it’s vulnerable when the data isn’t perfect.</p><p>2. <strong><em>Weak Prior :</em></strong></p><p>Let’s say we want our prior belief to stay consistent even if we reparameterize the model — basically, we don’t want the prior to change just because we view the problem differently. So, we go with a weak prior centered at the sample mean, but with a wide spread, like:</p><pre>weak_prior = norm.pdf(mu_vals, loc=np.mean(temp), scale=10)<br><br>weak_posterior = likelihoods * weak_prior<br><br>weak_posterior /= np.sum(weak_posterior)<br><br>plt.figure(figsize=(8, 5))<br>sns.lineplot(x=mu_vals, y=weak_posterior, color=&quot;green&quot;, linewidth=1)<br>plt.fill_between(mu_vals, weak_posterior, color=&quot;green&quot;, alpha=0.3)<br><br>plt.xlabel(&quot;Mean Body Temperature (°F)&quot;)<br>plt.ylabel(&quot;Posterior Density&quot;)<br>plt.title(&quot;Posterior Distribution with Weak Prior&quot;)<br>plt.grid(True)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*KavkAnlsFZM38IPFPca1_w.png" /></figure><p>It looks similar to the flat prior, let’s see the likelihood(which would be same) and weak prior’s distribution over mean value:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*C-RTxH2U3jNyZQFZTkRLBw.png" /></figure><p>See that the prior is not flat anymore, means we are not treating all the mean values same. Instead we assumed a prior with some information. The reason the posterior still looks the same, even though this prior has some information, is because the information we gave wasn’t strong enough for the model to push through the data. It believed in the data more because the likelihood was more convincing, given that our prior had more uncertainty (higher standard deviation).</p><pre>weak_mu = mu_vals[np.argmax(weak_posterior)]<br>likelihood_mu = mu_vals[np.argmax(likelihoods_norm)]<br><br>print(f&quot;Posterior mean (weak prior): {weak_mu:.3f}&quot;)<br>print(f&quot;Maximum Likelihood Estimate: {likelihood_mu:.3f}&quot;)<br>print(f&quot;Are they equal? {weak_mu == likelihood_mu}&quot;)</pre><pre>Posterior mean (weak prior): 98.246<br>Maximum Likelihood Estimate: 98.246<br>Are they equal? True</pre><p>You can also see this in the fact that the peak of the posterior is exactly at the same value as the likelihood — meaning the model didn’t resist the data at all. Even though we had some belief baked into the prior, it wasn’t strong enough to fight back. So, the model gave in to the data completely, landing at the same mean.</p><p><strong><em>Take Away from Weak Prior :</em></strong></p><p>With a weak prior, we’re not blindly assuming all values are equal like in a flat prior — we’re adding some belief, just not a strong one. But because this belief is uncertain (wide standard deviation), the model still ends up trusting the data more. Yeah, the prior had a shape, but it didn’t really push back. The posterior still peaks where the likelihood does, showing that the model didn’t take the prior too seriously. It’s like saying, “I have an opinion, but I’ll go with the crowd if they seem confident.”</p><p>3. <strong><em>Informative Prior :</em></strong></p><p>If we want the model to stick to what we believe and not get easily convinced by new data, we can drop in a strong prior — one that’s tight and confident. How strong that prior should be totally depends on the context of your task. For this example, let’s say our prior belief is that the true mean of the population should be somewhere around the mean of the cleaned data — the one where we removed the outliers.</p><pre>Q1 = np.percentile(temp, 25)<br>Q3 = np.percentile(temp, 75)<br>IQR = Q3 - Q1<br>lower = Q1 - 1.5 * IQR<br>upper = Q3 + 1.5 * IQR<br>temp_clean = temp[(temp &gt;= lower) &amp; (temp &lt;= upper)]<br><br>informative_prior = norm.pdf(mu_vals, loc=np.mean(temp_clean), scale=0.1)<br>normalized_informative_prior = informative_prior / np.sum(informative_prior)<br><br>informative_posterior = likelihoods * informative_prior<br>informative_posterior /= np.sum(informative_posterior) <br><br>plt.figure(figsize=(9, 5))<br>sns.lineplot(x=mu_vals, y=informative_posterior, label=&quot;Posterior&quot;, color=&quot;green&quot;, linewidth=2)<br>plt.fill_between(mu_vals, informative_posterior, color=&quot;green&quot;, alpha=0.3)<br>plt.xlabel(&quot;Mean Body Temperature (°F)&quot;)<br>plt.ylabel(&quot;Density&quot;)<br>plt.title(&quot;Posterior with Strong Informative Prior&quot;)<br>plt.legend()<br>plt.grid(True)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*F-5hflpUMwheoJ-pcOjSwg.png" /></figure><p>Since our prior is strong and already sharply focused around a certain mean, the posterior gets compressed into that region. And because it’s compressed, the peak has to shoot up to keep the total probability the same. So, stronger prior = more confident model = narrow but higher posterior. Let’s see how our prior is with the likelihood:</p><pre>sns.lineplot(x=mu_vals, y=normalized_informative_prior, label=&quot;Strong Prior&quot;, color=&quot;green&quot;, linestyle=&quot;-&quot;)<br>sns.lineplot(x=mu_vals, y=likelihoods_norm, label=&quot;Likelihood&quot;, color=&quot;blue&quot;, linestyle=&quot;-&quot;)<br>plt.fill_between(mu_vals, likelihoods_norm, color=&quot;blue&quot;, alpha=0.3)<br>plt.fill_between(mu_vals, normalized_informative_prior, color=&quot;green&quot;, alpha=0.3)<br><br>plt.xlabel(&quot;Mean Body Temperature (°F)&quot;)<br>plt.ylabel(&quot;Density&quot;)<br>plt.title(&quot;Likelihood with Strong Informative Prior&quot;)<br>plt.legend()<br>plt.grid(True)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*p-rcYMG2IDUZGKtOBciYyg.png" /></figure><p>We can see the prior has some strong information about where the mean could be.</p><pre>strong_mu = mu_vals[np.argmax(informative_posterior)]<br>likelihood_mu = mu_vals[np.argmax(likelihoods_norm)]<br><br>print(f&quot;Posterior mean (strong prior): {strong_mu:.3f}&quot;)<br>print(f&quot;Maximum Likelihood Estimate: {likelihood_mu:.3f}&quot;)<br>print(f&quot;Are they equal? {strong_mu == likelihood_mu}&quot;)</pre><pre>Posterior mean (strong prior): 98.257<br>Maximum Likelihood Estimate: 98.246<br>Are they equal? False</pre><p>We can also see that the outlier data tried to convince the model but failed due to our strong prior belief.</p><p><strong><em>Take Away from Informative Prior :</em></strong></p><p>When we give strong information in the prior, the model doesn’t just follow the data blindly anymore. It becomes harder to convince — the likelihood needs to really align with the prior for the posterior to shift. In our case, the posterior moved away from the outlier-influenced mean and leaned more towards what we believed. So you’re telling the model “I know what I’m doing, don’t get carried away by every new datapoint”</p><p>4. <strong><em>Robust Prior :</em></strong></p><p>We can also use robust priors, like in this particular case if i use a Cauchy distribution which is more forgiving in the tails compared to normal distribution, the model believe in something but it is open to data to correct the belief.</p><pre>robust_prior = t.pdf(mu_vals, df=1, loc=np.mean(temp_clean), scale=1)<br><br>robust_posterior = likelihoods * robust_prior<br><br>robust_posterior /= np.sum(robust_posterior)<br><br>plt.figure(figsize=(9, 5))<br>sns.lineplot(x=mu_vals, y=robust_posterior, label=&quot;Posterior&quot;, color=&quot;green&quot;, linewidth=2)<br>plt.fill_between(mu_vals, robust_posterior, color=&quot;green&quot;, alpha=0.3)<br>plt.xlabel(&quot;Mean Body Temperature (°F)&quot;)<br>plt.ylabel(&quot;Density&quot;)<br>plt.title(&quot;Posterior with robust_posterior Prior&quot;)<br>plt.legend()<br>plt.grid(True)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*GcyobjWC-BJhgBGekzvA9A.png" /></figure><p>The posterior is looking similar to flat and weak priors, this is because robust prior doesn’t exist to fix outliers, it exist to <strong>be forgiving when you don’t fully trust your prior OR your data.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*AVAQZS5XUD0Dov5GM3o-Gw.png" /></figure><p>The above plot shows that the prior don’t want to mislead by outliers and don’t want to mislead the posterior with an over confident prior either.</p><p>We can use robust prior if :</p><ul><li>You’re dealing with noisy or real-world data</li><li>You want your model to learn from data, but stay stable under outlier pressure.</li></ul><p><strong><em>Take Away from Robust Prior:</em></strong></p><p>Robust priors like the Cauchy aren’t trying to ignore outliers completely or force the model into a belief. They give some shape to the prior but are still open to data. That’s why in this case, the posterior looked similar to flat or weak priors.</p><p>It’s not meant to overpower the data like a strong prior. It’s used when the data might be noisy, and you want the prior to have some belief — but still let the likelihood speak.</p><p><strong>Conclusion :</strong></p><p>Bayesian inference just gives us a way to update what we believe when new data comes in. The prior decides how stubborn or flexible we are, and the likelihood tells us what the data is trying to say. Multiply them, normalize, and you get the posterior, that’s it.</p><p>What changed across the flat, weak, informative, and robust priors wasn’t the process — it was how much the model trusted the data versus how much it leaned on the prior. Flat and weak priors basically let the data take full control. Informative priors pushed back. Robust priors stayed balanced.</p><p>This kind of setup is useful when your data isn’t perfect, or when we actually have some knowledge we want to include. Unlike the point estimation which works on frequential probability, Bayesian Inference works on what our belief is before seeing data and the actual data. We get the entire distribution of the parameter, where we can find uncertainty of the parameter and credible intervals to support our results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e169d6353ab1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Maximum Likelihood Estimation: Finding The Optimal Parameter]]></title>
            <link>https://mohith-g.medium.com/maximum-likelihood-estimation-finding-the-optimal-parameter-9cebfbd5aa1d?source=rss-6575ac4820f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cebfbd5aa1d</guid>
            <category><![CDATA[error-function]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[maximum-likelihood]]></category>
            <category><![CDATA[point-estimation]]></category>
            <category><![CDATA[log-likelihood]]></category>
            <dc:creator><![CDATA[Mohith]]></dc:creator>
            <pubDate>Mon, 28 Jul 2025 07:09:38 GMT</pubDate>
            <atom:updated>2026-01-26T06:34:35.213Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Concise Guide to MLE, Likelihood Functions, and Data-Driven Estimation</h4><p><strong>Maximum Likelihood Estimation </strong>is one of the most widely used techniques for point estimation. The core idea is to find the parameter value under which the observed data is most probable. In other words, the parameter that best explains or fits the data is taken as the estimate of the true population parameter.</p><p><strong>But how do we measure how well a parameter fits the data ?</strong></p><blockquote>That’s where the <strong>Likelihood Function</strong> comes in:</blockquote><p>Let <em>f(X;θ)</em> be the probability density function(PDF) or probability mass function(PMF) of a distribution, where <em>X</em> is a random variable and <em>θ ∈ Θ</em> is the parameter space. Then the <strong>likelihood function</strong>, is defined as :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/151/0*bcmUaEppYFFmFijA" /></figure><p>The likelihood function is just the joint density of the data, except we <strong>treat it as a function of the parameter</strong>. Thus <em>Θ → [0, ∞)</em>. The likelihood function is not a density function : in general, it is not true that it integrates to 1 <em>w.r.t θ</em></p><p><strong>The maximum likelihood estimator MLE, denoted by <em>θ̂ₙ</em> is the value of<em> θ</em> that maximizes 𝓛(<em>θ</em>).</strong></p><p>Let’s see how likelihood function helps us find best parameter(mean μ in this case) with a sample data:</p><pre>import numpy as np<br><br>mean = 5<br>scale = 2<br>sample_data = np.random.normal(mean, scale, 5)<br>mu_vals = np.linspace(2, 6, 100) # list of parameters</pre><p><em>Note: The variance is fixed for this example</em></p><pre>from scipy.stats import norm<br><br>likelihood = lambda est: np.prod(norm.pdf(sample_data, loc=est, scale=scale))<br>lik_vals = [likelihood(i) for i in mu_vals]</pre><p>MLE is the value that maximizes the likelihood of observing data, the plot would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*vWc8vgRGUB9UyMB_Sz36SQ.png" /></figure><p>The lik_vals are normalized for better plot readability and comparison.</p><p>The likelihood is high near the mean of the sample data which is 4.59, it’s obvious that the sample mean is the best point estimate for the true population mean, but we’ll see what maximum likelihood estimation gives. Before doing that, we will increases the sample size and see what happens if I take likelihoods with the same mu_vals :</p><pre>large_sample_data = np.random.normal(mean, scale, 1000)<br>likelihood = lambda est: np.prod(norm.pdf(large_sample_data, loc=est, scale=scale))<br><br>lik_vals = [likelihood(i) for i in mu_vals]<br>print(np.unique(lik_vals))</pre><pre>[0.]</pre><p>The likelihood of every mu_vals came 0. This is because of numeric underflow. The pdf values of normal distribution for 1000 samples norm.pdf(large_sample_data, loc=est, scale=scale)are near 0:</p><pre>[0.14277481 0.07710592 0.19761    0.1987914  0.1809856  0.17778946<br> 0.19551692 0.12493172 0.18626636 0.19038473 0.18573295 0.15994864<br> 0.10805176 0.19569509 0.13841485 0.15290876 0.18476323 0.08735241<br> 0.19023953 0.07264875 0.18079387 0.16589513 0.19095156 0.19885624<br> ...]</pre><p>when we do np.prod(...) we are multiplying all data points that are near 0 resulting in 0 likelihood because the result of the multiplication is so small that it cannot be stored in its intended destination format without suffering a rounding error.</p><p><strong>But most of the real world data have larger sample size than the likelihood’s limit. How to avoid numeric underflow ?</strong></p><blockquote>That’s where the <strong>Log-Likelihood Function</strong> comes in:</blockquote><p>Using logarithm addition property, instead of multiplying the near zero values we can add the log of individual data point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/0*BpalyKISlg6kZvi0" /></figure><pre>large_sample_data = np.random.normal(mean, scale, 1000)<br><br>log_likelihood = lambda est: np.sum(norm.logpdf(large_sample_data, loc=est, scale=scale))<br><br>log_lik_vals = [log_likelihood(i) for i in mu_vals]</pre><p>The values of log_lik_vals for each mu_vals follows this plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*B6U9Rf9AsC1ImpMf4nSwnQ.png" /></figure><p>The maximum of log-likelihood occurs at the same place as the maximum of likelihood, so maximizing the log-likelihood is similar to maximizing the likelihood. Just for now, let’s take a small sample data to see if the maximum of both log-likelihood and likelihood are at the same point.</p><pre>sample_data = np.random.normal(mean, scale, 5)<br><br>likelihood = lambda est: np.prod(norm.pdf(sample_data, loc=est, scale=scale))<br>log_likelihood_comparison = lambda mu: np.sum(norm.logpdf(sample_data, loc=mu, scale=scale))<br><br>lik_vals = [likelihood(mu) for mu in mu_vals]<br>loglik_vals = [log_likelihood_comparison(mu) for mu in mu_vals]</pre><p>After normalizing and plotting the densities we would get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*sZcNA10Nwiofu0gBlL5Tnw.png" /></figure><p>The log-likelihood and likelihood peaks are at the same point 4.59, proving the statement.</p><p>Now we understood why log is used, let’s get back to seeing whether maximum likelihood estimation gives the same estimator as 5.03 which we got from large_sample_data</p><p>It’s clear that we took data from the normal distribution, so the log of normal distribution’s pdf :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/0*wpwqv7I3M9MW-b48" /></figure><p>From this, we can do:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/309/0*k9cyCPzzQ_cHFRnQ" /></figure><p>The first term is <strong>constant</strong> with respect to μ, The second term involves the sum of square of deviations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/92/0*DV_cS0Xh94TxcvkV" /></figure><p>Due to the presence of negative sign, the lower is the deviation, the higher is the log-likelihood. We know that we get the lower deviation at the sample mean. So:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/43/0*B97bZkV2dOdxiyeA" /></figure><p>Let’s prove it by taking derivative <em>w.r.t</em> mean Since our estimate is the mean:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/469/0*dAC1cs82fxY9lqvb" /></figure><p>Simplify:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/177/0*kqrz-dk6kuhpvNi9" /></figure><p>Set derivative to 0 to solve for the mean at the top of the curve:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/0*-UrB7eNxpEM61F8S" /></figure><p>Hence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/140/0*tmZBmnHL_OBjOYFL" /></figure><p>MLE is the <strong>sample mean</strong></p><p>For our large_sample_data case, let’s see the MLE matches the random estimators we took:</p><pre>print(f&quot;The MLE estimate:{np.mean(large_sample_data):.2f}\nThe max log-likelihood estimate : {mu_vals[np.argmax(log_lik_vals)]:.2f}&quot;)</pre><pre>The MLE estimate:5.02<br>The max log-likelihood estimate : 5.03</pre><p>The difference is because at start we took our estimtors from random sampling mu_vals = np.linspace(2, 6, 100) . If you know how to use MLE you would directly know that the best estimate is the sample mean from derivation so, sample mean as the parameter would give the maximum value of joint density.</p><p>Now we know the sample mean gives the maximum log-likelihood, <strong>why the sample mean is MLE</strong>?</p><p>Because in the value:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/0*M61fA0awlxhb1uSU" /></figure><p>only the sample mean perfectly fits the data in it’s center reducing the deviations.</p><p>What I mean is, we know that mu_vals[np.argmax(log_lik_vals)] is the most optimal value to fit the data perfectly but not as optimal as sample mean.</p><pre>print(f&quot;The log likelihood with sample mean = {-np.sum((large_sample_data - np.mean(large_sample_data))**2)}\nThe log likelihood with randomly took optimal mean estimate = {-np.sum((large_sample_data - mu_vals[np.argmax(log_lik_vals)])**2)}&quot;)</pre><pre>The log likelihood with sample mean = -3687.630561482434<br>The log likelihood with randomly took optimal mean estimate = -3687.7116260855855</pre><p>notice that with a small difference the sample mean gave higher log-likelihood than the most optimal of our 100 estimates.</p><p>Lets see how well our estimates fits the data :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*eLQ9cU5qg4nHYnUy1SBxUQ.png" /></figure><p>We can see that our MLE and mu_vals[np.argmax(log_lik_vals)] are very close, zooming in might help:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*PntQ4VunkxGeocpIs1dENQ.png" /></figure><p>Even the small difference in MLE helps increases the likelihood of observing data.</p><p>One of the key properties of the Maximum Likelihood Estimator (MLE) is <strong>consistency</strong>. This means that as the sample size increases, the MLE gets closer and closer to the true value of the parameter. In other words, given enough data, the estimator locks onto the correct value.</p><p>Here is the simulation showing how the MLE for the mean of a normal distribution converges to the true mean (μ = 5) as sample size increases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*zoErHwae813mCPozX80q2g.png" /></figure><p><strong>Conclusion:</strong></p><p>We started with a simple idea: find the parameter that makes the observed data most likely. That led us to the likelihood function. We saw how it works, what breaks it (numeric underflow), and how switching to log-likelihood fixes it.</p><p>Then we showed that the sample mean is not just a good guess but it’s the MLE when variance is known. We derived it, plotted it, compared it against other estimators, and confirmed that even a small difference in the estimate affects the likelihood.</p><p>Finally, we showed consistency: as sample size increases, the MLE gets closer to the true mean. MLE doesn’t just work well on small samples, it keeps improving as more data comes in.</p><p>So, MLE gives us a clear, consistent way to estimate parameters, and the best part is , it comes straight from the data and also with an assumption on the distribution of the data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cebfbd5aa1d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>