Supervised Classification Algorithms
Major supervised learning classification algorithms
Full landscape of supervised classification algorithms
Quick Exam-Friendly Grouping
- Linear: Logistic Regression, LDA, Ridge, Perceptron, Passive-Aggressive
- Probabilistic: Naïve Bayes, Gaussian Process
- Distance-based: k-NN
- Margin-based: SVM
- Tree-based: Decision Tree, Random Forest
- Boosting: AdaBoost, Gradient Boosting, XGBoost, LightGBM, CatBoost
- Deep Learning: MLP, CNN, RNN
Classification Types & Their Core Mathematical Foundations
To know more, start here: Classification
Core Mathematics Behind Classification Model Types
Core Mathematical Ideas Used Across Algorithms
Not all classification algorithms use the same math, BUT many of them share common mathematical foundations, especially:
- Optimization
- Loss minimization
- Information theory (cross-entropy)
Some algorithms are probabilistic, some are geometric, some are tree-based, and some are neural/gradient-based.
What Each Column Really Means?
Comparative Analysis of Classification Algorithms: Math, Loss, and Optimization
You should know before we jump into algorithms
classification types define the mathematical problem formulation, and classification models are built to solve these formulations using largely the same core mathematical foundations. However, models do not all use identical equations, but they reuse the same mathematical building blocks, arranged differently based on assumptions and requirements.
Classification types come FIRST
All classification models are built to solve these types. No classification model exists outside these problem types. for example:
- Logistic Regression → Binary / Multiclass
- SVM → Binary / Multiclass
- Trees → Binary / Multiclass
- Neural Networks → Any classification type
The same core math appears everywhere
Across models, you will repeatedly see:
- Linear algebra (dot products, matrices)
- Probability theory (Bernoulli, Multinomial)
- Loss functions (cross-entropy, hinge)
- Optimization (minimization/maximization)
- Regularization (complexity control)
This is not a coincidence — it’s because the problem definition demands it.
“Classification types define the mathematical structure of the problem, and all classification models are designed to solve these structures using shared mathematical foundations such as probability, linear algebra, and optimization, though each model applies them differently based on its assumptions.”
Classification types define what must be solved; models define how it is solved — using the same underlying mathematics.
This is the level where ML stops being a list of algorithms and becomes one coherent framework.
To better understand the classification models and algorithms listed above, it is important to first understand each classification type and its core mathematical foundation. This approach saves time when learning how different models work and allows us to move directly into practical implementation. Below, I have shared links to all the classification types — please review them and then return to begin our discussion on supervised classification algorithms
Links: Binary Classifications, Multiclass classification, Multilabel classification, Imbalance classification, Hierarchical classification.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
Linear Models for Supervised Classification
Linear Classifiers: Learning Through Linear Separation (Linear Decision Boundary–Based Classifiers)
Logistic Regression
Logistic Regression models the probability of a class using a linear combination of features, squashed by a sigmoid function, and learns parameters by minimizing cross-entropy loss. (predict the categorical dependent variable using a given set of independent variables)
What Problem Does Logistic Regression Solve?
Binary classification (can be extended to multiclass)
Output: Probability, not just a class, Example: spam / not spam, disease / no disease
Logistic Regression is the canonical binary classification model. Binary classification means:
- Target y ∈ {0,1}
- Output is a probability
- Uses a sigmoid
- Optimize binary cross-entropy
Algorithms that are natively binary (these were originally designed for binary problems)
We understand till Optimization of the model, the next step comes into the process is Regularization (Controlling Overfitting), before let's see optimization.
Types of Logistic Regression
- Binary Logistic Regression (Model)
- Multinomial (Multiclass) Logistic Regression (SoftMax Regression) (Model)
- One-vs-Rest (OvR) Logistic Regression: STRATEGY (Helper Method) (One-vs-Rest is a multiclass strategy that combines multiple binary classifiers; it is not a separate learning model.)
- Multilabel Logistic Regression: modeling approach / formulation, not a model.
- L1-Regularized Logistic Regression (Lasso)
- L2-Regularized Logistic Regression (Ridge)
- Elastic Net Logistic Regression (Model)
- Batch Logistic Regression: Optimization style
- Stochastic Logistic Regression
- Mini-batch Logistic Regression
Mathematical Steps of Logistic Regression (Binary Classification):
Step 1: Linear Combination of Inputs: compute a weighted sum of features:
Step 2: Apply Sigmoid Function: Convert the linear output into a probability:
- Output is in range (0,1)
- Represents P(y=1∣x)
Step 3: Define the Loss Function (Binary Cross-Entropy): Measure prediction error:
Step 4: Compute Total Cost: Average loss over all training samples:
Step 5: Add Regularization (Optional): Control overfitting:
- L2 (Ridge)
- L1 (Lasso)
Step 6: Optimize Parameters: Minimize the cost function using gradient descent:
Step 7: Make Predictions: Apply threshold to probabilities:
Remember the flow:
Linear score → Sigmoid (probability) → Cross-entropy loss → Regularization (optional) → Optimization (Gradient Descent) → Probability-based decision
Multinomial (Multiclass) Logistic Regression (SoftMax Regression)
Extension of logistic regression to multiclass classification. A supervised classification algorithm used when the target variable has more than two mutually exclusive classes.
Examples:
- Digit recognition (0–9)
- News topic classification
- Object category prediction
Mathematical Steps (Clean Flow)
Step 1: Linear Scores (One per Class)
K = number of classes, each class has its own weights
Step 2: Softmax
Step 3: Categorical Cross-Entropy Loss, penalizes incorrect class probability and uses one-hot encoded labels
Step 4: Add Regularization (Optional)
Step 5: Optimization
Step 6: Prediction, choose class with highest probability
Pipeline (Multiclass)
Linear scores → SoftMax → Cross-entropy → Regularization → Optimization → Argmax decision (Prediction)
One-vs-Rest (OvR) Logistic Regression
Another standard way to extend logistic regression to multiclass classification.
A multiclass classification “Strategy” where one binary logistic regression model is trained per class, each separating one class from all others. If there are K classes → train K binary logistic regression models. OvR reuses the same binary logistic regression math.
A model defines: Mathematical form, Loss function, Optimization
A strategy defines:
- How multiple models are combined
- How final class prediction is made
How OvR works conceptually, for each class k:
- Label class k as 1
- Label all other classes as 0
- Train a binary logistic regression
When Is OvR Used?
When:
- Number of classes is moderate
- Classes are not mutually exclusive in practice
- Simpler training is preferred
Why we need OvR when we have Multinomial Logistic Regression?
We need One-vs-Rest (OvR) because not all models natively support multiclass learning, and OvR allows us to reuse strong binary classifiers for multiclass problems.
Multinomial (softmax) logistic regression is only one specific model — OvR is a general solution.
The Core Reasons
Multinomial Logistic Regression is: Only for logistic regression, requires softmax + categorical cross-entropy and assumes mutually exclusive classes
Some powerful algorithms are naturally binary:
- SVM (originally binary)
- Perceptron
- Passive-Aggressive
- Binary logistic regression
OvR allows us to say: “No problem — we’ll just train one binary model per class.”
Classes Are Not Always Perfectly Competing
Softmax forces: P(A) + P(B) + P(C) = 1
OvR allows: Each class to be evaluated independently
Useful when:
- Classes overlap
- Data is noisy
- One class dominates
OvR lets you:
- Tune each class separately
- Use different thresholds
- Apply different class weights
Softmax treats all classes together, which can hurt minority classes.
Use softmax when:
- Classes are mutually exclusive
- Data is balanced
- You want true probability distributions
- You want a single clean model
Use OvR when:
- Your base model is binary-only
- You want flexibility per class
- You’re handling class imbalance
- You want interpretability
Mathematical Formulation
Step 1: Binary Model per Class, For class k:
First equation is just a Score:
- x = input features
- wk = weights learned for class k
- bk = bias for class k
This score answers: “How much does this input look like class k?” It is not a probability yet.
Second Equation is Sigmoid, converts score → number between 0 and 1 and makes the output interpretable (The probability that this sample belongs to class k)
Get Mausam’s stories in your inbox
Join Medium for free to get updates from this writer.
binary model is trained like this:
- Target = 1 → sample belongs to class kkk
- Target = 0 → sample belongs to any other class
so mathematically: p^k = P (y = k ∣ x) (these probabilities are independent)
- They do NOT need to sum to 1
Step 2: Loss Function (Per Class)
Binary cross-entropy:
(Optional L1/L2 regularization can be added.)
Step 3: Train All Models Independently
- Optimize each classifier separately
- No probability coupling between classes
Step 4: Prediction Rule
Choose the class with the highest predicted probability.
Let say, after training, for ONE input x, we get:
Now we simply ask: Which class is most confident?
Choose the class with the highest predicted probability.
max(0.72,0.18,0.55) = 0.72
Final Prediction: Class A.
OvR vs Softmax
Multilabel Logistic Regression
A classification approach where each sample can belong to multiple classes at the same time, using independent binary logistic regression models per label.
Multilabel logistic regression is simply binary logistic regression applied independently to each label, with threshold-based predictions.
If you understand multilabel classification, then you already understand the core math and logic behind multilabel logistic regression. It applies binary logistic regression independently to each label using a linear score per label, followed by a sigmoid activation and binary cross-entropy loss. The total loss is computed as the sum or mean across labels and optimized using gradient descent. During prediction, a default probability threshold of 0.5 is commonly used to assign labels, and this threshold can be tuned based on the problem.
- Each label is treated as its own binary problem
- No competition between labels
- Multiple labels can be predicted at once
Elastic Net Logistic Regression
Simply logistic regression with a combined regularization penalty (L1 + L2) to control overfitting and improve stability.
Elastic Net Logistic Regression (Model) = Logistic Regression + L1 (Lasso) + L2 (Ridge) regularization
It blends the strengths of both:
- L1 → feature selection (sparsity)
- L2 → stability and weight shrinkage
Why Do We Need Elastic Net?
Problems with L1 alone:
- Picks one feature from a group of correlated features
- Can be unstable
Problems with L2 alone:
- Keeps all features (no sparsity)
- Less interpretable
Elastic Net fixes both
Working Pipeline (Same as Logistic Regression)
- Linear score: z = w^Tx + b
- Sigmoid → probability: y^ = σ(z)
- Cross-entropy loss
- Add Elastic Net penalty
- Optimize (gradient descent / solver)
- Threshold → class label
The classification logic does NOT change, only the loss function does.
λ :-> Overall regularization strength
α :-> Balance between L1 and L2
α=1 :-> Pure L1 (Lasso)
α=0 :-> Pure L2 (Ridge)
0<α<1 :-> Elastic Net
When To Use Elastic Net?
Use it when:
- You have many features
- Features are highly correlated
- You want feature selection + stability
- You want better generalization than L1 or L2 alone
Batch Logistic Regression
Not a new model; it is a training method (optimization strategy) for logistic regression.
What “Batch” Means: It refers to how much data is used to update the model at each step during training.
Note: These training type will be covered separately.
Batch Logistic Regression (Workflow)
- Compute predictions for all training samples
- Compute total loss (cross-entropy + regularization)
- Compute gradient using all samples
- Update weights once per iteration
Mathematical View
It tells us: How bad is my model on the entire training dataset? and How should I change the weights to reduce the overall error?
This is called the empirical risk or average loss.
Where Did This Come From?
Step A: Start with Logistic Regression Loss (per sample)
For one data point:
This comes from maximum likelihood estimation (MLE).
Step B: We Have Many Samples → Combine Them
Real datasets have N samples, not one So we aggregate error over all samples:
Step C: Why the Average 1/N?
We divide by N to:
- Make loss independent of dataset size
- Keep gradients numerically stable
- Allow fair comparison across datasets
So, we get:
This is called Empirical Risk Minimization (ERM).
Step D: Minimize This Function → Gradient Descent
To reduce error, we move weights opposite to the gradient:
Substitute J(w):
That’s your final formula.
Final Mental Model
- Loss → how wrong one sample is
- Cost → average wrongness of all samples
- Gradient → direction to reduce wrongness
- Batch → use everything before moving
Batch Gradient Descent Says: “Let me look at the entire dataset, understand the overall direction of error, and then update weights carefully.”
Why Use Batch Training?
Pros
- Stable updates
- Exact gradient
- Smooth convergence
Cons
- Slow for large datasets
- High memory usage
Summary:
Batch Logistic Regression refers to training logistic regression using the full dataset in each gradient update; it is an optimization method, not a separate model.
Stochastic Logistic Regression
Stochastic = Randomly sampled
Stochastic Logistic Regression = Logistic Regression trained using Stochastic Gradient Descent (SGD).
The model does not change
Only the training method changes
When we say stochastic gradient descent, we mean: The gradient is computed using a randomly chosen data point instead of the entire dataset.
Mathematical Formulation:
Start From the Same Logistic Regression Loss
Batch vs Stochastic (Key Change)
Stochastic Gradient Update Rule
This is the core stochastic update.
Where Does This Come From?
That’s the mathematical justification.
Batch vs Stochastic
Why Use Stochastic Methods?
Because:
- Computing exact values is expensive
- Random sampling gives a good approximation
- Faster learning
- Better scalability
Mini-batch Logistic Regression
Mini-batch means using a small random subset of the data instead of the entire dataset or a single example. So, it sits between batch and stochastic.
Mini-batch Logistic Regression = Logistic Regression trained using mini-batch gradient descent.
Mathematical Logic
This is an approximation of the full batch loss.
Step 4: Mini-batch Gradient Update
What Difference Does Mini-batch Make?
Summary: Mini-batch logistic regression updates model parameters using gradients computed from small random subsets of the training data, balancing speed and stability.
Let’s Start with Implementation of Logistic Regression from Scratch
have already covered:
- Classification types (binary, multiclass, multilabel)
- Logistic regression math (linear score → sigmoid → loss)
- Cross-entropy loss
- Optimization (batch, stochastic, mini-batch)
- Regularization (L1, L2, Elastic Net)
- Strategies (OvR vs Multinomial)
- Evaluation intuition (probabilities, thresholds, errors)
How We Should Move Toward Python
Step 1: Implement Logistic Regression from scratch
Linear score
- Sigmoid
- Loss
- Gradient descent
- Training loop
This locks the math into intuition.
Step 2: Use scikit-learn
- Binary logistic regression
- Multiclass (OvR vs multinomial)
- Regularization (L1, L2, Elastic Net)
- Solvers
Step 3: Evaluation
- Accuracy
- Precision, Recall, F1
- Confusion Matrix
- ROC-AUC
- Threshold tuning
Step 4: Practical Scenarios
- Imbalanced data
- Cost-sensitive decisions
- Probability interpretation
Code wise implementation will be shared soon… . . .
