MULTI-CLASS CLASSIFICATION
where sigmoid must gracefully step aside, and SoftMax takes the throne.
Multiclass classification is a supervised learning task where each input belongs to exactly one of three or more classes.
Core Idea: To know “Which one is it?” Compares all classes simultaneously, and strongest wins.
Example:
Identify animal — Cat, Dog, leopard.
Detect category — Promotions, Social, Updates
Sentiment — Positive, Neutral, Negative
Mathematically, the goal is to find a function
where K is the number of classes. We train a model that outputs scores or probabilities for each class and then predict the class with the highest score.
Mathematical Foundation
In multiclass logistic regression, we assign a score to each class k.
w — weight/slope
x — Input Feature
b — bias
Then we convert these scores into probabilities using the SoftMax function. The SoftMax ensures that all probabilities are positive and they sum to 1
Note: Even though the task is classification, the model called logistic regression (and its multiclass version, SoftMax regression) uses regression-like math — linear functions of inputs — to model probabilities of discrete classes. So, the term “regression” here refers to how the model computes its outputs, not what kind of prediction it makes
- In linear regression, we treat z itself as the prediction (a real number).
- In logistic (or SoftMax) regression, we transform zzz through a nonlinear function (sigmoid or SoftMax) to interpret it as a probability.
Why can't we stick to Sigmoid?
SoftMax is the referee that enforces exclusivity — it makes every class compete with every other class.
In multiclass classification your question changes to:
“Is this photo of a cat, dog, rabbit, or elephant?”
Now there are 4 possible outcomes — and the model must pick exactly one.
If you used sigmoid, each output neuron would produce its own probability independently:
But here’s the problem: Those probabilities don’t know about each other and you could end up with:
cat: 0.8
dog: 0.9
rabbit: 0.7
elephant: 0.6
Total = 3.0
That’s nonsense — probabilities can’t add up to more than 1! The model’s basically saying, “everything looks kind of right,” which helps no one.
In multiclass classification (SoftMax world)
Now imagine you have three or more possible classes — cat, dog, rabbit.
The question changes. You can no longer just ask:
“Is this a cat?”
because you might get a strong “yes” for both cat and dog using independent sigmoid!
Example:
p(cat) = 0.8
p(dog) = 0.7
p(rabbit) = 0.6
If you used sigmoid here, you’d have three independent yeses.
That’s a multilabel interpretation — meaning “this image could be multiple things.” But in a multiclass problem, that makes no sense. The photo must be exactly one of them.
What SoftMax really gives you
SoftMax always produces numbers between 0 and 1, and their total sum is exactly 1. That’s not an accident — it’s baked into the formula itself:
Here’s the key:
- The numerator e^zi is always positive.
- The denominator is the sum of all such positive terms.
- So pi is a positive fraction of the whole — it must be between 0 and 1
- And when you add up all pi, the denominators cancel out neatly:
So, the sum = 1 is guaranteed by definition. That’s what turns those raw scores into a probability distribution.
A real example
Say our model produces raw scores (logits): cat: 4, leopard: 3, dog: 1
Compute SoftMax:
e4=54.6, e3=20.1, e1=2.7 (Each e_i is a real number — not a probability yet — just a “confidence score” from each neuron.)
Sum = 77.4 (SoftMax takes all those scores and runs this formula for each class i, i.e. p_cat+p_dog+p_leopard=1)
Now probabilities, apply SoftMax:
p_cat = 54.6/77.4≈0.705
p_leopard = 20.1/77.4≈0.259
p_dog = 2.7/77.4≈0.035
Sum = 1, All between 0 and 1
SoftMax effectively compresses and scales raw scores, so they make sense as probabilities.
What this means intuitively
You can think of SoftMax as an internal competition — each class neuron “bids” for the input, shouting:
“It looks like me!
Softmax listens to all their bids, amplifies the strongest voices, and turns them into relative probabilities. So, when you ask the model “Is this a cat?”
it’s not just looking at the “cat” neuron’s score in isolation. It’s also checking:
“Hmm… the dog neuron is slightly higher, so maybe not cat after all.”
That’s the magic: each probability depends on all other classes.
Why this is powerful
This “relative” behaviour is what makes SoftMax ideal for mutually exclusive categories. If one class grows stronger, all others automatically weaken —
no redundant “yes” votes, no double counting. It’s elegant, self-balancing, and mathematically smooth for gradient descent to train on.
Summary Table
Loss Function (Cross-Entropy Loss)
Cross-entropy is just a measure of how wrong your model’s predictions are.
- If your model is very confident and correct, cross-entropy is small.
- If it’s confident but wrong, cross-entropy is large.
So, it’s basically a penalty for bad confidence.
Get Mausam’s stories in your inbox
Join Medium for free to get updates from this writer.
For multiclass classification (say K classes), we represent:
- The true label as a one-hot vector
y=[0,0,1,0,…,0] - The predicted probabilities as
y^=[P1,P2,…,PK]
Then, the multiclass cross-entropy is:
For a single training example (x,y), where y is the true label:
Exactly like the binary case — just extended to K probabilities instead of one.
Example: Now let’s move to a 3-class problem:
We are classifying an image into 3 possible classes:
Our model outputs probabilities (after SoftMax):
Predicted y^=[P_cat, P_dog, P_rabbit]=[0.7,0.2,0.1]
the true label is Cat:
Write the true label as a one-hot vector: y = [1,0,0]
This means:
- Cat = 1 (correct class)
- Dog = 0
- Rabbit = 0
Since only one yk=1 for the correct class), the sum becomes just one term:
Loss = −log(y^true class)
The true class = Cat, (and the model’s predicted probability for “Cat” is 0.7)
Loss = −log(0.7)
If we use natural log (base e):
Loss=−(−0.357) = 0.357
So, the cross-entropy loss = 0.357
If the model was less confident (say 0.3 for cat):
Loss = −log (0.3) = 1.204
Now the loss is much higher (1.204 instead of 0.357) → because the model gave low probability to the true class. (Bigger loss → model did worse.)
What’s a One-Hot Vector?
It’s simply a way of writing down which class is correct when there are several options.
For our cat image:
Class [Cat, Dog, Rabbit]
True label (y) [1, 0, 0]
This is called a one-hot vector — one “1”, rest “0”.
Gradient Descent (Parameter Updates)
Multiclass (SoftMax Regression) Gradient, in multiclass classification, the model gives one score per class.
We optimize weights wk using gradient descent:
Example:
Let’s take 3 classes (Cat, Dog, Rabbit).
True label: “Dog” → one-hot = [0, 1, 0]
Predicted probabilities: y^ = [0.7,0.2,0.1]
So gradient descent:
- Reduces confidence for wrong classes (positive error)
- Increases confidence for correct class (negative error)
Each class’s weights are updated independently but simultaneously.
Gradient descent doesn’t check whether a single prediction is right or wrong. It minimizes the average loss (cross-entropy, MSE, etc.) across all training examples.
Even if one example is “perfect,” the model might still:
- Misclassify other examples
- Be too confident or not confident enough
- Not yet at the lowest possible loss
So, we continue updating until the loss function for the whole dataset stops decreasing.
When Do We Actually Stop?
So the goal isn’t just “correct answers,” but optimized confidence and generalization.
Think of training like learning basketball free throws:
- You might make a shot once (correct output).
- But that doesn’t mean you stop practicing — you keep adjusting to make more shots consistently and confidently (reducing loss).
Visit this link for Multilable Classification: Multilabel Classification — Mausam — Medium
