Gini Index & Entropy
How My Decision Tree Decides “Who Goes Where”
When I first heard the terms Gini Index and Entropy, I genuinely thought they were either desserts…
or tax rules.
Turns out,these two are the brains behind how a Decision Tree splits data.
Whenever the tree is deciding where to split, it asks itself:
“Which feature gives me the cleanest, least confusing groups?”
To answer that, it measures how messy each group is.
That “messiness” is exactly what Gini and Entropy calculate.
Entropy — The Chaos Meter
Entropy tells me how mixed up the data is.
More mixed → more chaos → higher entropy.
Here’s the formula (don’t panic):
Where
- pip_ipi = probability of each class
- log stuff = our model pretending to be intellectual
How I think about it wrt Iris Dataset:
- If a node has all Setosa → entropy = 0
(pure peace, like a group chat with only one person) - If a node has 33% each of Setosa, Versicolor, Virginica → entropy is max
(pure chaos, like an Indian family group chat)
Entropy loves measuring messiness.
Gini Index — The Impurity Meter
Gini Index does the same job but calculates impurity differently:
Gini=1−∑pi2Gini = 1 — \sum p_i²Gini=1−∑pi2
Get Aakansha A’s stories in your inbox
Join Medium for free to get updates from this writer.
Again:
- Lower Gini = cleaner, purer node
- Higher Gini = more mixed, more confusion
How I understand it:
- If the node has all samples from 1 species → Gini = 0
- If it’s a random mix → Gini becomes high
The tree always prefers lower Gini, because lower means the group is purer.
So How Does My Tree Use These?
Every time the tree tries a split (like “Petal length < 2.45?”), it calculates:
- Gini or Entropy before the split
- Gini or Entropy after the split
- The reduction in impurity
That reduction = Information Gain
This is what the tree is obsessed with.
More information gain = better split = “Yes let’s divide the data here!”
For the Iris dataset, the first split usually becomes:
Petal length < 2.45?Because this split gives maximum information gain , it separates Setosa almost perfectly from the rest.
In Simple Human Language
Here’s how I personally think about it:
- Entropy = how much drama is in the group
- Gini = how impure/confusing the group is
- Information Gain = how much clarity I get if I split here
And the decision tree?
It just wants peace.
It splits the data wherever the drama decreases the most.
