𝗘𝗻𝘀𝗲𝗺𝗯𝗹𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: 𝗪𝗵𝘆 𝗖𝗼𝗺𝗯𝗶𝗻𝗶𝗻𝗴 𝗠𝗼𝗱𝗲𝗹𝘀 𝗖𝗮𝗻 𝗕𝗲𝗮𝘁 𝗮 𝗦𝗶𝗻𝗴𝗹𝗲 𝗠𝗼𝗱𝗲𝗹

Why rely on a single model when you can combine several models and potentially get more accurate, stable, and robust predictions?

That's the core idea behind ensemble learning.

Instead of relying on one algorithm's predictions, we train multiple models and combine their outputs into a final prediction.

𝗔 𝗦𝗶𝗺𝗽𝗹𝗲 𝗘𝘅𝗮𝗺𝗽𝗹𝗲

Suppose we're building a classification system and train three different models:

🔹 Logistic Regression — captures linear relationships and is highly interpretable.
🔹 Random Forest — captures non-linear patterns and interactions using many decision trees.
🔹 XGBoost — builds trees sequentially, focusing on correcting the errors of previous trees.

Each model has different strengths and weaknesses.

One model might get a particular observation wrong while another gets it right. If their errors aren't perfectly correlated, combining their predictions can produce a stronger overall model.

𝗛𝗼𝘄 𝗖𝗮𝗻 𝗪𝗲 𝗖𝗼𝗺𝗯𝗶𝗻𝗲 𝗧𝗵𝗲𝗺?

𝟭. 𝗛𝗮𝗿𝗱 𝗩𝗼𝘁𝗶𝗻𝗴

Each model predicts a class, and the majority wins.

For three models:

Model A → Class 1

Model B → Class 1

Model C → Class 0

Final prediction → Class 1

Simple, but it ignores how confident each model is.

𝟮. 𝗦𝗼𝗳𝘁 𝗩𝗼𝘁𝗶𝗻𝗴

Instead of using class labels, combine predicted probabilities.

For example:

Logistic Regression → 0.62

Random Forest → 0.71

XGBoost → 0.68

Average:
(0.62 + 0.71 + 0.68) / 3 = 0.67

With a 0.5 threshold → Class 1

Unlike hard voting, soft voting preserves the models' confidence information.

💡 𝗡𝗼𝘁𝗲: In the case of regression problems, there are no classes or probabilities to vote on. Instead, the ensemble simply takes the mathematical average of the numerical predictions from all models.

𝟯. 𝗪𝗲𝗶𝗴𝗵𝘁𝗲𝗱 𝗔𝘃𝗲𝗿𝗮𝗴𝗶𝗻𝗴

Not every model needs to have equal influence.

Get Affan Ahmed’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

If XGBoost consistently performs better on validation data, we could assign it a larger weight.

For example:

Logistic Regression (Weight: 0.2) → 0.62

Random Forest (Weight: 0.3) → 0.71

XGBoost (Weight: 0.5) → 0.68

Calculation:
(0.2 × 0.62) + (0.3 × 0.71) + (0.5 × 0.68) = 0.677

With a 0.5 threshold → Class 1

The final prediction becomes a weighted combination of their outputs.

𝟰. 𝗦𝘁𝗮𝗰𝗸𝗶𝗻𝗴

Instead of manually deciding the weights, we can train another model to learn how to combine the base models.

The predictions from the first-layer models become inputs to a meta-model.

For example:
Logistic Regression + Random Forest + XGBoost → Meta-model → Final prediction

The meta-model learns which predictions to trust and how to combine them.

𝟱. 𝗕𝗮𝗴𝗴𝗶𝗻𝗴 & 𝗕𝗼𝗼𝘀𝘁𝗶𝗻𝗴

Some ensemble methods are already ensembles themselves.

🌳 Random Forest = Bagging + many decision trees
⚡ XGBoost = Gradient Boosting + many decision trees

So combining Random Forest, XGBoost, and Logistic Regression is effectively creating an ensemble containing different types of models — including ensemble models themselves.

𝗪𝗵𝘆 𝗗𝗼 𝗘𝗻𝘀𝗲𝗺𝗯𝗹𝗲𝘀 𝗪𝗼𝗿𝗸?

The key is diversity.

If several models make exactly the same mistakes, combining them won't provide much benefit.

But if their errors are different, one model can compensate for another's weaknesses.

This can lead to:
✅ Better predictive performance
✅ Greater robustness
✅ More stable predictions
✅ Reduced variance — particularly with techniques such as bagging
✅ Better handling of complex patterns when models capture complementary relationships

⚠️ 𝗧𝗵𝗲 𝗧𝗿𝗮𝗱𝗲-𝗢𝗳𝗳

Ensembling isn't automatically better.

More models mean:
→ More computational cost
→ More complexity
→ More difficult interpretation
→ More complicated deployment and monitoring

And combining several highly correlated models that make similar mistakes may provide little or no improvement.

The real power of an ensemble comes from combining models that are both reasonably strong and sufficiently diverse.

𝗧𝗵𝗲 𝗕𝗶𝗴 𝗣𝗶𝗰𝘁𝘂𝗿𝗲

Ensemble learning isn't about finding one perfect model.

It's about recognizing that every model has blind spots — and combining models in a way that prevents those blind spots from lining up.

One model gives you one perspective.
An ensemble gives you several perspectives — and a way to combine them.