Similar Models. BUT Different Perspectives on the Same Problem
Why I Compared Multiple Tree Models Instead of Optimizing One
“The experiments, analysis, and interpretations are based on my own work, ChatGPT was used to assist with editing, wording, and improving readability.”
Why not just optimize one model?
By the time I built my baseline, I already had four models to work with. I could have simply looked at the strongest result, chosen that model, and spent all subsequent effort optimizing it.
That would have been the simpler path.
I could have taken an even simpler route — select one well-established model with a strong track record and focused all my experiments on optimizing it.
So why didn’t I?
What was I actually trying to learn
Because the goal of the project gradually became more than finding the lowest score.
I wanted to understand what the experiments were actually telling me — and whether the patterns I observed belonged to the problem itself or only to one particular model.
The four models were not merely four chances to get a better score. They became four perspectives on the same problem.
Comparing their behavior gave me an opportunity to ask questions that one model alone could not answer:
- Why did an improvement happen?
- Why did some improvements fail to generalize?
- Was an observation genuinely useful, or specific to one model?
- Would the same conclusion survive a different learning behavior?
Why These Models — and Why the Same Broad Family?
If the goal were simply to compare as many machine learning approaches as possible, I could have selected a mix of models from different families.
But that was not what I wanted to learn.
My goal was to introduce meaningful variation without changing the entire type of learning problem.
The four models belong broadly to the same family — Tree-Based Gradient Boosting — but they do not learn in exactly the same way:
- how trees are grown
- how model complexity is controlled
- how certain types of features are handled.
That gave me an important balance:
Different enough to reveal different behavior, but similar enough to make those differences meaningful.
This was a little like brothers from the same family. They shared the same broad foundation, but each had its own characteristics, which meant they could react differently when faced with the same situation.
Why No Random Forest?
At this point, the obvious question is — Why wasn’t Random Forest included while comparing tree models?
Get JM tan’s stories in your inbox
Join Medium for free to get updates from this writer.
The reason was simply that it followed a different learning approach.
Random Forest is based on bagging, while the four models in this project were based on boosting.
I was not trying to compare every major tree-based approach. I wanted to keep the broad modeling direction relatively controlled, then observe whether meaningful differences could still emerge within it.
Control the broad direction, look for meaningful differences within it.
Why Comparing Multiple Models Became More Interesting
At first, one of the reasons for comparing four models was to reduce the risk of committing too early to the wrong starting point.
But as the project went on, the focus was no longer on which model performed the best. I started noticing that these similar models were responding differently to the same experimental changes:
- Feature Engineering: The same set of modified and additional features did not produce the same level of improvement across all four models.
- SHAP feature selection: Removing low importance features did not affect every model in the same way.
- Hyperparameter Tuning: The responses became even more contrasting — One model improved dramatically in CV but failed to generalize, another improved more modestly but reliably, while others showed much less room for improvement.
The experiments kept changing, but the four models rarely reacted in exactly the same way.
If I had optimized only one model, my conclusion might have been based on just one of their reactions:
- CatBoost said the large CV improvements were the main signal of successful tuning.
- LightGBM showed that careful tuning produced steady and reliable improvement.
- HistGradientBoosting suggested that there was limited room for further gains.
- XGBoost made an already strong baseline look difficult to improve further.
None of those conclusions was completely wrong, but each would have been incomplete.
One model could show me what happened, but four models helped me question whether that was the whole story.
What multiple models taught me
Working with multiple models helped me avoid judging the entire project by a single model’s reaction:
- An improvement in one model could have a different effect on the other models.
- High validation result could still fail to translate into robust generalization.
- If a model shows little sensitivity across varied setups, it might already be its optimal operating threshold.
- The strongest model at one stage was not necessarily the strongest at the next.
In other word:
Running parallel versions helped me to see whether an enhancement benefited the overarching project or simply worked well for one particular model.
So, why didn’t I just stick to optimizing one model?
A single version can only provide an isolated perspective, while comparing them against each other could tell whether I was seeing a broader pattern or simply one version of the story.
“This article is a follow-up to my Kaggle Ames Housing series, where I explored feature engineering, SHAP-based feature selection, hyperparameter tuning, and combining models and methods.”
