I Realized I Was Asking SHAP Too Early
Why I Tuned My Model Before Feature Selection
“The experiments, analysis, and interpretations are based on my own work, ChatGPT was used to assist with editing, wording, and improving readability.”
The Workflow I Was Taught
When I started to learn machine learning, one core idea appeared — Prepared your data completely before modeling. To me, this meant to complete all data processing steps first, includes data cleansing, feature engineering, and feature selection.
It seemed completely logic and intuitive.
Data cleaning removes meaningless or problematic information from the dataset. Feature engineering expands or transforms the available information. Feature selection then removes variables that appear less useful, leaving the model with a cleaner and often smaller feature space.
The dataset is now ready for modeling.
Only then comes hyperparameter tuning — finding the model settings that “match” better with the dataset.
With the feature set already established, the tuning process can focus on finding a suitable configuration without having to repeat the optimization every time the features change.
This ordering offers several practical advantages.
Reducing unnecessary features can simplify the learning problem, lower computational cost, and sometimes improve generalization. So, the workflow I had learned was essentially:
Data Cleaning → Feature Engineering → Feature Selection → Hyperparameter Tuning → Final Model
It made sense. It was orderly, efficient, and easy to follow.
At the time, I had no reason to question it.
But One Question Kept Bothering Me…
After selecting SHAP as my method for feature selection, one question came to mind.
If SHAP selects features based on the importance calculated by the model, why should I let an untuned model decide which features to keep, instead of first tuning the model so that its learning behavior is better suited to the dataset?
One analogy popped into my mind at that moment.
Imagine asking a young child:
“What do you want to be when you grow up?”
You’ll probably get an answer immediately.
But how much should we trust that answer?
Maybe the child wants to become a doctor because a parent is a doctor. Maybe an astronaut simply sounds exciting. Maybe they have never even thought about what those jobs actually involve.
Now ask the same question years later, after the child has learned about different subjects, explored different interests, understood what different occupations actually involve, and gained some real-world experience.
The question hasn’t changed.
But the person answering it has changed.
The answer becomes more reliable because the person answering the question has become more informed.
“Isn’t my model doing something similar?”
The analogy suddenly felt surprisingly close to what I was doing with SHAP.
The model already has enough information to calculate feature importance.
But it is still using its initial hyperparameters.
Hyperparameter tuning in later stage will changes those parameters.
It can change the way the model learns — including the way different features contribute to predictions.
If tuning changes the model, could it also change the feature importance that SHAP reveals?
At that point, tuning before SHAP no longer felt unreasonable.
I had a logical reason to try it!
Then Another Question Emerged
My first conclusion was straightforward: tune before SHAP.
But something didn’t quite make sense.
If tuning can influence which features SHAP considers are more important, what happens after SHAP removes some of those features?
I’ve now changed the feature space.
If I was questioning whether SHAP should rely on an untuned model, why should I assume that the first round of tuning is still optimal after the feature space has changed?
Shouldn’t the model’s learning behavior be reconsidered as well?
💡Why limit tuning to one round?
🤔What if I tune again based on the new feature space?
That led to a new workflow:
Data Cleaning → Feature Engineering → Hyperparameter Tuning → Feature Selection → Hyperparameter Tuning → Final Model
Let the model learn first, use SHAP on that better-tuned model, and then tune again based on the new feature space.
I Needed More Than an Analogy
My idea made logical sense. But did it make technical sense too?
So I started looking more closely at the relationship between the two methods — Maybe the way they affect each other could explain why this workflow made sense.
How Tuning Can Change What SHAP Sees
SHAP is used to explain a machine learning model.
That means the feature importance produced by SHAP is not simply a fixed ranking of the dataset —it describes how a particular model uses the available features.
While hyperparameters define how an estimator is configured, and tuning searches for configurations that perform well under the selected evaluation method.
And that gave me the first important connection to my earlier idea:
Tuning can change what SHAP sees.
If the model learns differently under different hyperparameters, then the way it uses individual features could also change.
This means the feature importance could be changed.
This doesnt mean that the default untuned model would produce a “wrong” SHAP explanation.
Just …
If I was going to change the model through hyperparameter tuning later, why not make that change before asking SHAP to help me decide which features to keep?
How SHAP Can Change What Needs to Be Tuned
Then I looked at their relationship from the other direction.
Get JM tan’s stories in your inbox
Join Medium for free to get updates from this writer.
Feature selection changes the inputs that are passed to the model, and the selected features become the new feature space on which the subsequent estimator is trained.
Suppose I tune a model that has access to one feature space.
Then SHAP removes some features and now we have a different set of inputs.
The model is no longer solving exactly the same problem as before.
It changes the learning problem.
And this led to another connection:
The hyperparameters that worked well for the original feature space may not necessarily remain optimal for the new one.
This isn’t because the first round of tuning was wrong but the optimization was for the old feature space.
Once that feature space changed, there was a reasonable possibility that the balance between model complexity, learning behavior, and generalization would needs to be reconsidered.
So the relationship goes both ways:
Tuning can change what SHAP sees.
SHAP can change what tuning needs to work with.
This technical reasoning made my new workflow feel reasonable enough to try:
Tune → SHAP → Tune
Why I Didn’t Keep Iterating Forever
If tuning can change SHAP, and SHAP can change tuning, couldn’t we just keep repeating the cycle and look for further improvements?
But would this make the workflow better — both in terms of performance and efficiency? Or where would the equilibrium point be?
With this question in mind, the kid analogy came back to me.
Imagine the child keeps learning about the world.
At first, learning more helps him build a foundation — languages, mathematics, science, history, and many other things.
As he gets older, he begins to understand his strengths and interests better. He will also know what he doesnt like and what may eventually make him lose interest. He may then choose a particular direction to explore more deeply.
He can definitely keep learning and exploring to become more confident in that decision. But when should he stop? After getting a degree? A master’s? At 30? At 50? And would continuing to learn really guarantee that he would make a decision he would never regret?
The point is, at some time, he still needs to make a decision.
He doesn’t need to know everything before choosing a direction, but needs to know enough to make a reasonable decision.
I started to see my workflow in a similar way.
The first tuning was not supposed to make the model perfect, but was meant to make the model more informed before SHAP made its feature-selection decision.
SHAP then narrowed the feature space, giving the model a more focused learning problem.
A second round of tuning made sense because the learning problem had changed.
After that, I could theoretically run SHAP again, tune again, and continue the cycle indefinitely.
But just because another round is possible doesn’t mean another round is necessary.
The Technical Boundary
My analogy explained why I didn’t keep repeating SHAP and tuning indefinitely. But there are also technical reasons not to continue the process forever.
- The Experiment Becomes Harder to Interpret
If the workflow keeps expanding: Tune → SHAP → Tune → SHAP → Tune…
Then every additional cycle will introduces another set of decisions:
- Which features were removed?
- Which hyperparameters were changed?
- Why were they changed?
- Which result justified the next iteration?
Eventually, it becomes harder to understand what actually caused the improvement.
2. Every Round Needs a Reason
As the workflow becomes more iterative, the expected benefit of an additional round may become smaller, while the cost and complexity remain.
The first extra tuning had a clear purpose —
Make the model more informed to select features.The second tuning had another clear purpose —
The learning problem had changed.But every extra cycle would need its own justification.
“Could the CV score improve?” was not sufficient. I needed to know what another round was supposed to achieve.
3. CV Improvements Need More Skepticism
A better CV score is not automatically a reason to continue iterating.
- If the improvement is small, we need to consider whether it reflects better generalization or simply better adaptation to the validation setup.
- If the improvement is unexpectedly large, we need to understand what caused it before assuming the gain is real.
The more times we iterated against the same validation framework, the more cautious we needed to become about interpreting CV improvements as evidence of better generalization.
For me, the goal was not to find the maximum number of times I could iterate. It was to find a point where another iteration was no longer justified by the information it could provide.
What This Changed About My Workflow
At first, I learned about the standard workflow: Feature Engineering → Feature Selection (SHAP) → Hyperparameter Tuning (Tune) → Final Model
Then, I decided to try: Feature Engineering → Tune → SHAP → Tune → Final Model
This was because each step had its own purpose and reason:
- Tune first because the model used for SHAP should be reasonably representative of how the model will learn.
- Run SHAP to select more relevant features.
- Tune again because the learning problem had changed.
- Stop when another iteration no longer had a clear purpose.
Actually, what changed wasn’t simply the number of tuning rounds. It was the way I thought about the workflow itself, and whether each step was reasonable.
“This article is a follow-up to my Kaggle Ames Housing series, where I explored feature engineering, SHAP-based feature selection, hyperparameter tuning, and combining models and methods. This time, I took the opportunity to revisit one of the decisions behind that workflow.”
Resources
< SHAP documentation >
< scikit-learn: Feature Selection >
< scikit-learn: Hyper-parameters Tuning>
