A Step-by-Step Guide to Solving an Imbalanced Classification Problem
During my AI Residency at Apziva, I worked on a machine learning project focused on predicting customer responses to banking marketing campaigns. The goal of this project was to build a reliable model that could help banks identify customers who are most likely to subscribe to a term deposit.
In this article, I will walk through the entire project step-by-step, explaining not just what I did, but also why each step was important. My aim is to provide a practical guide for anyone interested in solving real-world classification problems, especially those involving imbalanced data.
You can explore the complete implementation here:
GitHub Repository:
Project Overview
Banks often conduct telemarketing campaigns to promote term deposits. However, these campaigns usually result in low conversion rates because most customers decline the offer.
This creates several challenges:
- High operational costs
- Inefficient use of resources
- Low return on investment
Instead of contacting every customer, a predictive model can help identify customers who are more likely to respond positively. This project focuses on building such a model.
Problem Statement
The central question addressed in this project was:
“Which customers are most likely to subscribe to a term deposit during a marketing campaign?”
While this may seem straightforward, the dataset presented several practical challenges:
- Highly imbalanced target variable
- Risk of data leakage
- Need for interpretable features
- Requirement for business-relevant insights
Addressing these challenges required careful preprocessing, thoughtful feature selection, and structured model development.
Dataset Overview
The dataset used in this project contained approximately 40,000 customer records and 14 features.
The target variable was:
- y = yes → Customer subscribed
- y = no → Customer did not subscribe
Feature Categories
The dataset included features from multiple categories.
Demographic Features
- age
- job
- marital
- education
Financial Features
- default
- balance
- housing
- loan
Campaign Features
- day
- month
- campaign
Important Feature Decisions
Two features were removed early in the project:
- duration
- contact
This decision was particularly important.
The feature duration represents the length of the phone call, which is only known after the call is completed. Including this feature would introduce data leakage, allowing the model to use information that would not be available during real-world prediction.
Removing this feature ensured that the model remained realistic and suitable for deployment scenarios.
Step 1 — Exploratory Data Analysis (EDA)
The first step in the project was performing Exploratory Data Analysis (EDA). This stage helped me understand the structure of the dataset and identify patterns that could influence modeling decisions.
During EDA, I performed:
- Summary statistics analysis
- Distribution visualization
- Target relationship exploration
- Group-based analysis
Key Observations from EDA
Several patterns emerged during the analysis.
Numeric Features
- The average customer age was approximately 40 years.
- The balance feature showed a highly skewed distribution.
- The average number of campaign contacts was about 2.88.
Customer Behavior Patterns
Some customer groups showed noticeably higher subscription rates.
For example:
- Students had higher response rates
- Retired customers responded more frequently
- Customers with a tertiary education showed better engagement
- Customers with higher balances were more likely to subscribe
Seasonal Insights
Campaign performance varied across months.
Higher subscription rates were observed during:
- October
- March
- April
This suggested that customer behavior may be influenced by seasonal trends.
Step 2 — Feature Selection and Preprocessing
Once the dataset was understood, I moved on to preprocessing and feature preparation.
The following steps were performed:
- Dropped leakage-prone features
- Applied One-Hot Encoding to categorical variables
- Split data into training and testing sets
- Used stratified sampling to maintain class distribution
These steps ensured that the dataset was properly structured for machine learning models.
Step 3 — Handling Class Imbalance
One of the most important challenges in this project was class imbalance.
The target distribution showed:
- No Subscription: ~92%
- Subscription: ~8%
If not handled carefully, this imbalance can cause models to predict the majority class most of the time while ignoring minority cases.
To address this issue:
- Stratified sampling was applied
- Evaluation metrics beyond accuracy were considered
- Model performance was analyzed using recall and ROC-AUC
This helped ensure that the model captured minority class behavior effectively.
Step 4 — Building the Baseline Model
Before optimizing the model, I built a baseline model to establish initial performance.
To evaluate the baseline model, I used:
Stratified 5-Fold Cross Validation
Baseline Performance
The baseline model produced the following results:
- Accuracy: 75.8%
- Recall: 0.49
- F1 Score: 0.23
- ROC-AUC: 0.69
Although the accuracy appeared reasonable, the recall and F1 score indicated that improvement was needed, particularly in identifying minority class cases.
Step 5 — Hyperparameter Tuning
To improve model performance, I applied RandomizedSearchCV.
Get shincy mathew’s stories in your inbox
Join Medium for free to get updates from this writer.
I selected Randomized Search because:
- It is faster than Grid Search
- It efficiently explores large parameter spaces
- It improves model generalization
Parameters Tuned
The following parameters were optimized:
- n_estimators
- max_depth
- learning_rate
- subsample
- colsample_bytree
Optimized Model Results
After tuning, the model showed improved performance:
- Cross-validation accuracy increased to approximately 81.6%
- Model generalization improved
- Performance stability increased
Step 6 — Model Evaluation
After training and tuning, the model was evaluated using the unseen test dataset.
Final Model Performance
- Test Accuracy: ~81%
- ROC-AUC: ~0.73–0.74
- Improved Recall
- Stable performance across validation sets
These results indicated that the model successfully learned meaningful patterns without overfitting.
Confusion Matrix Interpretation
To better understand how the model performed, I analyzed the confusion matrix generated from the final predictions.
The results were:
- True Negatives: 6239
- True Positives: 262
- False Positives: 1182
- False Negatives: 317
This resulted in an overall accuracy of approximately 81%, which aligns with the earlier test performance metrics.
More importantly, the model correctly identified 243 customers who actually subscribed, while maintaining strong performance in identifying non-subscribers.
Although there were some false positives, this is acceptable in marketing scenarios where reaching a few additional customers is preferable to missing potential subscribers. Overall, the confusion matrix confirms that the model achieved a good balance between predictive performance and practical usability.
ROC Curve Analysis
To further evaluate the model’s performance, I plotted the Receiver Operating Characteristic (ROC) curve. The ROC curve illustrates the relationship between the True Positive Rate (Recall) and the False Positive Rate across different classification thresholds.
The Area Under the Curve (ROC-AUC) for the final model was approximately 0.73–0.74, indicating that the model has a good ability to distinguish between customers who subscribe and those who do not.
An ROC-AUC value above 0.7 generally represents acceptable model performance, particularly in imbalanced classification problems where detecting minority class behavior is challenging.
Overall, the ROC curve confirms that the model performs significantly better than random guessing and demonstrates reliable predictive capability for targeted marketing applications.
Step 7 — Feature Importance Analysis
Understanding which features influence predictions is critical in business-focused machine learning projects.
Feature importance analysis revealed several strong predictors.
Most Influential Features
- Month
- Marital Status
- Housing Loan
- Account Balance
- Campaign Day
Interpretation
These features suggest that customer decisions are influenced by:
- Timing of campaign
- Financial stability
- Household characteristics
Such insights are valuable for designing more effective marketing strategies.
Step 8 — Customer Segmentation Insights
Beyond prediction, this project also focused on customer segmentation.
Segmentation was performed using probability-based grouping.
The method used was:
df.groupby('feature')['y'].mean()This approach calculates the subscription probability for each group.
Key Segmentation Results
Some customer segments showed higher response probabilities.
Notably:
- Students showed approximately 15.6% response rate
- Retired customers showed approximately 10.5% response rate
- Customers with tertiary education showed higher engagement
- High-balance customers demonstrated stronger response behavior
These findings provide meaningful direction for targeted marketing.
Business Impact
From a business perspective, this model provides several practical advantages.
Improved Targeting
Marketing teams can prioritize:
- Students
- Retired customers
- Financially stable clients
Cost Reduction
Campaign resources can be focused on:
- High-probability customers
- Avoiding unnecessary outreach
Better Return on Investment
Campaign planning can emphasize:
- High-response months
- Customer segments with stronger engagement history
Lessons Learned from This Project
Working on this project reinforced several important machine learning principles.
Preventing Data Leakage is Critical
Removing the duration feature ensured realistic predictions and prevented misleading results.
Imbalanced Data Requires Careful Evaluation
Metrics such as recall, F1 score, and ROC-AUC provided deeper insights beyond accuracy.
Business Context Should Guide Technical Decisions
Modeling decisions were made with deployment and usability in mind, not just performance metrics.
Conclusion
This project demonstrates the complete lifecycle of a machine learning solution, from data exploration to final business insights.
Through structured analysis, careful feature selection, and model optimization, I developed a model capable of predicting customer subscription behavior with reliable performance.
More importantly, the project highlights how machine learning can support data-driven marketing strategies and improve campaign efficiency.
For Hiring Managers
This project demonstrates:
- End-to-end machine learning workflow execution
- Practical handling of imbalanced classification problems
- Prevention of data leakage through informed feature selection
- Hyperparameter Optimization Using Randomized Search
- Translation of model outputs into actionable business insights
