Predicting Term Deposit Subscriptions with Machine Learning
How I approached class imbalance, feature selection, and customer prioritization
I recently completed a machine learning project at Apziva focused on predicting whether a customer would subscribe to a term deposit following a marketing call. The objective was to help a European banking client improve customer outreach while keeping the model interpretable.
The task initially seemed straightforward! To use customer and campaign information to predict a yes-or-no outcome.
However, two details shaped the analysis: approximately 93% of customers did not subscribe, and I initially had an overfitted model, which I had to edit to make the model properly usable.
Here’s how I approached the problem, and how you can work through a similar classification task!
1. Understand the data and the decision
The dataset contains 13 predictors, including age, occupation, education, account balance, loan status, contact method, call duration, and the number of campaign contacts.
The target, y, indicates whether the customer subscribed.
I started, as always, by checking column types, missing values, categorical labels, and the target distribution:
import pandas as pd
df = pd.read_csv("term-deposit-marketing-2020.csv")
print(df.dtypes)
print(df.isna().sum())
print(df["y"].value_counts(normalize=True))The exploratory analysis found no conventional missing values and consistently reported categories. However, labels such as `unknown` still represent unavailable information and are thus examined separately.
The main issue was class imbalance. Only around 7% of customers subscribed, meaning that predicting “no” for everyone would already yield roughly 93% accuracy! That model would look sufficient while identifying none of the customers we were interested in finding.
To that extent, it was important to determine better diagnostics to actually represent the model’s performance.
2. Choosing appropriate metrics
To account for this imbalance, I prioritized balanced accuracy, which averages the model’s ability to identify subscribers and non-subscribers. An always-negative model would score 50% on this measure, meaning that
To see why, let’s write out the calculation:
Here, TP and TN represent correctly identified subscribers and non-subscribers, while FN and FP represent the respective false positives.
Suppose we have 1,000 customers: 70 subscribers and 930 non-subscribers. Predicting “no” for everyone gives:
Before fitting a model, make sure you ask what score you could achieve without learning anything from the predictors. That baseline tells you whether an apparently impressive result is actually useful.
I also examined recall, precision, F1, ROC AUC, and log loss. Recall was particularly relevant, answering the question: of the customers who actually subscribed, how many did the model identify?
Precision provides the other side of that question. If the model identifies more potential buyers but also recommends many unsuccessful calls, the client needs to understand that trade-off.
For a similar project, establish a majority-class baseline first. Then choose metrics based on what the business gains or loses from each type of error.
3. Preparing the data and fitting a baseline
I converted the target to binary values and one-hot encoded categorical predictors, removing one reference category from each. I then used an 80/20 train-test split and stratified five-fold cross-validation on the training data.
The following example prepares the inputs and creates the evaluation splits. I’ve added stratification to the initial train-test split so that both sets retain similar proportions of subscribers.
from sklearn.model_selection import train_test_split, StratifiedKFold
y = df["y"].map({"no": 0, "yes": 1})
X = pd.get_dummies(
df.drop(columns="y"),
drop_first=True,
dtype=int
)
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.20,
stratify=y,
random_state=23
)
cv = StratifiedKFold(
n_splits=5,
shuffle=True,
random_state=23
)Logistic regression then models a score between zero and one using the sigmoid function:
The coefficients determine how the inputs contribute to the model’s log-odds. Training estimates those coefficients by penalizing predictions that disagree with the observed outcomes.
With balanced class weights, observations from class (k) receive weight
where (n) is the total number of training observations and (n_k) is the number belonging to that class.
With approximately 7% subscribers, a subscriber receives a weight of about 7.14, compared with 0.54 for a non-subscriber. Each subscriber therefore contributes roughly 13 times as much to the training loss.
Class weighting changes how the model learns from mistakes through the loss function. We are penalizing the model for not having balanced predictions.
The initial unweighted logistic regression achieved roughly 93% accuracy, but its recall was only around 25%. It correctly identified many non-subscribers while missing most subscribers.
Using `class_weight=”balanced”` gave errors involving the minority class greater weight during training. This substantially improved recall and balanced accuracy, although precision and ordinary accuracy decreased.
That was a great direction for the project! Identifying potential buyers was central to the task. Thus, this penalty using balanced accuracy hopefully allows the model to better reflect the company’s needs.
4. Reduce complexity with L1 regularization
Once the weighted model performed well, I investigated whether fewer features could preserve its performance.
L1 regularization shrinks some coefficients to zero, allowing logistic regression to perform feature selection. I tested different values of `C`, where smaller values impose stronger regularization, and selected `C = 0.00329`.
Get Carlos Arturo Paez Gonzalez’s stories in your inbox
Join Medium for free to get updates from this writer.
The core pipeline was:
import numpy as np
from sklearn.model_selection import cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
# Inputs are already one-hot encoded, as in the repository.
model = Pipeline([
("scaler", StandardScaler()),
("logistic", LogisticRegression(
penalty="l1",
solver="liblinear",
C=0.00329,
class_weight="balanced",
max_iter=5000
))
])
scores = cross_validate(
model,
X_train,
y_train,
cv=cv,
scoring=["balanced_accuracy", "precision", "recall"]
)
for metric in ["balanced_accuracy", "precision", "recall"]:
values = scores[f"test_{metric}"]
print(f"{metric}: {values.mean():.3f} ± {values.std():.3f}")
# cross_validate fits copies; fit this model to inspect its coefficients.
model.fit(X_train, y_train)
coefficients = pd.Series(
model.named_steps["logistic"].coef_[0],
index=X_train.columns
)
selected = coefficients[np.abs(coefficients) > 1e-8]
print(f"Retained {len(selected)} of {X_train.shape[1]} features")
print(selected.sort_values())Conceptually, L1-regularized logistic regression balances two objectives:
A stronger penalty encourages a simpler model. In scikit-learn, smaller values of C correspond to stronger regularization. However, the precise scaling depends on the solver’s objective.
Something to wonder is why can L1 produce coefficients that are exactly zero? The absolute-value penalty has a sharp corner at zero. For some features, the improvement in prediction loss is too small to justify moving their coefficients away from that point.
Standardization matters because the penalty acts on coefficient sizes. A variable measured in euros and one measured in seconds can require very different coefficient magnitudes simply because of their units. Scaling makes those penalties more comparable.
Keeping scaling inside the pipeline also ensures that each validation fold is transformed using parameters learned only from its training fold. The printed “±” values describe variation across folds, rather than confidence intervals.
If you are attempting your own project like this, try a smaller and a larger C, then compare the number of nonzero coefficients with cross-validation performance. The aim is to find a useful balance between simplicity and predictive performance!
The selected model retained 21 of 36 encoded features, with little loss in predictive performance. These are encoded columns, rather than 21 original attributes.
Duration entered the regularization path first, followed by contact month and contact method. This helped explain which information the model relied on, although feature retention does not establish an explicit causal relationship with subscription, it can showcase potential meaningful correlations.
5. Remove information unavailable at prediction time
Duration was the most influential feature, but the final length of a call is unknown before it happens. A model using it cannot directly support the decision of who to call next.
I therefore repeated the analysis without duration. Balanced logistic regression fell to approximately 61% balanced accuracy, showing how much of the original performance depended on completed-call information.
I then tested histogram gradient boosting, random forests, and logistic regression with interaction terms. The interaction model retained approximately 79 terms without a meaningful improvement. Histogram gradient boosting performed best on balanced accuracy among the tested pre-call models, using a decision threshold near 0.08.
A lower threshold identifies more potential subscribers, but also produces more false positives. It should be selected using training-only validation and the client’s contact capacity, rather than assumed to work for every campaign.
The distinction between a score and a prediction is worth making explicit:
Here, (t) is the decision threshold. Changing it does not retrain the model or alter its ranking of customers. It changes which customers receive a positive prediction.
We can explore this trade-off using out-of-fold scores. Each customer is scored by a model trained without that customer’s fold:
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import (
balanced_accuracy_score, precision_score, recall_score
)
X_train_pre = X_train.drop(columns="duration")
boosting = HistGradientBoostingClassifier(
learning_rate=0.1,
max_iter=300,
max_leaf_nodes=15,
min_samples_leaf=100,
random_state=23
)
oof_scores = cross_val_predict(
boosting,
X_train_pre,
y_train,
cv=cv,
method="predict_proba"
)[:, 1]
rows = []
for threshold in [0.05, 0.08, 0.10, 0.20, 0.50]:
predictions = (oof_scores >= threshold).astype(int)
rows.append({
"threshold": threshold,
"balanced_accuracy": balanced_accuracy_score(
y_train, predictions
),
"precision": precision_score(
y_train, predictions, zero_division=0
),
"recall": recall_score(y_train, predictions),
"fraction_selected": predictions.mean()
})
print(pd.DataFrame(rows).round(3))Compare recall with the fraction of customers selected. A threshold that identifies more subscribers may also recommend more calls than the team can make.
Lowering the threshold cannot decrease recall on a fixed set of scores, but precision does not necessarily change monotonically. Examine the actual results rather than assuming every metric will move smoothly.
This table supports threshold selection using training data. Once a threshold is chosen, its final performance should be assessed on untouched data; selecting and reporting the best result from the same table would give an optimistic estimate.
The final repository README reports these five-fold cross-validation averages:
These are the repository’s reported results, not a new reproduction. Earlier runs have different approximate scores. The distinction remains consistent: performance is considerably stronger when duration is available.
For deployment, the same availability check should apply to campaign count, contact date, and contact method. Each field must reflect information known at the moment the customer is scored.
6. Translate the results into customer insights
Alongside modelling, I compared subscription rates across customer segments using lift, which is a group’s subscription rate divided by the overall rate.
Mathematically, for a group (denoted with S),
We estimate these probabilities using observed proportions. For example, a 14% subscription rate against a 7% overall rate gives a lift of 2.
The following code calculates lift by occupation using the training records:
segment_data = df.loc[X_train.index].copy()
segment_data["y"] = y_train
baseline_rate = segment_data["y"].mean()
segments = (
segment_data.groupby("job")["y"]
.agg(customers="size", subscribers="sum", rate="mean")
)
# Match the project's minimum segment-size rule.
segments = segments[segments["customers"] > 400].copy()
segments["lift"] = segments["rate"] / baseline_rate
print(segments.sort_values("lift", ascending=False).round(3))The original exploratory analysis used the full dataset, so this training-only example may produce different values.
Always examine segment size alongside lift. A high subscription rate based on a handful of customers is less stable than the same rate observed across hundreds.
Under a simple independent Bernoulli model, the approximate standard error of a segment’s subscription rate is
This shows why sample size matters! Uncertainty decreases approximately with (1/sqrt (n)). Four times as many observations roughly halves the standard error, assuming the underlying rate stays the same.
This expression concerns the subscription rate, not the lift ratio itself. It also does not account for repeated customers or the many segments explored. Those require additional care when assessing uncertainty.
The exploratory analysis found that students subscribed at roughly twice the overall rate, retired customers at around 1.45 times the rate, and customers contacted in April at around 2.29 times the rate. Management customers were only slightly above average but represented a large group, making their volume relevant as well.
These findings help communicate patterns to stakeholders, but they are observational. One good example of this is the April conversion rate. A high April conversion rate does not prove that moving calls to April will improve outcomes. It is totally possible that the campaign design and customer composition may explain part of the difference.
7. Validate the recommendation in practice
The pre-call model is best treated as a prioritization aid. A useful next step would be to rank customers, select the number the team can realistically contact, and compare conversion and subscriber capture against the existing outreach process.
Before that test, freeze model settings and the threshold, then evaluate on fresh data — ideally from a later campaign. The repository explores several models and examines test results during development, so a fresh evaluation would provide stronger evidence than the selected cross-validation scores alone.
The main takeaway for me was that model performance needs to be interpreted alongside feature availability and the business objective. Class weighting improved buyer detection, L1 regularization simplified the model, and removing duration clarified what could reasonably be predicted before a call.
That last step was especially useful. It turned a strong classification result into a more realistic recommendation about how the model could support customer outreach.
You can find the full analysis and scripts in my [GitHub repository](https://github.com/capg2002/Marketing-Call-Classification-with-Machine-Learning).
Start with `eda.py`, followed by `initial_logistic_regression.py` and `feature_selection.py`. The pre-call experiments are in `final_model_testing_wo_duration.py`. The scripts require `term-deposit-marketing-2020.csv`, which is not included in the repository.
