How to Predict and Prevent Customer Churn Using Machine Learning
Get a Better Understanding of What Data is Captured and How to Utilize that to Drive Customer Retention
As the cost of money is increasing and additional funding is coming with ‘additional strings attached’, it is now more important than ever for SaaS businesses to focus more religiously on customer retention. Investors are no longer buying the ‘growth at all cost’ story and are pushing to see a commitment to a path to profitability. The best way to increase your profitability is to retain and increase spend from your existing customer.
Increasing net retention in the current economic climate is tough. Coupled with tight liquidity, it is almost unthinkable to run a large customer acquisition campaign in this environment. By leveraging data and analytics, SaaS companies can gain insights into customer behavior, preferences, and satisfaction levels. This information can help businesses tailor their products and services to better meet customer needs, improving retention and reducing churn.
This article will focus on hands-on example of how you can very cheaply start implementing a machine learning model to anticipate and predict customers likely to churn and thereby efficiently and effectively utilizing your sales and marketing budget. Read my articles here: to know more about how to grow a SaaS business during recession and four ways to make your SaaS business profitable.
Garbage in = garbage out
To predict and prevent customer churn, you must first understand what data you are capturing. Data such as user activity, engagement metrics, demographics, and customer feedback are crucial for predicting churn.
The purpose of this article is not to delve deep into a data governance strategy as that is a topic of its own. If you do not have a define data model and a data governance strategy, you may broadly have to take the following steps:
- Identify your data sources
- Determine the data schema and data architecture that will be used to capture and store the data
- Establish data quality and data governance standards to ensure the accuracy and completeness of data
- Develop a data model and a data dictionary that defines the relationships between the data entities and attributes
If you are a small or medium sized business, chances are that the majority of the data can be acquired from your CRM, billing and ticketing systems. The key here to remember is that if you do have strong data governance and if your data is not correctly labeled for churned and ongoing customers then a machine learning model may not yield accurate results.
So how do we predict churn
For the purpose of this article, we would be using data on telecom business on Kaggle as there are not many publicly available datasets for an enterprise software business. However, the concepts would be applicable on a dataset for a SaaS business.
We are going to use Python and the following libraries to build our model. If you do not know Python, do not worry, you can follow along and leverage the code in this article to build your own simple machine learning model in no time.
1) Importing required libraries
#----main libraries----
import numpy as np
import pandas as pd
#----plotting libraries----
import seaborn as sns
import matplotlib.pyplot as plt
#----model and support imports----
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score, cross_val_predict
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score, recall_score, precision_score, f1_score, roc_auc_score, ConfusionMatrixDisplay
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier2) Loading data
tel_cus_df = pd.read_csv("/kaggle/input/telco-customer-churn/WA_Fn-UseC_-Telco-Customer-Churn.csv")3) Exploring data
Before you start building your machine learning model, it is always a good idea to explore the data to ensure that you are appropriately normalizing the data for any outliers, cleaning for any incomplete data, and also understanding the data and the interplay of the fields in your dataset.
The following code will help us visualize a sample of dataset:
tel_cus_df.head()
The following code will help us visualize the data type for all the fields in our data set:
tel_cus_df.info()
As we can see from the Dtype for TotalCharges that it is an object. We would have expected this field to be numeric. Let us try to convert this field to a numeric value and then check for any nulls/missing values in the dataset:
tel_cus_df['TotalCharges'] = tel_cus_df['TotalCharges'].replace(' ', np.nan, regex=True)
tel_cus_df['TotalCharges'] = pd.to_numeric(tel_cus_df['TotalCharges'])
tel_cus_df.isnull().sum()
4) Processing data
It seems that TotalCharges has 11 null values. We would need to either add a value to the field or remove the rows with null values to make our machine learning model function properly. We will come back to this later. Let us know check the number of unique values in each field/column:
columns = tel_cus_df.columns
print("----------- Numeric field ----------\n")
for i in range(len(columns)):
if tel_cus_df[columns[i]].dtypes!=object:
print("unique number of {} ---> {}".format(columns[i], len(tel_cus_df[columns[i]].unique())))
print("\n---------- Categorical field ----------\n")
for i in range(len(columns)):
if tel_cus_df[columns[i]].dtypes==object:
print("unique number of {} ---> {}".format(columns[i], len(tel_cus_df[columns[i]].unique())))
Let us know populate the missing values for TotalCharges. We can attempt to do that by estimating the TotalCharges based on the Contract and MonthlyCharges fields. Let us first check what values we have in the Contract field:
tel_cus_df['Contract'].unique()Given there are 3 unique values: “Month-to-month”, “One year” and “Two year”, we can utilize these to create a ContractMonth field in order to estimate TotalCharges:
tel_cus_df['ContractMonths'] = np.where(tel_cus_df['Contract']=="Month-to-month", 1, 0) + np.where(tel_cus_df['Contract']=="One year", 12, 0) + np.where(tel_cus_df['Contract']=="Two year", 24, 0)
tel_cus_df['ContractMonths'] = pd.to_numeric(tel_cus_df['ContractMonths'])
tel_cus_df.loc[tel_cus_df['TotalCharges'].isnull(), ['Contract','ContractMonths','MonthlyCharges','TotalCharges']]
We can now estimate the missing values for TotalCharges:
tel_cus_df['cal_TotalCharges'] = tel_cus_df['ContractMonths'] * tel_cus_df['MonthlyCharges']
tel_cus_df['TotalCharges'] = np.where(tel_cus_df['TotalCharges'].isnull(), tel_cus_df['cal_TotalCharges'], tel_cus_df['TotalCharges'])
tel_cus_df.isnull().sum()We do not have any null values now.
5) Dropping fields not required and visualizing data
Running a machine learning model is computationally expensive and if there are fields that do not add any predictive value to your model, then we should drop these fields:
tel_cus_df.drop(['customerID','ContractMonths','cal_TotalCharges'], axis=1, inplace=True)
tel_cus_df.columns.valuesLet us know visualize the data to understand it further and check if any further clean-up is required:
attributes = ['gender', 'SeniorCitizen', 'Partner', 'Dependents',
'PhoneService', 'MultipleLines', 'InternetService',
'OnlineSecurity', 'OnlineBackup', 'DeviceProtection',
'TechSupport', 'StreamingTV', 'StreamingMovies', 'Contract',
'PaperlessBilling', 'PaymentMethod']
fig, axes = plt.subplots(nrows = 8, ncols = 2, figsize = (22,20))
for i, item in enumerate(attributes):
if i < 8:
ax = tel_cus_df[item].value_counts().plot(kind= 'bar', ax=axes[i,0], rot =0, color=['black', 'red', 'green', 'blue', 'cyan'])
elif i >= 8:
ax = tel_cus_df[item].value_counts().plot(kind= 'bar', ax=axes[i-8,1], rot =0, color=['black', 'red', 'green', 'blue', 'cyan'])
ax.set_title(item)
fig.tight_layout()
A lot of fields seem to have data quality issues (i.e., no internet service instead of just yes or no). Let’s explore these further:
tel_cus_df.loc[tel_cus_df['OnlineSecurity'] == "No internet service", :].head()
It appears that ‘no internet service’ is downstream impact of a user not having an internet service (i.e., these are add-ons to the internet service). Therefore, we will not adjust for these in our model.
6) Model selection
Let us explore the field that we want to predict (churn in this case):
sns.countplot(x = "Churn", data = tel_cus_df)
tel_cus_df.loc[:, 'Churn'].value_counts()
The data is imbalanced (i.e., our data has more values where the customer had not churned and less values where the customer had churned). Training a machine learning model without correcting for data imbalance, will make our model biased towards predicting more customers as not churned (majority class).
We can either use approaches to oversample the minority class or we can use XGBoost and balance the positive and negative weights by setting scale_pos_weight to sum (negative instances) / sum (positive instances).
Do not worry, if you do not understand it now. All we are doing in the XGBoost is penalizing the model for predicting the majority class to balance the output. We will come to that later on how to do that.
spw = -(-tel_cus_df[tel_cus_df.Churn == "No"].shape[0]// tel_cus_df[tel_cus_df.Churn == "Yes"].shape[0])
print("XGBoost weight balancing ---> neg: {}, pos: {}, scale_pos_weight: {}".format(tel_cus_df[tel_cus_df.Churn == "No"].shape[0], tel_cus_df[tel_cus_df.Churn == "Yes"].shape[0], spw))You will see the following output: XGBoost weight balancing — > neg: 5174, pos: 1869, scale_pos_weight: 3
7) Model configuration
We will use three different models in this exercise (i) Logistic regression, (ii) Random Forest, and (iii) XGBoost. We will then also use TensorFlow to see if we get better performance using a Neural Network. We will run the first 2 models (Logistic Regression and Random Forest) without correcting for the data imbalance and hence we should expect it to be better at predicting when a customer has not churned with the downside of missing customers that did actually churn. We will use SMOTE (synthetic minority oversampling technique) to correct for imbalance while using TensorFlow.
In order to understand why a machine learning algorithm does that, it would be important to understand the mathematical model behind that, which would outside the scope of this article. Simply stating, the goal of any machine learning algorithm is to minimize the difference between its predicted output and the actual output. The algorithm achieves this by adjusting the model parameters or weights during the training process, based on the cost function, which measures the difference between the predicted and actual output.
The mathematical goal of a machine learning algorithm is to find the optimal model parameters that minimize the cost function, also known as the loss function, over the training data. This is typically done through an optimization algorithm such as gradient descent, which iteratively updates the model parameters to minimize the cost function.
Get Arsalan Pardesi’s stories in your inbox
Join Medium for free to get updates from this writer.
Let us first encode our dependent variable (i.e., convert it to 1 and 0):
le = LabelEncoder()
tel_cus_df['Churn'] = le.fit_transform(tel_cus_df['Churn'])We will now convert our categorical dependent variables into dummy variables (i.e., generate a separate column for each value and changing the values into 1 and 0):
df_dummy = pd.get_dummies(tel_cus_df)
df_dummy.head()
Let us now check correlation between the various variables:
matrix = np.triu(np.ones_like(df_dummy.corr()))
fig, ax = plt.subplots(figsize=(25,25),dpi=80, facecolor='w', edgecolor='k')
sns.heatmap(df_dummy.corr(), mask= matrix, cmap="coolwarm", annot = True, fmt = '.1f', center = 0,annot_kws={"fontsize":10})
It is typically good to spend some time here on feature selection and removing variables that might display multicollinearity, i.e., where independent variables are highly correlated to each other. We will skip this step but implementing this step may improve our model’s performance.
8) Splitting train test data
Let us now split the train test data:
X = df_dummy.drop("Churn", axis = 1)
y = df_dummy['Churn']
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 100, test_size = 0.3)
print(y_train.value_counts())Let us now applying scaling to the dependent variables:
scale = MinMaxScaler()
col = X_train.columns
X_train[col] = scale.fit_transform(X_train[col])
X_test[col] = scale.transform(X_test[col])9) Training and prediction
How to interpret results
Before we dive into the models, let’s understand what results we will generate and how to interpret these results:
Accuracy, recall, precision, cross-validation accuracy, and F1 score are metrics used to evaluate the performance of a machine learning model.
Accuracy: measures the proportion of correct predictions made by the model out of all predictions made. It is calculated as the ratio of correct predictions to the total number of predictions made.
Accuracy = (TP + TN) / (TP + TN + FP + FN)
where TP = true positive, TN = true negative, FP = false positive, and FN = false negative.
Recall: measures the proportion of true positive instances that are correctly identified by the model. It is calculated as the ratio of true positive instances to the total number of positive instances.
Recall = TP / (TP + FN)
Precision: measures the proportion of true positive predictions out of all positive predictions made by the model. It is calculated as the ratio of true positive instances to the total number of positive predictions made.
Precision = TP / (TP + FP)
Cross-validation accuracy: measures the accuracy of the model when applied to new, unseen data. It is typically calculated using k-fold cross-validation, where the data is split into k subsets and the model is trained and tested k times on different subsets.
F1 score: is the harmonic mean of precision and recall, and provides a balance between the two metrics. It is calculated as:
F1 Score = 2 * (Precision * Recall) / (Precision + Recall)
Logistic Regression
Logistic Regression is a supervised machine learning algorithm used for binary classification problems. It models the probability of the output variable taking a certain value, given the input features. It uses a sigmoid function to map the output to a probability value between 0 and 1.
Let us train our model using logistic regression and look at the results:
accuracy= []
recall =[]
roc_auc= []
precision = []
LR = LogisticRegression()
result_LR = LR.fit(X_train, y_train)
LR_y_pred = LR.predict(X_test)
cross_val_LR = cross_val_score(LR, X_train, y_train, scoring = 'accuracy')
cross_val_LR_f1 = cross_val_score(LR, X_train, y_train, scoring = 'f1')
accuracy.append(round(accuracy_score(y_test, LR_y_pred),4))
recall.append(round(recall_score(y_test, LR_y_pred),4))
roc_auc.append(round(roc_auc_score(y_test, LR_y_pred),4))
precision.append(round(precision_score(y_test, LR_y_pred),4))
model_names = ['Logistic Regression']
result_df = pd.DataFrame({'Accuracy':accuracy,'Recall':recall, 'Roc_Auc':roc_auc, 'Precision':precision, 'Cross_val_acc':cross_val_LR.mean(), 'Cross_val_f1':cross_val_LR_f1.mean()}, index=model_names)
result_df
Let us plot the confusion matrix for the results:
cm = confusion_matrix(y_test, LR_y_pred, labels=LR.classes_)
disp = ConfusionMatrixDisplay(confusion_matrix=cm,display_labels=LR.classes_)
disp.plot()
plt.show()
This model provides high accuracy but low recall and reasonable precision. Remember that accuracy can be higher driven by correct identification of true negatives. In certain cases, if the data is very skewed towards negative values, a model can get a high accuracy on a testing set by predicting the majority of values as negative. For our purposes, we want a model that has a higher recall as it is more important for us to have a model that has a better performance on true positive so that we can focus retention efforts towards those customers.
Random Forest
Random Forest is a supervised machine learning algorithm used for classification and regression problems. It creates a large number of decision trees, each trained on a random subset of the input features and a random subset of the training data. It then aggregates the predictions of the individual trees to make a final prediction.
Logistic regression is typically simpler and less prone to overfitting on a smaller dataset, if the relationship between input feature and the output variable is linear. Random forest is better than logistic regression when the relationship between the input features and the output variable is nonlinear and complex. It is also better suited for handling high-dimensional datasets with many input features, as it can capture more complex interactions between features.
Depending on how complex your data model is, you may want to start with a simple logistic regression before using other models.
Let us now implement a Random Forest classifier:
accuracy= []
recall =[]
roc_auc= []
precision = []
RF = RandomForestClassifier(criterion='entropy', min_samples_leaf=2,min_samples_split=6, n_estimators=1000, random_state=0)
result_RF = RF.fit(X_train, y_train)
RF_y_pred = RF.predict(X_test)
cross_val_RF = cross_val_score(RF, X_train, y_train, scoring = 'accuracy')
cross_val_RF_f1 = cross_val_score(RF, X_train, y_train, scoring = 'f1')
accuracy.append(round(accuracy_score(y_test, RF_y_pred),4))
recall.append(round(recall_score(y_test, RF_y_pred),4))
roc_auc.append(round(roc_auc_score(y_test, RF_y_pred),4))
precision.append(round(precision_score(y_test, RF_y_pred),4))
model_names = ['Random Forest']
result_df_RF = pd.DataFrame({'Accuracy':accuracy,'Recall':recall, 'Roc_Auc':roc_auc, 'Precision':precision, 'Cross_val_acc':cross_val_RF.mean(), 'Cross_val_f1':cross_val_RF_f1.mean()}, index=model_names)
result_df_RFUsing similar code to the above, we will generate a confusion matrix for the Random Forest Classifier model.
The results from Random Forest are similar, albeit, slightly worse than a simple Logistic Regression. The Random Forest model can be further tuned to improve performance and there are tools available to assist you with tuning. We will not spend further time on tuning the model and move on to XGBoost.
XGBoost
XGBoost is a powerful and popular gradient boosting algorithm used for classification and regression problems. It combines multiple weak predictive models to create a stronger, more accurate model. It is known for its speed, scalability, and ability to handle large, complex datasets. It also includes advanced features such as regularization, tree pruning, and early stopping to prevent overfitting. For the purpose of our example, we will not be using all the advanced features:
accuracy= []
recall =[]
roc_auc= []
precision = []
XGB = XGBClassifier(random_state=0,scale_pos_weight=spw)
result_XGB = XGB.fit(X_train, y_train)
XGB_y_pred = XGB.predict(X_test)
cross_val_XGB = cross_val_score(XGB, X_train, y_train, scoring = 'accuracy')
cross_val_RF_XGB = cross_val_score(XGB, X_train, y_train, scoring = 'f1')
accuracy.append(round(accuracy_score(y_test, XGB_y_pred),4))
recall.append(round(recall_score(y_test, XGB_y_pred),4))
roc_auc.append(round(roc_auc_score(y_test, XGB_y_pred),4))
precision.append(round(precision_score(y_test, XGB_y_pred),4))
model_names = ['XGBoost']
result_df_XGB = pd.DataFrame({'Accuracy':accuracy,'Recall':recall, 'Roc_Auc':roc_auc, 'Precision':precision, 'Cross_val_acc':cross_val_RF.mean(), 'Cross_val_f1':cross_val_RF_f1.mean()}, index=model_names)
result_df_XGBThe confusion matrix for the XGBoost and comparative results are below:
As anticipated XGBoost has better recall of the three models, however the overall accuracy is lower as we weighted the model to penalize negative predictions.
Neural Network (Keras on Tensor Flow)
Keras is a deep learning API, running on top of TensorFlow, a machine learning platform. It was developed with a focus on enabling fast performance.
Let us import the required libraries:
import tensorflow as tf
from tensorflow import keras
from imblearn.over_sampling import SMOTEWe will now correct for data imbalance using SMOTE:
X_train_sm = X_train.copy()
y_train_sm = y_train.copy()
print("Before OverSampling, counts of label '1': {}".format(sum(y_train == 1)))
print("Before OverSampling, counts of label '0': {} \n".format(sum(y_train == 0)))
sm = SMOTE(random_state = 2)
X_train_sm, y_train_sm = sm.fit_resample(X_train_sm, y_train_sm.ravel())
print('After OverSampling, the shape of X_train_sm: {}'.format(X_train_sm.shape))
print('After OverSampling, the shape of y_train_sm: {} \n'.format(y_train_sm.shape))
print("After OverSampling, counts of label '1': {}".format(sum(y_train_sm == 1)))
print("After OverSampling, counts of label '0': {}".format(sum(y_train_sm == 0)))You will see an output similar to below showing the additional sample created for the negative class.
Let us now set up our model. We will set up our model with 2 hidden layers and a sigmoid output for the output layer as we need a binary classification of 0 or 1 to predict churn.
We will also add a dropout rate of 20% to our model to avoid over-fitting.
model_tf = keras.Sequential([
keras.layers.Dense(16, input_shape=(X_train_sm.shape[1],), activation='relu'),
keras.layers.Dense(8, activation='relu'),
keras.layers.Dropout(0.2),
keras.layers.Dense(4, activation='relu'),
keras.layers.Dropout(0.2),
keras.layers.Dense(1, activation='sigmoid')
])Now that we have defined our model, let us now compile the model and run it. We will compile it with adam optimizer and default learning rate. For the loss function we will use binary_crossentropy and we will optimize our model for accuracy and recall.
model_tf.compile(optimizer = 'adam',
loss = 'binary_crossentropy',
metrics = ['accuracy', keras.metrics.Recall()])
model_tf.fit(X_train_sm, y_train_sm, epochs=100)As we ran for 100 Epochs, the model will optimize and run for 100 iterations. Once our model has finished running, we can now evaluate to check the performance of our model.
model_tf_eval = model_tf.evaluate(X_test, y_test)
Our model has an accuracy of 72.2% and recall of 75.8%. We will now create the result outputs and the confusion matrix for our model. As the output from our model returns values close to 0 and close to 1 but not exactly 0 or 1, we will incorporate an additional step to round the values to either 0 or 1.
accuracy= []
recall =[]
roc_auc= []
precision = []
tf_y_pred = model_tf.predict(X_test)
accuracy.append(round(model_tf_eval[1],4))
recall.append(round(model_tf_eval[2],4))tf_ypred_uns = []
for i in tf_y_pred:
if i>0.5:
tf_ypred_uns.append(1)
else:
tf_ypred_uns.append(0)roc_auc.append(round(roc_auc_score(y_test, tf_ypred_uns),4))
precision.append(round(precision_score(y_test, tf_ypred_uns),4))
model_names = ['tf']
result_df_tf = pd.DataFrame({'Accuracy':accuracy,'Recall':recall, 'Roc_Auc':roc_auc, 'Precision':precision, 'Cross_val_acc':"n/a", 'Cross_val_f1':"n/a"}, index=model_names)
result_df_tfLet us now create the confusion matrix and compare the results against other models.
cm = tf.math.confusion_matrix(labels=y_test, predictions=tf_ypred_uns)
plt.figure(figsize = (10,10))
sns.heatmap(cm, annot=True, fmt='d', cmap='viridis')
plt.xlabel('Predicted label')
plt.ylabel('True label')
Using TensorFlow, we get a higher recall but lower accuracy due to lower precision. The recall is significantly higher than what we achieved with XGBoost. As we want to identify all potential customer churn, we can probably afford a slightly lower precision in favor of higher recall. The model can be tuned further but we will not tune it further for the purpose of this article.
Conclusion
Predicting and preventing customer churn is critical for SaaS businesses to drive revenue growth and profitability. By understanding your data environment and utilizing machine learning models, you can identify at-risk customers cost effectively and take proactive measures to prevent churn.
You can access the full notebook (excl. TensorFlow) with code here or the notebook including TensorFlow here.
