Text Features for Churn Prediction: A Beginner’s Guide
How support tickets and reviews become sentiment scores and topic labels that a plain churn model can use, and how to tell if they helped.
A customer writes to support: “Third time asking. The export is still broken.” Three weeks later they cancel. Your churn model saw nothing coming, because all it looks at is logins, seat count and plan type. This post is about text features for churn prediction: how to turn messages like that one into numbers a simple model can use.
The words were sitting in your helpdesk the whole time. The model just had no way to read them.
So the obvious move is to add the text. But there is a trap, and I have walked into it myself. Adding text can make a model look much better on paper while it gets no better at all on real customers. By the end of this post you will know how to tell which of those two things happened.
What a feature is, and why a churn model only eats numbers
Some definitions first, one at a time.
Churn means a customer leaving: cancelling, not renewing, going quiet for good.
A churn model is a program that looks at a customer and estimates how likely they are to churn soon. The output is usually a probability, like 0.7, which means “about a 70% chance”.
A feature is one measurable fact about a customer that the model gets as input. “Logged in 4 times last month” is a feature. So is “on the annual plan” or “has 12 seats”.
Think of a feature as one column in a spreadsheet. Each row is a customer and each column is a fact about them. The model reads across the row and makes its guess.
Classic models like logistic regression or decision trees can only read numbers in those cells. They cannot read a sentence. “Third time asking” means nothing to them until it becomes something like 0.2 or 1.
Recap: a feature is one numeric column about a customer. Text has to be turned into columns like that before a classic model can use it.
Turning text features for churn prediction into numbers
There are many ways to do this. A practical guide on Towards Data Science about NLP for product data science walks through how text can strengthen churn, customer lifetime value and other traditional models. I like it because it treats language tools as feature makers for the models you already have, rather than as a replacement for them.
In my own projects, two kinds of text features have done most of the work. They are also the easiest to explain.
Sentiment score
A sentiment score is a number that says how positive or negative a piece of text is. A common scale runs from minus one (very negative) to plus one (very positive). Zero means neutral.
Some examples:
- “Love the new dashboard, thanks!” might score around 0.8
- “How do I add a user?” might score around 0
- “Third time asking. Still broken.” might score around minus 0.7
A customer can write many tickets, so you squash them into a few columns. For example:
- average sentiment over the last 90 days
- the lowest sentiment score in that window
- the change in sentiment, recent month compared to the month before
That last one is my favourite. In my experience a customer who goes from friendly to curt tells you more than a customer who has always been curt. Some people just write like that.
Topic labels
A topic label is a category that says what a message is about. Say you pick a short list: billing, bugs, missing feature, how to questions, praise.
Each ticket gets one label. Then you count per customer: number of billing tickets in the last 90 days, number of bug tickets, and so on. Each count becomes its own column.
You can assign labels with a simple keyword rule (“invoice” or “refund” means billing), with a small trained classifier, or by asking a language model to choose from your list. I built something very close to this for routing alerts as urgent, routine or noise. If you want the details on probabilities and confidence thresholds, I wrote about it here.
One warning if you use a language model for labelling: it may not give the same label twice for the same ticket, even at temperature 0. That is annoying for features, because your training data and your live data could be labelled slightly differently. I went deeper into why that happens in an earlier post. The fix that works for me is boring: label each ticket once, store the label, and never relabel it on the fly.
Recap: sentiment gives you “how do they feel”, topic labels give you “what are they talking about”. Both become a few plain numeric columns per customer.
Plugging the new columns into a simple churn model
Now your spreadsheet has the old columns (logins, seats, plan) plus some new ones (average sentiment, lowest sentiment, sentiment change, count of bug tickets, count of billing tickets).
You train the model the same way you did before. Nothing about the model changes. Only the inputs got richer.
Here is a made up example to build intuition. Two customers both logged in 10 times last month on the same plan. To the old model they look identical. With text features, one has average sentiment 0.5 and zero bug tickets. The other has sentiment minus 0.6 and four bug tickets. The model can now tell them apart.
Get András Kis’s stories in your inbox
Join Medium for free to get updates from this writer.
That is the whole promise: text adds information that usage numbers cannot see.
Recap: the model stays the same. You add a handful of text columns, and customers who looked identical can now look different.
Why more features can make the model worse
Text tempts you to go big. Why stop at five columns? You could add a column for every word in every ticket. That is a real technique, often called “bag of words”, and it can easily give you thousands of columns.
This is where overfitting shows up. Overfitting means the model memorises quirks of the data it learned from instead of learning patterns that hold for new customers.
An everyday comparison: a student who memorises the answers to last year’s exam scores 100% on last year’s exam. Give them a new exam and they struggle. They learned the paper, not the subject.
With thousands of word columns and only a few hundred churned customers, the model will find coincidences. Maybe three customers who churned all happened to write the word “Tuesday”. The model decides “Tuesday” predicts churn. It does not.
A rough rule of thumb I use (my own habit, not a law): the fewer churned customers you have, the fewer columns you should add. A handful of well chosen text features usually beats a giant pile of word counts when your data is small.
Recap: every new column is another chance for the model to memorise noise. Small, meaningful text features are safer than thousands of raw ones.
How to check whether the text actually helped
This is the question from the start: did the text really help, or does it just look that way?
You answer it with a fair test. Here are the steps.
Step 1: hold some customers back
Split your customers into two groups. The training set is the group the model learns from. The test set (also called the holdout set) is a group the model never sees while learning. Think of it as the new exam.
If your data covers time, split by time when you can: train on older customers, test on more recent ones. That is closer to how the model will be used for real.
Step 2: build two models
Build a baseline model with only the old columns. Then build a second model with the old columns plus the text features. Same algorithm, same split, same settings. The only difference is the text.
Step 3: score both on the test set
A common score for churn models is AUC. In plain words: pick one customer who churned and one who stayed, at random. AUC is the chance that the model gave the churner the higher risk score. An AUC of 0.5 is coin flipping. An AUC of 1.0 is perfect ranking.
The comparison is simple:
improvement = test AUC with text minus test AUC without text
If that number is clearly above zero, the text helped. If it is close to zero, the text added cost and complexity but no real signal.
Step 4: compare training and test scores
Look at both models’ scores on the training set too. If the text model scores much higher on training data but about the same on test data, you are looking at overfitting. The model memorised. It did not learn.
Step 5: check for leakage
This is the trap I mentioned at the start, and it has caught me more than once.
Leakage means a feature contains information that would not be available at the moment you need the prediction. With text, the classic case is the cancellation ticket itself. “Please cancel my account” is an incredibly strong churn feature. It is also useless, because by the time it arrives, the customer has already decided.
In one of my own early attempts, the text model looked brilliant. Then I noticed a big share of the “negative billing tickets” were refund requests sent after cancelling. Once I only counted tickets written before a cutoff date, most of the improvement disappeared. What was left was smaller but real.
So the rule is: for each customer, pick the date you would have made the prediction, and only use text written before that date.
Recap: hold customers back, build a model with and without text, compare test scores, watch the gap between training and test, and throw out any text written after the prediction date.
What to try today
This is how you know whether text helped your churn model. Do not go by how impressive the training score looks. Go by whether a fair test on unseen customers, using only text from before the prediction date, beats a model without text.
Sometimes it does, and then you have spotted unhappy customers your usage numbers could never see. Sometimes it does not, and that is useful too, because you just saved yourself a fragile pipeline.
If you later want richer text features than sentiment and topics, embeddings are the natural next step. They turn a whole sentence into a list of numbers that captures its meaning. I explained them from scratch in a separate beginner post on multimodal embeddings.
Your next step for today is small. Export 90 days of support tickets, pick five topic labels that make sense for your product, and count them per customer. Put those counts next to your existing churn data, train two models, one with them and one without, and compare the test AUC. That single comparison will teach you more than any amount of reading, including this post.
The full example notebook, with its code and results, is on GitHub. You can run it yourself in the browser: Open in Colab.
If you enjoyed this, you can read more of my writing on my Substack, and subscribe here so you don’t miss the next one.
