Who Will Leave Is Not the Same Question as Who Can Be Kept
What building a turnover model taught me about the gap between prediction and intervention
I’ve spent enough time with turnover risk scores that they don’t really look dramatic to me anymore. Someone lands in High Risk, someone else is Medium, a quarterly refresh moves a few people around, and after a while you get used to reading the probabilities for what they are: estimates based on patterns, not little countdown clocks above people’s heads. I’m probably more cautious about that distinction now than I was when I first started working on the model, mostly because the longer you live with predictive work, the easier it becomes to see how much people want the prediction to tell them. A risk score looks very decisive sitting in a spreadsheet. Put an 80 next to one employee and a 60 next to another, and the list practically begs to be sorted from largest to smallest.
That’s where I started getting stuck.
If one employee has an estimated 80% probability of voluntarily leaving and another is sitting at 60%, who should we worry about first? The obvious answer is the person at 80%. I don’t think that answer is wrong, exactly. I just started wondering what question it was answering. Say the person at 80% has already decided they’re moving across the country, changing industries, going back to school, or simply done with what the organization can realistically offer them. We could throw our very best retention effort at them and their likelihood of leaving may barely move. Meanwhile, the person at 60% could be sitting underneath a manager who suddenly has twice the team, hasn’t seen a meaningful development opportunity in a year, or has been doing work well above their role long enough that the novelty wore off several months ago. Change the right thing there and maybe 60 becomes 30.
I kept coming back to that scenario because it bothered the tidy logic of the risk list. The model could be completely right that the first person is more likely to leave and still tell me very little about which person we have a better chance of keeping. The ranking I had was about risk. The ranking I suddenly wanted was about something else.
The Question I Hadn’t Asked Yet
This wasn’t the first time the model had sent me down a road I wasn’t expecting. When I wrote Building a Turnover Model That Survives Contact With Reality, I was mostly preoccupied with getting predictive work to function inside an actual organization. The algorithm was one thing. The data, variable choices, politics, privacy, maintenance, stakeholder trust, and question of whether anyone could realistically carry the thing after it was built turned out to be most of the job. Somewhere in that article I acknowledged a problem I didn’t really know what to do with yet: historical validation could tell me whether the model recognized patterns that had preceded turnover, but it couldn’t tell me whether doing something because of the prediction would change what happened next.
Later, I wrote The Departures That Never Happened because I became slightly obsessed with the people the model flagged who never actually left. Technically, they’re false positives. You count them as errors, let them hurt the appropriate model-performance metric, and move on. But I couldn’t quite move on. Some are absolutely just wrong predictions. Still, I kept wondering whether some of those employees really had been moving toward an exit until something interrupted the pattern. A manager changed. A promotion finally came through. An internal move appeared. The team settled down. Maybe nothing happened that our systems knew how to record. I wanted to know what separated the high-risk people who stayed from the high-risk people who left, because throwing all of them into the “model was wrong” bucket felt like a very efficient way to discard potentially useful information.
The question bothering me now is related, but it’s not the same one. Finding out that something changed before an expected departure still doesn’t tell me whether that something caused the person to stay, and it definitely doesn’t tell me which employee would respond to that same intervention next time. That’s where my nice, straightforward turnover problem started getting considerably less straightforward.
Part of the trouble is that predictive models are perfectly happy using information that we have no business treating like a treatment plan. Tenure is an easy example. If people in a certain tenure window historically leave at higher rates, tenure can help the model recognize risk. Useful. Unfortunately, the retention application for “time keeps passing” remains limited. Division can be predictive too, but I’d prefer not to solve that with a mass organizational transfer. Manager span is more tempting because it feels actionable. If employees under managers with very large teams are leaving more often, it’s easy to start telling yourself a story about manager capacity, reduced coaching, less access, weaker connection, and eventual turnover. That story might even be true. It could also be that rapidly growing teams create both larger spans and more instability, or that certain kinds of work naturally operate with broader spans while competing in very different labor markets. The Random Forest doesn’t need to figure out which of those stories is right before manager span can help it predict. The rest of us do before we start changing anyone’s job.
I’ve started side-eyeing the word driver for the same reason. People Analytics uses it constantly: turnover drivers, engagement drivers, performance drivers. I use it, too. It’s convenient. But “driver” can quietly pick up more meaning than the analysis actually earned, and once something has been introduced as a turnover driver, somebody will understandably want to know what we’re doing about it. The statistical distinction underneath that discomfort is pretty important. A variable can help predict an outcome without being the thing that would change the outcome if we intervened on it. Prediction and causal questions are built for different purposes, even when they involve the same data and the same people.
The Other Ranking
I went looking to see whether I was simply overcomplicating a perfectly normal retention problem, and, reassuringly, other fields have already been making themselves miserable over this for years. Customer retention research turned out to be especially interesting. Eva Ascarza studied the very intuitive practice of predicting which customers were most likely to churn and then targeting those high-risk customers with retention efforts. It sounds almost too obvious to question. Her field experiments found that the highest-risk customers were not necessarily the people whose behavior changed the most because of the intervention. Targeting customers based on their sensitivity to the retention effort could work better than simply starting at the top of the churn-risk list.
I know. Employees are not customers. I have no desire to turn somebody’s career into the HR equivalent of a “we miss you, here’s 20% off” email. The analogy is useful because the analytical problem underneath it is so clean. One model is trying to tell you who is most likely to experience an outcome. The other thing you care about is whose outcome actually changes when you do something. I eventually found the word I had been circling: responsiveness.
Get Whitney Hosea’s stories in your inbox
Join Medium for free to get updates from this writer.
Once I had that word, my 80% and 60% employees looked different to me. I could imagine the person at 80% being almost completely unresponsive to anything the organization is prepared to offer, while the person at 60% has a problem we could genuinely influence. But then another annoying question shows up: how would I know? If someone receives an internal move and stays, I get to observe the version of their future where they received the move. I don’t also get the parallel version where they didn’t receive it. Maybe the move saved them. Maybe they were going to stay anyway. If someone gets promoted and leaves four months later, it’s easy to conclude the promotion didn’t work, but maybe they would have left four weeks later without it. We only get one timeline.
That missing timeline is where causal inference finally stopped feeling like an academic term I understood in theory and started feeling directly relevant to the work in front of me. The basic problem is almost unfairly simple: to know the effect of an intervention, what we really want is the difference between what happened because someone received it and what would have happened to that same person at that same moment without it. We never get both. So experiments and causal methods try, in different ways and under different assumptions, to create a credible comparison for the world we couldn’t observe. Susan Athey has written about this gap between prediction and decision-making, and the treatment-effect literature goes much further into estimating whether the same intervention works differently across people and circumstances.
That sent me into causal forests, which was inevitable once I learned they existed. My turnover model is built with a Random Forest, so naturally I was intrigued by a method that uses forest-based ideas to estimate differences in treatment effects rather than differences in outcome risk. Wager and Athey developed causal forests specifically to study heterogeneous treatment effects, essentially giving us a way to ask whether an intervention appears to work differently for different kinds of people. I’m still learning this area, and I’m glad I am, because it’s forcing me to separate several questions I used to let sit too close together. I also don’t think the practical lesson here is that every People Analytics team should immediately add causal forests to the roadmap and send everyone into a panic. Most organizations have a more basic problem first: we don’t do a particularly good job of remembering what we did.
We Remember the Outcome and Forget the Intervention
Think about how many retention-related actions happen inside a company without becoming usable analytical data. A manager has a real career conversation after months of vague check-ins. Someone gets an internal move. Responsibilities change. Compensation gets adjusted. A development opportunity opens up. A reporting relationship changes. A team gets additional support after losing several people. Sometimes those things happen because someone was considered a flight risk, and sometimes they happen as ordinary talent management. Either way, six months later leadership can ask whether the intervention “worked,” and the data usually knows exactly when the employee changed cost centers but has absolutely no recollection of the conversation that changed their mind about leaving.
I think that’s part of what I was reaching for in The Departures That Never Happened. I wanted to look backward at the high-risk employees who stayed and ask what changed around them. I still think that analysis is worth doing. But now I’d treat the result as the beginning of the investigation rather than the answer. If high-risk stayers received internal moves more often than high-risk leavers, that’s interesting. It gives me a hypothesis worth chasing. It doesn’t automatically mean the internal move caused the retention difference, because there are all sorts of reasons the people who received those opportunities may have been different before the intervention ever even happened.
What I’d want next is a much more deliberate record of retention actions themselves. Nothing wildly futuristic. What happened? When? Why? Who was eligible? Who actually received it? What did their situation look like beforehand? What happened later? Depending on the intervention, eventually there may be opportunities for randomized pilots, staggered rollouts, natural experiments, or carefully designed comparison groups. Some questions may never support a strong causal claim, and I’d rather know that than torture the available data until it gives leadership the answer everyone hoped for.
If we got good at that over time, the retention conversation could become much more useful. Instead of only saying, “Here are the employees with elevated turnover risk,” we might start learning that a certain kind of career intervention seems to matter most during a particular tenure period, or that additional manager support is especially useful after clustered team departures. We might find that something everyone assumes is a great retention lever does almost nothing for retention but improves mobility or performance enough that it’s still valuable. We may even discover that some interventions work beautifully for exactly the employees a traditional risk ranking would have placed further down the list.
Who Can Be Kept Isn’t the Same as Who Should Be
And then, because organizations never allow a problem to remain conveniently one-dimensional, we’d still have to decide whether the intervention was worth doing.
Keeping someone has a cost, even when the intervention doesn’t involve money. A compensation adjustment affects internal equity. A promotion uses a role or budget that could have gone somewhere else. A manager spending substantial time on one retention case is not spending that time somewhere else. A role redesign might solve one employee’s problem while creating workload for another team. Even if we became very good at estimating who could be retained, “can” would never automatically become “should.”
That matters to me for another reason, too. Employees have agency, and I don’t want better analytics to slowly turn retention into a game where every departure is treated as a preventable defect. Someone may genuinely be ready to go. They may want something the organization cannot give them, or they may simply have found an opportunity that makes more sense for their life. There are also interventions an employer shouldn’t pursue just because a model suggests they might change someone’s behavior. I want predictive analytics to help organizations notice problems sooner and make better decisions about the conditions people work in. I do not want us becoming extraordinarily efficient at convincing individuals to remain inside conditions we should have fixed instead.
That last distinction may be the one I care about most. If elevated risk keeps clustering around the same manager setup, lack of mobility, workload pattern, or team instability, the useful intervention may have very little to do with persuading the individual employee to stay. At some point, repeatedly “saving” people one at a time starts looking like an expensive workaround for an organizational problem we already have enough evidence to see.
What I Want the Model to Do Now
None of this has made me less interested in turnover prediction. I actually appreciate the model more now that I’m clearer about the job I want it to do. It gives us lead time. It can show us where patterns are concentrating while employees are still there, before everything becomes another retrospective turnover slide explaining what happened last quarter. It narrows the search and gives us places worth looking more closely.
I’ve just stopped expecting the risk score to finish the thought for me.
The model can tell me who looks more like the employees who left before. What I want to understand next is where our actions actually change that trajectory, where they don’t, and what separates the two. I don’t think that makes the original prediction less useful. However, I do think it means the question just gets more interesting once the model has answered it.
