DataScience@HF

Do social determinants scores matter?

Last time we wrote about the power of using machine learning to infer member’s social needs and the ways of validating our approach. Here, I wanted to talk about the analytic value the nine different risk scores we built. In healthcare, an approximation for how sick someone is is if a patient goes back to the hospital following a discharge.The importance of this from both the care and policy landscapes can’t be overstated . So a useful heuristic we developed when building model features across analytics@hf is “does X predict readmissions?” I’m glad you asked, dear reader. It actually creates a

LITERATURE-BEATING READMISSIONS MODEL

We built a literature-beating model with just our social determinants personas…and we did it in 20 minutes using DataRobot. Here it is — it’s a glm-ensemble of xgBoosted trees with early stopping, some anomaly detection, random forest with gini importance, Vowpal stagewise polynomial feature classifier (but that doesn’t really matter). We ran our analysis in 20 minutes courtesy of DataRobot!

Pretty charts aside, the point is I don’t even need other features to create an awesome readmissions model. In fact, when I add in my 800ish column modeling asset (on which I should write a blog) the auc creeps up to 0.80, which is definitely worth the work but the effort vs reward should be kept in mind by the data science project manager.

What do our 9 risk scores tell us?

Probably two things. Let’s see the correlation matrix:

A trick to doing PCA by eye, and don’t tell your stats teachers, is to order your correlation matrix by the angle of the eigenvalues.

Using the standard orthogonal transform in R’s Psych package’s PCA function we tried to figure out what latent factors underlie these 9 scores.

What do we find? Well we see two real drivers of variance (93% of the variance is explained by two factors). You could make a case that we should force 3 factors given housing circumstance loads equally on both, but we don’t get much more variance explained doing that. So we call the latent factors ‘Social Situation’and ‘Work-Related.’ And they make logical sense. Going forward, we’ll use the component scores for parsimony.

Can we create a persona view for social risk?

Let’s finish this exploration with a customer-segmentation type analysis, only for the social scores. According to the gap statistic method we should expect 18 clusters in our data (of 9 columns and 1.4 million rows). So I ran k-means on the risk scores and wrote down names for the groups based on their risk scores aggregates. I also created a severity score so a business unit might know which cohort to prioritize, using the descriptions to connect to social services like SNAP , PATH or food pantries. We’re currently using these scores to help our care managers connect members with these types of resources via a social connection platform called NowPow.

This is the kind of thing we can share with decision makers and clinicians to help them prioritize which members need which type of social support. Most of our members are low risk on most of the scores. The ultimate goal is a clear view of risk levels, intervention scores and recommended programs to help members get connected with services that can help them be healthier, safer and more stable lives.

Wrapping up

So we learned two things You can get industry-leading prediction results by incorporating social risk scores into your health outcomes models. We have two or three latent factors in the description of social risk and we can use them to create personas for more targeted intervention.

One last thing to discuss…why doesn’t everyone just do this? I’ll write more about this in a future post but to create all this, I used Python, R, Tableau, Excel, DataRobot and most importantly, the snipping tool. There’s no perfect package, there’s no one kind of beautiful soup, just get the job done as compellingly as possible as quickly as possible. Two variables (speed and completeness of story) to trade on some economist’s marginal curve; your value as analyst is where on the curve you lie.

Our social determinants personas was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Social determinants of health — factors outside of a person’s direct health status such as employment, where they live, and education level — drive at least twenty percent of health outcomes. At Healthfirst, members facing challenges related to these determinants go to the hospital 30% more, stay up to 2 days longer in the hospital when they go, and have 33% higher healthcare costs. The problem is we lack the data to systematically measure them. If this data were readily available, we’d enable new methods of connecting people to social and environmental services to help treat, prevent and cure medical conditions, improve quality of life and, reduce overall medical costs.

As an insurance provider with 1.4 million New Yorkers (1 in 8!), our data science and population health strategy revolve around analyzing clinical data and predicting health behavior and outcomes. Could it be that we were missing something so fundamental to understanding our members? Yes, definitely. In a city where three in ten households are below the federal poverty line (FPL), it seems superficially wrong that only one percent of our members’ medical claims contain an indicator of financial stress. This is why it’s good to have priors before you do anything analytically.

The good news is that machine learning can help. There has been plenty of unsupervised work done to classify neighborhoods and create risk indices. But we have labeled data! True positive labels come from those 1% of claims with a so-called Z-Code and we are reasonably sure of our true zeros (probably 1–3 in ten will be mislabeled based on our priors). Statisticians will cringe, but deep learning is robust against label noise. One-sided label noise is a little different because estimating the real class-conditional distributions require stricter theoretical assumptions (we will have an upcoming blog about this!). Using everything we know about a member (demographics, neighborhood, disease burden, utilization, medications) we can reasonably predict which members may be struggling with these same social factors as in the labeled set. We built 10 models (one for each set of diagnosis codes) and scored our entire membership with it. Armed with this new knowledge, our population health team, care managers and provider partners can more readily reach out to members who might be struggling with one or more social factor and connect them with helpful resources like PATH for housing and SNAP for food insecurity.

These risk scores are valuable on their own and also provide a rich new feature-set for further analysis. The health insurance industry hasn’t figured out how to incorporate them into a risk model yet, but that doesn’t stop them for using them to impact clinical outcomes. The financial and housing stress risk score is the second most important variable in our Plan All Cause Readmissions model, after “have you been readmitted in the past?” Indeed, the first principal component of all 10 scores acts as a decent readmissions model itself: AUC of 0.68 and Brier score of 0.19. Our goal is to use them as a standard descriptive feature in any analysis, much like you would age or ethnicity.

Here’s a density map of our members with known social determinant status

And here we have all members classified by our models, a factor of 10 more!

VALIDATION AND IMPACT

As a provider-sponsored health plan, we are lucky enough to have the ultimate validation set of data: something that was nowhere near the data scientists during model training. We have survey data from sponsor hospitals that tries to capture members struggling with social burdens. They built the survey not knowing we were modeling and we built the models not knowing they were going to survey. The results: a c-statistic of 0.71 and Brier Score of 0.16, which means we are directionally correct (You can practically hear the chorus of data scientists singing, “All models are wrong, some are useful”).

Our understanding of the social determinants needs to inform every clinical interaction we have with our members. If we notice members are struggling to afford food (or even if we have a good guess), we can connect them to important services they might not know about.

As we progress in our vision to better understand the needs of our membership, we will improve provider recognition of these conditions and partner with more sponsors; both of which gives us better labels which yield better models. Let’s definitely do that. But also, we can use the ideas developed here to immediately impact our members. That’s why I show up to work.

Predicting the social determinants of health was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.