Dissecting racial bias in an algorithm used to manage the health of populations — a structured, explained summary
Paper: Obermeyer et al., Science (2019)
TL;DR
This paper examines a widely used health-care risk algorithm and shows that it systematically underestimated the needs of Black patients. The key point is that the problem did not come from the algorithm explicitly using race. Instead, it came from the target it was trained to predict: future health-care cost, rather than future health need. Because Black patients with the same illness burden often generate lower medical spending than White patients, a model trained on cost can look accurate on its own target while still producing unequal and clinically misleading risk scores. At the same predicted risk level, Black patients were sicker, had more uncontrolled disease and were less likely to be identified for additional care-management support. The paper became so influential because it gave a concrete mechanism for algorithmic bias in medicine: a model can inherit structural inequities through the choice of label, even when race is excluded from the inputs.
If you prefer a talk version, Ziad Obermeyer also gave a video presentation on this work, which walks through the same main argument in lecture form.
Why this paper exists
Health systems use commercial prediction algorithms to decide which patients should receive extra resources such as care coordination, closer follow-up and other high-risk management services. These programs are expensive, so they are usually targeted only to patients judged most likely to benefit. In practice, this means an algorithmic score can directly influence who receives additional help and who does not. The authors studied one of the most prominent examples of this class of systems, which the paper describes as representative of an industry-wide approach affecting roughly 200 million people in the United States each year.
The paper also addresses a broader problem in algorithmic fairness research. It is often difficult to understand why real deployed systems are biased because the models are proprietary and outside researchers usually cannot inspect their targets, predictions and internal design. In this case, the authors had access not only to the outputs of the algorithm but also to enough information to understand what it had been trained to do. That made it possible to move beyond simply documenting disparity and instead identify the mechanism that produced it.
What the algorithm was doing in practice
The empirical analysis used primary care patients enrolled in risk-based contracts at a large academic hospital between 2013 and 2015. The main sample included 6,079 patients who self-identified as Black and 43,539 who self-identified as White, observed over 11,929 and 88,080 patient-years respectively. In the system studied, patients above the 97th percentile of risk were automatically identified for the care-management program, while those above the 55th percentile were referred to their physicians for possible enrollment review.
That operational detail matters because the score was not being used for a theoretical benchmark exercise. It was part of a resource-allocation workflow. The relevant question was therefore not only whether the algorithm predicted its target well, but whether patients at the same predicted risk level were similarly sick and similarly deserving of extra support.
Where the bias appeared
The central empirical finding is that Black patients were substantially sicker than White patients at the same algorithm-assigned risk score. Using the number of active chronic conditions as a summary measure of illness burden, the authors showed that at the 97th percentile of risk — the threshold for automatic enrollment — Black patients had 4.8 active chronic conditions on average compared with 3.8 among White patients, a 26.3% higher burden of illness.
The paper then shows why this difference mattered operationally. If the disparity were removed, the fraction of Black patients receiving additional help at that high-risk threshold would increase from 17.7% to 46.5%. That is not a small calibration issue at the margins. It means the algorithm was materially changing who got access to extra care resources.
Get Alex Yorov’s stories in your inbox
Join Medium for free to get updates from this writer.
The authors also looked beyond a comorbidity count and examined biomarkers that reflected disease severity. At the same risk score, Black patients showed worse values across several important indicators, including hypertension severity, diabetes severity, renal failure, cholesterol and anemia. The paper therefore argues that the disparity was not just an artifact of coding or one narrow outcome definition. It reflected a broader mismatch between predicted risk and actual health status.
Why predicting cost created the problem
The main explanation offered by the paper is straightforward but important. The algorithm was trained to predict future medical spending, not future illness burden. From the perspective of the developer, that choice was not irrational. Cost is easy to measure, strongly related to utilization and often treated as a convenient proxy for how much care a patient is likely to need. But in a health system where Black patients receive less care, face barriers to access or generate lower spending despite similar levels of illness, cost becomes a distorted proxy for health need.
That is exactly what the authors found. Conditional on health, Black patients generated lower costs than White patients. So a model that accurately predicts future spending can still systematically understate the needs of Black patients. The paper’s point is not that the model was poorly optimized. It is that the optimization target itself encoded an important structural inequity. In that setting, good prediction of cost does not imply fair or clinically appropriate prediction of need.
This is also why the paper became more influential than a standard audit showing unequal performance across groups. It shifted attention from whether race was included as a feature to whether the label was appropriate. The algorithm did not need to use race explicitly to produce racial bias. The bias entered through the target variable because cost was standing in for something different from what the program actually cared about.
What happens when the label changes
One of the strongest parts of the paper is that the authors did not stop at critique. They retrained alternative predictors using different labels, including total cost, avoidable cost and active chronic conditions. The goal was to see whether changing the target alone would change who was classified as highest risk.
The answer was yes, and by a large margin. Among patients placed in the highest-risk group, the fraction of Black patients was 14.1% for the cost-based predictor and 26.7% for the chronic-conditions predictor, with the avoidable-cost model in between. That result is important because it shows that the disparity was not simply a fixed property of the data. It depended strongly on what the model was asked to predict.
The paper also reports that the manufacturer later replicated the problem in a dataset of 3,695,943 commercially insured patients and that a relabeling strategy combining health prediction with cost prediction reduced one measure of bias by 84%. That does not mean the fairness problem is universally solved, but it does show that the paper identified a modifiable design choice rather than an abstract moral concern with no practical fix.
What this paper contributed
The paper’s main contribution was to show that algorithmic bias in medicine can arise from the selection of a convenient proxy outcome rather than from explicit use of race. That insight travels well beyond this particular case. Many real-world health algorithms are trained on outcomes such as spending, utilization, admission or readmission because those variables are easy to measure and operationally attractive. But if those outcomes are themselves shaped by unequal access, unequal treatment or other structural factors, then the resulting models may reproduce those distortions even when they appear accurate by conventional predictive standards.
The paper therefore changed the way many people think about fairness in clinical prediction. It is not enough to ask whether a protected attribute was included, whether calibration looks acceptable or whether performance is high on the target variable. One also has to ask whether the target reflects the clinical construct the health system actually cares about. In this case, the relevant construct was health need, while the training label was cost. The gap between those two is where the bias emerged.
Limitations and scope
The analysis is focused on one major health system and compares self-identified Black and White patients, so it does not provide a complete fairness analysis across all racial and ethnic groups or across intersecting dimensions such as socioeconomic status, geography or sex. The paper is also about one specific use case — identifying patients for care-management programs — rather than every possible medical AI application.
Even with those limits, the paper’s broader lesson remains strong because the mechanism is so clear and because the manufacturer independently replicated the problem at much larger scale. The paper should therefore be read not as proof that every health-care algorithm fails in this exact way, but as a warning that proxy labels can quietly carry structural inequities into otherwise competent prediction systems.
If you keep one idea from this paper
The most useful way to remember this paper is not as a case where a model explicitly used race and therefore became biased. The more important lesson is that a model can exclude race, perform well on its stated target and still produce serious racial disparities if the target itself is a distorted proxy for the clinical quantity of interest. In this case, predicting health-care spending was not the same as predicting who was sick and needed more help. That distinction is the real conceptual contribution of the paper, and it is why the paper still matters in discussions of medical AI evaluation and fairness.
