<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alon Hazan on Medium]]></title>
        <description><![CDATA[Stories by Alon Hazan on Medium]]></description>
        <link>https://medium.com/@alonhzn?source=rss-c2b3b4511cc3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*M6RcRHSJLW-vrbPFGj6cdQ.png</url>
            <title>Stories by Alon Hazan on Medium</title>
            <link>https://medium.com/@alonhzn?source=rss-c2b3b4511cc3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:15:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alonhzn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The fallacy of an ROC curve on a Test set]]></title>
            <link>https://alonhzn.medium.com/the-fallacy-of-an-roc-curve-on-a-test-set-c96d8fdf774?source=rss-c2b3b4511cc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c96d8fdf774</guid>
            <category><![CDATA[auc]]></category>
            <category><![CDATA[roc]]></category>
            <category><![CDATA[mls]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Alon Hazan]]></dc:creator>
            <pubDate>Sat, 08 May 2021 23:23:59 GMT</pubDate>
            <atom:updated>2026-08-04T16:50:49.144Z</atom:updated>
            <content:encoded><![CDATA[<p>Update: This concept was published on April 2024: <a href="https://www.nature.com/articles/s42256-024-00817-7">The curious case of the test set AUROC | Nature Machine Intelligence</a></p><p>TL;DR<br>I’m going to explain why ROC is not a fair performance metric and why calculating the AUC on your test set will create <strong>false expectations</strong> from your model.</p><p><strong>Introduction</strong><br>A Receiver operating characteristic (ROC) curve, despite it’s military origin, is the go to performance assessment tool for binary classification tasks in general and AI in particular (eg. cancer versus no cancer).<br>The area under curve (AUC) is arguably the most common performance metric that reduces the entire model to one scalar score, neatly described as a number between 0 and 1.<br>In the last decade I’ve been working on machine learning and AI in the medical domain and I see over and over again how very experienced scientists misunderstand the meaning of AUC on a test set and perceive it as a fair metric of performance, while in reality, it is false.</p><p><strong>ROC is not a fair metric</strong><br>Here, I don’t mean “metric” in the <a href="https://en.wikipedia.org/wiki/Metric_(mathematics)">mathematical</a> sense. I mean metric as a tool to estimate how a model will perform in real life. A fair metric will compare the prediction of a model against the ground truth and provide a score without using the ground truth to maximize the score. But that is exactly what ROC does! More on that later.</p><p><strong>Why do we have a test set?</strong><br>In real world application where the performance of the model is critical (such as the medical domain, self driving, etc.) it is important to estimate how our model is going to perform in the wild. That is the reason why it is common to split our entire data set into three parts:</p><ul><li><em>Train set</em>- this data will be used to fit the parameters of the model (e.g. backpropagation)</li><li><em>Validation set</em>- this data will be used to fit the hyper-parameters of the model (e.g. learning rate, learning epochs, model architecture, etc.)</li><li><em>Test set</em>- this data will be used to estimate the future performance of the model in production (i.e. “the wild” or “in real life”)</li></ul><p>When data scientists are asked why do we even need a test set, many will underestimate the importance of it and say that it is just to verify the performance on a new data set. While this is not entirely wrong, it misses one of the most important reasons to have a test set, which is to mitigate for <strong>hyper-parameter overfitting</strong>! You probably know all about overfitting, that is where the model starts to learn the specific train samples rather than the underlying distribution and then the performance on the validation drops. What many data scientists don’t realize , is that there is another kind of overfitting that is caused by repeatedly changing hyperparameters such as learning rate, architecture, optimizers, activations, augmentations and choosing the stopping criteria (such as epoch in AI) based on the validation set. Essentially overfitting these parameters to the validation set.<br>Many people ask why AIs trained to make medical decisions, that perform as good as human experts are not in clinical use? One good answer is that the data set used to train is not necessarily the of the same distribution as real world data in a different hospital (says <a href="https://spectrum.ieee.org/view-from-the-valley/artificial-intelligence/machine-learning/andrew-ng-xrays-the-ai-hype">Andrew Ng</a>). However, even if the data would have been the same, the model will not work in a real clinical setting as expected because the evaluation process that is commonly used today (i.e. AUC) is misleading and creating false expectations.</p><p><strong>It’s all about the threshold!<br></strong>Let’s assume that we trained our model to classify a chest x-ray to determine if a person is Healthy or Sick. The raw output of the model is a number between 0 and 1. It is not yet a clinical decision. To decide, we as the model designers need to choose a specific threshold and then all samples predicted above the threshold are predicted as Sick and all the others are predicted as Healthy. This threshold needs to be chosen before the model starts being used in a clinical setting obviously. The correct way to choose this operation threshold will be on an ROC curve calculated on the Validation set using some <a href="https://en.wikipedia.org/wiki/Sensitivity_and_specificity">Sensitivity and Specificity</a> criteria. If you could fly into the future, after your model has been deployed in a hospital and predicted 1000 patients, you would be able to run the ROC on those patients, and select a better threshold! but you can’t fly into the future and get back to the present. Choosing the operation threshold on the Test set ROC is equivalent to travelling in time, only that you are fooling yourself.</p><p><strong>Proof!<br></strong>Enough talk, let’s see some proof. Let’s assume that we trained a model and evaluated the ROC on the Validation set. Since the Test set is generally a completely different set of samples from the Validation set, we will get a different set of predictions on it. Ideally the distribution of the predictions on both sets will be the same and therefore the operation threshold that generated a certain Sensitivity and Specificity on the Validation set, would generate the same for the Test set. However, in reality there will be a difference. For convenience of this proof, let’s simulate the difference in predictions on the Test set as some function of the Validation predictions (of course in reality it would be random, but that doesn’t change the point). More specifically I will choose this function as <a href="https://mathworld.wolfram.com/StrictlyIncreasingFunction.html">Strictly Increasing</a> simply because it has the nice quality of <strong>not changing the ROC</strong>. Take a moment to think about it — even though the predictions on the Test set can be entirely different than the predictions on the Validation set, they will have the exact same AUC. But an operation threshold chosen on the Validation set (i.e. not seeing into the future) will give different Sensitivity and Specificity entirely.</p><p>Let’s simulate the change in distribution of the Test set as y = log(1+x) where x is a prediction on the Validation set and y is a prediction on the Test set (again, it could really be anything…)</p><figure><img alt="The fallacy of ROC on a Test set" src="https://cdn-images-1.medium.com/max/728/1*fAt5eJkpajxdql9H9zqxGw.png" /></figure><p>On the top right we see the box plot of the predictions on our Validation set. That set should help us choose the operation threshold such that we meet some Sensitivity and Specificity criteria. On the top left we see the box plot of our Test set predictions. If at that point we calculate the ROC curve on both Validation and Test sets, we seemingly would get the exact same ROC curve, hiding the fact that the distribution had changed!</p><p>On the bottom graph we can see a true depiction of the drift from Validation set to Test set. In a particular example, assume that we chose the operation point of our model to provide 90% Sensitivity and 76.2% Specificity using the Validation set. A year later, after this model was used in a clinical setting, we can estimate that it would get 60.6% Sensitivity and 96.5% Specificity. Which is very different than the operation point we aimed for. Using this drift calculation is the only fair estimation that we can give as to how much performance might drift in a future real life production environment (aka the wild).</p><p>Script used to create the graphs. Try to play with the function simulating the Test set predications ;-)</p><pre>import matplotlib.pyplot as plt<br>import pylab as p<br>from sklearn.metrics import roc_curve, auc<br>import numpy as np<br><br>np.random.seed(2021)<br><br>N = 1000<br>std = 0.3<br><br>truth = np.concatenate([np.zeros(N), np.ones(N)])<br>labels = [<strong>&#39;Negative class&#39;</strong>, <strong>&#39;Positive class&#39;</strong>]<br><br><em># Simulate some possible model that outputs a score from 0 to 1<br></em>val_prediction = np.concatenate([np.random.normal(0.2, std, N), np.random.normal(0.8, std, N)])<br>val_prediction -= np.min(val_prediction)<br>val_prediction /= np.max(val_prediction)<br><br>test_prediction = np.log(val_prediction + 1)  <em># Assume some different prediction on the test set<br># test_prediction = val_prediction / 1.1  # Assume some different prediction on the test set<br># test_prediction = val_prediction - 0.03  # Assume some different prediction on the test set<br><br></em>val_dist_ax = plt.subplot(3, 2, 1)<br>test_dist_ax = plt.subplot(3, 2, 2)<br>val_roc_ax = plt.subplot(3, 2, 3)<br>test_roc_ax = plt.subplot(3, 2, 4)<br><br><br>val_dist_ax.boxplot([val_prediction[truth == 0], val_prediction[truth == 1]], labels=labels, showfliers=False)<br>test_dist_ax.boxplot([test_prediction[truth == 0], test_prediction[truth == 1]], labels=labels, showfliers=False)<br>test_dist_ax.set_ylim([0, 1])<br>val_dist_ax.set_ylim([0, 1])<br>val_dist_ax.set_title(<strong>&#39;AI prediction on Validation set&#39;</strong>)<br>test_dist_ax.set_title(<strong>&#39;AI prediction on Test set&#39;</strong>)<br>val_dist_ax.set_ylabel(<strong>&#39;AI score&#39;</strong>)<br>test_dist_ax.set_ylabel(<strong>&#39;AI score&#39;</strong>)<br>val_dist_ax.grid(<strong>&#39;on&#39;</strong>)<br>test_dist_ax.grid(<strong>&#39;on&#39;</strong>)<br><br>val_set_fpr, val_set_tpr, validation_thresholds = roc_curve(truth, val_prediction)<br>rocauc = auc(val_set_fpr, val_set_tpr)<br>val_roc_ax.plot(val_set_fpr, val_set_tpr)<br>d_from_90 = np.abs(val_set_tpr-0.9)<br>sens_90_ind = np.where(d_from_90 == np.amin(d_from_90))[0][0]<br>validation_sens_90_th = validation_thresholds[sens_90_ind]<br>fpr_at_90sens=100*val_set_fpr[sens_90_ind]<br>val_roc_ax.set_title(<strong>f&#39;Validation ROC AUC = </strong>{round(rocauc, 3)}<strong>&#39;</strong>)<br>val_roc_ax.annotate(<strong>f&#39;90% sensitivity</strong>\n<strong>&#39;<br>                    f&#39;</strong>{round(100-fpr_at_90sens, 1)}<strong>% Specificity</strong>\n<strong>&#39;<br>                    f&#39;Threshold=</strong>{round(validation_sens_90_th,2)}<strong>&#39;</strong>,<br>                    xytext=(0.5, 0.5),<br>                    xy=(val_set_fpr[sens_90_ind],0.9),<br>                    arrowprops=dict(facecolor=<strong>&#39;black&#39;</strong>, shrink=0.01, width=1))<br>val_roc_ax.set_ylabel(<strong>&#39;Sensitivity&#39;</strong>)<br>val_roc_ax.set_xlabel(<strong>&#39;1-Specificity&#39;</strong>)<br><br>test_set_fpr, test_set_tpr, test_thresholds = roc_curve(truth, test_prediction)<br>rocauc = auc(test_set_fpr, test_set_tpr)<br>test_roc_ax.plot(test_set_fpr, test_set_tpr)<br>test_roc_ax.set_title(<strong>f&#39;&quot;False&quot; Test ROC AUC = </strong>{round(rocauc, 3)}<strong>&#39;</strong>)<br>d_from_90 = np.abs(test_set_tpr-0.9)<br>sens_90_ind = np.where(d_from_90 == np.amin(d_from_90))[0][0]<br>test_sens_90_th = test_thresholds[sens_90_ind]<br>fpr_at_90sens=100*test_set_fpr[sens_90_ind]<br>test_roc_ax.annotate(<strong>f&#39;90% sensitivity</strong>\n<strong>&#39;<br>                    f&#39;</strong>{round(100-fpr_at_90sens, 1)}<strong>% Specificity</strong>\n<strong>&#39;<br>                    f&#39;Threshold=</strong>{round(test_sens_90_th,2)}<strong>&#39;</strong>,<br>                    xytext=(0.5, 0.5),<br>                    xy=(test_set_fpr[sens_90_ind],0.9),<br>                    arrowprops=dict(facecolor=<strong>&#39;black&#39;</strong>, shrink=0.01, width=1))<br>test_roc_ax.set_ylabel(<strong>&#39;Sensitivity&#39;</strong>)<br>test_roc_ax.set_xlabel(<strong>&#39;1-Specificity&#39;</strong>)<br><br>def calc_tpr_fpr(pred, ground_truth, th):<br>    tp = np.sum(pred[ground_truth == 1] &gt;= th)<br>    tn = np.sum(pred[ground_truth == 0] &lt; th)<br>    fp = np.sum(pred[ground_truth == 0] &gt;= th)<br>    fn = np.sum(pred[ground_truth == 1] &lt; th)<br>    fpr = fp / (fp + tn)<br>    tpr = tp / (tp + fn)<br>    return fpr, tpr<br><br><br>def roc_drift(ax, ground_truth, test_set_prediction, val_set_thresholds, val_set_fpr, val_set_tpr):<br>    i=0<br>    for th, vfpr, vtpr in zip(val_set_thresholds,val_set_fpr, val_set_tpr):<br>        if i % 40 == 0 or i&gt;len(val_set_thresholds)-3:  <em># just for visualization, prune points to make the plot legible<br>            </em>fpr, tpr = calc_tpr_fpr(test_set_prediction, ground_truth, th)<br>            lin = ax.plot([vfpr, fpr],[vtpr,tpr])<br>            ax.plot(vfpr, vtpr, marker=<strong>&#39;.&#39;</strong>, color=lin[0].get_color())<br>            ax.plot(fpr, tpr, marker=<strong>&#39;^&#39;</strong>, color=lin[0].get_color())<br>        i += 1<br><br>test_true_roc_ax=plt.subplot(3, 1, 3)<br>roc_drift(test_true_roc_ax, truth, test_prediction, validation_thresholds, val_set_fpr, val_set_tpr)<br>test_true_roc_ax.grid(<strong>&#39;on&#39;</strong>)<br>test_true_roc_ax.set_title(<strong>&#39;True ROC drift from Validation to Test&#39;</strong>)<br>fpr, tpr = calc_tpr_fpr(test_prediction, truth, validation_sens_90_th)<br>msg = <strong>f&#39;&#39;&#39;90% Validation sensitivity<br></strong>{round(100-fpr_at_90sens, 1)}<strong>% Validation Specificity<br>Drift to:<br></strong>{round(tpr*100, 1)}<strong>% Test sensitivity<br></strong>{round(100-100*fpr, 1)}<strong>% Test Specificity&#39;&#39;&#39;<br></strong>test_true_roc_ax.annotate(msg,<br>                    xytext=(0.5, 0.5),<br>                    ha=<strong>&#39;center&#39;</strong>, va=<strong>&#39;center&#39;</strong>,<br>                    xy=(val_set_fpr[sens_90_ind],0.9),<br>                    arrowprops=dict(facecolor=<strong>&#39;black&#39;</strong>, shrink=0.01, width=1))<br>test_true_roc_ax.annotate(msg,<br>                    xytext=(0.5, 0.5),<br>                    ha=<strong>&#39;center&#39;</strong>, va=<strong>&#39;center&#39;</strong>,<br>                    xy=(fpr,tpr),<br>                    arrowprops=dict(facecolor=<strong>&#39;black&#39;</strong>, shrink=0.01, width=1))<br>test_true_roc_ax.plot([val_set_fpr[sens_90_ind], fpr],[0.9,tpr],color=<strong>&#39;red&#39;</strong>)<br>test_true_roc_ax.plot(val_set_fpr[sens_90_ind], 0.9, marker=<strong>&#39;.&#39;</strong>, color=<strong>&#39;red&#39;</strong>)<br>test_true_roc_ax.plot(fpr, tpr, marker=<strong>&#39;^&#39;</strong>, color=<strong>&#39;red&#39;</strong>)<br>test_true_roc_ax.set_ylabel(<strong>&#39;Sensitivity&#39;</strong>)<br>test_true_roc_ax.set_xlabel(<strong>&#39;1-Specificity&#39;</strong>)<br><br>plt.show()</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c96d8fdf774" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>