ROC-AUC vs Precision-Recall AUC: Which One Should You Use and When?
ROC-AUC vs Precision-Recall AUC: Which One Should You Use and When?
A practical guide for data scientists working on real-world classification problems
The Setup
You’ve just finished training your classification model. You run your evaluation and two numbers stare back at you — ROC-AUC and Precision-Recall AUC. Both claim to measure model performance. Both are between 0 and 1. Both go up when your model improves.
So which one do you trust? Which one do you show your stakeholders? And most importantly — which one should actually drive your model selection and threshold decisions?
The answer depends entirely on one thing: the distribution of your target variable.
Let me walk you through both metrics from first principles, show you exactly when each one lies to you, and give you a clear decision framework tied to real industry scenarios.
What is ROC-AUC?
The ROC curve plots two things as you vary your classification threshold from 1.0 down to 0:
Y axis — True Positive Rate (TPR/Recall): TPR = TP / (TP + FN) Out of all actual positives, what fraction did I correctly catch?
X axis — False Positive Rate (FPR): FPR = FP / (FP + TN) Out of all actual negatives, what fraction did I incorrectly flag as positive?
As you lower the threshold you catch more positives — TPR goes up — but you also flag more negatives incorrectly — FPR goes up. The curve moves from bottom left (strict threshold, few flags) to top right (loose threshold, flag everything).
AUC — the Area Under this Curve — summarizes the model in one number. The beautiful probabilistic interpretation: AUC-ROC is the probability that your model ranks a randomly chosen positive case higher than a randomly chosen negative case.
A diagonal line from (0,0) to (1,1) = AUC of 0.5 = random guessing. A curve hugging the top left corner = AUC close to 1.0 = near perfect separation.
What is Precision-Recall AUC?
The Precision-Recall curve plots:
Y axis — Precision: Precision = TP / (TP + FP) Out of everything I flagged as positive, what fraction was actually positive?
X axis — Recall: Recall = TP / (TP + FN) Out of all actual positives, what fraction did I catch?
As you lower the threshold recall increases — you catch more positives — but precision typically decreases — you also flag more negatives incorrectly. The tradeoff is direct and visible.
AUC-PR summarizes this curve in one number. A random classifier on an imbalanced dataset achieves AUC-PR equal to the prevalence of the positive class — say 1% for fraud. A perfect classifier achieves 1.0.
The Core Difference — And Why It Matters
Here is the fundamental insight that most people miss:
ROC-AUC uses FPR in its x axis. FPR has TN in its denominator.
When your dataset is severely imbalanced — say 1% positive, 99% negative — you have an enormous number of true negatives. This makes FPR look artificially small even when you’re making a huge number of false positive errors in absolute terms.
Let me show you exactly why this is dangerous with a concrete example.
The Numbers That Expose Everything
Say you have 100,000 transactions. 1,000 are fraud (1%). 99,000 are legitimate (99%).
Your model at threshold 0.5 produces:
TP = 800 (caught 800 actual fraud cases) FN = 200 (missed 200 actual fraud cases) FP = 9,000 (wrongly flagged 9,000 legitimate transactions) TN = 90,000 (correctly cleared 90,000 legitimate transactions)
Now calculate both metrics:
TPR = 800 / (800 + 200) = 80% — looks great FPR = 9,000 / (9,000 + 90,000) = 10% — looks acceptable
ROC point: (10%, 80%) — model looks strong on the ROC curve
But now look at Precision-Recall:
Recall = 800 / (800 + 200) = 80% — same as TPR Precision = 800 / (800 + 9,000) = 8.2% — catastrophically bad
You are wrongly flagging 9,000 legitimate customers for every 800 actual fraud cases you catch. That’s more than 11 false alarms for every correct fraud detection.
The ROC curve hid this completely. The Precision-Recall curve exposed it immediately.
This is the key insight: FPR is diluted by the massive TN pool in imbalanced data. Precision is not — it only looks at what you actually flagged.
Real Industry Scenarios — When to Use Which
Scenario 1 — Payments Fraud Detection
A payments platform processing millions of international currency conversions needs to flag suspicious transactions. Fraud rate is 0.1% — 1 in every 1,000 transactions.
Use: Precision-Recall AUC
Why: With 99.9% legitimate transactions, ROC-AUC will look impressive even for a mediocre model. The massive TN pool makes FPR look tiny. But the real business question is — out of every transaction I block, how many are genuinely fraudulent? Wrongly blocking a legitimate $50,000 international transfer destroys customer trust and may violate service agreements. Precision-Recall directly shows this cost.
Threshold decision: Set threshold where precision stays above say 85% — meaning at least 85 out of every 100 flagged transactions are genuine fraud — to protect legitimate customers while still catching the majority of fraud cases.
Scenario 2 — Ride-sharing Identity Verification
A ride-sharing platform needs to verify whether a new signup is a genuine user or a bot/fraudulent account attempting to abuse promotions. Fraud rate is approximately 5% of signups.
Get Manojkumar Rayana’s stories in your inbox
Join Medium for free to get updates from this writer.
Use: Both — start with ROC-AUC, validate with Precision-Recall
Why: At 5% fraud rate the class imbalance is moderate — not severe enough to completely invalidate ROC-AUC, but significant enough that you should validate with Precision-Recall before making final decisions. ROC-AUC gives you a good overall picture of model discrimination ability. Precision-Recall tells you the real cost of your threshold choice on user experience — wrongly blocking genuine new users means lost lifetime value.
Threshold decision: Balance based on LTV calculation. If average rider LTV is $500 and average fraud loss per account is $50, wrongly blocking 10 genuine users to catch 1 fraudster is a net loss. Set threshold accordingly.
Scenario 3 — Credit Card Transaction Fraud
A major credit card network processes billions of transactions. Fraud rate is approximately 0.05% — 5 in every 10,000 transactions.
Use: Precision-Recall AUC exclusively
Why: Extreme class imbalance makes ROC-AUC essentially useless as a decision tool. A model that flags nothing achieves ROC-AUC of 0.5 but Precision-Recall AUC of near zero — which correctly signals the model is worthless. Additionally the business cost asymmetry is complex — declining a legitimate transaction at a point of sale is immediately visible and embarrassing to the cardholder, while missed fraud often goes undetected for days. Precision-Recall directly quantifies the false alarm rate that drives customer complaints.
Threshold decision: Different thresholds for different transaction risk tiers. High-value transactions above $5,000 get a stricter threshold — higher precision required. Low-value everyday purchases get a looser threshold — higher recall acceptable since the cost of a wrong block (embarrassment at checkout) is higher relative to the fraud loss amount.
Scenario 4 — Seller Fraud on an E-commerce Marketplace
A marketplace platform needs to identify sellers who are misrepresenting products or manipulating reviews. Fraudulent seller rate is approximately 2–3%.
Use: Precision-Recall AUC as primary, ROC-AUC as secondary
Why: Wrongly suspending a legitimate high-volume seller causes immediate revenue loss and potential legal disputes — false positives are extremely costly. Precision must be high. However missing actual fraudulent sellers damages buyer trust and marketplace integrity — recall matters too. At 2–3% prevalence ROC-AUC is still somewhat meaningful as a secondary check but should never be the primary decision metric.
Threshold decision: Tiered by seller history. New sellers with no track record get a lower threshold — catch potential fraud early even at the cost of some false positives. Established sellers with years of positive history get a much higher threshold — require very strong evidence before flagging.
Scenario 5 — Mortgage Application Default Prediction
A bank needs to predict whether a mortgage applicant will default. Default rate is approximately 3–5%.
Use: ROC-AUC as primary, Precision-Recall as secondary
Why: This is one of the few financial use cases where ROC-AUC remains meaningful. Default rates of 3–5% while still imbalanced are not as extreme as fraud scenarios. More importantly regulatory requirements (Fair Lending laws) require the bank to demonstrate consistent, auditable decision-making across demographic groups — ROC-AUC’s threshold-independent nature makes it easier to demonstrate consistent discrimination ability across groups. Additionally rejecting a legitimate mortgage applicant and missing a likely defaulter have more comparable costs than in fraud scenarios — making the balanced view of ROC-AUC more appropriate.
Threshold decision: Set by regulatory capital requirements and risk appetite — typically calibrated to match historical default rates in the loan portfolio rather than optimized purely for statistical performance.
Scenario 6 — Email Spam Detection
An email provider needs to classify incoming emails as spam or not spam. Spam rate varies — could be 20–40% of all email.
Use: ROC-AUC
Why: At 20–40% spam rate the classes are relatively balanced — ROC-AUC is perfectly valid and not misleading. The FPR denominator (legitimate emails) is not so large that it distorts the metric. Additionally users care equally about both types of errors — missing spam (annoyance) and blocking legitimate email (missed important message) — and ROC-AUC’s balanced view of both error types maps well to this equal-cost scenario.
Threshold decision: Set conservatively — slightly favor letting spam through over blocking legitimate email, since missing a legitimate email from a client or employer is typically more damaging than receiving one unwanted message.
Scenario 7 — Medical Diagnosis (Cancer Detection)
A healthcare system uses ML to flag potential cancer cases from imaging data. Cancer prevalence in screening population is 0.5%.
Use: Precision-Recall AUC exclusively
Why: Extreme class imbalance and dramatically asymmetric costs. Missing a cancer case (false negative) could mean a patient doesn’t receive timely treatment — potentially life threatening. A false positive means additional tests and anxiety — costly and stressful but not life threatening. ROC-AUC at 0.5% prevalence would be severely misleading. Precision-Recall AUC directly quantifies the recall (are we catching all cancer cases?) versus precision (are we sending too many healthy patients for unnecessary follow-up?) tradeoff that actually drives clinical decision-making.
Threshold decision: Set aggressively low — maximize recall even at the cost of precision. Missing cancer is categorically worse than unnecessary follow-up tests. Regulatory and clinical standards typically mandate very high recall (sensitivity) thresholds for cancer screening tools.
The Decision Framework — One Simple Rule
Ask yourself one question first:Is my positive class rare (less than 10% of data)?YES → Use Precision-Recall AUC as primary metric
ROC-AUC will be misleadingly optimistic
NO → ROC-AUC is valid
Use Precision-Recall as a secondary check
When in doubt → Use both, but lead with Precision-Recall
when presenting to business stakeholders
since it's more interpretable in cost terms
The Threshold Setting Framework
Once you’ve chosen your evaluation metric, here’s how to set your threshold in practice:
Step 1 — Plot the full Precision-Recall curve across all thresholds.
Step 2 — Calculate F1 score at each threshold — F1 = 2 × (Precision × Recall) / (Precision + Recall). The threshold maximizing F1 is your mathematically balanced starting point.
Step 3 — Adjust based on business cost ratio. If missing a positive costs C_FN and a false alarm costs C_FP, your optimal threshold shifts toward higher recall when C_FN > C_FP and toward higher precision when C_FP > C_FN.
Step 4 — Validate on holdout data and confirm precision and recall numbers are acceptable to business stakeholders before deployment.
Step 5 — Monitor in production and recalibrate threshold periodically as the distribution of scores shifts over time.
Summary Table
The One Line You Should Remember
ROC-AUC tells you how well your model separates classes in a perfect world. Precision-Recall AUC tells you how well it performs in the real world where positive cases are rare and every false alarm has a real business cost.
When in doubt — and especially when your dataset is imbalanced — trust the Precision-Recall curve. It never lies to you the way ROC-AUC can.
