Stop Trusting Accuracy. Your Model Might Be Lying to You.

You trained your model. You ran it on the test set. It says 95% accuracy.

You feel good. You should, right?

Then your manager asks: “But how many fraudulent transactions did it actually catch?”

Silence.

That 95% accuracy was meaningless. Your dataset had 95% legit transactions and 5% fraudulent ones. Your model learned to just predict “not fraud” every single time — and still scored 95%. It caught zero actual fraud cases.

Welcome to the most common trap in machine learning. The fix? Understanding your classification metrics — really understanding them, not just memorizing definitions for an interview.

Let’s fix that.

The Building Block: Confusion Matrix

Before any metric, you need to understand the confusion matrix. Everything — every single metric we’ll cover — is just a combination of these four numbers.

Imagine you’re building a cancer detection model. You run it on 100 patients. Here’s what you get:

Predicted: Malignant    Predicted: Benign
Actual: Malignant     45  ← TP                5  ← FN
Actual: Benign        10  ← FP               40  ← TN

Four cells. Four terms that drive every metric:

TP (True Positive) — You said malignant, it was. Correct call. TN (True Negative) — You said benign, it was. Also correct. FP (False Positive) — You said malignant, it was benign. False alarm. FN (False Negative) — You said benign, it was malignant. You missed it.

That last one? In cancer detection, that’s the one that gets people killed.

1. Accuracy — The Metric Everyone Uses, Often Wrongly

Formula:

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Accuracy answers: “Out of everything, how often was I right?”

Using our example: (45 + 40) / 100 = 85%. Sounds decent. But here’s the trap.

If your dataset had 95 healthy patients and only 5 cancer cases, a model that screams “healthy!” every single time scores 95% accuracy — while missing every single cancer patient. That’s not a model. That’s a liability.

The Accuracy Trap: If your minority class is less than 20% of your data, accuracy is a misleading metric. Always pair it with something else.

Accuracy only makes sense when your classes are roughly balanced and both types of errors cost the same. That’s rarely true in the real world.

2. Precision — “When You Raise an Alarm, Can We Trust It?”

Formula:

Precision = TP / (TP + FP)

From our numbers: 45 / (45 + 10) = 81.8%

When the model says “malignant,” it’s right 82% of the time.

Think of precision as your model’s trustworthiness when it flags something. High precision means fewer false alarms. Low precision means you’re crying wolf constantly.

Real-world example — Spam Filter: A False Positive here means a legitimate email from your boss lands in spam. That’s bad. You want high precision — when the model flags something as spam, it better actually be spam.

Another example — Legal document review: If you’re using AI to flag potentially fraudulent contracts, you want high precision. You don’t want lawyers wasting hours reviewing hundreds of false hits.

When to prioritize Precision: When the cost of False Positives is high — when a wrong positive causes significant damage, cost, or wasted effort.

3. Recall — “How Much Did You Actually Catch?”

Formula:

Recall = TP / (TP + FN)

From our numbers: 45 / (45 + 5) = 90%

Out of all actual cancer cases, the model caught 90% of them. That 10% it missed? Those are patients who got sent home with a clean bill of health.

Recall measures your model’s ability to not miss things. It’s the metric that asks: “Of all the real positives out there, how many did I find?”

Real-world example — Cancer Detection: A False Negative here means you told a patient they’re fine — when they actually have cancer. They won’t get treatment in time. You want high recall here, even if it means some healthy patients get flagged for follow-up tests. False alarms are stressful but fixable. Missed cancer is not.

Another example — COVID screening at airports: You’d rather scan too many healthy people than let one infected person through. Recall is everything.

When to prioritize Recall: When the cost of False Negatives is high — when missing a positive case is dangerous, costly, or irreversible.

The Precision–Recall Tradeoff

Here’s the tension that trips everyone up: these two metrics pull against each other.

Get EasyPeasySaral’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

If you lower your decision threshold to catch more cancer cases (recall goes up), you’ll also flag more healthy patients as malignant (precision goes down). You can’t max both simultaneously — you have to decide which error is worse for your specific problem.

Lower threshold → More positives predicted → Recall ↑, Precision ↓
Higher threshold → Fewer positives predicted → Precision ↑, Recall ↓

This isn’t a bug. It’s the fundamental reality of classification. Your job is to tune the threshold based on what matters most in your context.

4. F1 Score — The Balanced Middle Ground

Formula:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

From our numbers: 2 × (0.818 × 0.90) / (0.818 + 0.90) = 85.7%

F1 is the harmonic mean of precision and recall. It gives you one number that respects both — and punishes models that are extreme in either direction.

Why harmonic mean and not regular average?

Because the arithmetic average would give a model with 100% precision and 0% recall a score of 50% — which sounds passable. The harmonic mean gives it 0% — which is what it deserves. It punishes extreme imbalances hard.

Here’s how all four metrics look on the same cancer model:

Use F1 when: You care about both precision and recall, your data is imbalanced, and there’s no strong business reason to favor one error type over the other. It’s the default metric for most NLP classification tasks.

5. ROC-AUC — The Big Picture View

ROC = Receiver Operating Characteristic Curve AUC = Area Under the Curve

All the metrics above evaluate your model at one specific decision threshold. ROC-AUC asks a different question: how good is your model across all possible thresholds?

The ROC curve sweeps through every threshold and at each one records:

• Y-axis (TPR): True Positive Rate = Recall = How many real positives did you catch?
• X-axis (FPR): False Positive Rate = FP / (FP + TN) = How many negatives did you wrongly flag?

AUC = 1.0  → Perfect model (hugs top-left corner)
AUC = 0.9  → Excellent
AUC = 0.7  → Decent
AUC = 0.5  → Random model (diagonal line — worthless)
AUC < 0.5  → Worse than random

What AUC actually tells you:

“If I randomly pick one positive case and one negative case, what’s the probability my model scores the positive higher?”

AUC = 0.92 means there’s a 92% chance your model ranks a real cancer patient above a healthy one. That’s threshold-independent, and that’s powerful.

When to use ROC-AUC:

• When your classes are relatively balanced
• When you want to evaluate overall ranking ability, regardless of any specific threshold
• When comparing multiple models head-to-head

When NOT to use ROC-AUC: On highly imbalanced datasets. ROC-AUC can look great (0.95+) while your model still performs terribly on the rare minority class. In those cases, use PR-AUC — it tells a much more honest story.

The Cheat Sheet: Which Metric, When?

If your classes are balanced and you just need a quick overview, Accuracy works fine. But the moment your data is imbalanced, switch to F1 or PR-AUC — accuracy will lie to you. When missing a positive case is catastrophic, like in cancer detection or fraud, prioritize Recall, because false negatives are the real danger. On the flip side, if false alarms are costly — think spam filters or legal document review — go with Precision, where being right when you flag something matters more. For comparing multiple models on balanced data, ROC-AUC is your go-to.

Putting It All Together: A Real Workflow

Here’s how a thoughtful ML practitioner actually evaluates a model:

Step 1. Check your class distribution. Imbalanced? Accuracy is out.

Step 2. Ask: what’s the real-world cost of FP vs FN?

• FP costly → optimize Precision
• FN costly → optimize Recall
• Both matter → use F1

Step 3. Need a threshold-independent view?

• Balanced data → ROC-AUC
• Imbalanced data → PR-AUC

Step 4. Always report multiple metrics. No single number tells the whole story. A good evaluation shows accuracy, precision, recall, F1, and AUC — and explains what they mean for the specific problem.

The Real Takeaway

Metrics aren’t just about measuring performance — they’re about defining what success means for your specific problem.

Two models with the exact same accuracy can behave completely differently on the cases that matter most. The best ML engineers aren’t the ones who know every formula. They’re the ones who ask: “What’s the cost of being wrong — and wrong in which direction?”

That question changes everything.

Thanks for reading! If this helped, share it with someone still trusting their accuracy score unconditionally.

I write about practical machine learning, AI/ML engineering, and building real-world systems. Follow for more.