METRICS

CLASSIFICATION METRICS

The Confusion Matrix:

• True Negative (TN): The model correctly predicted not spam.
• True Positive (TP): The model correctly predicted spam.
• False Positive (FP): The model predicted spam, but it was actually not spam (Type I error). This is also known as a “False Alarm.”
• False Negative (FN): The model predicted not spam, but it was actually spam (Type II error). This is also known as a “Miss.”

With these four values, we can calculate a wealth of insightful metrics:

1.Accuracy

Formula: (TP + TN) / (TP + TN + FP + FN)

What it tells you: The proportion of correct predictions out of all predictions.

When to use it: When your classes are roughly balanced.

Caveat: Can be misleading with imbalanced datasets. If 95% of emails are not spam, a model that always predicts “not spam” would have 95% accuracy but be completely useless for finding actual spam.

2. Precision (Positive Predictive Value)

Formula: TP / (TP + FP)

• What it tells you: Out of all the instances the model predicted as positive, how many were actually positive? It measures the quality of positive predictions.
• When to use it: When the cost of a False Positive is high.
• Example: If you’re classifying healthy patients for a follow-up, a high false positive rate means unnecessary anxiety and medical procedures.

3. Recall (Sensitivity, True Positive Rate)

Formula: TP / (TP + FN)

• What it tells you: Out of all the actual positive instances, how many did the model correctly identify? It measures the model’s ability to find all the positive cases.
• When to use it: When the cost of a False Negative is high.
• Example: In medical diagnosis for a serious disease, a high false negative rate means missing actual cases, which can have severe consequences.

4. F1-Score

Formula: 2 * (Precision * Recall) / (Precision + Recall)

• What it tells you: The harmonic mean of Precision and Recall. It provides a single score that balances both concerns.

• When to use it: When you need a balance between Precision and Recall, especially with uneven class distribution. A high F1-score means your model has both good precision and good recall.

5. Specificity (True Negative Rate)

Formula: TN / (TN + FP)

• What it tells you: Out of all the actual negative instances, how many did the model correctly identify?
• When to use it: When correctly identifying negative cases is crucial.

6. ROC Curve and AUC (Area Under the Curve)

• ROC Curve: Plots the True Positive Rate (Recall) against the False Positive Rate (1 — Specificity) at various threshold settings.
• AUC: The area under the ROC curve.
• What it tells you: AUC provides a single value to compare classifier performance across different thresholds. An AUC of 1 means a perfect classifier, while 0.5 means a random classifier.
• When to use it: When you need to evaluate the overall discriminative power of your model, especially when class distributions are imbalanced.

REGRESSION METRICS

Let yi be the actual value and y^i be the predicted value for the i-th data point.

1.Mean Absolute Error (MAE)

Formula: (1/n) * \sum_{i=1}^{n} |yi — y^i |

Get Mattaparthi Mohana Sree’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

What it tells you: The average magnitude of the errors in a set of predictions, without considering their direction.

When to use it: When you want a simple, interpretable metric that is robust to outliers. The unit of MAE is the same as the target variable.

Caveat: Treats all errors equally, regardless of their magnitude.

2. Mean Squared Error (MSE)

Formula: (1/n) * \sum_{i=1}^{n} (yi — y^i)²

What it tells you: The average of the squared differences between predicted and actual values. It penalizes larger errors more heavily than MAE.

When to use it: When you want to penalize large errors more.

Caveat: Because errors are squared, MSE is not in the same unit as the target variable, making it less interpretable than MAE.

3. Root Mean Squared Error (RMSE)

Formula: sqrt{(1/n) * \sum_{i=1}^{n} (yi — y^i)²}

What it tells you: The square root of MSE. It brings the error unit back to the same unit as the target variable, making it more interpretable than MSE while still penalizing larger errors.

When to use it: It’s a very common and generally good default metric for regression.

Caveat: Still more sensitive to outliers than MAE.

4. R-squared (R², Coefficient of Determination)

Formula: 1 — (\sum (y_i — y^i)² / \sum (y_i — \bar{y})²) where bar{y} is the mean of actual values.

What it tells you: The proportion of the variance in the dependent variable that is predictable from the independent variables. In simpler terms, it indicates how well the independent variables explain the variation in the target variable.

Range: Typically between 0 and 1. A higher R² indicates a better fit. • When to use it: When you want to understand the explanatory power of your model. An R² of 0.8 means 80% of the variance in the target variable can be explained by your model.

Caveat: R² always increases or stays the same when you add more independent variables, even if they are not truly helpful. For this, we often use Adjusted R squared.

5. Adjusted R-squared

Formula: 1 — [(1 — R²) * (n — 1) / (n — p — 1)] where n is the number of data points and p is the number of predictors.

• What it tells you: A modified version of R² that accounts for the number of predictors in the model. It only increases if the new predictor improves the model more than would be expected by chance.

• When to use it: When comparing models with different numbers of features to avoid overfitting.

Choosing the Right Metrics

There’s no single “best” metric. The most appropriate evaluation metric depends entirely on your specific problem, the nature of your data, and the business or real-world consequences of different types of errors.

• For Classification: Consider the costs of False Positives vs. False Negatives. If missing a positive case is critical (e.g., disease detection), prioritize Recall. If false alarms are very expensive (e.g., flagging innocent people as criminals), prioritize Precision.

• For Regression: Think about how sensitive you are to large errors. If extreme errors are particularly bad, MSE or RMSE might be more suitable. If all errors are roughly equally problematic, MAE might be more transparent.

By understanding these metrics, you can move beyond a superficial understanding of your model’s performance and truly grasp its strengths and weaknesses, leading to more informed decisions and better machine learning solutions.