When evaluating a classification model, reporting "98% accuracy" sounds impressive. But in machine learning, high accuracy can be dangerously misleading.
If you are building a fraud-detection model where only 1 out of every 1,000 transactions is fraudulent, a trivial dummy model that labels every single transaction as legitimate will achieve 99.9% accuracy—while failing completely at its only job.
To properly evaluate classification algorithms, especially on imbalanced datasets, you must look beyond raw accuracy and master the quartet: Accuracy, Precision, Recall, and the F1 Score.
1. The Foundation: The Confusion Matrix
Every classification metric is derived from four fundamental outcomes when comparing model predictions against ground truth:
| Actual \ Predicted | Predicted Positive | Predicted Negative |
|---|---|---|
| Actual Positive | True Positive (TP)<br>(Correctly flagged hit) | False Negative (FN)<br>(Type II Error: Missed hit) |
| Actual Negative | False Positive (FP)<br>(Type I Error: False alarm) | True Negative (TN)<br>(Correctly rejected) |
- True Positive (TP): Model predicts positive, and the actual label is positive.
- True Negative (TN): Model predicts negative, and the actual label is negative.
- False Positive (FP): Model predicts positive, but the actual label is negative (Type I Error).
- False Negative (FN): Model predicts negative, but the actual label is positive (Type II Error).
2. Accuracy: When It Works and When It Fails
Accuracy is the ratio of correct predictions (both positive and negative) to the total number of input samples.
$$\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}$$
When to Use Accuracy:
- Your target classes are balanced (e.g., 50% cat images and 50% dog images).
- Every type of misclassification carries roughly equal real-world cost.
When to Avoid Accuracy:
- Your target classes are imbalanced (e.g., disease diagnosis, cybersecurity threat detection, anomaly detection). In these scenarios, high accuracy simply reflects the majority class distribution.
3. Precision: Minimizing False Alarms
Precision answers: "Out of all samples the model predicted as positive, how many were actually positive?"
$$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$$
Precision measures the reliability of the model's positive claims. If your model has high precision, you can trust that whenever it sounds the alarm, it is almost certainly right.
When Precision Is the Primary Metric:
Prioritize precision when the cost of a False Positive (Type I Error) is unacceptable.
- Email Spam Filtering: A false positive means a critical email from a client gets sent to the spam folder and missed. It is safer to let an occasional spam email through (lower recall) than to misroute an urgent invoice (high precision required).
- Video Recommendation / Search Results: Showing irrelevant recommendations degrades user trust quickly.
4. Recall (Sensitivity): Catching Every Target
Recall (also known as Sensitivity or True Positive Rate) answers: "Out of all actual positive cases in the dataset, how many did the model successfully identify?"
$$\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$$
Recall measures the model's ability to find all relevant instances without letting critical targets slip through undetected.
When Recall Is the Primary Metric:
Prioritize recall when the cost of a False Negative (Type II Error) is catastrophic or dangerous.
- Cancer Detection: A false negative means a sick patient is told they are healthy, delaying life-saving treatment. Flagging a healthy patient for follow-up testing (false positive) is vastly preferable to missing a malignant tumor.
- Airport Security Screening: Missing a dangerous item in baggage is unacceptable, even if it causes a few extra manual bag checks.
5. The Precision-Recall Tradeoff
You cannot simply maximize both metrics independently. Raising the classification threshold makes the model conservative:
- Higher Threshold: Precision increases (fewer false positives), but recall drops (more false negatives).
- Lower Threshold: Recall increases (catches nearly all positives), but precision drops (more false alarms).
Low Classification Threshold High Classification Threshold
(e.g., 0.20) (e.g., 0.80)
<-------------------------------------------------------------------------------->
High Recall / Low Precision High Precision / Low Recall
Catches almost all targets Rarely flags, but almost always right
Higher rate of false alarms Misses subtle positive targets
