Chapter 22 — Confusion Matrix and Basic Metrics
Accuracy • Precision • Recall • Specificity • F1-score • Multiclass Averages
Turning classification predictions into interpretable measures of error and success
| BRIDGE FROM THE MODEL CHAPTERS Classification models do not become useful simply because they produce predictions. We also need to understand what kinds of mistakes they make. The confusion matrix is the foundation: it separates correct and incorrect predictions by class and leads directly to accuracy, precision, recall, specificity, and F1-score. |
Chapter overview
A single classification score can hide important behavior. Two models can have the same accuracy while making very different types of errors. In fraud detection, a false negative may mean a fraudulent transaction is missed. In spam filtering, a false positive may mean a legitimate message is incorrectly blocked. The evaluation metric should therefore reflect the practical cost of errors.
This chapter begins with the binary confusion matrix and develops the most common basic metrics from its four cells. Students learn when each metric is appropriate, why accuracy can be misleading for imbalanced data, and how precision and recall emphasize different error types. The chapter then extends these ideas to multiclass problems through macro, micro, and weighted averaging.
The practical lab uses a fixed confusion matrix for manual calculations and then verifies the results with scikit-learn. A second multiclass exercise demonstrates how classification_report summarizes per-class metrics and averages.
Learning objectives
- Identify true positives, true negatives, false positives, and false negatives from predictions.
- Construct and interpret a binary confusion matrix.
- Calculate accuracy, precision, recall, specificity, and F1-score manually.
- Explain why accuracy can be misleading on imbalanced datasets.
- Choose precision when false positives are particularly costly.
- Choose recall when false negatives are particularly costly.
- Interpret specificity as the ability to correctly identify negative observations.
- Explain the balance represented by the F1-score.
- Differentiate macro, micro, and weighted averages in multiclass classification.
- Verify manual calculations using scikit-learn evaluation functions.
Table 22.1. Chapter structure
| Section | Main question | Core idea |
|---|---|---|
| 22.1 | What did the classifier get right and wrong? | Confusion matrix: TP, TN, FP, FN. |
| 22.2 | How often was the classifier correct overall? | Accuracy. |
| 22.3 | When the model predicts positive, how often is it right? | Precision. |
| 22.4 | How many real positives did the model find? | Recall / sensitivity. |
| 22.5 | How many real negatives did the model reject correctly? | Specificity. |
| 22.6 | How can precision and recall be summarized together? | F1-score. |
| 22.7 | How are metrics summarized across several classes? | Macro, micro, and weighted averages. |
| Lab | Do the manual and library calculations agree? | Calculate by hand, then verify with scikit-learn. |
22.1 Confusion matrix
A confusion matrix cross-tabulates the true class labels and the predicted class labels. For binary classification, it contains four counts. These four counts describe every possible outcome for one observation and form the basis of most basic classification metrics.
Table 22.2. Binary confusion matrix
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | True Positive (TP) | False Negative (FN) |
| Actual Negative | False Positive (FP) | True Negative (TN) |
True positive
A true positive is an observation that is actually positive and is correctly predicted as positive. Example: a fraudulent transaction is fraudulent and the model flags it as fraud.
True negative
A true negative is an observation that is actually negative and is correctly predicted as negative. Example: a legitimate transaction is legitimate and the model leaves it unflagged.
False positive
A false positive occurs when the observation is actually negative but the model predicts positive. This is also called a false alarm or a Type I classification error in some contexts. The practical cost depends on the application.
False negative
A false negative occurs when the observation is actually positive but the model predicts negative. This is a missed positive case. In safety, fraud, or screening applications, false negatives can be particularly costly.
| ALWAYS DEFINE THE POSITIVE CLASS The meanings of TP, FP, FN, and TN depend on which class is designated as positive. Before interpreting any metric, state what “positive” means in the application. |
Worked confusion-matrix example
Suppose a classifier is evaluated on 110 observations and produces the following counts: TP = 42, TN = 50, FP = 8, and FN = 10. There are therefore 52 actual positives and 58 actual negatives.
Table 22.3. Worked binary example
| Predicted Positive | Predicted Negative | Total | |
|---|---|---|---|
| Actual Positive | 42 (TP) | 10 (FN) | 52 |
| Actual Negative | 8 (FP) | 50 (TN) | 58 |
| Total | 50 | 60 | 110 |
PYTHON • Create and display a confusion matrix import numpy as np |
| SCIKIT-LEARN LAYOUT For binary labels ordered as 0 then 1, confusion_matrix returns [[TN, FP], [FN, TP]]. Always verify the label order before unpacking the matrix. |
22.2 Accuracy
Accuracy measures the proportion of all evaluated observations that were classified correctly. It counts both correctly predicted positives and correctly predicted negatives.
Accuracy = (TP + TN) / (TP + TN + FP + FN) Fraction of all predictions that are correct. |
Worked calculation
Accuracy = (42 + 50) / 110 = 92 / 110 ≈ 0.836 The worked classifier is correct on about 83.6% of observations. |
Appropriate use
Accuracy is useful when the classes are reasonably balanced and false-positive and false-negative errors have similar practical consequences. It is intuitive and can provide a useful first summary, especially when combined with a confusion matrix.
Limitations with imbalanced data
Accuracy can become dangerously optimistic when one class dominates the dataset. If 990 observations are negative and only 10 are positive, a model that predicts every observation as negative achieves 99% accuracy while detecting none of the positive cases.
Table 22.4. Why accuracy can mislead
| Scenario | Correct predictions | Accuracy | Recall for positive class |
|---|---|---|---|
| Predict every case as negative | 990 of 1,000 | 99% | 0% |
| Interpretation | Looks excellent by accuracy | Misleading | All positives are missed |
| EVALUATION RULE Never use accuracy alone when the target is strongly imbalanced or when the two error types have very different costs. Inspect class-specific metrics and the confusion matrix. |
PYTHON • Calculate accuracy from sklearn.metrics import accuracy_score |
22.3 Precision
Precision focuses on the observations predicted as positive. It asks: among everything the model labeled positive, what fraction was truly positive?
Precision = TP / (TP + FP) The denominator contains all predicted positives. |
Precision = 42 / (42 + 8) = 42 / 50 = 0.840 In the worked example, 84% of positive predictions are correct. |
Meaning
High precision means the model produces relatively few false alarms. A low-precision classifier may flag many observations as positive, but a large fraction of those alerts are incorrect.
Use when false positives are costly
Precision deserves special attention when a positive prediction triggers an expensive, disruptive, or limited intervention. Examples include manual investigations, sending scarce resources, blocking legitimate transactions, or contacting customers unnecessarily.
Table 22.5. Precision-oriented thinking
| Question | Focus |
|---|---|
| What is in the denominator? | Everything predicted positive: TP + FP |
| Which error lowers precision? | False positives |
| When is it especially important? | When false alarms are costly |
| How can precision be increased? | Often by using a stricter decision threshold, at the possible cost of recall |
PYTHON • Calculate precision from sklearn.metrics import precision_score |
22.4 Recall
Recall focuses on the observations that are actually positive. It asks: among all real positive cases, what fraction did the classifier successfully identify? Recall is also called sensitivity or the true positive rate.
Recall = TP / (TP + FN) The denominator contains all actual positives. |
Recall = 42 / (42 + 10) = 42 / 52 ≈ 0.808 The classifier detects about 80.8% of the actual positives. |
Sensitivity
The word sensitivity is commonly used when the positive class represents an event that must be detected. High sensitivity means few positive observations are missed.
Use when false negatives are costly
Recall is central when missing a positive case is more harmful than producing some additional false alarms. Examples include fraud screening, equipment-failure warning, security detection, or other applications where an undetected positive event can create substantial risk.
Table 22.6. Recall-oriented thinking
| Question | Focus |
|---|---|
| What is in the denominator? | All actual positives: TP + FN |
| Which error lowers recall? | False negatives |
| When is it especially important? | When missed positives are costly |
| How can recall be increased? | Often by using a more permissive decision threshold, at the possible cost of precision |
PYTHON • Calculate recall from sklearn.metrics import recall_score |
22.5 Specificity
Specificity measures the ability to correctly identify negative observations. It asks: among all actual negatives, what fraction did the classifier correctly classify as negative? Specificity is also called the true negative rate.
Specificity = TN / (TN + FP) The denominator contains all actual negatives. |
Specificity = 50 / (50 + 8) = 50 / 58 ≈ 0.862 The classifier correctly rejects about 86.2% of the actual negatives. |
Specificity versus recall
Recall and specificity examine opposite classes. Recall treats the positive class as the event to be found; specificity asks how reliably the negative class is rejected. A complete binary evaluation often reports both when both types of recognition matter.
Table 22.7. Recall and specificity side by side
| Metric | Numerator | Denominator | Main error |
|---|---|---|---|
| Recall / Sensitivity | TP | TP + FN | False negatives |
| Specificity | TN | TN + FP | False positives |
PYTHON • Calculate specificity from the confusion matrix from sklearn.metrics import confusion_matrix |
22.6 F1-score
Precision and recall can move in opposite directions as the classification threshold changes. The F1-score summarizes them using their harmonic mean. It becomes high only when both precision and recall are reasonably high.
F1 = 2 × (Precision × Recall) / (Precision + Recall) Harmonic mean of precision and recall. |
F1 ≈ 2 × (0.840 × 0.808) / (0.840 + 0.808) ≈ 0.824 The worked classifier has an F1-score of about 82.4%. |
Why the harmonic mean?
The arithmetic mean can remain relatively high when one component is much larger than the other. The harmonic mean penalizes imbalance between precision and recall more strongly. Consequently, F1 is useful when both false positives and false negatives matter and a single precision-recall summary is needed.
| IMPORTANT LIMITATION F1 does not use true negatives. If correctly identifying the negative class is important, report specificity, accuracy, or another suitable metric alongside F1. |
PYTHON • Calculate the F1-score from sklearn.metrics import f1_score |
Table 22.8. Basic binary metrics at a glance
| Metric | Formula emphasis | Best question to ask |
|---|---|---|
| Accuracy | Correct predictions / all predictions | How often is the classifier correct overall? |
| Precision | TP / predicted positives | Can I trust a positive prediction? |
| Recall | TP / actual positives | How many real positives did I find? |
| Specificity | TN / actual negatives | How many real negatives did I reject? |
| F1-score | Harmonic mean of precision and recall | Are precision and recall jointly strong? |
22.7 Macro, micro, and weighted averages
In multiclass classification, precision, recall, and F1 can be calculated separately for each class by treating that class as positive and the remaining classes as negative. A dataset with three classes therefore produces three class-specific precision values, three recall values, and three F1 values. Averaging determines how those class-specific results are summarized.
Macro average
The macro average computes the metric independently for each class and then takes the simple arithmetic mean. Every class receives equal weight, regardless of how many observations it contains. Macro metrics therefore make poor performance on a small class visible rather than allowing a large class to dominate the summary.
Macro metric = (Metric₁ + Metric₂ + ··· + Metric_K) / K Each of the K classes contributes equally. |
Micro average
The micro average first aggregates the true-positive, false-positive, and false-negative counts across classes and then computes the metric from those global totals. Individual observations, rather than classes, receive equal weight. For standard single-label multiclass classification, micro precision, micro recall, and micro F1 are equal to overall accuracy.
Weighted average
The weighted average computes a metric for each class and weights each class by its support, meaning the number of true observations belonging to that class. Large classes therefore contribute more to the final value. Weighted averages reflect class frequency but can hide weak performance on rare classes.
Weighted metric = Σₖ (supportₖ / N) × Metricₖ Each class contribution is proportional to its number of true observations. |
Table 22.9. Macro, micro, and weighted averages
| Average | How it is formed | Effect of class imbalance | Useful when |
|---|---|---|---|
| Macro | Unweighted mean of class metrics | Gives minority and majority classes equal importance | You care equally about performance on every class |
| Micro | Aggregate counts before calculating metric | Dominated by the total number of observations | You want an observation-level global summary |
| Weighted | Mean weighted by class support | Majority classes contribute more | You want a class metric that reflects observed class frequencies |
Multiclass interpretation example
Table 22.10. Example per-class F1 values
| Class | Support | F1-score |
|---|---|---|
| Class A | 700 | 0.95 |
| Class B | 250 | 0.82 |
| Class C | 50 | 0.40 |
The macro F1 would be strongly reduced by the weak result on Class C because all classes receive equal weight. The weighted F1 would remain much closer to the strong Class A result because Class A contains most observations. Reporting both can reveal whether a good overall result is masking a weak minority class.
PYTHON • Obtain multiclass metrics with classification_report from sklearn.metrics import classification_report |
| DO NOT CHOOSE AN AVERAGE MECHANICALLY Macro, micro, and weighted averages answer different questions. Keep the per-class results visible, especially when one class is rare or operationally important. |
Practical lab — Manual metrics and scikit-learn verification
The lab begins with a known binary confusion matrix so every metric can be calculated by hand. Students then create equivalent y_true and y_pred arrays, verify all results with scikit-learn, and finish with a multiclass extension.
| LAB DATA Use TP = 42, TN = 50, FP = 8, and FN = 10. Total observations = 110. |
Step 1 — Store the confusion-matrix counts
PYTHON • Define TP, TN, FP, and FN tp = 42 |
Step 2 — Calculate every metric manually
PYTHON • Manual formulas accuracy = (tp + tn) / (tp + tn + fp + fn) |
Table 22.11. Expected manual results
| Metric | Expected value |
|---|---|
| Accuracy | 0.8364 |
| Precision | 0.8400 |
| Recall | 0.8077 |
| Specificity | 0.8621 |
| F1-score | 0.8235 |
Step 3 — Build labels that reproduce the same confusion matrix
The following arrays are constructed by concatenating 42 true positives, 10 false negatives, 8 false positives, and 50 true negatives. Their order does not affect the counts, although real datasets naturally contain observations in arbitrary order.
PYTHON • Construct y_true and y_pred import numpy as np |
Step 4 — Verify the confusion matrix with scikit-learn
PYTHON • Check the four cells from sklearn.metrics import confusion_matrix |
Step 5 — Verify accuracy, precision, recall, and F1
PYTHON • Use scikit-learn metric functions from sklearn.metrics import ( |
Step 6 — Verify specificity
PYTHON • Compute specificity from the verified matrix specificity_skl = tn_skl / (tn_skl + fp_skl) |
Specificity can also be viewed as recall for the negative class when class 0 is treated as the class of interest. Explicit calculation from TN and FP is often the clearest teaching approach.
Step 7 — Display a classification report
PYTHON • Generate a compact binary report from sklearn.metrics import classification_report |
The report provides precision, recall, F1-score, and support for each class. Notice that the recall shown for the Negative class corresponds to binary specificity when Negative is treated as the class of interest.
Step 8 — Visualize the confusion matrix
PYTHON • ConfusionMatrixDisplay import matplotlib.pyplot as plt |
Step 9 — Explore an imbalanced-data failure
PYTHON • A 99% accurate classifier that finds no positives y_true_imbalanced = np.array([0] * 990 + [1] * 10) |
| OBSERVATION The model reaches 0.99 accuracy but 0.00 recall for the positive class. This is why class balance and error costs must be considered before selecting the main metric. |
Step 10 — Multiclass extension
PYTHON • Compare macro, micro, and weighted F1 from sklearn.metrics import f1_score |
Step 11 — Inspect per-class performance
PYTHON • Print the multiclass classification report print( |
Student analysis questions
1. Using TP = 42, TN = 50, FP = 8, and FN = 10, calculate accuracy without Python.
2. Why is the precision denominator TP + FP rather than TP + FN?
3. Why is recall especially sensitive to false negatives?
4. What practical question does specificity answer?
5. Why can F1 decrease even if precision improves?
6. Why does the 99%-accuracy imbalanced classifier still have unacceptable positive-class performance?
7. In a fraud-detection problem, when might recall be prioritized over precision?
8. In a costly manual-review system, when might precision deserve more emphasis?
9. What is the main difference between macro and weighted averaging?
10. Why should per-class metrics remain visible even when one multiclass average is reported?
Chapter summary
Table 22.12. Metric selection summary
| Metric | Primary focus | Error most directly emphasized | Common use |
|---|---|---|---|
| Accuracy | All predictions | FP and FN contribute equally to error count | Balanced classes and similar error costs |
| Precision | Predicted positives | False positives | False alarms are expensive |
| Recall | Actual positives | False negatives | Missed positives are expensive |
| Specificity | Actual negatives | False positives | Correct rejection of negatives matters |
| F1-score | Precision + recall | FP and FN through their effect on P/R | Need one score balancing positive-class precision and recall |
| Macro average | Equal class importance | Poor minority-class performance remains visible | Every class matters equally |
| Micro average | All observations together | Global aggregate mistakes | Overall observation-level summary |
| Weighted average | Class-frequency-weighted results | Majority classes have greater influence | Summary reflecting observed support |
Key takeaways
- The confusion matrix is the foundation for interpreting binary classification errors.
- TP and TN are correct predictions; FP and FN are the two kinds of classification mistakes.
- Accuracy measures overall correctness but can be misleading on imbalanced datasets.
- Precision focuses on the reliability of positive predictions and decreases when false positives increase.
- Recall measures how many actual positives are found and decreases when false negatives increase.
- Specificity measures how many actual negatives are correctly rejected.
- F1 is the harmonic mean of precision and recall and does not use true negatives directly.
- Macro averaging treats every class equally; weighted averaging reflects class support.
- Micro averaging aggregates decisions across classes before computing the metric.
- Metric choice must be guided by class balance and the real-world cost of different errors.
| CONNECTION TO THE NEXT EVALUATION TOPICS Basic confusion-matrix metrics summarize predictions at one decision rule or threshold. More advanced evaluation compares behavior across thresholds and examines ranking quality, probability quality, and model selection under uncertainty. |
Quick knowledge check
1. What is the difference between a false positive and a false negative?
2. Write the accuracy formula using TP, TN, FP, and FN.
3. Which metric answers: “Of all positive predictions, how many were correct?”
4. Which metric is also called sensitivity?
5. What is the formula for specificity?
6. Why can F1 be preferable to accuracy on an imbalanced problem?
7. What does macro averaging do with rare classes?
8. Why can a weighted average hide weak performance on a minority class?
Suggested student deliverable
Submit a notebook or short report containing the hand calculations for TP = 42, TN = 50, FP = 8, and FN = 10; the scikit-learn verification; the confusion-matrix visualization; the imbalanced-data experiment; the multiclass macro, micro, and weighted comparison; and a short paragraph explaining which metric you would prioritize in one real classification application and why.