Chapter 23 — Probability-Based Classification Evaluation
Scores • Thresholds • ROC AUC • Precision–Recall • Log Loss • Calibration
Evaluating not only the class decision, but also the confidence and operating point behind it
| BRIDGE FROM CHAPTER 22 A confusion matrix evaluates predictions after a decision threshold has already converted model scores into class labels. Probability-based evaluation looks one level earlier: it studies the scores themselves, how performance changes as the threshold moves, and whether predicted probabilities can be trusted as probabilities. |
Chapter overview
Many classifiers produce more information than a final class label. They may output probabilities such as 0.82, continuous decision scores such as 2.7, or both. These values allow the practitioner to compare models across many possible decision thresholds rather than accepting one fixed operating point.
This chapter develops the main tools for probability-based evaluation. ROC and precision–recall curves evaluate ranking behavior over thresholds. Log loss and the Brier score evaluate the quality of probabilistic predictions. Calibration curves test whether predicted probabilities correspond to observed event frequencies. Finally, threshold selection converts technical evaluation into an operational decision by explicitly considering the relative cost of false positives and false negatives.
Learning objectives
- Differentiate predict(), predict_proba(), and decision_function().
- Explain why a decision threshold is an operational choice rather than a universal constant.
- Predict how lowering or raising a threshold changes false positives and false negatives.
- Construct and interpret ROC curves, true-positive rate, false-positive rate, and ROC AUC.
- Explain why precision–recall curves are especially informative for imbalanced positive classes.
- Interpret average precision as a summary of a precision–recall curve.
- Use log loss to evaluate the quality of probability assignments.
- Define calibration and interpret reliability diagrams and calibration curves.
- Use the Brier score as a probability error measure.
- Select a classification threshold from validation data using an explicit operational cost.
Table 23.1. Chapter structure
| Section | Main question | Core idea |
|---|---|---|
| 23.1 | What numerical output does the classifier provide? | Labels, probabilities, decision scores, and calibration. |
| 23.2 | Where should the positive/negative cutoff be placed? | Thresholds trade false positives against false negatives. |
| 23.3 | How well does the model rank positives above negatives? | ROC curve and ROC AUC. |
| 23.4 | How does precision behave as recall changes? | Precision–recall curve and average precision. |
| 23.5 | Are probability assignments numerically good? | Log loss penalizes poor probability forecasts. |
| 23.6 | Does 0.8 really mean about 80%? | Calibration curves, Brier score, and calibration methods. |
| Lab | Which threshold minimizes operational cost? | Choose on validation data, then evaluate once on test data. |
23.1 Prediction scores and probabilities
A classifier can expose several forms of output. The correct evaluation method depends on which output is available and what the application needs. Class labels are useful for a fixed decision rule, while probabilities and continuous decision scores support threshold analysis and ranking metrics.
predict()
predict() returns the final class selected by the estimator. In binary classification the result is commonly 0 or 1. The classifier has already applied its internal decision rule, so predict() does not show how close an observation was to the boundary.
predict_proba()
predict_proba() returns estimated class probabilities for estimators that support probabilistic prediction. In binary classification, each row normally contains the probability of class 0 and the probability of class 1, and the two values sum to one. The column corresponding to the positive class is commonly used for ROC, precision–recall, log-loss, calibration, and threshold analysis.
decision_function()
decision_function() returns a continuous score related to the classifier decision boundary. Larger scores usually indicate stronger evidence for the positive class. These scores do not have to lie between 0 and 1 and should not automatically be interpreted as probabilities. They can nevertheless be used for ranking metrics such as ROC AUC and precision–recall evaluation.
Table 23.2. Common classifier outputs
| Output | Typical values | Best use | Important caution |
|---|---|---|---|
| predict() | Class labels | Evaluate one fixed operating point | Loses information about confidence and alternative thresholds. |
| predict_proba() | Probabilities in [0, 1] | Thresholds, log loss, calibration, ROC/PR curves | Probabilities may be poorly calibrated. |
| decision_function() | Unbounded or model-specific scores | Ranking and threshold analysis | Scores are not necessarily probabilities. |
PYTHON • Inspect labels, probabilities, and decision scores from sklearn.datasets import load_breast_cancer |
Probability calibration
A probability is calibrated when its numerical value matches observed frequency over many similar predictions. If a well-calibrated model assigns probability 0.80 to a large set of cases, approximately 80% of those cases should actually belong to the positive class. A model can rank observations very well while still producing overconfident or underconfident probabilities.
| RANKING IS NOT CALIBRATION ROC AUC can be excellent even when probability values are systematically too extreme or too conservative. Use calibration-specific diagnostics when the probability itself will drive pricing, triage, prioritization, or risk communication. |
23.2 Decision thresholds
A continuous probability or score must be converted into a class decision when an operational action is required. The threshold defines where that conversion occurs. For a probability p and threshold t, the basic binary rule is: predict positive when p ≥ t; otherwise predict negative.
Predict positive if p ≥ t; predict negative if p < t t is the chosen decision threshold. |
Default threshold
For many binary probabilistic classifiers, a threshold of 0.5 is the familiar default. It is convenient, but it is not automatically the best threshold for a real application. The best operating point depends on prevalence, model behavior, capacity constraints, and the relative consequences of errors.
Lowering the threshold
Lowering the threshold makes it easier for an observation to be classified as positive. More cases are flagged. This usually increases recall because fewer real positives are missed, but it can also increase false positives and reduce precision.
Increasing the threshold
Raising the threshold makes the positive decision more selective. Fewer cases are flagged. Precision often increases because the selected cases are more convincing, but recall can decrease because some real positives no longer cross the threshold.
Table 23.3. Typical threshold effects
| Threshold change | Predicted positives | Recall | Precision | Common operational effect |
|---|---|---|---|---|
| Lower threshold | Usually more | Usually increases | May decrease | Catch more positives, accept more false alarms. |
| Raise threshold | Usually fewer | Usually decreases | May increase | Act only on stronger cases, miss more positives. |
PYTHON • Apply custom probability thresholds import numpy as np |
Operational threshold selection
Threshold selection should translate a business, scientific, or safety requirement into a measurable rule. Examples include maximizing recall while maintaining precision above 0.70, limiting false positives to the number that a review team can inspect, or minimizing an explicit monetary or risk cost.
Operational cost(t) = Cost_FN × FN(t) + Cost_FP × FP(t) The confusion-matrix counts change as the threshold t changes. |
| EXPERIMENTAL DISCIPLINE Do not optimize a threshold repeatedly on the final test set. Choose the threshold with training/validation data or cross-validation, freeze the rule, and use the protected test set only for final evaluation. |
23.3 ROC curve
The receiver operating characteristic (ROC) curve evaluates a binary classifier over many thresholds. Each threshold produces one true-positive rate and one false-positive rate. Plotting those pairs shows the trade-off between detecting positives and incorrectly flagging negatives.
True-positive rate
The true-positive rate is the same quantity introduced as recall or sensitivity in Chapter 22. It measures the fraction of actual positives that are detected.
TPR = TP / (TP + FN) True-positive rate = recall = sensitivity. |
False-positive rate
The false-positive rate measures the fraction of actual negatives that are incorrectly classified as positive. It is one minus specificity.
FPR = FP / (FP + TN) = 1 − Specificity Lower FPR means fewer negative cases are falsely flagged. |
Threshold variation
At a very strict threshold, both TPR and FPR are often low because few observations are classified as positive. As the threshold is relaxed, both quantities can rise. The ROC curve records this sequence of operating points from strict to permissive decisions.
ROC AUC
ROC AUC summarizes the ROC curve as a single number. It can be interpreted as a ranking measure: the probability that a randomly selected positive case receives a higher score than a randomly selected negative case. AUC = 1 represents perfect ranking. AUC near 0.5 represents random-like ranking for a balanced interpretation of pairwise order.
Table 23.4. Practical ROC AUC interpretation
| ROC AUC | Interpretation |
|---|---|
| 1.00 | Perfect separation on the evaluated data. |
| 0.90–1.00 | Very strong ranking, subject to context and uncertainty. |
| 0.80–0.90 | Good ranking in many applications. |
| 0.70–0.80 | Moderate discrimination. |
| ≈ 0.50 | Random-like ranking. |
| < 0.50 | Ranking is systematically reversed or labels/scores require investigation. |
| DO NOT TREAT RANGES AS UNIVERSAL GRADES The practical value of an AUC depends on the domain, dataset difficulty, uncertainty, class prevalence, and consequences of error. Compare against realistic baselines and operational requirements. |
PYTHON • Compute and plot an ROC curve import matplotlib.pyplot as plt |
ROC curves are useful for comparing ranking performance, but they do not directly tell us whether probability values are calibrated or which threshold minimizes real-world cost. A high AUC therefore does not eliminate the need for threshold selection or calibration analysis.
23.4 Precision-recall curve
A precision–recall (PR) curve plots precision against recall as the decision threshold changes. It focuses directly on performance for the positive class and is often especially informative when positive observations are rare.
Importance for imbalanced classification
In a highly imbalanced dataset, the number of true negatives can be very large. ROC false-positive rate divides false positives by all actual negatives, so a seemingly small FPR can still correspond to many false alarms. Precision exposes this burden directly because false positives appear in its denominator.
Precision = TP / (TP + FP) How trustworthy are positive predictions? |
Recall = TP / (TP + FN) How many actual positives are detected? |
Precision-recall trade-off
A lower threshold normally retrieves more positives, increasing recall, but may admit more false positives and reduce precision. A higher threshold usually produces fewer, stronger positive predictions, often improving precision while reducing recall. The PR curve makes this trade-off visible across many thresholds.
Average precision
Average precision (AP) summarizes the precision–recall relationship across thresholds. In scikit-learn it is calculated as a weighted mean of precisions, where increases in recall determine the weights. Higher AP indicates that high precision is maintained as recall expands.
AP = Σₙ (Rₙ − Rₙ₋₁) Pₙ Pₙ and Rₙ are precision and recall at successive operating points. |
| USEFUL BASELINE For a random ranking, the expected precision is approximately the positive-class prevalence. When positives are rare, compare the PR curve and AP against that prevalence rather than against 0.5. |
PYTHON • Compute and plot a precision-recall curve import matplotlib.pyplot as plt |
Table 23.5. ROC versus precision–recall evaluation
| Aspect | ROC curve | Precision–recall curve |
|---|---|---|
| Axes | FPR versus TPR | Recall versus precision |
| Uses true negatives? | Yes, through FPR | No direct TN term |
| Very imbalanced positives | Can look optimistic if many negatives dominate | Often reveals false-positive burden more clearly |
| Summary | ROC AUC | Average precision (AP) |
| Main purpose | Ranking over thresholds | Positive-class retrieval quality over thresholds |
23.5 Log loss
ROC AUC and average precision evaluate ranking. Log loss evaluates the probability values themselves. It rewards assigning high probability to the correct class and strongly penalizes confident probability assignments to the wrong class.
Log loss = −(1/N) Σ [ yᵢ log(pᵢ) + (1−yᵢ) log(1−pᵢ) ] Binary log loss; lower values are better. |
Evaluating probability quality
Suppose two models classify the same observation correctly as positive. Model A assigns probability 0.55 and Model B assigns 0.95. Log loss gives more credit to the confident correct prediction from Model B. But if the observation is actually negative, the 0.95 prediction is penalized much more heavily than 0.55.
Penalizing confident incorrect predictions
Table 23.6. Probability confidence and log-loss behavior
| True class | Predicted P(positive) | Qualitative effect |
|---|---|---|
| Positive | 0.95 | Very small penalty: confident and correct. |
| Positive | 0.55 | Larger penalty: correct but uncertain. |
| Negative | 0.55 | Moderate penalty: incorrect but uncertain. |
| Negative | 0.95 | Very large penalty: confidently incorrect. |
| LOWER IS BETTER Unlike accuracy, ROC AUC, and average precision, log loss is an error measure. Better probability forecasts produce smaller values. |
PYTHON • Calculate log loss from sklearn.metrics import log_loss |
23.6 Calibration
Calibration asks whether predicted probabilities correspond to empirical event frequencies. This property matters whenever the probability value itself is interpreted as risk, confidence, expected demand, or priority rather than being used only for ranking.
Well-calibrated probabilities
For a well-calibrated model, cases assigned probabilities near 0.2 should contain roughly 20% positives, cases near 0.7 should contain roughly 70% positives, and so on. Perfect calibration would follow the diagonal relationship observed frequency = predicted probability.
Well calibrated: P(Y = 1 | predicted probability ≈ p) ≈ p Probability forecasts agree with observed frequencies. |
Reliability diagrams and calibration curves
A calibration curve groups probability predictions into bins. For each bin, it compares the mean predicted probability with the observed fraction of positive cases. Plotting those pairs produces a reliability diagram. Points above the diagonal indicate that positives occur more often than predicted; points below the diagonal indicate that probabilities are too high for the observed frequency.
PYTHON • Display a calibration curve import matplotlib.pyplot as plt |
Brier score
The Brier score is the mean squared difference between the predicted probability and the binary outcome. Lower scores indicate smaller probability errors. Like log loss, it is a proper probabilistic scoring rule; however, a single Brier score reflects both calibration and discrimination, so the calibration plot should still be inspected when calibration itself is the main concern.
Brier score = (1/N) Σ (pᵢ − yᵢ)² Binary probability error; lower is better. |
PYTHON • Calculate the Brier score from sklearn.metrics import brier_score_loss |
Probability calibration methods
Post-hoc calibration learns a mapping from a model score or raw probability to a better calibrated probability. Common approaches include sigmoid calibration (often associated with Platt scaling) and isotonic regression. Sigmoid calibration is smooth and relatively data-efficient. Isotonic regression is more flexible but can overfit when the calibration dataset is small. Calibration must be learned on data that are not simultaneously used to fit the underlying classifier.
Table 23.7. Common calibration methods
| Method | Shape | Strength | Caution |
|---|---|---|---|
| Sigmoid | Smooth S-shaped mapping | Stable with moderate calibration data | May be too restrictive for complex miscalibration. |
| Isotonic | Flexible monotonic mapping | Can fit non-sigmoid calibration patterns | Needs more calibration data; can overfit small samples. |
PYTHON • Calibrate a classifier with cross-validation from sklearn.calibration import CalibratedClassifierCV |
| CALIBRATION IS NOT A FREE IMPROVEMENT A calibrated model may improve probability reliability without improving ranking, and limited calibration data can introduce variance. Evaluate discrimination and calibration separately after calibration. |
Which metric answers which question?
Table 23.8. Probability-based metrics at a glance
| Tool | Primary question | Depends on threshold? | Direction |
|---|---|---|---|
| ROC AUC | How well are positives ranked above negatives? | No single fixed threshold | Higher is better |
| Average precision | Can high precision be maintained as recall increases? | Summarizes many thresholds | Higher is better |
| Log loss | Are the assigned probabilities numerically good? | No fixed classification threshold | Lower is better |
| Brier score | How close are probabilities to binary outcomes? | No fixed classification threshold | Lower is better |
| Calibration curve | Do predicted probabilities match observed frequencies? | No | Closer to diagonal is better |
| Operational cost | Which threshold best reflects real consequences? | Yes | Lower cost is better |
Practical lab — Select a threshold from operational cost
In this lab, the positive class represents a high-risk event requiring follow-up. Missing a real high-risk case is assumed to be ten times as costly as unnecessarily reviewing a negative case. Students train a probabilistic classifier, choose the threshold on validation data, and then apply the frozen threshold to a protected test set.
| OPERATIONAL COST RULE False negative cost = 10 units. False positive cost = 1 unit. Choose the threshold that minimizes 10 × FN + 1 × FP on the validation set. |
Step 1 — Create an imbalanced classification dataset
PYTHON • Generate the learning dataset from sklearn.datasets import make_classification |
Step 2 — Create train, validation, and test sets
PYTHON • Protect the final test set from sklearn.model_selection import train_test_split |
Step 3 — Train a probabilistic classifier
PYTHON • Fit logistic regression in a scaling pipeline from sklearn.linear_model import LogisticRegression |
Step 4 — Establish threshold-independent baselines
PYTHON • Evaluate ranking and probability quality from sklearn.metrics import ( |
Step 5 — Define the operational cost function
PYTHON • Calculate cost at one threshold from sklearn.metrics import confusion_matrix |
Step 6 — Search candidate thresholds on validation data
PYTHON • Find the minimum-cost validation threshold import numpy as np |
Step 7 — Compare the selected threshold with 0.50
PYTHON • Inspect precision, recall, and cost from sklearn.metrics import precision_score, recall_score |
Because false negatives cost ten times more than false positives, the minimum-cost threshold will often be lower than 0.50. The exact value depends on the generated data and trained model. Students should interpret the result rather than expecting one universal threshold.
Step 8 — Visualize cost over thresholds
PYTHON • Plot validation cost import matplotlib.pyplot as plt |
Step 9 — Freeze the threshold and evaluate the test set
PYTHON • Final evaluation on protected test data from sklearn.metrics import ( |
Step 10 — Compare probability-based diagnostics on the test set
PYTHON • Report ranking, probability, and threshold metrics from sklearn.metrics import ( |
Lab interpretation questions
1. Why should the threshold be selected on validation data rather than on the final test set?
2. How did the chosen threshold compare with 0.50, and why did the cost ratio influence that result?
3. What happened to recall when the threshold was lowered? What happened to false positives?
4. Could two models have the same ROC AUC but different log loss? Explain.
5. Why might average precision be more informative than ROC AUC when the positive class is rare?
6. If the review team can inspect only 80 cases per day, how would you change the threshold-selection rule?
7. If predicted probabilities are used as risk estimates, what additional diagnostic should be checked?
8. How would the selected threshold change if false positives became more expensive than false negatives?
Common mistakes to avoid
Table 23.9. Probability-evaluation mistakes
| Mistake | Why it is a problem | Better practice |
|---|---|---|
| Treating 0.50 as universally optimal | It ignores asymmetric costs and operational constraints. | Choose threshold from explicit requirements on validation data. |
| Using predict() for ROC AUC | Class labels discard score ranking information. | Use predict_proba() or decision_function(). |
| Calling decision_function() a probability | Decision scores may be unbounded and uncalibrated. | Use probabilities only when the estimator/calibration supports them. |
| Using ROC AUC as a calibration metric | AUC measures ranking, not probability reliability. | Inspect calibration curves and proper scoring rules. |
| Optimizing threshold on the test set | It leaks test information into model decisions. | Select on validation data; evaluate once on test. |
| Reporting only one summary score | Different metrics answer different questions. | Report ranking, probability quality, calibration, and operating-point metrics as needed. |
Chapter summary
- predict() returns class labels, predict_proba() returns estimated probabilities, and decision_function() returns continuous decision scores.
- Decision thresholds determine the operational balance between false positives and false negatives.
- Lower thresholds generally increase recall and false positives; higher thresholds generally increase selectivity and may improve precision.
- ROC curves plot true-positive rate against false-positive rate over thresholds, while ROC AUC summarizes ranking ability.
- Precision–recall curves focus on the positive class and are particularly useful when positives are rare.
- Average precision summarizes precision across recall changes without choosing one fixed threshold.
- Log loss evaluates probability quality and strongly penalizes confident incorrect predictions.
- Calibration asks whether predicted probabilities agree with observed event frequencies.
- Reliability diagrams, the Brier score, and post-hoc calibration methods help assess and improve probability reliability.
- Operational thresholds should be chosen using validation data and explicit costs or constraints, then frozen before final test evaluation.
| KEY TAKEAWAY A classifier is not fully evaluated by asking only “Was the class correct?” Strong practice also asks: “How were cases ranked?”, “Were probability values trustworthy?”, and “Which threshold best serves the real decision?” |
Short knowledge check
1. What is the main difference between predict_proba() and decision_function()?
2. What usually happens to recall when the classification threshold is lowered?
3. Write the formulas for true-positive rate and false-positive rate.
4. What does ROC AUC primarily measure: calibration, ranking, or threshold cost?
5. Why is the precision–recall curve useful for a rare positive class?
6. Why does log loss strongly penalize a probability of 0.99 assigned to the wrong class?
7. What would a perfectly calibrated reliability diagram look like?
8. Why should a threshold not be tuned on the final test set?