Explaining why a trained model produced one particular prediction — and knowing when that explanation should not be trusted.

Learning objectives

• Distinguish global explanations from local explanations and choose the correct scope for a question.
• Describe an individual prediction using a baseline, positive contributions, and negative contributions.
• Explain the central idea of Shapley values and how SHAP attributes a prediction to individual features.
• Use local SHAP visualizations to investigate correct, incorrect, and low-confidence predictions.
• Apply simple counterfactual reasoning without confusing association with causality.
• Recognize the effects of correlated variables, model instability, visualization choices, and domain assumptions on explanations.

Chapter map

Table 35.1. Chapter structure

35.1 Global versus local explanations

Global interpretation summarizes how a fitted model behaves across a population of observations. Local interpretation narrows the question to one prediction: what evidence inside the fitted model pushed this observation toward its final score or class? The two scopes answer different questions and should not be substituted for each other.

Table 35.2. Global and local explanation scopes

A useful mental model

Think of the prediction as a journey from a reference value to the observation-specific output. A local explanation asks which features moved the model upward, which moved it downward, and by how much within the chosen explanation framework.

35.2 Explaining a single prediction

Baseline prediction

A baseline is the reference prediction before the observation-specific feature contributions are applied. In additive explanation methods, local feature effects are interpreted relative to this reference. The baseline may be an average model output over a background dataset rather than the prevalence of the positive class itself.

Feature contributions

A local feature contribution describes how a particular feature value changes the model output relative to the baseline under the explanation method. A positive contribution pushes the explained output upward; a negative contribution pushes it downward. The sign must always be interpreted relative to the output being explained.

Table 35.3. Reading a local contribution report

Prediction confidence

For probabilistic binary classifiers, local explanation should be paired with the prediction score. A high-confidence prediction lies far from the operational threshold; a low-confidence prediction lies near it. Low confidence does not imply the explanation is wrong, but it often means small perturbations could change the predicted class.

Counterfactual reasoning

Counterfactual reasoning asks how the model prediction would change if one or more input values were different while other information stayed fixed. This can reveal decision sensitivity and suggest what separates two model outcomes. However, a counterfactual input may be unrealistic, impossible, ethically inappropriate, or outside the data distribution.

Table 35.4. Counterfactual questions

35.3 SHAP concepts

SHAP (SHapley Additive exPlanations) applies ideas from cooperative game theory to model explanation. Features are treated as contributors to a prediction, and Shapley-value reasoning distributes the difference between a baseline and the explained output among those features according to their marginal contributions across feature coalitions.

Shapley values

A Shapley value is based on the average marginal contribution of a feature across possible coalitions of other features. Exact enumeration becomes expensive as the number of features grows, so practical SHAP implementations use model-specific algorithms or approximations.

Table 35.5. SHAP vocabulary

Local contribution

For a single observation, a waterfall plot is especially useful because it explicitly connects the baseline to the model output. Students should read the largest absolute contributions first, verify their direction, and then connect the feature values to domain expectations.

Global summaries from local explanations

Local SHAP values can also be aggregated across many observations. Mean absolute SHAP values provide a global importance summary, while beeswarm plots preserve both contribution magnitude and direction. This links local and global interpretation: the global view is built from many local explanations.

Dependence plots

A SHAP dependence or scatter plot relates a feature value to its local contribution across observations. It can reveal nonlinear patterns and possible interactions. The vertical axis is an explanation contribution, not the observed target, and the pattern remains model-dependent.

Computational considerations

Table 35.6. Choosing an explanation strategy

35.4 Explanation limitations

Explanation is not causality

Local explanation describes how the fitted model transforms the supplied input into an output. It does not establish that the important feature causes the real-world outcome. Causal claims require a causal design, assumptions, and domain evidence beyond predictive explanation.

Correlated features

When features carry overlapping information, attribution can be distributed in unintuitive ways. Two highly correlated variables may share credit, one may dominate depending on the model, or the explanation method may depend strongly on how missing-feature coalitions are represented.

Model instability

An explanation is only as stable as the fitted model. If small changes in training data produce very different models, local explanations can also change. Explanation stability should therefore be considered alongside prediction stability, especially for borderline observations.

Misleading visualizations

Table 35.7. Common explanation mistakes

Domain validation

A technically correct explanation can still be operationally misleading. Domain experts should verify that influential features are available at prediction time, have valid definitions, are not proxies for forbidden information, and behave plausibly for the intended population.

Practical lab — Explain three individual predictions

Goal: train one classifier, identify three diagnostically different test observations, and create a local explanation report for each: one correct prediction, one incorrect prediction, and one low-confidence prediction. The same model and explanation method must be used for all three cases so differences come from the observations rather than from the analysis procedure.

Step 1 — Load the data and create a protected test set

Step 2 — Train a deliberately interpretable-size Random Forest

Step 3 — Locate the three cases

Step 4 — Build a compact case table

Step 5 — Create SHAP explanations

Step 6 — Plot the correct prediction

Student interpretation prompts:

• Which three features have the largest absolute contributions?
• Which contributions push toward class 1 and which push away from it?
• Is the prediction far from or close to the 0.50 threshold?
• Do the dominant feature values look plausible for the fitted model and dataset?

Step 7 — Plot the incorrect prediction

The goal is not to “justify” the error. Instead, identify what evidence the model relied on, what evidence opposed the final output, whether correlated variables may have shared attribution, and whether this observation lies in a difficult part of feature space.

Step 8 — Plot the low-confidence prediction

A low-confidence case often contains competing contributions: some features push the score upward while others push it downward. Students should explicitly connect this contribution balance to the small distance from the decision threshold.

Step 9 — Inspect the strongest local contributions as a table

Step 10 — Perform cautious counterfactual reasoning

Choose one feature with a strong contribution. Replace it with a plausible value drawn from the training distribution, recompute the model probability, and describe how the model changes. This is a sensitivity experiment, not a causal recommendation.

Step 11 — Compare explanation stability

Optional extension: refit the Random Forest with several random seeds and repeat the explanation for the low-confidence observation. Record whether the prediction and top local features remain similar. Large changes indicate that the local explanation is sensitive to model instability.

Step 12 — Write the local explanation report

Table 35.8. Required report structure for each case

Discussion questions

1. Why can the same globally important feature have a small SHAP value for one observation?

2. What does an incorrect prediction reveal that a correct prediction may hide?

3. Why are low-confidence predictions especially useful for robustness analysis?

4. How could correlated features change the apparent ranking of local contributions?

5. Why is a model counterfactual not automatically an actionable real-world recommendation?

6. Which information should accompany a waterfall plot in a formal report?

Chapter summary

Table 35.9. Key takeaways