Chapter 34 — Global Model Interpretation
Understanding how a trained model behaves across the dataset — not only how accurately it predicts.
| CHAPTER GOAL Learn to interpret global model behavior using coefficients, odds ratios, tree-based feature importance, permutation importance, and partial dependence — while recognizing the limits of each method. |
Learning objectives
- Explain why interpretation is important for trust, debugging, leakage detection, regulation, and scientific understanding.
- Interpret linear-model coefficient signs and magnitudes while accounting for feature units and scaling.
- Convert logistic-regression coefficients to odds ratios and communicate them carefully.
- Explain impurity-based feature importance and its common biases.
- Compute and interpret permutation importance on held-out data.
- Use partial dependence plots to inspect average nonlinear model responses and interactions.
- Compare two global feature-importance methods and explain why rankings may differ.
Chapter map
Section | Main question | Key output |
|---|---|---|
| 34.1 Why interpretation matters | Why inspect a model after evaluating it? | Interpretation goals and safeguards |
| 34.2 Linear models | What do coefficients mean globally? | Signs, magnitudes, odds ratios |
| 34.3 Tree importance | Which features reduce impurity most? | MDI feature ranking |
| 34.4 Permutation importance | Which features matter to held-out score? | Model-agnostic score drops |
| 34.5 Partial dependence | How does average prediction change with a feature? | 1D/2D response plots |
| Practical lab | Do two importance methods tell the same story? | Comparison table and interpretation report |
34.1 Why interpretation matters
Predictive performance answers whether a model works on the evaluation data. Interpretation asks a different question: what patterns is the model using, how does it react to important variables, and are those patterns credible for the intended use? Global interpretation summarizes behavior across many observations rather than explaining one single prediction.
Table 34.1. Main reasons for global interpretation
Reason | What interpretation can reveal | Example question |
|---|---|---|
| Understand behavior | Dominant relationships and directions used by the model. | Which variables influence predictions most strongly? |
| Detect leakage | Suspicious variables with unexpectedly dominant influence. | Is a post-outcome field driving the model? |
| Build user trust | A high-level explanation of what the system relies on. | Does the model use variables stakeholders expect? |
| Debugging | Encoding, scaling, feature-engineering, or pipeline errors. | Why is an ID-like field ranked highly? |
| Regulatory requirements | Evidence about model logic, sensitive variables, and governance. | Can the organization justify the main drivers? |
| Scientific understanding | Hypotheses about associations and nonlinear patterns. | What relationships deserve further investigation? |
| IMPORTANT Interpretation describes the behavior of a fitted model on a particular data distribution. It does not automatically identify causal effects or prove that the learned relationships are scientifically correct. |
Global versus local interpretation
Table 34.2. Two interpretation scopes
Scope | Question | Typical methods |
|---|---|---|
| Global | How does the model behave overall? | Coefficients, global feature importance, permutation importance, partial dependence. |
| Local | Why did the model make this prediction for one observation? | Local surrogate explanations, SHAP-like contribution methods, counterfactual analysis. |
This chapter focuses on global interpretation. Local interpretation can be added later when individual decisions must be explained.
34.2 Linear model interpretation
Linear and logistic models expose coefficients directly, which makes them natural starting points for interpretation. A coefficient describes how the model output changes when one feature changes while the other modeled features are held fixed. The numerical meaning depends on the feature scale and the model link function.
ŷ = β₀ + β₁x₁ + β₂x₂ + ··· + βₚxₚ For linear regression, βⱼ is the model-implied change in the prediction for a one-unit increase in xⱼ, holding other inputs fixed. |
Coefficient sign
- A positive coefficient means the prediction increases as the feature increases, all else equal.
- A negative coefficient means the prediction decreases as the feature increases, all else equal.
- A coefficient near zero indicates little linear contribution after accounting for the other included features.
- The sign can become unstable when predictors are strongly correlated.
Magnitude and feature units
Raw coefficient magnitudes cannot be compared fairly when features use different units. A coefficient of 0.5 per millimeter and a coefficient of 0.02 per euro describe different unit changes. Standardization places numeric features on a common scale and makes magnitude comparisons more meaningful, although correlation and regularization still affect the coefficients.
Table 34.3. Standardized versus unstandardized coefficients
Input representation | Coefficient meaning | Best use |
|---|---|---|
| Original units | Effect of a one-unit change in the original measurement. | Operational interpretation in natural units. |
| Standardized inputs | Effect of a one-standard-deviation change in a feature. | Relative comparison of coefficient magnitudes. |
PYTHON • Inspect standardized logistic-regression coefficients import numpy as np |
Odds ratios for logistic regression
Logistic regression is linear in log-odds, not directly in probability. Exponentiating a coefficient converts it to an odds ratio. When inputs are standardized, the odds ratio corresponds to a one-standard-deviation increase in that feature, holding other modeled inputs fixed.
Odds ratioⱼ = exp(βⱼ) OR > 1 increases modeled odds of the positive class; OR < 1 decreases them; OR = 1 indicates no modeled odds change. |
PYTHON • Convert logistic coefficients to odds ratios coef_table["odds_ratio"] = np.exp(coef_table["coefficient"]) |
| INTERPRET CAREFULLY An odds ratio is not the same as a probability ratio, and a fitted association is not automatically causal. Correlated predictors, regularization, sampling variation, and omitted variables can change coefficient values. |
34.3 Tree-based feature importance
Decision trees and tree ensembles commonly expose impurity-based feature importance, often called mean decrease in impurity (MDI). Each split reduces an impurity criterion. The model accumulates the weighted reductions attributed to each feature across the tree or ensemble and normalizes them into a global importance ranking.
Importance(feature j) ∝ Σ weighted impurity decrease from splits using feature j Larger values indicate that the fitted trees used the feature more strongly to improve their internal split criterion. |
Strengths
- Available directly after fitting many scikit-learn tree estimators.
- Fast to compute because no additional predictions are required.
- Useful for a first global ranking within the fitted tree model.
- Can reveal variables that repeatedly participate in important splits.
Limitations and bias
- Importance is model-specific: it explains the fitted tree ensemble, not the intrinsic value of a variable.
- Impurity importance can favor continuous or high-cardinality variables because they offer many possible split points.
- Correlated predictors can share or substitute for each other, making rankings unstable.
- A feature can appear important in training yet contribute little to held-out generalization.
- Rankings can change across random seeds, folds, or modest dataset changes.
PYTHON • Fit a random forest and obtain impurity importance from sklearn.ensemble import RandomForestClassifier |
| DIAGNOSTIC IDEA If an identifier, timestamp created after the outcome, or operational field unavailable at prediction time dominates the ranking, investigate leakage before trusting the model. |
34.4 Permutation importance
Permutation importance measures how much a chosen model score decreases after one feature is randomly shuffled. Shuffling destroys the relationship between that feature and the target while leaving the other columns unchanged. If the score drops substantially, the fitted model relied on that feature for performance on the evaluated dataset.
Permutation importanceⱼ = baseline score − average score after shuffling feature j Repeat the shuffle several times to estimate both the mean score reduction and its variability. |
Why it is useful
- Model-agnostic: it can be applied to linear models, trees, ensembles, kernels, and other fitted estimators.
- Can be computed on held-out validation or test-like data to focus on contribution to generalization.
- Uses a user-selected scoring metric, so importance can be tied to ROC AUC, F1, R², or another operational objective.
- Repeated permutations provide an empirical variability estimate.
PYTHON • Compute held-out permutation importance from sklearn.inspection import permutation_importance |
Correlated-feature limitation
Permutation importance can underestimate the apparent importance of correlated predictors. If two variables contain nearly the same information, shuffling one may cause only a small score drop because the model can still rely on the other. Therefore, “low permutation importance” does not always mean “no useful information.”
Table 34.4. Impurity importance versus permutation importance
Property | Impurity-based (MDI) | Permutation importance |
|---|---|---|
| Model scope | Tree models | Model-agnostic |
| Extra computation | Very low after fitting | Requires repeated predictions |
| Typical dataset | Training structure inside fitted trees | Training or preferably held-out data |
| High-cardinality bias | Can be substantial | Less direct, but data/model effects remain |
| Correlated features | Importance can be split/substituted | Score drop can be diluted by substitutes |
| Operational metric | Uses tree impurity criterion | Can use selected scoring metric |
| BEST PRACTICE Interpret importance only after verifying that the model itself has acceptable predictive performance. Feature importance from a poor model can be precise about the wrong model behavior. |
34.5 Partial dependence
Feature importance ranks variables but does not show the direction or shape of their influence. Partial dependence plots (PDPs) address that question by varying one feature (or a pair of features), generating predictions, and averaging the model response over the observed dataset.
PDⱼ(z) = average over observations of model prediction with feature j set to z The curve summarizes the average modeled response as feature j changes. |
What partial dependence can reveal
- Approximately linear or monotonic relationships.
- Thresholds, saturation, plateaus, and other nonlinear responses.
- Regions where the model prediction changes rapidly.
- Two-feature interactions through two-dimensional partial dependence surfaces.
PYTHON • Plot one-way partial dependence for important features import matplotlib.pyplot as plt |
PYTHON • Inspect a two-feature interaction top_two = perm_df.head(2)["feature"].tolist() |
Correlation risk
PDPs conceptually replace a feature with a grid of values while other feature values remain as observed. With strongly correlated predictors, this can create combinations that are rare or impossible in the real population. The resulting curve may therefore describe model behavior in regions where there is little data support.
| INTERPRETATION RULE Before interpreting a PDP causally or operationally, inspect feature distributions, correlations, and data support. Treat the curve as an average model response, not as proof that changing the feature will cause the prediction or real-world outcome to change. |
Practical lab — Compare two global feature-importance methods
Students will train a random-forest classifier on the Breast Cancer Wisconsin dataset and compare impurity-based importance with held-out permutation importance. The goal is not merely to generate rankings, but to explain why the rankings agree or disagree and what each method is actually measuring.
Lab objectives
- Evaluate the classifier before interpreting it.
- Generate impurity-based and permutation importance rankings.
- Compare ranks, magnitudes, and uncertainty.
- Investigate correlation among highly ranked predictors.
- Create a partial dependence plot for one important feature.
- Write a short global-interpretation report with limitations.
Step 1 — Prepare data and fit the model
PYTHON • Data, split, model, and baseline performance from sklearn.datasets import load_breast_cancer |
| CHECKPOINT Do not interpret importance before checking that the fitted model has adequate held-out performance. Interpretation is conditional on the model being worth interpreting. |
Step 2 — Method A: impurity-based importance
PYTHON • Build the MDI ranking mdi = pd.DataFrame({ |
Step 3 — Method B: held-out permutation importance
PYTHON • Build the permutation ranking with variability from sklearn.inspection import permutation_importance |
Step 4 — Compare the rankings
PYTHON • Join both methods and calculate rank disagreement comparison = mdi.merge(perm, on="feature") |
Step 5 — Check correlation as a possible explanation
PYTHON • Inspect correlations among top-ranked features top_features = ( |
Students should look for highly correlated feature pairs and discuss whether substitute information could explain a low permutation score or an unstable rank.
Step 6 — Add direction with partial dependence
PYTHON • Partial dependence for the most important permutation feature import matplotlib.pyplot as plt |
Step 7 — Optional stability check across random seeds
PYTHON • Measure how stable the top MDI features are seeds = [1, 7, 21, 42, 99] |
Student interpretation report
Table 34.5. Required lab report
Report item | What to include |
|---|---|
| Model quality | Held-out ROC AUC and a sentence confirming the model is suitable for interpretation. |
| MDI ranking | Top five features and what MDI measures. |
| Permutation ranking | Top five features, mean score drop, and variability. |
| Agreement | Features ranked highly by both methods. |
| Disagreement | At least two large rank gaps and plausible explanations. |
| Correlation check | Any highly correlated top features and how correlation affects interpretation. |
| Partial dependence | Shape and direction of the average model response for one feature. |
| Limitations | At least three caveats, including non-causality and data-distribution dependence. |
| LAB CONCLUSION A strong interpretation report does not search for one “true” importance ranking. It explains what each method measures, where the methods agree, why they can disagree, and what evidence is still missing before drawing domain conclusions. |
Common interpretation mistakes
Table 34.6. Mistakes to avoid
Mistake | Why it is problematic | Better practice |
|---|---|---|
| Treating importance as causality | Models learn associations and may exploit proxies. | Use causal design when the question is causal. |
| Interpreting a poor model | Importance may describe noise or misspecification. | Validate predictive performance first. |
| Comparing raw linear coefficients across units | Magnitude partly reflects measurement scale. | Standardize or interpret in natural units. |
| Trusting one tree ranking | High-cardinality bias and instability can distort MDI. | Compare with held-out permutation importance. |
| Ignoring correlated predictors | Importance can be split or diluted across substitutes. | Inspect correlation and grouped/domain features. |
| Reading PDPs outside data support | Feature replacement can create unrealistic combinations. | Check distributions/correlations and restrict conclusions. |
Discussion questions
1. Why can a feature have high impurity importance but low permutation importance on held-out data?
2. Why does standardization help when comparing linear-model coefficient magnitudes?
3. How should an odds ratio below 1 be interpreted in logistic regression?
4. What happens to permutation importance when two predictors contain nearly interchangeable information?
5. Why can a partial dependence curve be misleading for strongly correlated features?
6. How could feature importance help detect data leakage?
7. Why is “feature importance” not a causal statement?
Chapter summary
Table 34.7. Global interpretation methods at a glance
Method | Best question | Main caution |
|---|---|---|
| Linear coefficients | What direction and linear strength does the model assign to each input? | Units, correlation, regularization, and non-causality. |
| Logistic odds ratios | How do modeled odds change for a unit / SD increase? | Odds are not probabilities; association is not causation. |
| Tree impurity importance | Which features drive split-quality improvements inside the fitted trees? | High-cardinality bias and ranking instability. |
| Permutation importance | Which features matter to a chosen predictive score on this dataset? | Correlated substitutes can dilute importance. |
| Partial dependence | What is the average modeled response as a feature changes? | Correlated inputs can create unrealistic combinations. |
| NEXT STEP Global interpretation explains overall model behavior. In many real applications the next question is local: why did the model make this particular prediction, and what changes would alter that decision? |