Chapter 25 — Residual Analysis
Residual plots • Model diagnostics • Error segmentation • Drift
Finding where a regression model fails, not only how much it fails
| BRIDGE FROM CHAPTER 24 Regression metrics summarize error with one number. Residual analysis opens that number back up and asks where the errors occur, whether they are systematic, and which observations or groups need attention. |
Chapter overview
A low MAE or a respectable R² does not guarantee that a regression model is behaving well everywhere. Errors can cancel, concentrate in a subgroup, grow with the target magnitude, follow a curved pattern, or become worse over time. Residual analysis studies individual prediction errors so these structures become visible.
This chapter develops a practical diagnostic workflow: compute residuals, visualize them against predictions and features, detect systematic patterns, segment errors across operational groups, and turn the findings into an actionable model-error report.
Learning objectives
- Compute residuals and interpret their sign and magnitude.
- Create residual-versus-prediction, residual-distribution, feature, and time-order plots.
- Recognize patterns associated with bias, heteroscedasticity, nonlinearity, outliers, subgroup effects, and temporal drift.
- Segment MAE, RMSE, bias, and sample counts across business-relevant groups.
- Avoid misleading conclusions from very small segments.
- Use demographic or sensitive-group segmentation only when legally, ethically, and operationally appropriate.
- Write an error-analysis report that identifies where the model performs poorly and what to investigate next.
Table 25.1. Residual-analysis workflow
| Section | Diagnostic question | Main evidence |
|---|---|---|
| 25.1 Residuals | What is the error for each observation? | Signed and absolute residuals |
| 25.2 Residual plots | Does error structure appear visually? | Prediction, feature, distribution, and time plots |
| 25.3 Model problems | What pattern might explain the errors? | Bias, fan shapes, curves, outliers, drift |
| 25.4 Error segmentation | Where is performance weakest? | Grouped MAE, RMSE, bias, and counts |
| Practical lab | Can findings be turned into action? | Structured error-analysis report |
25.1 Residuals
A residual is the observed target minus the model prediction for one observation. It preserves both the size and the direction of the error.
eᵢ = yᵢ − ŷᵢ Positive residual: the model underpredicted. Negative residual: the model overpredicted. |
Reading the sign
Table 25.2. Residual sign and interpretation
| Residual | Meaning | Example |
|---|---|---|
| eᵢ > 0 | Actual value is above the prediction | Actual 120, predicted 100 → residual +20 |
| eᵢ < 0 | Actual value is below the prediction | Actual 80, predicted 100 → residual −20 |
| eᵢ = 0 | Prediction is exact | Actual 100, predicted 100 → residual 0 |
Residuals versus absolute errors
Signed residuals reveal direction and systematic bias. Absolute residuals remove the sign and are useful for ranking observations by error magnitude. Squared residuals emphasize unusually large misses. A strong diagnostic workflow often keeps all three views available.
| IMPORTANT CONVENTION This chapter defines residual = actual − predicted. Some software or teams may use the opposite sign. Always state the convention before interpreting positive and negative residuals. |
PYTHON • Create residual columns import numpy as np |
25.2 Residual plots
Residual plots convert a long list of errors into patterns that can be inspected. No single plot is sufficient. A useful minimum set examines residuals against predictions, the residual distribution, important features, and observation order or time.
Residuals versus predictions
Plot predicted values on the horizontal axis and residuals on the vertical axis. A healthy pattern is often an approximately structureless cloud centered near zero. Curvature, sloped bands, or a widening fan can reveal model problems.
PYTHON • Residuals versus predictions import matplotlib.pyplot as plt |
Residual distribution
A histogram or density-like view shows whether residuals are centered near zero, strongly skewed, heavy-tailed, or dominated by a few extreme values. Symmetry is not required in every application, but a shifted distribution can indicate systematic underprediction or overprediction.
PYTHON • Inspect the residual distribution plt.hist(residuals, bins=30, edgecolor="black") |
Residuals versus important features
A residual plot against an influential feature can expose a missing nonlinear term, an interaction, or a region of feature space with weak coverage. If residuals become increasingly positive as a feature rises, the model may systematically underpredict that range.
PYTHON • Residuals versus one feature feature_name = "usage" |
Time-ordered residuals
When observations have a meaningful order, plot residuals by time. A run of positive residuals, a gradual shift, or increasing volatility can indicate changing conditions that the training data no longer represents.
PYTHON • Plot residuals in time order ordered = results.sort_values("date") |
Table 25.3. Common residual-plot signals
| Plot | What to look for | Possible interpretation |
|---|---|---|
| Residual vs prediction | Curve or slope | Missing nonlinearity or systematic bias |
| Residual vs prediction | Fan / changing spread | Heteroscedasticity |
| Residual histogram | Shift from zero | Overall directional bias |
| Residual vs feature | Pattern within feature ranges | Missing transformation or interaction |
| Residual vs time | Runs, trend, abrupt shift | Temporal drift or process change |
25.3 Detecting model problems
Systematic bias
A model is systematically biased in an evaluation slice when residuals are consistently positive or consistently negative. The mean residual is a simple signed-bias statistic. A positive mean residual means average underprediction under this chapter’s convention.
Mean residual = (1/n) Σᵢ₌₁ⁿ eᵢ |
Heteroscedasticity
Heteroscedasticity means error variance changes across the prediction range or feature space. A classic residual plot shows a narrow band for small predictions that widens into a fan for large predictions. The model may still be useful, but one global RMSE then hides very different uncertainty levels.
Nonlinearity
Curved residual structure suggests the model has not captured the relationship between inputs and target. A linear model fitted to a curved process often alternates between overprediction and underprediction across the feature range.
Outliers
Observations with very large absolute residuals deserve investigation. They may represent valid rare cases, data-quality problems, a missing explanatory variable, a process change, or a population not represented during training. Removing them automatically is usually a mistake.
Unmodeled subgroups
A model can have acceptable global error while performing poorly for one region, product type, customer segment, device class, or other operational subgroup. Group-level bias is especially important because positive and negative residuals from different groups can cancel in the overall mean.
Temporal drift
If residuals worsen or change sign over time, the relationship learned from historical data may be drifting. Drift can originate from price changes, policy changes, seasonality, equipment aging, new customer behavior, or upstream data changes.
Table 25.4. Diagnostic patterns and possible responses
| Observed symptom | Potential cause | Next investigation |
|---|---|---|
| Residual mean far from zero | Directional bias | Calibration, omitted group effect, shifted target |
| Fan-shaped spread | Heteroscedasticity | Transform target, segment uncertainty, alternative loss |
| Curved residual band | Missing nonlinearity | Polynomial/interaction features or nonlinear model |
| Few extreme residuals | Outliers or rare cases | Check data quality and rare-case coverage |
| One group much worse | Unmodeled subgroup | Group features, interactions, sample coverage |
| Errors worsen over time | Temporal drift | Retraining window, monitoring, process changes |
| DIAGNOSIS, NOT PROOF A residual pattern is evidence that deserves investigation; it is not, by itself, proof of a single cause. Several mechanisms can produce similar plots. |
Quantitative residual diagnostics
Plots should be paired with summary statistics. Useful diagnostics include mean residual for direction, MAE for typical magnitude, RMSE for large-error sensitivity, residual quantiles, and the count of observations in each segment.
PYTHON • Summarize residual diagnostics import numpy as np |
25.4 Error segmentation
Error segmentation compares model behavior across meaningful slices of the evaluation data. It answers a different question from global evaluation: not “How good is the model overall?” but “Where is the model strongest, weakest, biased, or insufficiently supported?”
Useful segmentation dimensions
- Customer group — for example new versus established customers, when such a distinction is operationally relevant.
- Region — geographic, service, branch, or market region.
- Product category — product family, service type, equipment class, or application type.
- Time period — month, quarter, season, pre/post process change, or model-age window.
- Target range — low, medium, and high actual values or business-defined bands.
- Demographic or operational group — only where legally permitted, ethically justified, appropriately protected, and relevant to the evaluation purpose.
What to calculate per segment
Table 25.5. Recommended segment diagnostics
| Statistic | Purpose |
|---|---|
| Count | Shows whether the segment has enough observations to interpret |
| Mean residual | Measures directional underprediction or overprediction |
| MAE | Typical absolute error in the segment |
| RMSE | Highlights large segment-level failures |
| P90 absolute error | Describes a high-error tail without relying only on the maximum |
| SMALL-SEGMENT CAUTION A segment with only a few observations can produce an extreme MAE or bias by chance. Always report sample count, and avoid strong conclusions from tiny slices. |
Ethical and legal care
Sensitive-attribute analysis can be important for detecting harmful disparities, but it requires governance. Use only data that may lawfully and ethically be processed for the stated purpose, protect privacy, restrict access, document the justification, and avoid interpreting group differences as causal without evidence.
| RESPONSIBLE PRACTICE Do not create or infer sensitive demographic attributes merely to make a segmentation table. When fairness or compliance analysis is required, follow the applicable organizational, legal, and ethical review process. |
PYTHON • Reusable grouped error summary def segment_metrics(frame, group_col): |
A practical residual-analysis workflow
Table 25.6. End-to-end residual-analysis checklist
| Step | Action | Question answered |
|---|---|---|
| 1 | Confirm evaluation data and residual convention | Are the errors computed correctly? |
| 2 | Review global MAE/RMSE/R² | How large is overall error? |
| 3 | Plot residuals vs predictions | Is there bias, curvature, or changing variance? |
| 4 | Inspect distribution and largest errors | Are tails, skew, or outliers important? |
| 5 | Plot residuals vs important features | Where in feature space does the model fail? |
| 6 | Inspect time order | Is performance drifting? |
| 7 | Segment errors | Which groups or ranges are weak? |
| 8 | Form hypotheses and actions | What should be checked, changed, or monitored? |
What an error-analysis report should contain
- Evaluation dataset, time window, and model version.
- Residual definition and primary global metrics.
- Two to four diagnostic plots with written interpretations.
- Largest-error observations or ranges, reviewed for data quality and representativeness.
- Segment table with counts, bias, MAE, and RMSE.
- Evidence of drift or changing variance when relevant.
- Prioritized hypotheses rather than unsupported causal claims.
- Recommended follow-up actions and monitoring indicators.
| ACTION ORIENTATION A good residual report ends with decisions: collect more data, add a feature, investigate a data pipeline, change model class, recalibrate, retrain, monitor a subgroup, or accept a known limitation. |
Practical lab — Build an error-analysis report
Students create a small operational regression problem with regions, product categories, customer groups, time, and numeric predictors. The target intentionally contains nonlinearity, a subgroup interaction, heteroscedastic noise, and late-period drift. A simple linear model is then analyzed to discover these weaknesses.
Lab objectives
- Generate a reproducible regression dataset with operational segments.
- Train a linear regression pipeline on earlier observations and test on later observations.
- Calculate residuals and global regression metrics.
- Create residual plots for predictions, features, distribution, and time.
- Segment errors by region, product category, customer group, month, and target range.
- Identify the worst-performing slices and propose evidence-based follow-up actions.
- Produce a concise error-analysis report.
Step 1 — Generate the operational dataset
PYTHON • Create reproducible synthetic data import numpy as np |
PYTHON • Create target with structured effects base = 80 + 2.4 * data["usage"] + 5.0 * data["service_level"] |
Step 2 — Create a time-based train/test split
The model trains on months 1–18 and is evaluated on months 19–24. This mimics a real deployment where the future is not available during training and also allows temporal drift to appear in the residual analysis.
PYTHON • Separate historical training data from later test data train_data = data[data["month"] <= 18].copy() |
Step 3 — Train a deliberately simple linear model
PYTHON • Build preprocessing and LinearRegression pipeline from sklearn.compose import ColumnTransformer |
Step 4 — Build the residual table
PYTHON • Store predictions and residual diagnostics results = test_data.copy() |
Step 5 — Compute global metrics
PYTHON • Measure overall test performance from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score |
Step 6 — Create the core residual plots
PYTHON • Residuals versus predictions and usage import matplotlib.pyplot as plt |
PYTHON • Residual distribution and time order plt.hist(results["residual"], bins=30, edgecolor="black") |
Step 7 — Review the largest errors
PYTHON • Inspect high-error observations columns = [ |
| STUDENT TASK Do the largest errors share a region, product, target range, month, or unusually high usage? Write down patterns before changing the model. |
Step 8 — Segment error by operational groups
PYTHON • Compare region and product performance def segment_metrics(frame, group_col): |
PYTHON • Compare customer group and month print("By customer group") |
Step 9 — Segment by target range
PYTHON • Create target bands and compare errors results["target_range"] = pd.qcut( |
Step 10 — Investigate a two-way subgroup
PYTHON • Find weak region-product combinations two_way = results.groupby(["region", "product"]).agg( |
| EXPECTED DISCOVERY Because the synthetic process contains an extra South × Product C effect that the linear model does not explicitly represent, that combination should show unusually poor or biased residual behavior. |
Step 11 — Build a concise error-analysis report
Table 25.7. Required lab report structure
| Report section | What students should write |
|---|---|
| Global performance | MAE, RMSE, R², mean residual, evaluation period |
| Residual plots | Two or more visible patterns and what they may indicate |
| Largest errors | Shared characteristics or data-quality concerns |
| Segmentation | Worst groups with count, bias, MAE, and RMSE |
| Temporal behavior | Whether errors improve, worsen, or shift over months |
| Hypotheses | Plausible causes supported by evidence, not causal claims |
| Actions | Feature/model/data/monitoring changes to test next |
Lab questions
1. Is the overall mean residual close to zero? What does its sign imply?
2. Does residual spread remain constant across predicted values and usage?
3. Which observations have the largest absolute errors, and what do they have in common?
4. Which region has the highest MAE? Does it also have the largest signed bias?
5. Which product category performs worst, and is its sample count large enough for interpretation?
6. Does performance change across months 19–24? What evidence suggests temporal drift?
7. Which target range has the largest error? What could explain that pattern?
8. Which region-product combination is weakest? What model change could represent that interaction?
9. Write three prioritized recommendations for the next modeling iteration.
Interpreting findings without overclaiming
Residual analysis is exploratory diagnosis. If the South–Product C segment has a large positive residual, the evidence shows that the model underpredicts that segment on the evaluation data. It does not automatically show why. The cause might be an interaction, sample imbalance, a missing variable, a data pipeline issue, or a process difference.
From finding to follow-up test
Table 25.8. Turning residual evidence into experiments
| Finding | Reasonable next test |
|---|---|
| Curved residuals vs usage | Add polynomial/interaction feature or compare a nonlinear model |
| High error in one product-region pair | Add explicit interaction or inspect coverage/data quality |
| Errors grow with target | Consider transformation, weighted loss, or target-range modeling |
| Bias shifts after a date | Check upstream changes and retrain with recent data |
| Tiny group has extreme MAE | Collect more observations before a strong conclusion |
| GOOD SCIENTIFIC HABIT Change one hypothesis at a time when possible, evaluate on protected data, and verify that fixing one segment does not degrade other important groups. |
Chapter summary
- A residual is the observed target minus the predicted target: eᵢ = yᵢ − ŷᵢ.
- Residual-versus-prediction plots can reveal bias, curvature, and changing error variance.
- Residual distributions expose skew, heavy tails, and extreme prediction failures.
- Feature and time-ordered residual plots help detect missing structure and temporal drift.
- Global metrics can hide subgroup weaknesses because errors from different segments are aggregated.
- Segment analysis should report sample count together with bias, MAE, RMSE, or tail statistics.
- Sensitive-group analysis requires a legitimate purpose, lawful and ethical data use, privacy safeguards, and careful interpretation.
- Residual patterns generate hypotheses; they do not by themselves prove a causal explanation.
- The goal of residual analysis is an actionable error report that guides model, data, and monitoring improvements.
| FINAL TAKEAWAY Regression evaluation becomes much more useful when the question changes from “What is the model’s score?” to “Where, how, and under what conditions does the model make mistakes?” |
Knowledge check
1. What does a positive residual mean under the convention used in this chapter?
2. What residual-plot pattern is commonly associated with heteroscedasticity?
3. Why can a near-zero global mean residual coexist with strong subgroup bias?
4. What should accompany every segment-level error metric?
5. How can time-ordered residuals reveal temporal drift?
6. Why should large-residual observations be investigated rather than automatically deleted?