Chapter 24 — Regression Metrics
MAE • MSE • RMSE • R² • MAPE • Metric Selection
Measuring prediction error from several complementary perspectives
| BRIDGE FROM CHAPTER 21 A regression model can produce accurate-looking predictions while still failing on the errors that matter most. This chapter turns residuals into quantitative evaluation criteria and connects each metric to a different operational question. |
Chapter overview
Regression evaluation is not summarized by one universal score. Mean absolute error measures typical absolute deviation, squared-error metrics emphasize large mistakes, R² compares a model with a mean-prediction baseline, and percentage metrics express error relative to the target magnitude. Each view can support a different conclusion about the same predictions.
This chapter develops the interpretation, strengths, and limitations of the most common regression metrics. The emphasis is not only on computing a number, but on choosing a metric that matches the target distribution and the real cost of prediction errors.
Learning objectives
- Compute and interpret MAE in the original target units.
- Explain why MSE gives disproportionate weight to large residuals.
- Interpret RMSE in target units and compare it with MAE.
- Explain R² as improvement over a mean-prediction baseline and interpret negative R².
- Use MAPE carefully and identify zero and near-zero target problems.
- Match a regression metric to business cost, outliers, scale, and relative-error requirements.
- Explain why two metrics can rank the same models differently.
- Evaluate a regression model with several metrics in scikit-learn.
Table 24.1. Chapter structure
| Section | Main question | Core idea |
|---|---|---|
| 24.1 MAE | How large is a typical absolute error? | Average |residual| in target units |
| 24.2 MSE | How strongly should large errors be punished? | Average squared residual |
| 24.3 RMSE | Can squared-error emphasis be returned to target units? | Square root of MSE |
| 24.4 R² | How much better is the model than predicting the mean? | Relative reduction in squared error |
| 24.5 MAPE | How large is the error relative to the actual value? | Average absolute percentage error |
| 24.6 Selection | Which metric fits the decision? | Align metric with costs and data characteristics |
24.1 Mean absolute error
For each observation, the residual is the difference between the actual target and the prediction. Mean absolute error removes the residual sign, adds the error magnitudes, and averages them over all observations.
MAE = (1/n) Σᵢ₌₁ⁿ |yᵢ − ŷᵢ| yᵢ = observed target; ŷᵢ = predicted target; n = number of observations |
Interpretation in target units
MAE has the same unit as the target. If house prices are measured in thousands of euros and MAE = 18, the model is wrong by about 18 thousand euros per prediction on average, when error direction is ignored. This direct unit interpretation is one of MAE’s main advantages.
Robustness compared with squared errors
MAE grows linearly with the size of an error. Doubling an error doubles its contribution. A very large residual matters, but it is not squared. Consequently, MAE is usually less dominated by a few extreme prediction failures than MSE or RMSE.
Table 24.2. Linear versus squared error growth
| Residual magnitude | MAE contribution | Squared-error contribution |
|---|---|---|
| 2 | 2 | 4 |
| 5 | 5 | 25 |
| 10 | 10 | 100 |
| 20 | 20 | 400 |
| INTERPRETATION RULE Use MAE when a roughly linear cost per unit of error is reasonable and when you want a metric that remains understandable in the original target units. |
PYTHON • Compute MAE from arrays and with scikit-learn import numpy as np |
24.2 Mean squared error
Mean squared error squares every residual before averaging. The squaring operation removes the sign and makes large errors increase much faster than small errors.
MSE = (1/n) Σᵢ₌₁ⁿ (yᵢ − ŷᵢ)² |
Higher penalty for large errors
Because residuals are squared, a prediction error of 20 contributes sixteen times as much as an error of 5. MSE therefore gives strong influence to large mistakes. This is useful when large errors are disproportionately costly, but it also makes the metric sensitive to outliers and unusual observations.
MSE is expressed in squared target units. If the target is measured in dollars, MSE is measured in dollars squared. That mathematical property is convenient for optimization, but it makes direct business interpretation less intuitive than MAE or RMSE.
| OPTIMIZATION CONNECTION Ordinary least squares chooses coefficients by minimizing a sum of squared residuals. MSE therefore aligns naturally with the training objective of standard linear regression. |
PYTHON • Compute MSE and compare two error patterns import numpy as np |
| KEY OBSERVATION Two models can have the same MAE but different MSE. The model that concentrates error into one large failure receives a much larger squared-error penalty. |
24.3 Root mean squared error
Root mean squared error takes the square root of MSE. The square root restores the original target unit while preserving the squared-error emphasis on large residuals.
RMSE = √MSE = √[(1/n) Σᵢ₌₁ⁿ (yᵢ − ŷᵢ)²] |
Interpretation in target units
If sales are measured in units and RMSE = 42, the error score is also expressed in units. Unlike MAE, however, RMSE is not simply the average absolute miss; the squaring step makes large residuals disproportionately influential before the square root is applied.
Sensitivity to outliers
When a few large errors occur, RMSE often becomes noticeably larger than MAE. The gap between RMSE and MAE can therefore provide a useful clue that the error distribution contains heavy tails or substantial outliers.
Table 24.3. Interpreting MAE and RMSE together
| Situation | MAE behavior | RMSE behavior | Interpretation |
|---|---|---|---|
| Errors similar in size | Close to RMSE | Close to MAE | Residuals are relatively uniform |
| A few very large errors | Rises moderately | Rises strongly | Large failures dominate squared error |
| Need direct typical miss | Very natural | Less direct | MAE often easier to explain |
| Large failures are costly | May under-emphasize them | Emphasizes them | RMSE can match the objective better |
PYTHON • Compute RMSE from sklearn.metrics import mean_squared_error |
24.4 Coefficient of determination
R² evaluates squared-error performance relative to a simple baseline that always predicts the mean of the observed target values. It is therefore a relative goodness-of-fit measure rather than an error expressed in target units.
R² = 1 − [Σ(yᵢ − ŷᵢ)² / Σ(yᵢ − ȳ)²] ȳ is the mean of the observed target values |
Explained variation
An R² of 1 indicates perfect predictions on the evaluated data. R² = 0 means the model performs no better, in squared-error terms, than always predicting the evaluation-set mean. Values between 0 and 1 indicate partial improvement over that baseline.
Negative R²
R² can be negative. A negative value means the model has a larger sum of squared errors than the mean-prediction baseline on the evaluated data. This can happen when a model generalizes poorly, when the test distribution changes, or when an unsuitable model is applied.
Table 24.4. Interpreting R²
| R² value | Typical interpretation |
|---|---|
| 1.00 | Perfect predictions on the evaluated observations |
| 0.70 | Substantial reduction in squared error relative to the mean baseline |
| 0.00 | No squared-error improvement over predicting the mean |
| < 0 | Worse than the mean-prediction baseline |
Limitations of R²
- R² does not express the typical error in the target units.
- A high R² does not prove that errors are operationally small enough.
- R² does not establish causality or validate modeling assumptions.
- R² can be misleading when compared across datasets with very different target variability.
- Adding predictors can increase training R² even when generalization does not improve.
| USE AN ERROR METRIC TOO Report R² together with MAE or RMSE whenever stakeholders need to know both relative fit and the practical size of prediction errors. |
PYTHON • Compute R² and compare against the mean baseline import numpy as np |
24.5 Mean absolute percentage error
Mean absolute percentage error scales each absolute residual by the magnitude of the corresponding actual target. It is attractive because it can be communicated as an average relative error rather than an error in a domain-specific unit.
MAPE = (100/n) Σᵢ₌₁ⁿ |(yᵢ − ŷᵢ) / yᵢ| Often reported as a percentage; software APIs may return a proportion that must be multiplied by 100 |
Percentage interpretation
A MAPE of 8% suggests that the absolute prediction error is about 8% of the actual value on average. Because the measure is scale-free, it can be useful when relative error matters more than absolute error or when results must be compared across units with different magnitudes.
Problems with zero and near-zero targets
MAPE divides by the actual target. If an actual value is zero, the percentage error is undefined. If the actual value is very close to zero, even a small absolute error can create an enormous percentage error and dominate the average.
MAPE also treats relative errors asymmetrically in some situations. For example, overprediction and underprediction around the same scale do not always have symmetric percentage behavior. These properties make MAPE inappropriate for many targets that cross zero, contain zeros, or frequently approach zero.
Table 24.5. When MAPE is and is not suitable
| Target situation | MAPE suitability | Reason |
|---|---|---|
| Strictly positive and comfortably away from zero | Often useful | Relative errors are numerically stable |
| Contains exact zeros | Poor | Division by zero is undefined |
| Contains very small positive values | Risky | Small denominators create huge percentages |
| Contains negative values | Usually inappropriate | Percentage interpretation becomes difficult |
| Absolute business cost matters | Secondary metric | Unit-based MAE/RMSE is often more meaningful |
PYTHON • Compute MAPE and inspect near-zero behavior import numpy as np |
| SCIKIT-LEARN DETAIL mean_absolute_percentage_error returns a relative value: 0.08 represents 8%. Multiplying by 100 converts it to percentage form for reporting. |
24.6 Selecting the appropriate metric
Metric selection should be decided from the decision problem, not from whichever score makes a model look best. A useful metric reflects the cost structure of errors, the target scale, the data distribution, and the way predictions will be used.
Business cost
If every unit of error has approximately the same cost, MAE is a natural choice. If very large mistakes trigger disproportionate losses, RMSE or MSE can reflect that risk more strongly. In many real systems, a custom cost function is more faithful than any standard symmetric metric.
Target distribution and outliers
Heavy-tailed targets and unusual observations can make squared-error metrics unstable. MAE is more resistant to a small number of extreme residuals, whereas RMSE intentionally makes those residuals prominent.
Relative versus absolute errors
An absolute error of 10 may be negligible when the target is 10,000 but severe when the target is 20. Percentage metrics capture relative magnitude, while MAE and RMSE preserve the original unit. The correct perspective depends on the task.
Asymmetric errors
Standard MAE, MSE, and RMSE treat an underprediction and an overprediction of the same magnitude equally. Many applications do not. Under-forecasting inventory can lose sales, while over-forecasting can create storage cost. Such settings may require an asymmetric custom loss or separate analysis of positive and negative residuals.
Table 24.6. Choosing a regression metric
| Evaluation need | Useful metric | Why |
|---|---|---|
| Typical absolute error in real units | MAE | Direct, robust, easy to communicate |
| Strongly punish large failures | MSE / RMSE | Squared errors emphasize extremes |
| Error in real units + large-error emphasis | RMSE | Same target unit after square root |
| Relative improvement over mean baseline | R² | Measures squared-error improvement |
| Relative error as a percentage | MAPE | Scale-free when targets stay away from zero |
| Different cost for over- and underprediction | Custom asymmetric metric | Standard metrics are symmetric |
| MODEL SELECTION PRINCIPLE Choose the primary metric before comparing models. Otherwise, metric shopping can turn evaluation into a post-hoc search for the most favorable score. |
Practical lab — Evaluate regression predictions from several perspectives
In this lab, students train a linear regression model on the Diabetes dataset, evaluate it on a protected test set, and interpret five metrics. The lab then introduces a deliberately large prediction error to show why MAE and RMSE can react differently.
Lab objectives
- Create a reproducible train/test split.
- Train a baseline linear regression model.
- Compute MAE, MSE, RMSE, R², and MAPE.
- Inspect residuals and compare absolute and squared-error conclusions.
- Demonstrate the effect of one unusually large prediction error.
- Explain why no single metric completely describes regression performance.
Step 1 — Load and split the dataset
PYTHON • Load Diabetes data and create a protected test set from sklearn.datasets import load_diabetes |
Step 2 — Train the regression model
PYTHON • Fit LinearRegression and predict the test set from sklearn.linear_model import LinearRegression |
Step 3 — Calculate the metrics
PYTHON • Calculate MAE, MSE, RMSE, R², and MAPE from sklearn.metrics import ( |
| STUDENT INTERPRETATION Do not write only “lower is better” or “higher is better.” Explain the unit, reference point, and error behavior represented by each metric. |
Step 4 — Inspect residual behavior
PYTHON • Summarize residuals and large errors import numpy as np |
Step 5 — Visualize residuals
PYTHON • Plot actual vs predicted values and residuals import matplotlib.pyplot as plt |
Step 6 — Stress test with one large prediction error
The next experiment leaves all but one prediction unchanged. One prediction is moved far from its actual value. Students then compare how much MAE and RMSE change.
PYTHON • Create one large error and recompute MAE/RMSE import numpy as np |
| EXPECTED CONCLUSION Both metrics get worse, but RMSE should react more strongly because the deliberately large residual is squared before averaging. |
Step 7 — Compare with a mean-prediction baseline
PYTHON • Relate R² to the mean baseline import numpy as np |
Step 8 — Explain the metrics in words
Table 24.7. Lab interpretation checklist
| Metric | Student explanation should include |
|---|---|
| MAE | Typical absolute prediction miss and target unit |
| MSE | Squared unit and strong penalty for large residuals |
| RMSE | Target unit plus sensitivity to large residuals |
| R² | Performance relative to the mean-prediction baseline |
| MAPE | Relative error percentage and zero/near-zero caution |
Lab questions
1. Which metric is easiest to explain directly to a domain stakeholder? Why?
2. Is RMSE noticeably larger than MAE? What does that suggest about the residual distribution?
3. How did the deliberately large prediction error change MAE and RMSE?
4. What does the model’s R² say relative to the mean-prediction baseline?
5. Would MAPE remain trustworthy if the target frequently approached zero? Explain.
6. If underprediction cost twice as much as overprediction, which standard metric would fully represent that requirement?
7. Choose one primary metric for this dataset and justify the choice in two or three sentences.
Extension — When metrics rank models differently
A model can make many modest errors while another makes mostly small errors plus a few severe failures. MAE may prefer the second model because its typical absolute error is small, while RMSE may prefer the first because it avoids catastrophic residuals. Neither conclusion is automatically correct: the preferred model depends on the real cost of those failures.
Table 24.8. Why metric rankings can differ
| Model pattern | MAE may favor | RMSE may favor | Operational question |
|---|---|---|---|
| Many moderate errors vs. few severe errors | Few severe-error model | Moderate-error model | How costly are rare large failures? |
| Good absolute accuracy on large targets vs. good percentages on small targets | Absolute-accuracy model | Depends on large errors | Do absolute or relative errors matter? |
| High R² but large unit errors | May reveal large misses | May reveal large misses | Is relative fit enough for deployment? |
Chapter summary
- MAE is the average absolute residual and is expressed in target units.
- MSE squares residuals, making large prediction failures disproportionately influential.
- RMSE preserves squared-error sensitivity while returning the score to target units.
- R² measures squared-error improvement relative to predicting the target mean and can be negative.
- MAPE expresses relative error but is unstable or undefined for zero and near-zero targets.
- Metric selection should reflect operational costs, outliers, scale, and whether absolute or relative error matters.
- Multiple metrics can legitimately produce different conclusions because they emphasize different error properties.
| FINAL TAKEAWAY A regression metric is a definition of what counts as a bad prediction. Choosing the metric is therefore part of defining the problem, not merely a reporting step. |
Knowledge check
1. Why does RMSE usually increase more than MAE when one very large error is introduced?
2. What does R² = 0 mean on an evaluation set?
3. How can R² become negative?
4. Why can MAPE become extremely large for a small absolute error?
5. When would MAE be more appropriate than RMSE?
6. Give one example of a problem that requires an asymmetric cost rather than a standard regression metric.