Chapter 19 — Linear Regression
Linear Model • Coefficients • Residuals • Assumptions • Polynomial Features
Predicting continuous outcomes with an interpretable least-squares model
Chapter overview
Linear regression is one of the most important baseline models in supervised machine learning. It predicts a continuous target by representing the expected response as a weighted sum of input features. Despite its simple form, linear regression provides a foundation for understanding prediction, model parameters, residuals, loss functions, regularization, diagnostics, and the distinction between association and causation.
The strength of linear regression is not only prediction. Its coefficients can often be interpreted directly, its assumptions can be investigated through residual diagnostics, and its failure modes are instructive. When the relationship between the inputs and target is not perfectly linear, polynomial features and interaction terms can expand the model while preserving a linear relationship in the learned coefficients.
This chapter emphasizes both mathematical understanding and practical workflow. Students will fit linear models, interpret intercepts and coefficients, quantify errors with residuals and squared loss, inspect diagnostic plots, recognize multicollinearity and outlier problems, and build polynomial regression pipelines without leaking information from the test set.
| BRIDGE FROM CHAPTER 18 Support Vector Machines focused on classification boundaries. Chapter 19 begins the regression part of the course: instead of predicting a class label, the model predicts a numerical value on a continuous scale. |
Learning objectives
- Write and interpret the multiple linear regression equation.
- Distinguish the intercept, coefficients, predictions, residuals, and sum of squared errors.
- Explain least squares as the criterion used by ordinary linear regression.
- Recognize the main practical assumptions behind linear regression and diagnose important violations.
- Interpret coefficient signs and magnitudes while respecting feature units and scaling.
- Explain why predictive coefficients do not automatically have a causal meaning.
- Recognize multicollinearity and understand how it can destabilize coefficient estimates.
- Use residual plots to examine linearity, changing variance, unusual observations, and systematic structure.
- Generate polynomial and interaction features to model nonlinear relationships.
- Compare training and test performance to detect overfitting as polynomial degree increases.
- Train and evaluate a complete linear regression model in scikit-learn.
- Analyze residuals and coefficients in a reproducible practical lab.
Table 19.1. Chapter structure
| Section | Focus | Student outcome |
|---|---|---|
| 19.1 | Linear model | Connect a continuous prediction to an intercept plus weighted feature values. |
| 19.2 | Model components | Compute and interpret predictions, residuals, and squared error. |
| 19.3 | Assumptions and interpretation | Use diagnostics to identify situations where a linear model may be unreliable. |
| 19.4 | Coefficient interpretation | Explain sign, magnitude, units, scaling effects, and causal limits. |
| 19.5 | Polynomial features | Represent curvature and feature interactions while controlling overfitting. |
| Lab | Model + residual analysis | Train a model, evaluate predictions, inspect residuals, and interpret coefficients. |
19.1 Linear model
A regression problem has a numerical target. Examples include predicting a house price, electricity demand, delivery time, temperature, sales volume, or a laboratory measurement. Linear regression assumes that the prediction can be expressed as an additive weighted combination of the available features.
ŷ = β₀ + β₁x₁ + β₂x₂ + ⋯ + βₚxₚ Multiple linear regression model |
Table 19.2. Meaning of the symbols
| Symbol | Meaning | Example |
|---|---|---|
| ŷ | Predicted value produced by the model | Predicted monthly sales |
| β₀ | Intercept: baseline prediction when all feature values equal zero | Baseline sales level |
| βⱼ | Coefficient attached to feature xⱼ | Change associated with one unit of advertising |
| xⱼ | Observed value of feature j | Advertising budget in thousands |
| p | Number of input features | Age, income, usage, tenure → p = 4 |
From simple to multiple regression
With one feature, the model defines a straight line. With two features, the model defines a plane. With more than two features, it defines a hyperplane in a higher-dimensional feature space. The mathematical structure remains the same: each feature contributes its value multiplied by a learned coefficient.
FEATURES | → | WEIGHTED SUM | → | INTERCEPT | → | PREDICTION |
| KEY IDEA The word linear refers to the way the coefficients enter the model. A model can use transformed inputs such as x² and still be linear in its coefficients. |
How the coefficients are learned
Ordinary least squares chooses coefficient values that make the model predictions as close as possible to the observed targets according to squared error. For each observation, the difference between the observed target and prediction is the residual. Squaring the residuals prevents positive and negative errors from canceling and gives greater weight to large errors.
SSE = ∑ᵢ (yᵢ − ŷᵢ)² Ordinary least squares minimizes the sum of squared errors over the training data |
Minimizing squared error has useful mathematical properties and leads to an efficient solution. However, because large residuals are squared, unusual observations can have a strong influence on the fitted line. This is one reason residual diagnostics and outlier checks are important.
PYTHON • Fit a first linear regression model from sklearn.datasets import load_diabetes |
| PRACTICE NOTE LinearRegression in scikit-learn fits ordinary least squares by default. Feature scaling is not required to obtain the least-squares predictions, although scaling changes coefficient units and can make coefficient magnitudes easier to compare. |
Prediction as a deterministic calculation
After training, prediction is simply a calculation. For a new row of feature values, the model multiplies each value by its coefficient, adds all contributions, and then adds the intercept. There is no iterative search at prediction time.
ŷᵢ = β₀ + ∑ⱼ₌₁ᵖ βⱼxᵢⱼ Prediction for observation i |
PYTHON • Generate predictions on unseen test data y_pred = model.predict(X_test) |
19.2 Model components
A useful way to understand linear regression is to separate the fitted model into five core components: the intercept, coefficients, predicted values, residuals, and the aggregate squared-error criterion. Each component answers a different question about how the model works and how well it fits the observed data.
Intercept
The intercept β₀ is the model prediction when every feature equals zero. Whether that value has a meaningful real-world interpretation depends on the feature definitions. If zero is outside the realistic range of the data, the intercept still plays an important mathematical role but may not represent a plausible observation.
| INTERPRET WITH CONTEXT Do not force a substantive interpretation of the intercept when the all-zero feature combination is impossible or far outside the observed data. |
Coefficients
Each coefficient βⱼ controls the contribution of one feature. Holding all other features fixed, increasing xⱼ by one unit changes the prediction by βⱼ units. This “holding other features constant” condition is essential in multiple regression because several features contribute simultaneously.
Predicted value
The predicted value ŷᵢ is the model output for observation i. Prediction quality is evaluated by comparing these fitted or test predictions with the corresponding observed targets yᵢ.
Residual
eᵢ = yᵢ − ŷᵢ Residual = observed target − predicted target |
A positive residual means the observed value is above the prediction, so the model underpredicted that case. A negative residual means the observed value is below the prediction, so the model overpredicted that case. Residuals are not merely errors to summarize; their pattern is a diagnostic signal.
Table 19.3. Reading residuals
| Observed y | Predicted ŷ | Residual y − ŷ | Interpretation |
|---|---|---|---|
| 200 | 180 | +20 | Underprediction by 20 units |
| 150 | 165 | −15 | Overprediction by 15 units |
| 120 | 120 | 0 | Exact prediction for this observation |
Sum of squared errors
The sum of squared errors aggregates training residuals into one nonnegative quantity. Ordinary least squares finds the intercept and coefficients that minimize this value. A smaller SSE means the fitted predictions are closer to the training targets in squared-error terms, but SSE by itself depends on the number of observations and target scale.
SSE = e₁² + e₂² + ⋯ + eₙ² Equivalent expression using residuals |
Related evaluation metrics
Table 19.4. Common regression metrics
| Metric | Definition / idea | Interpretation |
|---|---|---|
| MSE | Mean of squared residuals | Lower is better; strongly penalizes large errors. |
| RMSE | Square root of MSE | Lower is better; expressed in the same unit as the target. |
| MAE | Mean absolute residual magnitude | Lower is better; less dominated by very large errors than MSE. |
| R² | Fraction of target variance explained relative to a mean-only baseline | 1 is perfect; 0 matches the mean baseline; negative values are possible on test data. |
PYTHON • Compute residuals and regression metrics import numpy as np |
| METRIC DISCIPLINE Evaluate a model on observations that were not used to fit its coefficients. Training error describes fit to known data; test error estimates predictive performance on unseen data. |
A residual table for inspection
Before plotting, it is often useful to create a small table containing observed values, predictions, and residuals. Sorting by absolute residual highlights the cases the model predicts most poorly and can reveal data-quality issues, rare subgroups, nonlinear behavior, or influential observations.
PYTHON • Build a residual inspection table import pandas as pd |
19.3 Assumptions and interpretation
Linear regression is often described through a list of assumptions. In machine learning, these assumptions should be treated as diagnostic guides rather than as a ritual checklist. Some assumptions are especially important when interpreting coefficients or constructing statistical confidence intervals, while predictive performance can still be acceptable under moderate departures. The central practical question is whether the residuals reveal systematic structure that the model is failing to represent.
Approximate linear relationship
The expected target should be reasonably approximated by an additive linear combination of the features included in the model. If residuals show a curved pattern against fitted values or against an important feature, the relationship may require transformation, polynomial terms, interaction terms, or a different model family.
Table 19.5. Typical diagnostic signals
| Pattern | Possible meaning | Possible response |
|---|---|---|
| Curvature in residual plot | The mean relationship is not adequately linear | Transform a feature, add polynomial terms, or try a nonlinear model. |
| Fan / funnel shape | Residual variance changes with prediction level | Transform the target, reconsider features, or use methods appropriate for changing variance. |
| Clusters or bands | Unmodeled groups, categories, or time effects | Add relevant variables or use a grouped/time-aware model. |
| A few extreme residuals | Outliers, data errors, rare cases, or missing structure | Validate records and assess influence rather than deleting automatically. |
Independent observations
The usual linear regression framework assumes that observations provide independent information. This assumption is often violated by repeated measurements from the same person, multiple rows from the same customer, spatially neighboring measurements, or time-series observations. Dependence also affects data splitting: rows from the same group should not be carelessly divided between training and test sets.
| LEAKAGE CONNECTION If multiple rows belong to the same entity, use group-aware splitting. If observations are ordered in time, preserve temporal order. A random split can make performance look unrealistically strong. |
Constant error variance
Constant error variance, also called homoscedasticity, means that the vertical spread of residuals is roughly similar across the prediction range. A clear increase or decrease in residual spread is called heteroscedasticity. It can indicate that the target becomes intrinsically harder to predict at certain levels or that the chosen scale is unsuitable.
Residual behavior
For a well-specified predictive linear model, residuals should be centered around zero without an obvious systematic pattern. Approximate residual normality is particularly relevant to classical statistical inference; it is not a strict prerequisite for generating least-squares predictions. In machine-learning practice, patterns, extreme tails, and instability often matter more than forcing a perfect bell shape.
PYTHON • Plot residuals against fitted values import matplotlib.pyplot as plt |
Desired pattern: a roughly horizontal cloud centered around zero. Curvature suggests missing nonlinear structure; a funnel shape suggests changing variance; isolated points deserve investigation.
Multicollinearity
Multicollinearity occurs when predictors contain strongly overlapping information. The model may still predict well, but individual coefficients can become unstable because several correlated features compete to explain the same variation in the target. Small changes to the data can then produce large changes in coefficient values or even coefficient signs.
Table 19.6. Prediction and interpretation under multicollinearity
| Situation | Prediction | Coefficient interpretation |
|---|---|---|
| Features weakly correlated | Usually stable if relationship is appropriate | Individual effects are easier to distinguish. |
| Features strongly correlated | Can remain accurate | Individual coefficients may be unstable or counterintuitive. |
| Nearly duplicate features | Little predictive benefit from redundancy | Attribution between the duplicate features becomes unreliable. |
| IMPORTANT DISTINCTION A model can have useful predictive accuracy while its individual coefficients are unstable. Predictive performance and coefficient interpretability are related but different objectives. |
Influence of outliers
An outlier is an observation with an unusual target value, unusual feature values, or both. Because least squares squares the residual, a point with a large residual can contribute disproportionately to the objective. A high-leverage point with unusual predictor values can also pull the fitted relationship toward itself. Influential observations therefore deserve verification and sensitivity analysis.
- Data error: correct the source record if the value is demonstrably wrong.
- Valid rare case: do not remove it automatically; decide whether the deployed model must handle similar cases.
- Model mismatch: an extreme residual may reveal a missing nonlinear term, interaction, group effect, or omitted feature.
- Sensitivity: refit with and without a questionable point to see whether conclusions change substantially.
A practical diagnostic workflow
1. Start with held-out performance: Evaluate MAE, RMSE, and R² on validation or test data.
2. Plot predictions versus observations: Check whether predictions follow the overall target range and whether errors grow at the extremes.
3. Plot residuals versus fitted values: Look for curvature, funnels, clusters, and isolated observations.
4. Inspect the largest absolute residuals: Verify data quality and look for missing structure.
5. Review feature relationships: Check duplicate or highly correlated variables before interpreting individual coefficients.
6. Compare alternatives: Try transformations, polynomial features, interactions, or another model family and validate the change.
PYTHON • Observed versus predicted diagnostic plot plt.figure(figsize=(5.5, 5.5)) |
19.4 Coefficient interpretation
Linear regression is attractive because its parameters are directly connected to the prediction formula. However, coefficient interpretation must respect the units of the features, the presence of other features in the model, feature transformations, scaling, multicollinearity, and the observational nature of the data.
Positive and negative effects
A positive coefficient means that, holding the other modeled features constant, larger values of that feature are associated with larger predictions. A negative coefficient means larger feature values are associated with smaller predictions. A coefficient near zero means the model assigns little linear contribution per unit of that feature, but this does not prove that the feature is irrelevant in every possible model.
Table 19.7. Reading coefficient signs
| Coefficient | Model statement | Caution |
|---|---|---|
| βⱼ > 0 | Increasing xⱼ by one unit increases ŷ by βⱼ units, other features held fixed. | Association can be distorted by correlated predictors. |
| βⱼ < 0 | Increasing xⱼ by one unit decreases ŷ by |βⱼ| units, other features held fixed. | A negative sign does not prove a harmful causal effect. |
| βⱼ ≈ 0 | The fitted model uses little linear contribution per unit of xⱼ. | Nonlinear or interaction effects may still exist. |
Feature units matter
Coefficient magnitude cannot be compared fairly across features measured on very different scales. A coefficient of 0.002 for income measured in currency units may represent a large effect over thousands of units, while a coefficient of 5 for a proportion measured from 0 to 1 may describe a smaller real-world range.
Δŷ = βⱼ · Δxⱼ A coefficient must be interpreted together with a meaningful change in the feature |
| UNIT-AWARE INTERPRETATION Instead of asking only whether a coefficient is numerically large, ask: What prediction change corresponds to a realistic change in this feature? |
Effect of scaling
Standardization changes the feature unit from the original measurement to standard deviations. The model predictions can remain equivalent for ordinary linear regression when scaling is performed consistently, but the coefficient values change because one unit now represents one standard deviation instead of one original unit.
Table 19.8. Raw versus standardized coefficients
| Model input | Coefficient meaning | Use case |
|---|---|---|
| Original units | Target change for a one-unit feature increase | Business/domain interpretation in native units. |
| Standardized features | Target change for a one-standard-deviation feature increase | Rough comparison of feature contributions on a common feature scale. |
PYTHON • Compare raw and standardized coefficient scales from sklearn.pipeline import make_pipeline |
Limits of causal interpretation
A regression coefficient describes a conditional association in the fitted model. It does not, by itself, establish that changing the feature will cause the target to change. Confounding variables, reverse causality, measurement choices, selection effects, and omitted variables can all produce associations that are predictive but not causal.
| PREDICTION IS NOT CAUSATION “The model predicts higher outcomes when x is higher, holding included variables fixed” is a predictive statement. “Increasing x will cause the outcome to rise” is a causal claim and requires a suitable causal design or assumptions beyond ordinary predictive regression. |
Inspecting coefficients programmatically
A coefficient table should include feature names and, when possible, domain units. Sorting by absolute magnitude is useful for inspection, but only after accounting for scale. For correlated features, coefficient magnitude should not be interpreted as a reliable standalone measure of global feature importance.
PYTHON • Create a coefficient table from sklearn.datasets import load_diabetes |
Interpreting a coefficient carefully
1. State the feature and unit: Identify what a one-unit increase actually means.
2. State the sign: Explain whether the fitted association raises or lowers the prediction.
3. State the magnitude: Translate the coefficient into target units for a realistic feature change.
4. Condition on the other features: Remember that the coefficient is a partial association in the multivariable model.
5. Check scaling and transformations: A coefficient for x², log(x), or a standardized variable has a different interpretation.
6. Avoid causal language: Unless the analysis is supported by a causal design, describe association or prediction rather than intervention effects.
19.5 Polynomial features
A straight-line relationship is often too restrictive. Polynomial feature expansion allows a linear regression estimator to represent curvature by adding transformed versions of the original inputs. The estimator is still linear in its learned coefficients even though the prediction becomes nonlinear as a function of the original feature values.
Representing nonlinear relationships
ŷ = β₀ + β₁x + β₂x² Quadratic regression: linear in β₀, β₁, β₂ but curved in x |
For one feature, a degree-2 expansion creates x and x². A degree-3 expansion adds x³. With multiple features, polynomial expansion can also create cross-products such as x₁x₂, which represent interactions between features.
Polynomial degree
Table 19.9. Effect of polynomial degree
| Degree | Typical representation | Flexibility / risk |
|---|---|---|
| 1 | x₁, x₂, … | Ordinary linear model; lowest flexibility. |
| 2 | x₁, x₂, x₁², x₁x₂, x₂², … | Captures curvature and pairwise interactions. |
| 3 | Adds cubic terms and more interactions | More flexible, but feature count and overfitting risk increase rapidly. |
| High degree | Many powers and interactions | Can fit training noise and behave erratically outside observed regions. |
Interaction terms
An interaction means that the contribution of one feature depends on the value of another. A term x₁x₂ allows the slope associated with x₁ to change as x₂ changes. Interactions are useful when domain reasoning suggests combined effects, but indiscriminately generating many interactions can greatly expand the feature space.
ŷ = β₀ + β₁x₁ + β₂x₂ + β₃x₁x₂ The interaction coefficient β₃ modifies the joint contribution of x₁ and x₂ |
Using PolynomialFeatures safely
In scikit-learn, PolynomialFeatures creates the expanded design matrix. The transformer should be placed inside a Pipeline so that feature generation is part of the fitted workflow. A pipeline also makes it easier to cross-validate the degree and to add scaling or regularization later.
PYTHON • Fit a quadratic regression pipeline from sklearn.pipeline import make_pipeline |
Feature-space growth
Polynomial expansion can create many more columns than the original dataset. The growth is especially strong when both degree and number of features increase. This raises computation, memory use, multicollinearity, and overfitting risk. Therefore, degree should be treated as a hyperparameter rather than increased until training error becomes tiny.
| OVERFITTING WARNING A high-degree polynomial can pass close to many training points yet generalize poorly. Choose complexity using validation or cross-validation, not by minimizing training error. |
Compare degrees with held-out data
PYTHON • Compare polynomial degrees using the same split from sklearn.metrics import mean_squared_error |
If training RMSE continues to fall while test RMSE begins to rise, additional polynomial complexity is fitting training-specific noise rather than generalizable structure. This is a direct manifestation of the bias–variance trade-off.
Table 19.10. Linear versus polynomial regression
| Question | Linear features | Polynomial features |
|---|---|---|
| Decision surface / function | Flat hyperplane in the original feature space | Curved response in original variables. |
| Interpretability | Usually simpler | More difficult because powers and interactions multiply. |
| Feature count | Original p features | Can grow rapidly with p and degree. |
| Overfitting risk | Lower baseline flexibility | Higher as degree and interactions increase. |
| Best practice | Use as a strong baseline | Validate degree; consider regularization when expansion is large. |
Practical lab — Train a linear regression model and analyze residuals and coefficients
In this lab, students use the scikit-learn diabetes regression dataset. The target is a quantitative disease-progression measure one year after baseline. The ten baseline features have already been numerically prepared in the dataset. The goal is not to make a medical claim, but to practice a complete regression workflow with a compact real dataset.
| LAB OBJECTIVE Build a reproducible linear regression baseline, measure held-out performance, inspect prediction errors, create residual diagnostics, and interpret the fitted coefficient table without making causal claims. |
Lab learning outcomes
- Load a built-in regression dataset with feature names.
- Create one protected train/test split.
- Fit LinearRegression on the training subset only.
- Generate predictions for the test subset.
- Calculate MAE, RMSE, and R².
- Construct a residual table and identify the largest prediction errors.
- Plot observed versus predicted values.
- Plot residuals versus predicted values and describe the pattern.
- Build and rank a coefficient table.
- Explain why coefficient magnitude and sign do not automatically imply causal importance.
Step 1 — Load and inspect the dataset
PYTHON • Load the diabetes dataset as pandas objects from sklearn.datasets import load_diabetes |
Students should verify that the target is numerical, identify the ten feature names, and check the number of observations. This confirms that the task is regression rather than classification.
Step 2 — Create a protected test split
PYTHON • Split the data once for final evaluation from sklearn.model_selection import train_test_split |
| EXPERIMENTAL DISCIPLINE Do not repeatedly change the model after looking at the final test result. In larger projects, use a separate validation set or cross-validation for model selection and reserve the test set for the final estimate. |
Step 3 — Train the baseline model
PYTHON • Fit ordinary linear regression from sklearn.linear_model import LinearRegression |
LinearRegression estimates one coefficient per input feature plus an intercept. Because the dataset features are already numerically prepared, no categorical encoding step is required in this particular lab.
Step 4 — Predict and evaluate
PYTHON • Calculate MAE, RMSE, and R² import numpy as np |
Table 19.11. Lab metric interpretation
| Metric | Question to answer |
|---|---|
| MAE | On average, how large is the absolute prediction error? |
| RMSE | What is the typical error scale when larger errors receive more weight? |
| R² | How much better is the model than predicting the training-target mean as a baseline? |
Step 5 — Build a residual table
PYTHON • Inspect the largest residuals import pandas as pd |
For the largest errors, students should avoid immediately labeling the observations “bad data.” The correct next step is investigation: Are the observations valid? Do they occupy unusual feature regions? Do they reveal nonlinear behavior or missing explanatory variables?
Step 6 — Plot observed versus predicted
PYTHON • Visualize overall prediction quality import matplotlib.pyplot as plt |
Points close to the diagonal indicate accurate predictions. Systematic deviations from the diagonal can reveal compression toward the mean, range limitations, bias at low or high target values, or nonlinear structure not captured by the model.
Step 7 — Analyze residuals
PYTHON • Plot residuals versus predictions residuals = y_test.to_numpy() - y_pred |
Students should describe the residual plot in words. Is the cloud approximately centered around zero? Is there obvious curvature? Does the vertical spread grow with the prediction? Are there isolated residuals that are much larger than the rest? A diagnostic plot is useful only when the observed pattern is explicitly interpreted.
Step 8 — Analyze coefficients
PYTHON • Create and rank the coefficient table coef_table = pd.DataFrame({ |
| INTERPRETATION BOUNDARY The diabetes dataset is suitable for teaching regression mechanics, but the fitted coefficients should not be presented as causal medical effects. They are coefficients in a predictive model trained on observational baseline variables. |
Step 9 — Optional polynomial extension
As an extension, students can compare a degree-2 polynomial model with the ordinary linear model. Because all pairwise interactions and squared terms are generated, the expanded model is much more flexible. The comparison should use the same train/test split.
PYTHON • Compare linear and quadratic feature spaces from sklearn.pipeline import make_pipeline |
A more flexible model is not automatically better. Students should judge the quadratic model by held-out performance and not by training fit alone. If performance worsens, the additional terms increased variance without adding useful generalizable structure.
Lab questions
Table 19.12. Questions for student analysis
| # | Question |
|---|---|
| 1 | What numerical quantity is the model trying to predict? Why is this a regression problem? |
| 2 | What do MAE and RMSE reveal that R² alone does not? |
| 3 | Which test observations have the largest absolute residuals? What should be investigated before calling them outliers? |
| 4 | Does the residual plot show curvature, changing variance, clusters, or unusually large errors? |
| 5 | Which coefficients are positive and which are negative? State the interpretation as a conditional predictive association. |
| 6 | Why is comparing raw coefficient magnitude potentially misleading when feature scales differ? |
| 7 | Why should these coefficients not be described automatically as causal effects? |
| 8 | Does the quadratic model improve held-out RMSE or R²? What does the result suggest about model complexity? |
Suggested student conclusion template
| WRITE-UP TEMPLATE “The linear regression baseline achieved a test MAE of ___, RMSE of ___, and R² of ___. The residual plot showed ___. The largest coefficient magnitudes were associated with ___, but these coefficients describe conditional predictive associations rather than causal effects. Compared with the linear baseline, the quadratic model ___, suggesting that additional nonlinear flexibility was ___ for this split.” |
Common mistakes to avoid
- Using classification metrics such as accuracy for a continuous target.
- Interpreting R² as the percentage of individual predictions that are correct.
- Judging model quality only from training error.
- Ignoring residual patterns after reporting a single summary metric.
- Comparing coefficient magnitudes without considering feature scale.
- Treating a coefficient sign as proof of causation.
- Deleting observations only because they have large residuals.
- Increasing polynomial degree until training error is nearly zero.
- Selecting polynomial degree by repeatedly inspecting the final test set.
Chapter summary
- Linear regression: predicts a continuous target as an intercept plus a weighted sum of feature values.
- Least squares: chooses coefficients that minimize the sum of squared training residuals.
- Residuals: are observed minus predicted values and are central to both error measurement and diagnostics.
- Assumptions: linearity, independence, stable error variance, and sensible residual behavior should be assessed in context.
- Multicollinearity: can leave prediction useful while making individual coefficients unstable.
- Outliers and influence: matter because squared loss gives large residuals substantial weight.
- Coefficient interpretation: must respect sign, units, scaling, conditioning on other variables, and the limits of causal claims.
- Polynomial features: add powers and interactions so a linear estimator can represent nonlinear response shapes.
- Model complexity: should be selected with validation or cross-validation because higher polynomial degree increases overfitting risk.
- Practical workflow: fit on training data, evaluate held-out predictions, inspect residuals, and interpret coefficients carefully.
Knowledge check
Table 19.13. Quick knowledge check
| Prompt | Expected idea |
|---|---|
| What is a residual? | Observed target minus predicted target. |
| What does least squares minimize? | The sum of squared residuals on the training data. |
| What does a positive coefficient mean? | A positive conditional association with the prediction, holding modeled features fixed. |
| Why inspect residual plots? | To detect systematic structure, changing variance, unusual cases, or model mismatch. |
| Why is multicollinearity a problem? | It can make individual coefficients unstable even when predictions remain useful. |
| Does scaling change coefficient values? | Yes; it changes the feature unit and therefore the coefficient unit. |
| Does a regression coefficient prove causation? | No. Predictive association alone is not causal evidence. |
| How can linear regression represent curvature? | Add transformed inputs such as squared, cubic, or interaction features. |
| Why validate polynomial degree? | Higher degree lowers training bias but can increase variance and overfitting. |
| NEXT STEP After ordinary linear regression, the course can naturally move to regularized linear models such as Ridge and Lasso, which control coefficient size and become especially useful with many or correlated predictors. |