Chapter 20 — Regularized Regression
Ridge • Lasso • Elastic Net • Scaling • Coefficient Shrinkage
Controlling model complexity while preserving predictive performance
| BRIDGE FROM CHAPTER 19 Ordinary linear regression chooses coefficients that minimize squared prediction error. Chapter 20 adds a second objective: keep the coefficient vector under control. Regularization trades a small amount of flexibility for improved stability, reduced variance, and better generalization. |
Chapter overview
Linear regression can become unstable when a dataset contains many predictors, strongly correlated variables, noisy features, or limited training observations. The model may still fit the training data well, but its coefficients can become unnecessarily large and its predictions can change sharply when the sample changes.
Regularization modifies the regression objective by penalizing coefficient size. Ridge regression uses an L2 penalty, Lasso uses an L1 penalty, and Elastic Net combines both. These methods are especially valuable when the feature space is complex and prediction on unseen data matters more than perfectly fitting the training sample.
This chapter emphasizes the relationship between the penalty, model flexibility, feature scaling, coefficient interpretation, and the bias-variance trade-off. Students will finish by comparing ordinary linear regression, Ridge, Lasso, and Elastic Net on the same dataset.
Learning objectives
- Explain why ordinary least squares can overfit or produce unstable coefficients.
- Describe how regularization adds a penalty to the regression objective.
- Distinguish L2 shrinkage in Ridge from L1 sparsity in Lasso.
- Explain how Elastic Net combines L1 and L2 penalties.
- Interpret the role of alpha and l1_ratio in scikit-learn models.
- Explain why regularized regression should be trained on comparably scaled features.
- Use pipelines so scaling is learned only from the training data.
- Compare models using MAE, RMSE, R², coefficient magnitude, and residual behavior.
- Recognize that feature selection by Lasso is predictive rather than automatically causal.
- Select a regularized model using cross-validation rather than the test set.
Table 20.1. Chapter structure
| Section | Main question | Core idea |
|---|---|---|
| 20.1 | Why regularize? | Control variance, coefficient magnitude, and overfitting. |
| 20.2 | What does Ridge do? | Shrink all coefficients with an L2 penalty. |
| 20.3 | What does Lasso do? | Use an L1 penalty that can set coefficients exactly to zero. |
| 20.4 | Why Elastic Net? | Combine sparsity with stability for correlated predictors. |
| 20.5 | Why scale first? | Make the penalty comparable across features. |
| Lab | Which model works best? | Compare Linear, Ridge, Lasso, and Elastic Net fairly. |
20.1 Why regularization is needed
Ordinary least squares minimizes prediction error on the training data. When the feature set is simple and informative, this can work extremely well. As model complexity increases, however, several related problems can appear.
MORE FEATURES | CORRELATION | LARGE COEFFICIENTS | HIGH VARIANCE | OVERFITTING |
Too many features
Adding predictors gives a regression model more ways to fit the observed sample. If many predictors contain little signal, the model may use accidental patterns that do not repeat in new data. This risk becomes especially important when the number of features is large relative to the number of observations.
| KEY IDEA More features do not automatically mean more useful information. Regularization discourages the model from depending too strongly on weak or noisy predictors. |
Correlated variables
When two or more predictors carry similar information, many different coefficient combinations can produce nearly the same fitted values. Ordinary least squares may therefore assign one large positive coefficient and another large negative coefficient even though the combined prediction remains reasonable.
This is a coefficient-stability problem. Small changes in the training sample can produce large changes in individual coefficients. Ridge is particularly useful in this situation because it tends to distribute weight more smoothly across correlated predictors.
High variance and large coefficients
A high-variance model is sensitive to the exact observations used for training. One symptom is a coefficient vector with very large magnitudes. Large coefficients can amplify small changes in feature values and make predictions more sensitive to noise.
Training objective = prediction error + regularization penalty Regularization asks the model to fit the data while also keeping complexity under control. |
Overfitting
Overfitting occurs when a model learns sample-specific noise instead of generalizable structure. Training error may continue to decrease while validation or test error becomes worse. Regularization intentionally limits flexibility, which can increase training error slightly while improving performance on unseen data.
Table 20.2. Common warning signs
| Warning sign | What it may indicate | Regularization response |
|---|---|---|
| Training score much better than validation score | High variance / overfitting | Increase regularization strength. |
| Very large coefficient magnitudes | Sensitivity to scale, noise, or collinearity | Shrink coefficients. |
| Coefficients change sharply between samples | Unstable estimation | Prefer Ridge or Elastic Net. |
| Many weak predictors | Unnecessary complexity | Lasso may remove some predictors. |
| Many correlated predictors | Redundant information | Ridge or Elastic Net often behaves more stably. |
Bias-variance trade-off
Regularization adds bias because it deliberately moves coefficients away from the unconstrained least-squares solution. The benefit is lower variance. A well-chosen penalty can therefore produce better test performance even though the training fit is not as close.
| INTERPRETATION The goal is not to make coefficients as small as possible. The goal is to choose enough regularization to improve generalization without erasing useful signal. |
20.2 Ridge regression
Ridge regression adds the squared magnitude of the coefficients to the ordinary least-squares objective. Because the penalty uses squared coefficients, large values are penalized strongly.
min Σᵢ(yᵢ − ŷᵢ)² + α Σⱼ βⱼ² Ridge regression: least-squares error plus an L2 penalty. |
The L2 penalty
The L2 penalty is the sum of squared coefficients. The hyperparameter α controls how strongly the model is penalized. When α is zero, the objective approaches ordinary linear regression. As α increases, the model accepts more bias in exchange for smaller coefficients and lower variance.
Table 20.3. Effect of Ridge alpha
| Alpha | Typical behavior | Risk |
|---|---|---|
| Very small | Close to ordinary linear regression | May not reduce variance enough. |
| Moderate | Useful coefficient shrinkage | Often good bias-variance balance. |
| Very large | Coefficients pushed strongly toward zero | Underfitting can occur. |
Coefficient shrinkage
Ridge usually reduces coefficient magnitudes continuously rather than eliminating features. A predictor that had a coefficient of 25 in ordinary regression might become 12, 5, or 1.5 as regularization becomes stronger. The exact path depends on the data and scaling.
| IMPORTANT Ridge normally retains all predictors. Coefficients can become very small, but they are generally not forced exactly to zero. |
Handling correlated predictors
Suppose two variables provide nearly the same information. Ordinary least squares can struggle to decide how the effect should be divided. Ridge stabilizes this allocation because a solution with several moderate coefficients can receive a smaller L2 penalty than a solution with one extremely large coefficient.
PYTHON • Fit Ridge inside a scaling pipeline from sklearn.pipeline import Pipeline |
Choosing alpha
Alpha should be selected using validation data or cross-validation. The test set should remain untouched until the model and its hyperparameters have been selected. A logarithmic search grid is often useful because meaningful alpha values can span several orders of magnitude.
PYTHON • Tune Ridge alpha with cross-validation from sklearn.model_selection import GridSearchCV |
| PRACTICE NOTE The double underscore in ridge__alpha tells a scikit-learn pipeline to tune the alpha parameter inside the step named ridge. |
20.3 Lasso regression
Lasso regression uses the absolute values of the coefficients instead of their squares. This apparently small mathematical change produces an important practical difference: Lasso can set some coefficients exactly to zero.
min Σᵢ(yᵢ − ŷᵢ)² + α Σⱼ |βⱼ| Lasso regression: least-squares error plus an L1 penalty. |
The L1 penalty
The L1 penalty grows linearly with coefficient magnitude. During optimization, the geometry of this penalty makes exact zeros possible. This creates a sparse coefficient vector in which only a subset of predictors remains active.
Table 20.4. Ridge versus Lasso
| Property | Ridge | Lasso |
|---|---|---|
| Penalty | L2: squared coefficients | L1: absolute coefficients |
| Coefficient behavior | Shrinks all coefficients | Can set coefficients to zero |
| Feature selection | No automatic hard selection | Yes, through zero coefficients |
| Correlated predictors | Often distributes weight | May choose one and suppress others |
| Primary strength | Stability | Sparsity and simpler models |
Sparse coefficients and feature selection
When a Lasso coefficient becomes zero, that feature no longer contributes to the prediction. This can produce a compact model and may help when the original dataset contains many weak or irrelevant predictors.
| CAUTION A zero coefficient means that the feature was not useful enough under this model, penalty, sample, and set of competing predictors. It does not prove that the feature is scientifically irrelevant. |
Limitations with correlated variables
If several predictors are highly correlated, Lasso may keep one and remove another even when both are meaningful. Which feature survives can change when the sample changes. This instability is one reason Elastic Net is often preferred when sparsity is desired in the presence of correlated predictors.
PYTHON • Fit a Lasso model from sklearn.linear_model import Lasso |
Counting selected features
PYTHON • Inspect nonzero Lasso coefficients import numpy as np |
| DIAGNOSTIC QUESTION If increasing alpha removes many variables and test error rises sharply, the model is probably being regularized too aggressively. |
20.4 Elastic Net
Elastic Net combines L1 and L2 regularization. It is useful when we want some of Lasso’s sparsity but also want Ridge-like stability, especially when predictors are correlated.
min Σᵢ(yᵢ − ŷᵢ)² + α[ ρ Σⱼ|βⱼ| + (1−ρ) Σⱼβⱼ² ] Conceptual Elastic Net objective. In scikit-learn, ρ corresponds to l1_ratio. |
Combining L1 and L2 penalties
The total regularization strength is controlled by alpha. The mixture between the L1 and L2 parts is controlled by l1_ratio in scikit-learn. A value near 1 emphasizes Lasso-like sparsity. A value closer to 0 emphasizes Ridge-like shrinkage.
Table 20.5. Elastic Net hyperparameters
| Hyperparameter | Controls | Interpretation |
|---|---|---|
| alpha | Overall penalty strength | Higher values produce stronger regularization. |
| l1_ratio | Mix of L1 and L2 | 1.0 is Lasso-like; smaller values add more L2 behavior. |
| max_iter | Optimization budget | Increase if the solver has not converged. |
| tol | Stopping tolerance | Controls convergence precision. |
Balancing sparsity and stability
Elastic Net can keep groups of correlated predictors more effectively than pure Lasso while still driving some coefficients to zero. This makes it a practical choice for high-dimensional tabular datasets, engineered feature sets, and applications where correlated variables are common.
LASSO-LIKE | ELASTIC NET | RIDGE-LIKE |
PYTHON • Fit Elastic Net in a pipeline from sklearn.linear_model import ElasticNet |
Jointly tuning alpha and l1_ratio
PYTHON • Cross-validate the two main Elastic Net hyperparameters elastic_grid = { |
| INTERPRETATION Elastic Net is not automatically better than Ridge or Lasso. Its advantage is flexibility: it can search between sparse and smooth coefficient patterns. |
20.5 Importance of scaling
Regularized models require comparable feature scales because the penalty is applied directly to coefficient values. If one feature is measured in thousands and another in fractions, the coefficient magnitudes needed to represent similar predictive effects can be very different.
Why scale affects the penalty
Consider two predictors: annual income measured in dollars and a ratio measured between 0 and 1. A tiny income coefficient can have a large predictive effect because the income values themselves are large. A ratio may need a much larger coefficient to create a similar effect. Penalizing the raw coefficients would therefore treat the two predictors unfairly.
| RULE Standardize numerical predictors before Ridge, Lasso, or Elastic Net unless you have a specific, justified preprocessing design that already places them on comparable scales. |
Standardization
z = (x − μ) / σ StandardScaler centers each feature and scales it by its training-set standard deviation. |
After standardization, numerical features are expressed in comparable units. Regularization can then penalize coefficients according to their contribution rather than according to arbitrary measurement units.
Use a pipeline to avoid leakage
The scaler must be fit using only training data. A Pipeline ensures that cross-validation and final model fitting learn scaling parameters inside each training fold instead of using information from validation observations.
PYTHON • Recommended scaling pattern from sklearn.pipeline import Pipeline |
Scaling and coefficient interpretation
After standardization, coefficients correspond to changes measured in standard-deviation units rather than original units. This can make coefficient magnitudes easier to compare across predictors, but it changes their direct unit-based interpretation. If domain interpretation in original units is important, document the preprocessing clearly.
Table 20.6. Practical scaling decisions
| Situation | Recommended action | Reason |
|---|---|---|
| Ridge / Lasso / Elastic Net | Scale numeric features | Penalty depends on coefficient magnitude. |
| Mixed units | Scale before regularization | Avoid unit-driven penalty imbalance. |
| Pipeline with cross-validation | Put scaler inside pipeline | Prevents validation leakage. |
| Categorical one-hot features | Consider preprocessing design carefully | Binary indicators have a different natural scale. |
| Already standardized dataset | Pipeline is still safe and explicit | Keeps workflow reusable and leakage-resistant. |
Practical lab — Compare Linear Regression, Ridge, Lasso, and Elastic Net
In this lab, students compare four regression models on the same train/test split. The goal is not only to identify the lowest error. Students also examine coefficient magnitude, sparsity, residual behavior, and the effect of hyperparameter selection.
Lab objectives
- Load a regression dataset and create a protected train/test split.
- Build leakage-safe scaling pipelines.
- Train ordinary Linear Regression, Ridge, Lasso, and Elastic Net.
- Evaluate MAE, RMSE, and R² on the same test observations.
- Inspect how regularization changes coefficient magnitude.
- Count zero coefficients for sparse models.
- Analyze residuals and compare generalization behavior.
- Tune regularization hyperparameters using cross-validation.
Dataset
The lab uses scikit-learn’s built-in diabetes regression dataset. It contains numerical predictors and a continuous disease-progression target. The dataset is convenient for classroom work because it requires no external download. We still use StandardScaler inside each pipeline to reinforce the correct reusable workflow for regularized regression.
Step 1 — Imports and data split
PYTHON • Load data and protect the test set import numpy as np |
| EXPERIMENTAL DISCIPLINE Do not use the test set to choose alpha or l1_ratio. Hyperparameter selection belongs inside the training data through cross-validation. |
Step 2 — Define the four models
PYTHON • Create comparable pipelines models = { |
Step 3 — Train and evaluate
PYTHON • Compare MAE, RMSE, and R² results = [] |
Table 20.7. How to read the evaluation metrics
| Metric | Better direction | Meaning |
|---|---|---|
| MAE | Lower | Average absolute prediction error in target units. |
| RMSE | Lower | Penalizes large errors more strongly than MAE. |
| R² | Higher | Fraction of target variation explained relative to a mean baseline. |
Step 4 — Compare coefficient magnitude and sparsity
PYTHON • Extract coefficients from each fitted pipeline coefficient_rows = [] |
| EXPECTED PATTERN Ridge should reduce coefficient magnitude without intentionally creating zeros. Lasso and Elastic Net may create exact or near-zero coefficients depending on alpha and the data. |
Step 5 — Inspect feature coefficients
PYTHON • Build a coefficient comparison table coef_table = pd.DataFrame(index=X.columns) |
Step 6 — Plot coefficient profiles
PYTHON • Visualize how penalties change coefficients coef_table.plot(kind="bar", figsize=(11, 5)) |
Step 7 — Analyze residuals
PYTHON • Residual plot for each model for name, model in models.items(): |
A useful residual plot should look like an unstructured cloud around zero. Curvature, funnels, or extreme isolated residuals can indicate problems that regularization alone does not solve.
Step 8 — Tune Ridge, Lasso, and Elastic Net
Fixed alpha values are useful for learning, but a fair predictive comparison should tune the regularization strength with cross-validation. The next code searches multiple candidates using only the training data.
PYTHON • Cross-validated search for regularized models from sklearn.model_selection import GridSearchCV |
Step 9 — Final test comparison after tuning
PYTHON • Evaluate only the selected models on the test set final_models = { |
Student analysis questions
1. Which model obtains the lowest test RMSE after tuning?
2. Does the best regularized model improve substantially over ordinary linear regression?
3. Which model produces the smallest overall coefficient magnitude?
4. Does Lasso set any coefficients to zero? How does this change as alpha increases?
5. How does Elastic Net differ from Lasso when l1_ratio is reduced?
6. Are the largest coefficients also the most trustworthy or causal predictors? Explain why not.
7. Do the residual plots show remaining structure that regularization cannot fix?
8. Why would it be incorrect to choose alpha by repeatedly checking the test-set RMSE?
Optional extension — Regularization paths
Students can fit the same model over a sequence of alpha values and plot how each coefficient changes. A regularization path makes the shrinkage process visible and is especially useful for comparing Ridge’s smooth shrinkage with Lasso’s transition to exact zeros.
PYTHON • Explore a simple Lasso coefficient path alphas = np.logspace(-2, 1, 40) |
Chapter summary
Table 20.8. Regularized regression at a glance
| Model | Penalty | Feature selection | Best suited to |
|---|---|---|---|
| Linear Regression | None | No | Simple baseline with stable predictors. |
| Ridge | L2 | No | Many predictors and correlated features. |
| Lasso | L1 | Yes | Sparse solutions and feature reduction. |
| Elastic Net | L1 + L2 | Yes | Sparse models with correlated predictors. |
Key takeaways
- Regularization adds a complexity penalty to the regression objective.
- Ridge shrinks coefficients smoothly and is often stable with correlated predictors.
- Lasso can force coefficients to zero and therefore performs embedded feature selection.
- Elastic Net combines L1 sparsity with L2 stability.
- Stronger regularization increases bias but can reduce variance and improve test performance.
- Feature scaling is essential because penalties act directly on coefficient magnitudes.
- Pipelines protect against preprocessing leakage during cross-validation.
- Hyperparameters must be selected on training/validation data, not on the final test set.
- Coefficient sparsity is a modeling result, not proof of causal importance.
- Residual diagnostics remain necessary because regularization does not correct every modeling problem.
| CONNECTION TO THE NEXT STEP Regularized linear models provide a strong baseline for continuous prediction. Later regression chapters can compare them with nonlinear tree ensembles and other models that capture interactions without manually constructing them. |
Quick knowledge check
1. Why can correlated predictors make ordinary least-squares coefficients unstable?
2. What is the main difference between the L1 and L2 penalties?
3. Why does Ridge usually keep every feature?
4. Why can Lasso be unstable when several predictors are strongly correlated?
5. What does l1_ratio control in Elastic Net?
6. Why must feature scaling occur inside the cross-validation workflow?
7. What happens if alpha is made excessively large?
8. Why is the model with the lowest training error not automatically the best model?
Suggested student deliverable
Submit a short notebook or report containing the model-comparison table, best hyperparameters, coefficient comparison, one residual plot for each final model, and a short conclusion explaining which model you would keep and why.