Chapter 21 — Tree-Based Regression
Decision Trees • Random Forests • Gradient Boosting • Nonlinear Regression
Learning nonlinear patterns and interactions from tabular data
| BRIDGE FROM CHAPTER 20 Regularized regression controls a linear model by shrinking coefficients. Tree-based regression takes a different approach: it partitions the feature space into regions and predicts from those regions. This makes nonlinear relationships and interactions possible without manually constructing polynomial terms. |
Chapter overview
Many regression problems are not well described by one global straight-line relationship. The effect of a predictor can change at different values, variables can interact, and the response can contain thresholds or plateaus. Tree-based regressors are designed to capture this kind of structure directly from data.
A decision tree builds a sequence of if-then splits and predicts a constant value inside each terminal leaf. Random forests average many randomized trees to reduce variance. Gradient boosting builds trees sequentially so that each new tree focuses on the remaining prediction errors. Together, these methods form some of the most important nonlinear baselines for tabular regression.
This chapter emphasizes not only predictive performance but also practical behavior: scaling requirements, overfitting, hyperparameter sensitivity, feature interactions, residual analysis, and the important limitation that tree-based models usually extrapolate poorly outside the range represented in the training data.
Learning objectives
- Explain how a regression tree chooses splits for a continuous target.
- Describe why each terminal leaf predicts a mean target value.
- Interpret piecewise-constant predictions and their consequences.
- Explain how random forests reduce variance by averaging many randomized trees.
- Describe how tree ensembles capture nonlinear patterns and feature interactions.
- Explain gradient boosting as sequential correction of residual errors.
- Interpret learning rate, number of estimators, tree depth, and loss functions.
- Explain why standard feature scaling is usually unnecessary for tree-based regression.
- Recognize overfitting, limited interpretability, and poor extrapolation as important limitations.
- Compare linear regression and tree-based regressors using one protected train/test split.
Table 21.1. Chapter structure
| Section | Main question | Core idea |
|---|---|---|
| 21.1 | How does a regression tree predict? | Recursive splits create leaves containing local mean predictions. |
| 21.2 | Why use a random forest? | Average many randomized trees to reduce variance. |
| 21.3 | How does boosting improve trees? | Add weak trees sequentially to correct remaining errors. |
| 21.4 | Why are tree models attractive? | Nonlinearity, interactions, and little need for scaling. |
| 21.5 | What are the trade-offs? | Interpretability, overfitting, extrapolation, and tuning. |
| Lab | Which approach works best? | Compare Linear, Tree, Random Forest, and Gradient Boosting. |
21.1 Decision tree regression
A regression tree predicts a numerical target by recursively splitting the feature space. At each internal node, the algorithm evaluates candidate feature thresholds and chooses a split that makes the target values inside the resulting child nodes more homogeneous.
TRAINING DATA | CHOOSE A SPLIT | CREATE CHILD NODES | REPEAT | LEAF PREDICTIONS |
Splitting numerical targets
For classification, a tree tries to create nodes containing observations from similar classes. For regression, the target is continuous, so the tree instead tries to create nodes whose target values are close to one another. A candidate split is useful when it reduces prediction error inside the two child nodes.
For a feature xⱼ, a split can be written conceptually as xⱼ ≤ t versus xⱼ > t. The threshold t is selected from the observed feature values. The tree evaluates many such candidates and keeps the one that gives the largest reduction in impurity or loss.
Mean prediction in leaves
Once an observation reaches a terminal leaf, the regression tree must return a numerical prediction. Under the usual squared-error criterion, the optimal constant prediction for a leaf is the mean target value of the training observations in that leaf.
ŷ_leaf = (1 / n_leaf) Σ yᵢ A regression leaf predicts the mean target value of the training observations assigned to that leaf. |
| INTERPRETATION Two new observations that fall into the same leaf receive exactly the same prediction, even if their feature values are not identical. |
Mean squared error criterion
A common regression-tree criterion is squared error. The algorithm prefers splits that reduce the sum of squared deviations between target values and their node mean. In scikit-learn, DecisionTreeRegressor uses squared_error by default, which corresponds to variance reduction and minimizes L2 loss using the mean of each terminal node.
SSE(node) = Σᵢ (yᵢ − ȳ_node)² A useful split produces child nodes whose combined squared error is lower than that of the parent node. |
Piecewise constant predictions
Because every leaf outputs one constant value, a decision tree produces a step-shaped regression function. This is very different from linear regression, which produces a smooth hyperplane. Trees can approximate curved relationships by creating many small regions, but the fitted function remains piecewise constant.
Table 21.2. Linear regression versus one regression tree
| Property | Linear Regression | Decision Tree Regression |
|---|---|---|
| Functional form | One global linear relationship | Piecewise-constant regions |
| Nonlinearity | Must be engineered explicitly | Captured through recursive splits |
| Interactions | Must be added explicitly | Discovered naturally through split sequences |
| Scaling sensitivity | Predictions usually unaffected, coefficients depend on units | Very low |
| Extrapolation | Can extend the fitted linear trend | Usually cannot extend beyond learned leaf values |
| Main risk | Underfitting nonlinear structure | High variance / overfitting |
Controlling tree complexity
An unrestricted tree can keep splitting until leaves contain very few observations. Such a tree may reproduce the training data closely but generalize poorly. Important controls include max_depth, min_samples_split, min_samples_leaf, max_leaf_nodes, and cost-complexity pruning through ccp_alpha.
Table 21.3. Important DecisionTreeRegressor controls
| Hyperparameter | Effect | Typical reason to tune |
|---|---|---|
| max_depth | Limits the number of split levels | Prevent overly complex trees. |
| min_samples_split | Requires enough observations before splitting | Avoid fragile splits. |
| min_samples_leaf | Requires enough observations in each leaf | Smooth predictions and reduce variance. |
| max_leaf_nodes | Limits the number of terminal regions | Directly control model size. |
| ccp_alpha | Prunes branches using a complexity penalty | Simplify a fully grown tree. |
PYTHON • Train a decision tree regressor from sklearn.tree import DecisionTreeRegressor |
| PRACTICAL NOTE For regression trees, min_samples_leaf is often an especially useful regularizer because larger leaves average more observations and create less jagged predictions. |
21.2 Random forest regression
A single decision tree is flexible but unstable: a small change in the training data can produce a different sequence of splits. Random forest regression reduces this variance by fitting many trees and averaging their predictions.
BOOTSTRAP SAMPLES | RANDOM FEATURES | MANY TREES | AVERAGE PREDICTIONS | STABLE OUTPUT |
Averaging predictions
If a forest contains B trees and tree b predicts ŷ⁽ᵇ⁾ for a new observation, the forest prediction is the average of those individual predictions. Averaging smooths out some of the accidental variation of individual trees.
ŷ_forest = (1 / B) Σᵦ ŷ⁽ᵦ⁾ Random forest regression averages the predictions produced by its individual trees. |
Variance reduction
Bagging works best when the individual models are reasonably accurate but not perfectly correlated. Random forests encourage diversity in two ways: each tree is trained from a bootstrap sample, and each split considers only a random subset of predictors. The resulting trees make different errors, so averaging can reduce overall variance.
| KEY IDEA A random forest is not powerful merely because it contains many trees. Its strength comes from averaging trees that are deliberately made different from one another. |
Nonlinear patterns
Because each tree partitions the feature space, a random forest can represent thresholds, plateaus, curved responses, and other nonlinear patterns. Increasing the number of trees generally stabilizes the ensemble rather than making each individual tree more complex.
Feature interactions
A feature interaction occurs when the effect of one variable depends on the value of another. Trees capture interactions naturally because later splits are conditional on earlier splits. For example, a model may first split on age and then use a blood-pressure threshold only for observations in one age region.
Out-of-bag evaluation
When bootstrap sampling is enabled, each tree leaves out some training observations. These out-of-bag observations can be predicted by the trees that did not train on them, giving an internal estimate of generalization performance. In scikit-learn, this can be enabled with oob_score=True.
PYTHON • Train a random forest regressor from sklearn.ensemble import RandomForestRegressor |
Feature importance
Random forests can report impurity-based feature importance, which summarizes how much each feature contributed to reducing the tree criterion across the ensemble. These values are convenient but should be interpreted cautiously: they describe the fitted model, not causality, and can be biased toward features that offer many possible split points.
| BETTER INTERPRETATION When feature importance matters, complement impurity importance with permutation importance or other model-inspection tools evaluated on held-out data. |
21.3 Gradient boosting regression
Random forests build trees largely independently and average them. Gradient boosting uses a different strategy: trees are added sequentially. Each new tree is trained to improve the errors that remain after the current ensemble prediction.
INITIAL PREDICTION | COMPUTE ERRORS | FIT SMALL TREE | ADD CORRECTION | REPEAT |
Sequential residual correction
With squared-error loss, the first model can begin with a simple constant prediction such as the training-target mean. Residuals are then computed. A small regression tree is fitted to those residuals, and its predictions are added to the existing model. Repeating this process gradually improves the fit.
Fₘ(x) = Fₘ₋₁(x) + η hₘ(x) Each stage adds a new tree hₘ scaled by the learning rate η. |
The residual interpretation is exact for squared-error boosting. More generally, gradient boosting fits each new learner to the negative gradient of the chosen loss function, which is why the method can support losses beyond ordinary squared error.
Weak learners
Boosting usually uses shallow trees rather than fully grown trees. Each tree is intentionally limited and contributes a modest correction. The ensemble becomes expressive through the accumulation of many small improvements.
Learning rate
The learning_rate parameter controls how much each new tree contributes. A smaller learning rate usually requires more estimators but can improve generalization because the model learns more gradually. A very large learning rate can cause the ensemble to fit too aggressively.
Number of trees
The n_estimators parameter controls the number of boosting stages. Too few trees can underfit. Too many trees can overfit, especially when the learning rate is high or the individual trees are deep. Learning rate and number of estimators therefore need to be considered together.
Tree depth
The depth of each boosting tree controls the complexity of each correction. Depth 1 trees model one-dimensional threshold effects. Slightly deeper trees can represent interactions. Very deep trees are often unnecessary because boosting already gains complexity by adding many stages.
Table 21.4. Main GradientBoostingRegressor controls
| Hyperparameter | Main effect | Typical trade-off |
|---|---|---|
| n_estimators | Number of boosting stages | More stages add capacity and computation. |
| learning_rate | Contribution of each tree | Smaller values usually require more trees. |
| max_depth | Complexity of each base tree | Deeper trees learn stronger interactions but may overfit. |
| min_samples_leaf | Minimum observations in a leaf | Larger values smooth each correction. |
| subsample | Fraction of observations used per stage | Values below 1 introduce stochastic boosting. |
| loss | Objective optimized by boosting | Choose according to error behavior and robustness needs. |
Loss functions
Squared error is a common default because it strongly penalizes large residuals and is easy to interpret. Alternatives can be more robust to unusual observations or can target different aspects of the conditional response. The loss should reflect the practical cost of prediction errors rather than being selected only by habit.
PYTHON • Train a gradient boosting regressor from sklearn.ensemble import GradientBoostingRegressor |
| TUNING PRINCIPLE A smaller learning rate combined with more shallow trees is often a safer starting point than a large learning rate with a few highly influential trees. |
21.4 Advantages of tree-based regression
Tree-based regressors are popular because they relax many of the structural assumptions of linear models. They can learn complex relationships directly from tabular predictors and often require less feature engineering.
Nonlinear modeling
Trees do not assume that a one-unit change in a predictor has the same effect everywhere. Different regions of the feature space can have different prediction rules, allowing thresholds, saturation, and local behavior to emerge naturally.
Minimal scaling requirements
A tree split depends on the ordering of feature values, not on Euclidean distances or coefficient magnitudes. Multiplying a feature by a positive constant usually changes the numerical threshold but not the ordering of observations. Standardization is therefore not normally required for decision trees, random forests, or gradient boosting trees.
| WORKFLOW SIMPLIFICATION If a pipeline contains only numerical tree-based models, StandardScaler is usually unnecessary. Scaling may still be needed when other preprocessing or other model families are part of the same workflow. |
Interaction discovery
Interactions appear automatically because a prediction path can contain splits on multiple features. This is particularly useful when domain knowledge suggests that predictor effects are conditional but the exact interaction form is unknown.
Strong performance on tabular datasets
Tree ensembles are strong general-purpose methods for structured numerical and categorical-derived features. Random forests provide a robust baseline with relatively forgiving tuning. Gradient boosting often reaches higher predictive accuracy when its hyperparameters are tuned carefully.
Table 21.5. When the main models are attractive
| Model | Main strength | Good starting use |
|---|---|---|
| Decision Tree | Simple nonlinear rules | Interpretable baseline and visualization. |
| Random Forest | Stable nonlinear performance | Robust ensemble baseline. |
| Gradient Boosting | High predictive accuracy | Carefully tuned tabular regression. |
| Linear Regression | Simple global relationship | Transparent reference baseline. |
21.5 Limitations
Tree-based methods solve important problems that linear regression cannot solve directly, but they introduce their own limitations. These limitations should influence both model selection and how results are communicated.
Reduced interpretability
A small decision tree can be read directly, but a forest containing hundreds of trees or a boosting model containing many sequential stages is not transparent in the same way as a linear equation. Feature importance, partial dependence, permutation importance, and local explanation methods can help, but they provide summaries rather than a single simple rule.
Risk of overfitting
Deep decision trees can overfit dramatically. Random forests reduce this risk through averaging but can still overfit noisy datasets or poorly chosen settings. Gradient boosting is especially sensitive to excessive model capacity because later stages may begin fitting residual noise.
Poor extrapolation beyond the training range
This limitation is fundamental. Tree regressors predict from learned leaves, and those leaves contain target values observed during training. If a new predictor value lies far beyond the observed training range, the model typically continues returning the value associated with the outermost learned region rather than extending a trend.
| IMPORTANT LIMITATION If the application requires forecasting beyond the range of observed predictors, always test extrapolation behavior explicitly. A tree ensemble can perform very well inside the training domain and still fail outside it. |
Hyperparameter sensitivity
Random forests are relatively forgiving, but depth, leaf size, number of features considered at each split, and number of trees still matter. Gradient boosting is more sensitive because learning rate, number of stages, tree depth, subsampling, and loss interact. Hyperparameters should be tuned using cross-validation or a validation set rather than the final test set.
Table 21.6. Common failure modes and responses
| Observed problem | Possible cause | Possible response |
|---|---|---|
| Training R² near 1 but test R² much lower | Tree too complex | Reduce depth; increase min_samples_leaf. |
| Forest predictions vary across runs | Too few trees or unstable settings | Increase n_estimators; set random_state. |
| Boosting improves training loss but test loss worsens | Too many stages / too much capacity | Reduce depth or learning rate; use validation. |
| Predictions flatten outside observed range | Tree extrapolation limitation | Use a model with appropriate trend structure or add domain constraints. |
| Feature importance seems dominated by one variable | Importance bias or real dominance | Verify with permutation importance and domain checks. |
Practical lab — Comparing linear and tree-based regression
In this lab, students use the scikit-learn diabetes regression dataset. All four models are trained on the same training observations and evaluated on the same protected test set. The goal is to compare not only test metrics but also prediction behavior, residuals, and model complexity.
| LAB PRINCIPLE Do not repeatedly tune models on the test set. Use cross-validation on the training set for model selection, then evaluate the selected models once on the protected test set. |
Step 1 — Import the libraries
PYTHON • Imports for the lab import numpy as np |
Step 2 — Load and inspect the dataset
PYTHON • Load the diabetes regression dataset diabetes = load_diabetes(as_frame=True) |
The target is a continuous disease-progression measure. The dataset is useful for model comparison because it contains several numerical predictors and a target with enough variation to expose differences between linear and nonlinear methods.
Step 3 — Create one protected split
PYTHON • Use the same split for every model X_train, X_test, y_train, y_test = train_test_split( |
Step 4 — Define the four baseline models
PYTHON • Create the comparison models models = { |
Step 5 — Train and evaluate every model
PYTHON • Compute MAE, RMSE, and R² results = [] |
Table 21.7. Interpreting the lab metrics
| Metric | Direction | Interpretation |
|---|---|---|
| MAE | Lower is better | Average absolute prediction error in target units. |
| RMSE | Lower is better | Penalizes large errors more strongly than MAE. |
| R² | Higher is better | Fraction of target variation explained relative to a mean baseline. |
Step 6 — Compare predicted and observed values
PYTHON • Prediction-versus-observation plots for name, y_pred in predictions.items(): |
Step 7 — Analyze residuals
PYTHON • Residual-versus-prediction plots for name, y_pred in predictions.items(): |
A useful model should produce residuals that are centered around zero without a strong systematic pattern. Curvature in the linear model residuals can suggest that nonlinear structure remains. Large isolated residuals may indicate difficult observations or outliers.
Step 8 — Compare training and test performance
PYTHON • Look for overfitting generalization = [] |
| INTERPRETATION A very large training-test gap is a warning sign of high variance. A small gap does not guarantee a good model, because both scores may be poor if the model underfits. |
Step 9 — Tune the tree models with cross-validation
PYTHON • Define compact search spaces search_spaces = { |
PYTHON • Run cross-validation searches best_models = {} |
Step 10 — Final test comparison after tuning
PYTHON • Evaluate selected models once on the test set final_models = { |
Step 11 — Inspect random forest feature importance
PYTHON • Rank the forest features forest = best_models["Random Forest"] |
| CAUTION Feature importance is not a causal effect and is not directly comparable to a linear regression coefficient. It measures how a fitted model used a feature to improve its splits. |
Optional extension — Demonstrate poor extrapolation
Create a simple one-feature dataset with a visible increasing trend, train a tree only on the central range, and then predict beyond that range. Compare the result with linear regression. Students should observe that the tree prediction becomes flat once new values remain in the outermost leaf.
PYTHON • Simple extrapolation experiment x = np.linspace(0, 10, 120).reshape(-1, 1) |
Student analysis questions
1. Which baseline model obtains the lowest test RMSE?
2. Does the decision tree show a larger training-test R² gap than the other models?
3. Does the random forest improve over the single tree? Explain the variance-reduction mechanism.
4. Does gradient boosting obtain better predictive accuracy than the random forest on this split?
5. Which model residual plot shows the clearest remaining structure?
6. Why was StandardScaler not required for the tree-based models?
7. Which hyperparameters changed after cross-validation, and what do those changes imply about model complexity?
8. Why should impurity-based feature importance not be interpreted as a causal effect?
9. What happens in the extrapolation experiment once x exceeds the range observed during training?
10. Based on predictive performance, stability, interpretability, and computation, which final model would you recommend for this dataset?
Chapter summary
Table 21.8. Tree-based regression at a glance
| Model | How predictions are formed | Main strength | Main weakness |
|---|---|---|---|
| Decision Tree | One leaf mean after recursive splits | Simple nonlinear structure | High variance and stepwise predictions |
| Random Forest | Average predictions from many randomized trees | Robust variance reduction | Less interpretable and more computationally expensive |
| Gradient Boosting | Sequentially add trees that correct current errors | Often high predictive accuracy | More sensitive to tuning and overfitting |
| Linear Regression | Global linear equation | Transparent baseline and extrapolating trend | Cannot capture complex nonlinear structure automatically |
Key takeaways
- Regression trees partition the feature space and predict the mean target value in each terminal leaf.
- Their fitted function is piecewise constant rather than globally linear.
- A single tree is flexible but can have high variance and overfit the training data.
- Random forests reduce variance by averaging many diverse trees trained with bootstrap and feature randomness.
- Tree sequences automatically represent nonlinear patterns and feature interactions.
- Gradient boosting builds trees sequentially so that each stage improves the current ensemble.
- Learning rate, number of trees, and base-tree depth must be tuned together.
- Standard feature scaling is normally unnecessary for tree-based regression.
- Tree ensembles can perform strongly on tabular data but are less transparent than simple linear models.
- Poor extrapolation beyond the training range is a fundamental practical limitation of tree-based regressors.
| CONNECTION TO MODEL SELECTION No regression family is automatically best. Linear and regularized models provide transparent trend-based baselines, while tree ensembles capture nonlinear structure and interactions. A strong workflow compares them under the same validation design and keeps the model whose generalization behavior best matches the application. |
Quick knowledge check
1. Why does a regression leaf usually predict the mean target value?
2. What makes a decision tree prediction piecewise constant?
3. How does averaging reduce the variance of a random forest?
4. Why does random feature selection make forest trees less correlated?
5. What does a boosting tree attempt to correct?
6. How are learning_rate and n_estimators related?
7. Why is feature scaling usually unnecessary for tree splits?
8. Why can a tree ensemble fail when asked to extrapolate beyond the training range?
Suggested student deliverable
Submit a notebook or short report containing the baseline and tuned model-comparison tables, training-versus-test R² analysis, residual plots, the random-forest feature-importance figure, the extrapolation experiment, and a short conclusion explaining which model you would deploy and why.