0. Introduction

Gradient Boosting is an ensemble learning method that builds many weak learners, usually shallow decision trees, one after another. Each new tree tries to correct the errors made by the previous trees. In MATLAB, the main functions are fitcensemble for classification and fitrensemble for regression. For regression, the standard boosted-tree method is LSBoost. For classification, MATLAB supports boosted tree methods such as AdaBoostM1, LogitBoost, and RUSBoost, among others.

This tutorial explains what Gradient Boosting is, how it works, when to use it, and how to implement it in MATLAB for both classification and regression. Because you wrote “Random Forests Gradient Boosting,” I’m interpreting that as Gradient Boosting in the same format as the Random Forest tutorial. Random Forests and Gradient Boosting are both tree ensembles, but Random Forests build trees independently with bagging, while boosting builds them sequentially to reduce previous errors.

1. What is Gradient Boosting?

Gradient Boosting is a supervised learning technique that combines many weak models into one strong model. Instead of training all trees independently, the algorithm trains them sequentially. Each new tree focuses on the mistakes or residuals left by the current ensemble. MATLAB documents boosted classification and regression ensembles through fitcensemble and fitrensemble.

In simple terms:

• start with a weak model,
• measure its errors,
• train a new small tree to reduce those errors,
• add that tree to the ensemble,
• repeat many times.

For regression, MATLAB uses least-squares boosting (LSBoost) as the boosted regression ensemble type.

2. Why use Gradient Boosting?

Gradient Boosting is popular because it often gives:

• strong predictive accuracy,
• good performance on nonlinear relationships,
• automatic handling of interactions between variables,
• strong results with tabular data.

In MATLAB’s ensemble framework, boosting and bagging are major ensemble families for tree-based models, and boosted ensembles are standard options for both classification and regression workflows.

3. Main ideas behind Gradient Boosting

3.1 Weak learners

Boosting usually uses small trees as weak learners. MATLAB notes that when boosting decision trees, the defaults are shallow trees; older fitensemble documentation says boosting grows stumps by default, while current boosted regression ensemble docs note that fitrensemble grows shallow trees by default.

3.2 Sequential learning

Unlike Random Forests, Gradient Boosting does not train trees independently. Each tree is trained after the previous ones and tries to improve the combined model. That sequential correction behavior is the defining idea of boosting in MATLAB’s ensemble algorithm framework.

3.3 Learning rate

Boosting often uses a learning rate, also called shrinkage, to control how much each new tree contributes. MathWorks examples for boosted regression trees explicitly use a shrinkage value, such as 0.2, with fitrensemble.

3.4 Number of learners

A boosting model is built from many trees, often called learning cycles in MATLAB. In fitcensemble and fitrensemble, this is controlled by NumLearningCycles.

3.5 Tree depth

Boosting often works best with shallow trees, not very deep ones. MATLAB’s boosted tree workflows commonly use shallow tree learners by default, and you can customize them with templateTree.

4. MATLAB tools you need to know

The most important MATLAB tools for Gradient Boosting are:

• fitcensemble → boosted ensembles for classification,
• fitrensemble → boosted ensembles for regression,
• templateTree → customize tree learners,
• predict → make predictions,
• loss → compute model loss,
• Classification Learner → app-based classification workflow,
• Regression Learner → app-based regression workflow. 

Part I — Gradient Boosting for Classification

5. First boosted classification example

Let us begin with a small binary classification example.

clc;
clear;
close all;

% Example data
X = [1 2;
     2 3;
     2 1;
     3 2;
     6 7;
     7 8;
     8 7;
     7 6];

Y = categorical([1;1;1;1;2;2;2;2]);

% Train boosted classification ensemble
Mdl = fitcensemble(X, Y, ...
    'Method', 'AdaBoostM1', ...
    'NumLearningCycles', 100);

% Predict on training data
YPred = predict(Mdl, X);

disp('Predicted labels:');
disp(YPred);

MATLAB documents fitcensemble as the main command-line function for classification ensembles and shows boosted classification examples using methods such as AdaBoostM1.

6. Train/test split for classification

In machine learning, we should evaluate on unseen data.

clc;
clear;
close all;

% Dataset
X = [1 2;
     2 3;
     2 1;
     3 2;
     6 7;
     7 8;
     8 7;
     7 6;
     1.5 2.5;
     6.5 7.5];

Y = categorical([1;1;1;1;2;2;2;2;1;2]);

% Split data
rng(1);
cv = cvpartition(Y, 'HoldOut', 0.3);

XTrain = X(training(cv), :);
YTrain = Y(training(cv), :);

XTest = X(test(cv), :);
YTest = Y(test(cv), :);

% Train boosted model
Mdl = fitcensemble(XTrain, YTrain, ...
    'Method', 'AdaBoostM1', ...
    'NumLearningCycles', 100);

% Predict
YPred = predict(Mdl, XTest);

% Accuracy
accuracy = mean(YPred == YTest) * 100;
fprintf('Test Accuracy = %.2f%%\n', accuracy);

This follows the standard MATLAB ensemble workflow of training with fitcensemble and predicting with predict.

7. Confusion matrix

A confusion matrix is useful for understanding classification performance.

cm = confusionmat(YTest, YPred);
disp('Confusion Matrix:');
disp(cm);

confusionchart(YTest, YPred);
title('Gradient Boosting Confusion Matrix');

This gives more detail than accuracy alone by showing which classes are confused with each other.

8. Classification loss

MATLAB ensemble classifiers support loss.

clc;
clear;
close all;

load ionosphere

Mdl = fitcensemble(X, Y, ...
    'Method', 'AdaBoostM1', ...
    'NumLearningCycles', 100);

L = loss(Mdl, X, Y);

fprintf('Classification Loss = %.4f\n', L);
fprintf('Approximate Accuracy = %.2f%%\n', (1 - L) * 100);

MathWorks documents classification ensemble objects and examples of boosted classification models, where loss is a standard evaluation function.

9. MATLAB default boosting behavior for classification

A useful detail: for binary classification, fitcensemble(X,Y) defaults to a boosted tree ensemble using LogitBoost with 100 classification trees. MathWorks states this directly in its “Train Classification Ensemble” example.

clc;
clear;
close all;

load ionosphere

% Default fitcensemble behavior for binary classification
Mdl = fitcensemble(X, Y);

YPred = predict(Mdl, X);
acc = mean(strcmp(YPred, Y)) * 100;

fprintf('Training Accuracy = %.2f%%\n', acc);

10. Comparing boosting methods

MATLAB supports several classification boosting algorithms, including:

• AdaBoostM1
• LogitBoost
• RUSBoost
• RobustBoost 

A simple comparison example:

clc;
clear;
close all;

load ionosphere

methods = {'AdaBoostM1', 'LogitBoost'};
acc = zeros(numel(methods),1);

for i = 1:numel(methods)
    Mdl = fitcensemble(X, Y, ...
        'Method', methods{i}, ...
        'NumLearningCycles', 100);
    YPred = predict(Mdl, X);
    acc(i) = mean(strcmp(YPred, Y)) * 100;
end

disp(table(methods', acc, 'VariableNames', {'Method','Accuracy'}));

When boosting becomes too focused on hard or noisy observations, MathWorks notes that robust methods such as RobustBoost can sometimes help.

Part II — Gradient Boosting for Regression

11. What is boosted regression?

For regression, boosting trains trees sequentially to reduce residual error in predicting a continuous target. MATLAB states that the only available boosted regression ensemble type is LSBoost.

12. First regression example with fitrensemble

clc;
clear;
close all;

% Simple regression dataset
X = (1:10)';
Y = [1.2; 1.9; 2.8; 3.9; 5.1; 5.9; 7.0; 8.1; 8.9; 10.2];

% Train boosted regression ensemble
Mdl = fitrensemble(X, Y, ...
    'Method', 'LSBoost', ...
    'NumLearningCycles', 100);

% Predict
YPred = predict(Mdl, X);

disp(table(X, Y, YPred));

MathWorks documents fitrensemble for regression ensembles and identifies LSBoost as the boosted regression-tree method.

13. Plotting regression predictions

clc;
clear;
close all;

X = (1:10)';
Y = [1.2; 1.9; 2.8; 3.9; 5.1; 5.9; 7.0; 8.1; 8.9; 10.2];

Mdl = fitrensemble(X, Y, ...
    'Method', 'LSBoost', ...
    'NumLearningCycles', 100);

YPred = predict(Mdl, X);

plot(X, Y, 'o', 'MarkerSize', 8, 'LineWidth', 1.5);
hold on;
plot(X, YPred, '-s', 'LineWidth', 1.5);
xlabel('X');
ylabel('Y');
title('Gradient Boosting Regression');
legend('Original Data', 'Predicted Values');
grid on;

14. Regression metrics: MAE, MSE, RMSE

clc;
clear;
close all;

X = (1:10)';
Y = [1.2; 1.9; 2.8; 3.9; 5.1; 5.9; 7.0; 8.1; 8.9; 10.2];

Mdl = fitrensemble(X, Y, ...
    'Method', 'LSBoost', ...
    'NumLearningCycles', 100);

YPred = predict(Mdl, X);

MAE = mean(abs(Y - YPred));
MSE = mean((Y - YPred).^2);
RMSE = sqrt(MSE);

fprintf('MAE  = %.4f\n', MAE);
fprintf('MSE  = %.4f\n', MSE);
fprintf('RMSE = %.4f\n', RMSE);

For regression ensembles, these metrics are the usual way to evaluate predictive performance.

15. Real regression example with carsmall

clc;
clear;
close all;

load carsmall

X = [Horsepower Weight];
Y = MPG;

% Remove missing values
idx = all(~isnan(X),2) & ~isnan(Y);
X = X(idx,:);
Y = Y(idx);

% Train boosted regression model
Mdl = fitrensemble(X, Y, ...
    'Method', 'LSBoost', ...
    'NumLearningCycles', 100);

% Predict
YPred = predict(Mdl, X);

% RMSE
RMSE = sqrt(mean((Y - YPred).^2));
fprintf('RMSE = %.4f\n', RMSE);

plot(Y, YPred, 'o');
xlabel('Actual MPG');
ylabel('Predicted MPG');
title('Boosted Regression Trees on carsmall');
grid on;

MathWorks uses carsmall in its regression ensemble examples and documents LSBoost as the default boosted regression approach there.

Part III — Customizing the Tree Learners

16. Why customize the tree depth?

Boosting usually benefits from small to moderate trees. MATLAB lets you customize the underlying tree learner by using templateTree. MathWorks examples show this for boosted regression trees with settings such as MaxNumSplits, MinLeafSize, and shrinkage.

clc;
clear;
close all;

load carsmall

X = [Horsepower Weight];
Y = MPG;

idx = all(~isnan(X),2) & ~isnan(Y);
X = X(idx,:);
Y = Y(idx);

t = templateTree('MaxNumSplits', 10, 'MinLeafSize', 5);

Mdl = fitrensemble(X, Y, ...
    'Method', 'LSBoost', ...
    'Learners', t, ...
    'NumLearningCycles', 100);

YPred = predict(Mdl, X);
RMSE = sqrt(mean((Y - YPred).^2));

fprintf('RMSE = %.4f\n', RMSE);

17. Using shrinkage

Shrinkage reduces the contribution of each new tree and often improves generalization when paired with more trees.

clc;
clear;
close all;

load carsmall

X = [Horsepower Weight];
Y = MPG;

idx = all(~isnan(X),2) & ~isnan(Y);
X = X(idx,:);
Y = Y(idx);

t = templateTree('MaxNumSplits', 10);

Mdl = fitrensemble(X, Y, ...
    'Method', 'LSBoost', ...
    'Learners', t, ...
    'NumLearningCycles', 150, ...
    'LearnRate', 0.1);

YPred = predict(Mdl, X);
RMSE = sqrt(mean((Y - YPred).^2));

fprintf('RMSE with shrinkage = %.4f\n', RMSE);

MathWorks’ boosted regression-tree example explicitly uses a learning rate for shrinkage in the training setup.

Part IV — Classification Learner and Regression Learner

18. Classification Learner

MATLAB’s Classification Learner app lets you train and compare ensemble models, including boosted tree classifiers. The classification ensembles page points users to Classification Learner for interactive exploration.

classificationLearner

Typical workflow:

1. import your dataset,
2. choose the response variable,
3. select boosted tree models,
4. train and compare them,
5. inspect validation results,
6. export the best model.

19. Regression Learner

For regression, MATLAB’s Regression Learner supports ensemble workflows as well, including boosted regression trees through the regression ensemble framework.

regressionLearner

Typical workflow:

1. import the data,
2. choose the numeric target,
3. select boosted trees,
4. train and compare them,
5. export the best model.

Part V — End-to-End Mini Projects

20. Classification project: predict pass or fail

clc;
clear;
close all;

% Example student data
StudyHours = [1;2;2;3;4;5;5;6;7;8];
Attendance = [50;55;60;65;70;75;80;85;90;95];
Pass = categorical([0;0;0;0;0;1;1;1;1;1]);

X = [StudyHours Attendance];
Y = Pass;

% Train/test split
rng(2);
cv = cvpartition(Y, 'HoldOut', 0.3);

XTrain = X(training(cv), :);
YTrain = Y(training(cv), :);

XTest = X(test(cv), :);
YTest = Y(test(cv), :);

% Train boosted classifier
Mdl = fitcensemble(XTrain, YTrain, ...
    'Method', 'AdaBoostM1', ...
    'NumLearningCycles', 100);

% Predict
YPred = predict(Mdl, XTest);

% Accuracy
accuracy = mean(YPred == YTest) * 100;
fprintf('Test Accuracy = %.2f%%\n', accuracy);

% Confusion matrix
confusionchart(YTest, YPred);
title('Boosted Trees: Pass/Fail');

% Predict a new student
newStudent = [6 82];
newClass = predict(Mdl, newStudent);

disp('Predicted class for new student:');
disp(newClass);

This mini-project uses the same core MATLAB boosted-classification workflow documented for fitcensemble.

21. Regression project: predict house prices

clc;
clear;
close all;

% Example house dataset
Size = [50; 60; 70; 80; 90; 100; 110; 120];
Rooms = [2; 3; 3; 4; 4; 5; 5; 6];
Age = [20; 18; 15; 12; 10; 8; 5; 3];
Price = [100; 120; 135; 150; 170; 190; 210; 230];

X = [Size Rooms Age];
Y = Price;

% Train/test split
rng(3);
idx = randperm(length(Y));

trainIdx = idx(1:6);
testIdx = idx(7:8);

XTrain = X(trainIdx, :);
YTrain = Y(trainIdx);

XTest = X(testIdx, :);
YTest = Y(testIdx);

% Train boosted regressor
t = templateTree('MaxNumSplits', 10);
Mdl = fitrensemble(XTrain, YTrain, ...
    'Method', 'LSBoost', ...
    'Learners', t, ...
    'NumLearningCycles', 100, ...
    'LearnRate', 0.1);

% Predict
YPred = predict(Mdl, XTest);

% RMSE
RMSE = sqrt(mean((YTest - YPred).^2));
fprintf('Test RMSE = %.4f\n', RMSE);

disp(table(YTest, YPred, 'VariableNames', {'ActualPrice','PredictedPrice'}));

This follows MATLAB’s documented regression-ensemble workflow using LSBoost.

Part VI — Common mistakes beginners make

22. Confusing boosting with bagging

Random Forests use bagging, where trees are trained independently. Gradient Boosting trains trees sequentially to correct earlier errors. MATLAB documents both families under ensemble learning, but they are different strategies.

23. Using trees that are too deep

Boosting often works best with shallow trees. MATLAB’s boosted-tree defaults and examples reflect that.

24. Using too high a learning rate

A large learning rate can make the model unstable or overfit. MathWorks’ boosted regression example explicitly pairs shrinkage with a moderate number of trees.

25. Looking only at training accuracy

Boosting can fit training data very well. You still need holdout testing or app-based validation to estimate real performance. MATLAB’s learner apps are designed for this compare-and-validate workflow.

26. Forgetting that boosted regression in MATLAB is LSBoost

For regression, MATLAB documents LSBoost as the boosted regression ensemble type.

Part VII — When should you use Gradient Boosting?

Gradient Boosting is a good choice when:

• you want high predictive performance on tabular data,
• relationships may be nonlinear,
• interactions between predictors matter,
• you are comfortable tuning ensemble settings.

You might avoid it when:

• you need the simplest possible interpretable model,
• training speed must be minimal,
• a quick baseline is enough.

These are practical consequences of how boosted ensembles are trained and tuned in MATLAB’s ensemble framework.

Part VIII — Summary

Gradient Boosting is one of the most powerful ensemble methods for tabular machine learning. In MATLAB, fitcensemble handles boosted classification trees, while fitrensemble handles boosted regression trees through LSBoost. MATLAB also supports these workflows through Classification Learner and Regression Learner, and it provides customization through templateTree, NumLearningCycles, and LearnRate.

A practical workflow is:

1. prepare the data,
2. choose classification or regression,
3. start with shallow trees,
4. choose the number of learning cycles,
5. tune the learning rate,
6. evaluate on unseen data,
7. compare results and export the best model.

Part IX — MATLAB cheat sheet

Boosted classification trees

Mdl = fitcensemble(X, Y, ...
    'Method', 'AdaBoostM1', ...
    'NumLearningCycles', 100);
YPred = predict(Mdl, Xnew);

Default binary boosted classifier

Mdl = fitcensemble(X, Y);

Boosted regression trees

Mdl = fitrensemble(X, Y, ...
    'Method', 'LSBoost', ...
    'NumLearningCycles', 100);
YPred = predict(Mdl, Xnew);

Custom tree learner

t = templateTree('MaxNumSplits', 10, 'MinLeafSize', 5);
Mdl = fitrensemble(X, Y, ...
    'Method', 'LSBoost', ...
    'Learners', t);

With shrinkage

Mdl = fitrensemble(X, Y, ...
    'Method', 'LSBoost', ...
    'NumLearningCycles', 150, ...
    'LearnRate', 0.1);

These patterns align with current MathWorks documentation for boosting in MATLAB.

Practice exercises

Exercise 1

Train a boosted classification ensemble on a small binary dataset and compute test accuracy

clc;
clear;
close all;

% Small binary dataset
X = [1 2;
     2 3;
     2 1;
     3 2;
     6 7;
     7 8;
     8 7;
     7 6;
     1.5 2.5;
     6.5 7.5];

Y = categorical([1;1;1;1;2;2;2;2;1;2]);

% Split data into training and test sets
rng(1);
cv = cvpartition(Y, 'HoldOut', 0.3);

XTrain = X(training(cv), :);
YTrain = Y(training(cv), :);

XTest = X(test(cv), :);
YTest = Y(test(cv), :);

% Train boosted classification ensemble
Mdl = fitcensemble(XTrain, YTrain, ...
    'Method', 'AdaBoostM1', ...
    'NumLearningCycles', 100);

% Predict on test set
YPred = predict(Mdl, XTest);

% Compute accuracy
accuracy = mean(YPred == YTest) * 100;
fprintf('Test Accuracy = %.2f%%\n', accuracy);

Exercise 2

Use the ionosphere dataset to train a boosted classification model and compute classification loss

clc;
clear;
close all;

% Load dataset
load ionosphere

% Train boosted classifier
Mdl = fitcensemble(X, Y, ...
    'Method', 'AdaBoostM1', ...
    'NumLearningCycles', 100);

% Compute classification loss
L = loss(Mdl, X, Y);

fprintf('Classification Loss = %.4f\n', L);
fprintf('Approximate Accuracy = %.2f%%\n', (1 - L) * 100);

Exercise 3

Compare AdaBoostM1 and LogitBoost on a binary classification dataset

clc;
clear;
close all;

% Load dataset
load ionosphere

methods = {'AdaBoostM1', 'LogitBoost'};
accuracies = zeros(length(methods),1);

for i = 1:length(methods)
    Mdl = fitcensemble(X, Y, ...
        'Method', methods{i}, ...
        'NumLearningCycles', 100);

    YPred = predict(Mdl, X);
    accuracies(i) = mean(strcmp(YPred, Y)) * 100;
end

% Display comparison
disp(table(methods', accuracies, ...
    'VariableNames', {'Method','Accuracy'}));

Exercise 4

Train a boosted regression ensemble on a simple numeric dataset and compute RMSE

clc;
clear;
close all;

% Simple regression dataset
X = (1:10)';
Y = [1.2; 1.9; 2.8; 3.9; 5.1; 5.9; 7.0; 8.1; 8.9; 10.2];

% Train boosted regression ensemble
Mdl = fitrensemble(X, Y, ...
    'Method', 'LSBoost', ...
    'NumLearningCycles', 100);

% Predict on training data
YPred = predict(Mdl, X);

% Compute RMSE
RMSE = sqrt(mean((Y - YPred).^2));
fprintf('RMSE = %.4f\n', RMSE);

% Plot actual and predicted values
plot(X, Y, 'o', 'MarkerSize', 8, 'LineWidth', 1.5);
hold on;
plot(X, YPred, '-s', 'LineWidth', 1.5);
xlabel('X');
ylabel('Y');
title('Boosted Regression Trees');
legend('Original Data', 'Predicted Values');
grid on;

Exercise 5

Build a small end-to-end Gradient Boosting project with train/test split, evaluation, and prediction for a new observation

clc;
clear;
close all;

% Example student dataset
StudyHours = [1;2;2;3;4;5;5;6;7;8];
Attendance = [50;55;60;65;70;75;80;85;90;95];
Pass = categorical([0;0;0;0;0;1;1;1;1;1]);

X = [StudyHours Attendance];
Y = Pass;

% Train/test split
rng(2);
cv = cvpartition(Y, 'HoldOut', 0.3);

XTrain = X(training(cv), :);
YTrain = Y(training(cv), :);

XTest = X(test(cv), :);
YTest = Y(test(cv), :);

% Train boosted classifier
Mdl = fitcensemble(XTrain, YTrain, ...
    'Method', 'AdaBoostM1', ...
    'NumLearningCycles', 100);

% Predict on test set
YPred = predict(Mdl, XTest);

% Compute accuracy
accuracy = mean(YPred == YTest) * 100;
fprintf('Test Accuracy = %.2f%%\n', accuracy);

% Confusion matrix
confusionchart(YTest, YPred);
title('Gradient Boosting: Pass/Fail');

% Predict a new student
newStudent = [6 82];
newClass = predict(Mdl, newStudent);

disp('Predicted class for new student:');
disp(newClass);

Short recap

Exercise 1

Train a boosted binary classifier and compute test accuracy.

Exercise 2

Use ionosphere and compute classification loss.

Exercise 3

Compare AdaBoostM1 and LogitBoost.

Exercise 4

Train a boosted regressor with LSBoost and compute RMSE.

Exercise 5

Build a complete mini-project with train/test split, evaluation, and new prediction.