Boosting is an ensemble technique that combines predictors (usually Decision Trees) sequentially.

It starts by training a decision tree known as a weak learner. This tree is then passed as input to the next tree to improve on it.

In Gradient Boosting, individual trees train upon the residuals (the difference between the prediction and the actual results) of the previous tree. Instead of aggregating trees, gradient boosted trees learns from errors during each boosting round.

Trees are added one at a time, and existing trees in the model are not changed. At any step m, the ensemble fm is given by the equation

f_m(x) = f_{m - 1}(x) + \gamma* h_m(x)

where γ is known as the learning rate.

hm(x) is known as the weak learner at step m.

Mathematically, the loss function L when doing gradient boosting is given by the equation

L(y, f_m) = (y - f_m)^2

and the weak learner hm(x) is given by the partial derivative of L w.r.t the ensemble at step (m-1):

h_m(x) = -\frac{\partial L}{\partial f_{m-1}} = 2(y - f_{m-1})

Hence, hm(x) corresponds to the residuals from the previous tree.

Here’s a visual guide to the concept of gradient boosting.

What is XGBoost?

XGBoost stands for Extreme Gradient Boosting (XGBoost) and is an open-source library that provides an efficient and effective implementation of the gradient boosting algorithm.

The documentation can be found at https://xgboost.readthedocs.io/en/latest/index.html.

Link to Code (as HTML file)

This tutorial demonstrates how to do XGBoosting in sklearn. To do that, you need to install the xgboost module first (you can do that by running the command pip install xgboost in “Anaconda Prompt”).

Click the link below for the sklearn code for XGBoosting.

XGBoost.html