Gradient Boost Explained
All about Gradient Boost in one article !
⬅️ Previous: [Ensemble Learning, Bagging and Random Forests]
Boosting
Boosting is a general ensemble method in machine learning that aims to create a strong classifier or regressor by combining the predictions of several weaker models. The idea is to build the strong model incrementally, by sequentially adding weak models that are trained to correct the mistakes made by the existing ensemble. Learn the mistakes made by previous models, thus next model make less mistakes and it reduces the “High Bias & Low Variance” of the model to “Low Bias & Low Variance”. It is also used in competitive Machine Learning like Kaggle Compition.
Q. How Boosting is different from Bagging?
- all models are trained sequentially (thus no n_jobs parameter).
- all models are trained on same dataset.
- We use High Bais and Low Variance model example Shallow Decision Tree, linear Models.
Types of Boosting Models
- Gradient Boosting :
- Adaboost :
- Xgboost :
- Lightgbm :
- Catboost :
Gradient Boosting
Gradient Boosting is a machine learning ensemble technique that aims to build a strong predictive model by combining the predictions of several weaker models (model with High Bias and Low Variance) using the concept of Additive Training, typically decision trees. The method works by iteratively adding models to the ensemble, with each new model trained to correct the mistakes made by the combined ensemble of existing models. It is a powerful and flexible method that can be used for both regression and classification tasks. It is highly effective for building ranking systems, commonly known as Learning to Rank (LTR), particularly for search engines and recommendation systems. It works by using models like LightGBM, XGBoost, or CatBoost to optimize for ranking-specific metrics (like NDCG) rather than simple classification accuracy, producing superior, high-performance ordered results.
It is particularly effective when the data has complex, non-linear relationships, and it often performs well even with little hyperparameter tuning. Its popularity in real-world applications and machine learning competitions is testament to its effectiveness, and it has implementations in most major machine learning libraries, such as scikit-learn, XGBoost, LightGBM, and CatBoost.
“Gradient Boosting relies on an additive training process. Instead of trying to build one massive, perfect model all at once, the target variable is predicted by summing the outputs of several individual, weaker models (typically decision trees). The final model can be represented as: F(X) = F_0(X) + f_1(X) + f_2(X) + … + F_N(X)
In this approach, X represents the full set of input features. Each new function (f) is a distinct model trained specifically to learn and correct the errors (residuals) made by the combination of all the previous models. By sequentially adding these models together, the algorithm captures complex, non-linear relationships in the data to produce a highly accurate final prediction.”
Initial γ will be constant and equal to mean of the output column i.e. y_bar (on differentiation of initial function, f0(x) = 0 to get minimum γ).
N is the number of rows and M is the number of Decision Trees we want to train. Here, Rim is called Pseudo Residual. Input column of the Decision Trees are the real input column but the output column is replaced with the Pseudo Residual. Terminal Regions, Rjm are the regions created after Splitting of Data during Decision Tree training.
slope, Rim tell direction in which Loss Function will minimize after differentiation. If slope is positive, residual will be negative and loss will be smaller in negative of slope direction.
Working of Gradient Boosting with example.
We have a dataset with Experience as input column and Salary as output column in a Regression Problem. we have to build a Gradient Boosting model to find the relationship between experience and salary. Initial model, f0(x) gives the mean of y column irrespective of x.
The Initial model, f0(x) is not don’t perform well, so we train first Decision Tree, h1(x) with residuals (y - f0(x)) as output and x as input. We apply first model, f1(x) = f0(x) + h1(x) on original y vs x.
The first model, f1(x) is still make mistakes but lower than pervious one, so we train second Decision Tree, h2(x) with residuals (y — f1(x)) as output and x as input. We apply second model, f2(x) = f1(x) + h2(x) on original y vs x.
The second model, f2(x) is still make mistakes but lower than pervious one, so we train third Decision Tree, h3(x) with residuals (y — f2(x)) as output and x as input. We apply third model, f3(x) = f2(x) + h3(x) on original y vs x.
Repeate the same multiple times till get the desired result.
After training 10 Decision Trees, The 10th model is doing very less mistakes then all earlier version and base line model.
“Gradient Boosting is performing Gradient Descent in Function Space.”
Q. Difference Between Gradient Boosting and Gradient Descent?
- Function Space Vs Parameter Space: The relationship between Loss Function and predicted variable fi(x) from i = 1 to n will be n + 1 dimention where n is number of rows because there will be n fi(x) and one Loss Function. Here, in Decision Tree, we have to find the function for which Loss Function is minimum and it is called Function Space. It is Non-Parametric and function the data follow in unknown. Gradient Boosting work in Parameter Space. While in Linear Regression, we find the slope, m for which Loss Function is minimum and it is called Parameter Space. It is Parametric and Function the data follow is already assumed. Parameter Space is subset of Function Space. Gradient Descent work in Parameter Space.
- Update the Function: If we do new function = old function — slope like (m_new = m_old - η*slope in linear regression) then there will be overfitting (fi(x) = yi). So, we train a weak decision tree, hi(x) on Residual vs x. thus, fm(x) = fm-1(x) + η*hm(x) where η represents the learning rate. Here, don’t use slope directly but weak prediction of slope.
Advantages of Gradient Boosting
- Work with any Loss Funcction like MSE, MAE, LogLoss etc.
- We can use any sub algorithm like Decision Tree, etc.
Working of Gradient Boosting with Classification
- The Loss Function: Gradient Boosting for Classification follow the same algorithm as followed by Regression but with different Loss Function i.e. Log Loss.
The above Loss Function give Prediction Output in term of Probability of the Positive Class. But, we want Prediction Output in the term of Log of Odds of Positive Class instead.
Get Mr. Vivek Kumar’s stories in your inbox
Join Medium for free to get updates from this writer.
Q. Why we use log of odds/Log(odds) instead of Probability?
- Unconstrained Prediction Space: Log odds can span the entire real line (−∞ to +∞), while probabilities are constrained between 0 and 1. Algorithms like gradient boosting involve adding corrections (via the weak learners) to the predictions iteratively. If you’re working in the log odds space, there’s no need to worry about your predictions going out of bounds or negative side.
- Better Gradients: When computing gradients (which guide the addition of new trees in boosting), the gradients can be more informative and have better magnitudes in the log odds space than in the probability space, especially when probabilities are near 0 or 1.
This is the Actual Loss Function we use in the Classification problem for Gradient Boosting and it give ouput in term of Log of Odds, not probability.
- Finding initial F_0(x) : We have to find the value of gamma
γorlog(odds)hat minimzes below Expression containing the above Loss Function whereγ = log(odds). It gives output in terms of log of odds.
On differentiating the F_0(x) equation with respect to γ or log(odds), F_0(x) = γ = log(odds)
The initial prediction F_0(x) is the log-odds of the mean probability.
2. (a) We have to find Pseudo Residuals/Errors/r_im.
For model F_1(x), in the below equation F(x_i) = F_0(x) and we know F_0(x) = γ = log(odds)
On differentiating the above Loss function with respect to log(odds). for each row, Pseudo Residuals/Errors/r_im = y_i - p_i.
2. (b) Train Regression Tree : We train weak Regression Decision Tree with with original features X againt above r_im as y giving terminal regions. The output of the Regression Tree’s leaf node is not usefull and it is average errors for the terminal region.
Q. Why we use Regression instead of Classification Decision Tree in Classification of Gradient Boosting ?
Answer : Regression because Residual is Numerical not Categorical.
2. (c) we have to calculate the gamma, γ_jm for each leaf nodes of the previous regression tree that minimizes the below equation such that the input feature’s row, x_i belongs to r_jm the node. γ_jmis Log of Odds.
For model F_1(x), in the below equation Fm-1(x_i) = F_0(x_i) and we know F_0(x) = γ = log(odds)
On applying Taylor Series and differentiating above equation with respect to γ, Output from leaf node, γ_jm =
2. (d) Update the model :
For first model, F_1(x) = F_0(x) + γ_jm. We train M weak Regression Decision Tree models like that.
3. Final Model: F(x) = F_m(x) = F_m-1(x) + last decision tree’s output γ_jm
To find out which class a row belongs to, you need to convert that final Log of Odds value back into something actionable like p is Probability Output calculated using log of Odds formula. One can do this by passing F_m(x) through the Sigmoid (or Logistic) Function:
Example :
To better visualize the data, we have converted 2D to 3D.
Our goal is to build a gradient boosting model that classifies those two classes. The first step is making a uniform prediction on a probability of class 1 (we will call it p) for all the data points. The most reasonable value for the uniform prediction might be the proportion of class 1 which is just a mean of y. p = P(y=1) = y_bar
Here is a 3D representation of the data and the initial prediction. At this moment, the prediction is just a plane which has the uniform value p=mean(y) on the y axis all the time.
But, for actual classification, we use log of odds then calculate it’s probability for visualization.
In our data, the mean of y is 0.56. As it is bigger than 0.5, everything is classified into class 1 with this initial prediction. Some of you might feel that this uniform value prediction does not make sense, but don’t worry. We will improve our prediction as we add more weak models to it.
To improve our prediction quality, we might want to focus on the residuals (i.e. prediction error) from our initial prediction as that is what we want to minimize. The residuals are defined as rᵢ = yᵢ−p (i represents the index of each data point). In the figure below, the residuals are shown as the brown lines that are the perpendicular lines from each data point to the prediction plane.
To minimize these residuals, we are building a regression tree model with both x₁ and x₂ as its features and the residuals r as its target. If we can build a tree that finds some patterns between x and r, we can reduce the residuals by utilizing that created tree. To simplify the demonstration, we are building very simple trees each of that only has one split and two terminal nodes which is called “stump”. Please note that gradient boosting trees usually use shallow trees (often between 4 and 32 leaves depending on the complexity of the data). Here we are creating the first tree predicting the residuals with two different values r={0.1,−0.6}.
Now calculating residuals
By substituting actual values for the variables in the right side of the above equation, we get our updated prediction F₁(x).
The purple-colored plane is the initial prediction p₀ and it is updated to the red and yellow plane p₁.
We can notice that the residuals have reduced.
In the next step, we are creating a regression tree again using the same x1 and x2 as the features and the updated residuals r as its target.
We iterate these steps until the model prediction stops improving. The figures below show the optimization process from 0 to 4 iterations.
You can see the combined prediction p(x) (red and yellow plane) is getting closer to our target y as we add more trees into the combined model. This is how gradient boosting works to predict complex targets by combining multiple weak models.
Example:
# Classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.datasets import load_iris
import pandas as pd
data = load_iris(as_frame=True)
df = data.frame
X = df.drop(columns=['target'])
y = df['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = GradientBoostingClassifier(random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(f"Classification Accuracy: {accuracy_score(y_test, y_pred)}") # 1.0# Regression
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_diabetes
from sklearn.metrics import r2_score
import pandas as pd
data = load_diabetes(as_frame=True)
df = data.frame
X = df.drop(columns=['target'])
y = df['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = GradientBoostingRegressor(random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(f"Regression R-squared Score: {r2_score(y_test, y_pred)}") # 0.4529343➡️ Next Up: [XGBoost (eXtreme Gradient Boosting)]
If you found value in this content, I invite you to stay connected for more insightful posts. Your time and interest are greatly appreciated. Thank you for reading!
