Ensemble Learning, Bagging and Random Forests

The Wisdom of the Crowd: Making Better Predictions with Machine Learning

Photo by Enfocus Collective on Unsplash

⬅️ Previous: [Decision Trees Explained]

Ensemble Learning

The “wisdom of the crowd” refers to the concept that the collective intelligence, opinions, and estimates of a large, diverse group of people are often more accurate than those of individual experts. Popularized by James Surowiecki, this phenomenon works best when participants offer independent, varied viewpoints, which allows individual errors to cancel each other out.

Example 1: Product reviews on shopping websites. By aggregating hundreds of reviews, the overall rating usually reflects the true quality of the product.

Example 2: In Kaun Banega Crorepati (the Indian version of Who Wants to Be a Millionaire?), the “Ask the Audience” lifeline is almost always correct. However, this works best when the audience is diverse and represents multiple domains like science, history, and math.

NOTE: The wisdom of the crowd only works well if the sample data truly represents the broader population (all people should not be doctor from profession in KBC example).

The word ensemble traditionally means a small group of musicians, dancers, or actors who perform together to create a unified piece of art.

In data science, Ensemble Learning is a machine learning technique that combines multiple individual models (often called “weak learners”) to create a single, stronger model that produces more accurate and reliable predictions. By aggregating predictions — such as through voting or averaging — it successfully reduces errors like bias, variance, and overfitting.

During prediction, the ensemble uses either multiple different ML models or the same ML model trained on different subsets of data.

• For classification problems: The final prediction is determined by a majority vote (the mode) from all the underlying models for a given data point.
• For regression problems: The final prediction is calculated as the average (mean) of all the models’ outputs.

The most commonly used base models for ensembles are Decision Trees and K-Nearest Neighbors (KNN).

https://machinelearningmastery.com/how-ensemble-learning-works/

Q. When should I use Ensemble Learning?

Answer. It is generally a best practice to apply ensemble learning at the end of the model development pipeline to squeeze out maximum performance and stability.

Advantage and Disadvantage of Ensemble Learning

Disadvantage:

• Large Training and Prediction Time: Training multiple models naturally takes longer than training one.
• High Complexity: The final model is harder to interpret and deploy.

Advantage:

• Improvement in Performance: Often yields higher accuracy than single models.
• Reduction in Bias and Variance: Can shift a model from “Low Bias and High Variance” or “High Bias and Low Variance” into the ideal “Low Bias and Low Variance” territory.
• Robustness: The overall model is much more resilient to noise and outliers.

Types of Ensemble Learning Techniques

1. Voting Ensemble: The base models are different algorithms entirely, but they are all trained on the exact same dataset.
2. Bagging: Uses n instances of the same model algorithm, but each model is trained on a different, random subset of the data. An example is a Random Forest (a collection of Decision Trees).
3. Boosting: Uses n instances of the same model trained sequentially on the same data. However, the mistakes made by one model are passed on to the next model so it can learn from the previous errors. Examples include AdaBoost, Gradient Boosting, and XGBoost.
4. Stacking: Similar to Voting, but instead of taking a simple average or majority vote, a “meta-model” is trained on the predictions (not on actual data) of the base models to output the final prediction.

Bagging (Bootstrap Aggregating)

Bagging is an ensemble technique (not a standalone algorithm) that involves training multiple models independently using the same algorithm, but on different subsets of the training data selected randomly with replacement. These models are then combined through averaging or voting to make predictions. Bagging reduces variance, improves stability, and helps avoid overfitting, making it a highly popular choice.

Because the final output relies on the majority vote of multiple models, it is resistant to small changes or noise in the data, keeping the variance low.

• Bootstrap sampling is the process of generating multiple, random subsets from a training dataset with replacement. This allows duplicate data points within a subset, creating diverse datasets for each model.
• Aggregating is the subsequent step of combining predictions from models trained on these subsets — typically via voting (classification) or averaging/median (regression).

Q. If Ensemble Learning itself reduces variance, is Pruning still needed for Decision Trees?

Answer. Yes. While ensemble learning significantly reduces variance, it might not always eliminate it entirely. In cases where overfitting persists, applying pruning hyperparameters to the base decision trees is still highly beneficial.

Types of Bagging (Based on Subset Creation)

• 1. Bagging: Row sampling is done with replacement.

from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.datasets import make_classification, make_regression
from sklearn.ensemble import BaggingClassifier, BaggingRegressor
from sklearn.metrics import accuracy_score, r2_score
from sklearn.linear_model import LinearRegression
from sklearn.neighbors import KNeighborsRegressor
from sklearn import datasets
from sklearn.svm import SVC

X, y = make_classification(n_samples=10000, n_features=10, n_informative=3)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Bagging Classification
# max_samples=0.5 means 50% of total data
# n_estimators=500 means number of models in Bagging
# bootstrap=True means sampling with replacement
bag = BaggingClassifier(estimator=DecisionTreeClassifier(), n_estimators=500, max_samples=0.5, bootstrap=True, random_state=42)
bag.fit(X_train, y_train)

y_pred = bag.predict(X_test)
print(accuracy_score(y_test, y_pred)) # 0.9415
# Number of rows for each model out of 500
print(bag.estimators_samples_[0].shape) # (4000,)
# Number of columns
print(bag.estimators_features_[0].shape) # (10,)


# Bagging Regression
X_r, y_r = make_regression(n_samples=10000, n_features=10, n_informative=3)
X_train_r, X_test_r, y_train_r, y_test_r = train_test_split(X_r, y_r, test_size=0.2, random_state=42)

bag_r = BaggingRegressor(estimator=DecisionTreeRegressor(), n_estimators=500, max_samples=0.5, bootstrap=True, random_state=42)
bag_r.fit(X_train_r, y_train_r)

y_pred_r = bag_r.predict(X_test_r)
print(r2_score(y_test_r, y_pred_r)) # 0.9888980915053766
# Alternative way to calculate r2_score in one line using bag_r.score
print(bag_r.score(X_test_r, y_test_r)) # 0.9888980915053766

• 2. Pasting: Row sampling is done without replacement.

# Classification
# bootstrap=True means sampling without replacement
# n_jobs=-1 means to make mode training fast
bag = BaggingClassifier(estimator=DecisionTreeClassifier(), n_estimators=500, max_samples=0.25, bootstrap=False, random_state=42, verbose = 1, n_jobs=-1)
bag.fit(X_train,y_train)

y_pred = bag.predict(X_test)
print("Pasting classifier",accuracy_score(y_test,y_pred)) # 0.94

# Regression
bag_r = BaggingRegressor(estimator=DecisionTreeRegressor(), n_estimators=500, max_samples=0.25, bootstrap=False, random_state=42, verbose = 1, n_jobs=-1)
bag_r.fit(X_train_r, y_train_r)

y_pred_r = bag_r.predict(X_test_r)
print("Pasting regressor", r2_score(y_test_r, y_pred_r)) # 0.94

• 3. Random Subspaces: Only column (feature) sampling is done, with or without replacement.

# Classification
# max_features=0.5 means only 50% of column or feature will be used
bag = BaggingClassifier(estimator=DecisionTreeClassifier(), n_estimators=500, max_samples=1.0, bootstrap=False, max_features=0.5, bootstrap_features=True, random_state=42)
bag.fit(X_train,y_train)

y_pred = bag.predict(X_test)
print("Random Subspaces classifier",accuracy_score(y_test,y_pred)) # 0.938

bag.estimators_samples_[0].shape # (8000,)
bag.estimators_features_[0].shape # (5,)


# Regression
bag_r = BaggingRegressor(estimator=DecisionTreeRegressor(), n_estimators=500, max_samples=1.0, bootstrap=False, max_features=0.5, bootstrap_features=True, random_state=42)
bag_r.fit(X_train_r, y_train_r)

y_pred_r = bag_r.predict(X_test_r)
print("Random Subspaces regressor", r2_score(y_test_r, y_pred_r)) # 0.938

• 4. Random Patches: Both Row and Column sampling is done simultaneously.

# Classification
# bootstrap_features=True 
bag = BaggingClassifier(estimator=DecisionTreeClassifier(), n_estimators=500, max_samples=0.25, bootstrap=True, max_features=0.5, bootstrap_features=True, random_state=42)
bag.fit(X_train,y_train)

y_pred = bag.predict(X_test)
print("Random Patches classifier",accuracy_score(y_test,y_pred)) # 0.934


# Regression
bag_r = BaggingRegressor(estimator=DecisionTreeRegressor(), n_estimators=500, max_samples=0.25, bootstrap=True, max_features=0.5, bootstrap_features=True, random_state=42)
bag_r.fit(X_train_r, y_train_r)

y_pred_r = bag_r.predict(X_test_r)
print("Random Patches regressor", r2_score(y_test_r, y_pred_r)) # 0.934

OOB Score (Out-Of-Bag Samples)

When we perform row sampling with replacement, there is a chance that some rows are never passed to any model, while others are passed to multiple times. Statistically, only about 63% of unique rows get sampled for a given tree, leaving roughly 37% (often cited around 1/3) of the data unseen by that specific tree. These unseen data points are called Out-Of-Bag (OOB) samples.

Get Mr. Vivek Kumar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

We can use these OOB samples as a built-in validation set. We pass them through the specific trees that didn’t see them during training and evaluate the predictions. Comparing these predictions to the actual values gives us an “out-of-bag score,” which estimates the prediction error on unseen data.

This is incredibly useful because it allows us to estimate real-world performance without needing to carve out a separate validation set, which is a lifesaver when working with small datasets.

# oob_score=True means OOB is enabled
bag = BaggingClassifier(estimator=DecisionTreeClassifier(), n_estimators=500, max_samples=0.25, bootstrap=True, oob_score=True, random_state=42)
bag.fit(X_train,y_train)

y_pred = bag.predict(X_test)
print("Accuracy", accuracy_score(y_test,y_pred)) # 0.9395
# Accuracy on OOB Samples
print("OOB score", bag.oob_score_) # 0.9375

Key Tuning Notes for Bagging:

• Bagging generally yields better results than Pasting.
• Optimal results usually appear around the 25% to 50% row sampling mark (max_samples).
• Random Patches and Random Subspaces are highly recommended when dealing with high-dimensional data (lots of features).
• Use GridSearchCV or RandomizedSearchCV to find the perfect hyperparameter combinations.

# GridSearchCV Classifier
params = {
    'estimator': [None, BaggingClassifier()],
    'n_estimators': [50,100,500],
    'max_samples': [0.1,0.4,0.7,1.0],
    'max_features' : [0.1,0.4,0.7,1.0],
    'bootstrap' : [True,False],
    'bootstrap_features': [True, False]
    }

search = GridSearchCV(BaggingClassifier(random_state=1, n_jobs=-1), param_grid = params, cv=5, n_jobs=-1, verbose=1)
search.fit(X_train, y_train)

search.best_score_ # 0.94312
search.best_params_
# { 'estimator': None
# 'bootstrap': True,
#  'max_features': 0.7,
#  'max_samples': 1.0,
#  'n_estimators': 500}


# GridSearchCV Regressor
params = {'estimator': [None, LinearRegression(), KNeighborsRegressor(), DecisionTreeRegressor()],
          'n_estimators': [20,50,100],
          'max_samples': [0.5,1.0],
          'max_features': [0.5,1.0],
          'bootstrap': [True, False],
          'bootstrap_features': [True, False]}

search_r = GridSearchCV(BaggingRegressor(random_state=1, n_jobs=-1), param_grid =params, cv=3, n_jobs=-1, verbose=1)
search_r.fit(X_train_r, y_train_r)

print(search_r.best_estimator_.score(X_train_r, y_train_r)) # 0.983
print(search_r.best_estimator_.score(X_test_r, y_test_r)) # 0.802
print(search_r.best_score_) # 0.870
print(search_r.best_params_)
# {'estimator': None, 
# 'bootstrap': True, 
# 'bootstrap_features': False, 
# 'max_features': 1.0, 
# 'max_samples': 1.0, 
# 'n_estimators': 50}

Random Forest

Random Forest is a versatile, widely used machine learning algorithm that belongs to the class of ensemble methods. Specifically, it is a specialized type of bagging technique that relies exclusively on Decision Trees as its base models.

Like generic Bagging, Random Forests perform row sampling (and often column sampling) with or without replacement. Each Decision Tree in a Random Forest will likely overfit its specific subset of data. However, because the trees are trained on different data and features, their errors are not concentrated in the same areas. When they are all averaged together, the correct predictions dominate, and the localized errors are ignored.

Ultimately, Random Forest takes a group of “Low Bias & High Variance” models (as single Decision Tree is Low Bias & High Variance model) and transforms them into a single “Low Bias & Low Variance”.

The Secret to Success: The success of a Random Forest lies in decorrelating the individual trees from each other. The more randomness we introduce, the more independent the errors become, resulting in better overall predictions.

Feature Importance of the Random Forest is just the avearge of the feature importance of individual Decision Trees.

Q. Is Bagging with a Decision Tree exactly the same as a Random Forest?

Answer. No. While they are very similar, there is a core difference:

• In Bagging, feature sampling is done at the tree level (the subset of features is chosen before the tree starts growing). It works with any algorithm including Decision Tree.
• In a Random Forest, feature sampling is done at the node level (a random subset of features is considered at every single split within the tree). It only works with Decision Trees.

Because of this, a Random Forest has much more randomness injected into it than a standard Bagging classifier, usually leading to better performance.

Random Forest Classifier works better than Decision Tree Classifier.

Random Forest Regressor works better than Decision Tree Regressor

Extremely Randomized Trees (Extra Trees)

Extra Trees is a modification of the Random Forest algorithm that changes the way the splitting points for decision tree branches are chosen.

In traditional decision trees (and Random Forests), the algorithm calculates the mathematically optimal split point for each feature at a given node. In the Extra Trees algorithm, a split point for each feature is chosen completely at random. The algorithm then evaluates these randomly generated splits and picks the best one to divide the node. This adds an extra layer of randomness to the model, hence the name “Extremely Randomized Trees”. Because of this arbitrary splitting, Extra Trees tend to be deeper and have more branches than Random Forests. Depending on the dataset, this extra randomness can sometimes yield better performance, especially on tasks where the data may not have clear optimal split points and faster training times than a standard Random Forest.

from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score

X, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

et = ExtraTreesClassifier(random_state=42)
et.fit(X_train, y_train)
et_predictions = et.predict(X_test)

rf = RandomForestClassifier(random_state=42)
rf.fit(X_train, y_train)
rf_predictions = rf.predict(X_test)

# Compute the accuracy of the classifiers
print(accuracy_score(y_test, rf_predictions)) # 0.9
print(accuracy_score(y_test, et_predictions)) # 0.935

Advantages and Disadvantages of Random Forest

Advantages of Random Forest

• Robustness to Overfitting: Random Forests are less prone to overfitting compared to individual decision trees, because they average the results from many different trees, each of which might overfit the data in a different way.
• Handling Large Datasets: They can handle large datasets with high dimensionality effectively. so, no need to train one model with whole dataset.
• Less Pre-processing: Random Forests can handle both categorical and numerical variables without the need for scaling or normalization. They can also handle missing values.
• Variable Importance: They provide insights into which features are most important in the prediction.
• Parallelizable: The training of individual trees can be parallelized, as they are independent of each other. This speeds up the training process.
• Non-Parametric: Random Forests are non-parametric, meaning they make no assumptions about the functional form of the transformation from inputs to output. This makes them very flexible and able to model complex, non-linear relationships.

Disadvantages of Random Forest

• Model Interpretability: One of the biggest drawbacks of Random Forests is that they lack the interpretability of simpler models like linear regression or decision trees. While you can rank features by their importance, the model as a whole is essentially a black box.
• Performance with Unbalanced Data: Random Forests can be biased towards the majority class when dealing with unbalanced datasets. This can sometimes be mitigated by balancing the dataset prior to training.
• Predictive Performance: Although Random Forests generally perform well, they may not always provide the best predictive performance. Gradient boosting machines, for instance, often outperform Random Forests .If the relationships within the data are linear, a linear model will likely perform better than a Random Forest.
• Inefficiency with Sparse Data: Random Forests might not be the best choice for sparse data or text data where linear models or other algorithms might be more suitable.
• Parameters Tuning: Although Random Forests require less tuning than some other models, there are still several parameters (like the number of trees, tree depth, etc.) that can affect model performance and need to be optimized.
• Difficulty with High Cardinality Features: Random Forests can struggle with high cardinality categorical features (features with a large number of distinct values). These types of features can lead to trees that are biased towards the variables with more levels, and may cause overfitting.
• Can’t Extrapolate: This is because they do not predict beyond the range of the training data, and that they may not predict as accurately as other regression models.

➡️ Next Up: [Gradient Boost Explained]

If you found value in this content, I invite you to stay connected for more insightful posts. Your time and interest are greatly appreciated. Thank you for reading!