In the previous lessons, we mainly introduced single-model machine learning methods. In fact, combining several single models into one composite model has already become a mainstream method in modern machine learning. This method is called ensemble learning. The goal of ensemble learning is to build a strong learner by combining multiple weak learners, that is, models whose classification effect is only a little better than random guessing. In this way, the final model can overcome the limits of a single model and get better generalization ability. It is usually used in scenes where high-accuracy prediction is especially important.
Ensemble-learning algorithms are mainly divided into the following kinds:
-
Bagging. Multiple subsets are created by random sampling from the training data. A model is trained on each subset, and then their results are combined. The most typical example is the random forest that we discussed earlier.
-
Boosting. Multiple models are trained iteratively. In each round, the algorithm pays more attention to the samples that were predicted incorrectly in the previous round. Classic algorithms include AdaBoost, Gradient Boosting, and XGBoost.
-
Stacking. Several first-level models are trained, then their predictions are used as new features and passed into another model, usually called a second-level model, which makes the final prediction.
The basic idea of Boosting is this: at the beginning, all samples are given the same weight; when the first model is trained, the weights of wrongly classified samples become larger; when the next model is trained, it focuses more on the samples that the earlier model classified wrongly; finally, the results of all models are combined by weights, and the models with better performance get larger weights. To say it simply, Boosting trains a series of weak classifiers one after another, so that samples classified wrongly by earlier weak classifiers get more attention later, and finally combines these classifiers into the best strong classifier.
AdaBoost, short for Adaptive Boosting, was proposed by Yoav Freund and Robert Schapire in 1996. It is a classic ensemble method. Its idea is simple:
- increase the weight of the samples misclassified in the previous round;
- give higher weight to the weak learners that perform better.
Its training process is iterative, and the key steps are below:
- Initialize sample weights. Each training sample starts with the same weight. If there are
Nsamples, the initial weight is:
- Train a weak learner. At each round, AdaBoost trains a weak classifier according to the current sample weights and computes its weighted error:
- Update the classifier weight. The weight of the
t-th weak learner is:
When the classifier makes fewer mistakes, \alpha_t becomes larger, which means that classifier matters more.
- Update sample weights. Samples classified incorrectly get larger weights, while correctly classified samples get smaller weights:
- Normalize the weights. Make the sum of all sample weights equal to
1. - Build the final classifier. The final result is the weighted combination of all weak learners:
Here, sign is the sign function:
For example, if there are 3 weak learners and their outputs are +1, -1, and +1, and their weights are 0.5, 0.3, and 0.2, then the weighted sum is 0.4. Because 0.4 is positive, the sign function outputs +1, which means the final predicted class is the positive class.
We still use the iris dataset as an example and apply the AdaBoost ensemble-learning algorithm to build a classification model. The full code is shown below.
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.metrics import classification_report
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, random_state=3)
base_estimator = DecisionTreeClassifier(max_depth=1)
model = AdaBoostClassifier(base_estimator, n_estimators=50)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))Output:
precision recall f1-score support
0 1.00 1.00 1.00 10
1 0.90 0.90 0.90 10
2 0.90 0.90 0.90 10
accuracy 0.93 30
macro avg 0.93 0.93 0.93 30
weighted avg 0.93 0.93 0.93 30
Here we should pay attention to several hyperparameter settings:
n_estimators: the number of weak learners to train.learning_rate: the contribution of each weak learner to the final model.algorithm: the training algorithm of AdaBoost. There are two main training modes,'SAMME'and'SAMME.R'.base_estimator: the base learner used in each round. By default it is a depth-1 decision tree.
If you increase n_estimators, model performance may improve, but it may also cause overfitting. Increasing learning_rate can speed up training, but it may also make the model unstable. Usually, we need to use cross-validation to find the best combination.
GBDT, short for Gradient Boosting Decision Trees, is also a powerful ensemble-learning algorithm. Compared with AdaBoost, models in the GBDT family are used more widely in real work. GBDT is based on the idea of gradient boosting. It combines the strengths of decision trees and gradually improves the model through a series of weak classifiers, usually decision trees. In each round of training, it improves prediction performance by reducing the error of the previous model, that is, by fitting residuals.
For classification, a common loss function is log loss. If the current model output is F(x), and p(x) is the probability after the sigmoid transform:
then the gradient of the loss with respect to F(x) is:
That gradient tells us the difference between the current prediction and the true label. In each round:
- we compute the residuals;
- we train a new decision tree to fit those residuals;
- we update the model with the new tree:
Here, \eta is the learning rate and h_m(x) is the weak learner trained in the current round.
Using GradientBoostingClassifier in scikit-learn is straightforward:
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import classification_report
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, random_state=3)
model = GradientBoostingClassifier(n_estimators=32)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))Output:
precision recall f1-score support
0 1.00 1.00 1.00 10
1 1.00 1.00 1.00 10
2 1.00 1.00 1.00 10
accuracy 1.00 30
macro avg 1.00 1.00 1.00 30
weighted avg 1.00 1.00 1.00 30
Some important hyperparameters are:
loss: the loss function. The default is log loss.learning_rate: how much each tree contributes to the final result.n_estimators: the number of trees.subsample: the fraction of samples used when training each tree.criterion: the rule used to evaluate splits.validation_fraction: the size of the validation set used for early stopping.
Usually, when you increase n_estimators, you also decrease learning_rate so the model does not overfit too easily.
Even though GBDT is already strong, there is still room to improve accuracy, speed, and generalization. XGBoost, proposed by Tianqi Chen, is one of the best-known gradient-boosting frameworks. Compared with plain GBDT, it improves optimization, adds regularization, handles sparse data better, and makes training faster through engineering optimizations.
To use it, install the package first:
pip install xgboostThen the code can look like this:
import matplotlib.pyplot as plt
import xgboost as xgb
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, random_state=3)
dm_train = xgb.DMatrix(X_train, y_train)
dm_test = xgb.DMatrix(X_test)
params = {
'booster': 'gbtree',
'objective': 'multi:softmax',
'num_class': 3,
'gamma': 0.1,
'max_depth': 6,
'lambda': 2,
'subsample': 0.8,
'colsample_bytree': 0.8,
'eta': 0.001,
'seed': 10,
'nthread': 16,
}
model = xgb.train(params, dm_train, num_boost_round=200)
y_pred = model.predict(dm_test)
print(classification_report(y_test, y_pred))
xgb.plot_importance(model)
plt.grid(False)
plt.show()The feature-importance output is shown below.
Some XGBoost parameters that deserve attention are booster, objective, eta or learning_rate, alpha, lambda, scale_pos_weight, gamma, num_class, and the feature-sampling parameters such as colsample_bytree. Besides the model parameters in params, some parameters of the train function are also worth attention, such as num_boost_round, early_stopping_rounds, feval, obj, evals, eval_results, verbose_eval, xgb_model, and callbacks.
LightGBM is another top Boosting framework. It was open-sourced by Microsoft in 2017. It is still based on GBDT, but it is designed for large-scale datasets and high efficiency. It uses tricks such as histogram-based splitting, gradient-based one-side sampling, exclusive feature bundling, and leaf-wise tree growth. These choices make the framework lighter and faster. Of course, on smaller datasets, the advantage of LightGBM is not so obvious, and both XGBoost and LightGBM also have problems in interpretability and hyperparameter tuning.
pip install lightgbmimport lightgbm as lgb
import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, random_state=3)
train_data = lgb.Dataset(X_train, label=y_train)
test_data = lgb.Dataset(X_test, label=y_test, reference=train_data)
params = {
'objective': 'multiclass',
'num_class': 3,
'metric': 'multi_logloss',
'boosting_type': 'gbdt',
'num_leaves': 31,
'learning_rate': 0.05,
'feature_fraction': 0.75,
'early_stopping_rounds': 10,
}
model = lgb.train(params=params, train_set=train_data, num_boost_round=200, valid_sets=[test_data])
y_pred = model.predict(X_test, num_iteration=model.best_iteration)
y_pred_max = np.argmax(y_pred, axis=1)
print(classification_report(y_test, y_pred_max))Here we still only briefly talk about the model hyperparameters. For more details, you can check the official documentation. Important ones include objective, metric, boosting_type, num_leaves or max_depth, lambda_l1 or lambda_l2, max_bin, feature_fraction, and early_stopping_rounds.
Ensemble learning reduces the bias and variance of models by combining multiple models, so it usually gets better prediction results than a single model. Because it combines multiple base models, it can effectively reduce the overfitting problem that a single model may have, and it can also handle abnormal data and noisy data better, so it is more stable than a single model. Of course, ensemble learning also has problems such as large computational cost, weak model interpretability, and complex hyperparameter tuning. Besides XGBoost and LightGBM, there is also a famous Boosting algorithm called CatBoost, which is known for handling categorical features. Even now, when deep learning is very popular, Boosting algorithms represented by XGBoost, LightGBM, and CatBoost still have broad practical use.