Skip to main content

Bagging and random forests

Examples

Averaging trees

If we had many training sets, we could fit a tree to each and average the trees. Each tree has a low bias and a high variance. The average keeps the low bias and loses much of the variance.

Thin lines: up to ten of the trees, each with a minimum leaf size of 3. Thick line: the average of number of trees trees. With a new training set, every tree sees 40 new points of the generator of the previous page. With a bootstrap sample, every tree sees \(n = 40\) points drawn with replacement from one training set of 40 points, shown as dots. Test errors on 20 000 new points; the irreducible error is 0.09.

With new training sets the average of 20 trees is close to \(\sin(x)\), and its test error, 0.096, is close to the irreducible error.

Bagging

With one training set, each tree is fitted to a bootstrap sample: \(n\) points drawn with replacement from the \(n\) training points (bagging). Switch the panel to a bootstrap sample. The test error of one tree is 0.211, and of the average of 200 trees 0.104, not quite as low as with new training sets.

Random forests

A random forest also considers only max_features randomly chosen predictors at each split. The default is \(\sqrt p\) in RandomForestClassifier and all \(p\) predictors, which is bagging, in RandomForestRegressor.

Out-of-bag error

Predicting every training point only with the trees whose bootstrap sample did not contain it gives the out-of-bag error.

Code
import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestRegressor

rng = np.random.default_rng(1)

def data_generator(n):
    x = rng.uniform(-4, 4, n)
    return x.reshape(-1, 1), np.sin(x) + 0.3 * rng.standard_normal(n)

X, y = data_generator(200)
Xtest, ytest = data_generator(20000)
Bs = [10, 20, 50, 100, 300, 1000]
oob, test = [], []
for B in Bs:
    forest = RandomForestRegressor(n_estimators=B, min_samples_leaf=3,
                                   oob_score=True, random_state=0).fit(X, y)
    oob.append(np.mean((forest.oob_prediction_ - y) ** 2))
    test.append(np.mean((forest.predict(Xtest) - ytest) ** 2))

fig, ax = plt.subplots(figsize=(6.0, 2.8))
ax.plot(Bs, test, "o-", label="test")
ax.plot(Bs, oob, "s-", label="out-of-bag")
ax.axhline(0.09, color="#7a838b", ls="--", lw=1, label="irreducible error")
ax.set(xscale="log", xlabel="number of trees", ylabel="MSE")
ax.legend()
plt.show()
Figure 50.1: Out-of-bag error and test error of a random forest against the number of trees, for 200 training points of the same generator.

Both curves level off. The out-of-bag error is computed on the 200 training points, whose noise happens to be smaller than average: the mean of \((y_i - \sin(x_i))^2\) over these points is 0.073, not 0.09. It is therefore lower than the test error here, which is measured on 20 000 points. With another training set it could just as well be higher. It is an estimate with the uncertainty of 200 points, like a cross-validation error.