A noisy copy of x1, as two genes that are almost always expressed together.
2
The last two columns are unrelated to \(y\).
Two kinds of importance
Impurity importance (feature_importances_): the decrease of the training loss in all splits on a predictor, in all trees, normalized to sum to 1.
Permutation importance (permutation_importance): the increase of the loss when the values of one predictor are shuffled.
Code
import matplotlib.pyplot as pltfrom sklearn.ensemble import RandomForestRegressorfrom sklearn.inspection import permutation_importanceforest = RandomForestRegressor(n_estimators=300, random_state=0).fit(Xtrain, ytrain)perm = permutation_importance(forest, Xtest, ytest, n_repeats=10, random_state=0, scoring="neg_mean_squared_error")fig, axes = plt.subplots(1, 2, figsize=(7.2, 2.6), sharey=True)names = X.columns[::-1]axes[0].barh(names, forest.feature_importances_[::-1])axes[0].set(title="impurity, training data", xlabel="share of the decrease")axes[1].barh(names, perm.importances_mean[::-1], color="C1")axes[1].set(title="permutation, test data", xlabel="increase of the MSE")plt.show()
Figure 52.1: Impurity importance on the training data, and permutation importance on the test data (the increase of the MSE, mean of 10 shuffles), of one random forest.
Both put x1 first and x2 second. They differ in two places.
The continuous noise gets about a third of the impurity importance of x3, because some of its many thresholds reduce the training loss by chance. The binary noise gets almost nothing, and on test data shuffling either changes nothing.
The copy of x1 takes away some of the importance of x1: when x1 is shuffled, the forest still has the copy.
forest_alone = RandomForestRegressor(n_estimators=300, random_state=0).fit( Xtrain.drop(columns="x1 copy"), ytrain)perm_alone = permutation_importance(forest_alone, Xtest.drop(columns="x1 copy"), ytest, n_repeats=10, random_state=0, scoring="neg_mean_squared_error")print("permutation importance of x1, with the copy: ", round(perm.importances_mean[0], 2))print("permutation importance of x1, without the copy:", round(perm_alone.importances_mean[0], 2))
permutation importance of x1, with the copy: 7.14
permutation importance of x1, without the copy: 8.57
An important predictor helps to predict \(y\). This does not mean that changing it would change \(y\).