Skip to main content

Which predictors matter

Examples

A data set where we know the answer

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(3)
n = 1000
x1, x2, x3 = rng.standard_normal((3, n))
X = pd.DataFrame({"x1": x1,
                  "x1 copy": x1 + 0.3 * rng.standard_normal(n),
                  "x2": x2,
                  "x3": x3,
                  "noise": rng.standard_normal(n),
                  "noise, binary": rng.integers(0, 2, n)})
y = 2 * x1 + x2 + np.sin(2 * x3) + rng.standard_normal(n)
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.5, random_state=0)
1
A noisy copy of x1, as two genes that are almost always expressed together.
2
The last two columns are unrelated to \(y\).

Two kinds of importance

Impurity importance (feature_importances_): the decrease of the training loss in all splits on a predictor, in all trees, normalized to sum to 1.

Permutation importance (permutation_importance): the increase of the loss when the values of one predictor are shuffled.

Code
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestRegressor
from sklearn.inspection import permutation_importance

forest = RandomForestRegressor(n_estimators=300, random_state=0).fit(Xtrain, ytrain)
perm = permutation_importance(forest, Xtest, ytest, n_repeats=10, random_state=0,
                              scoring="neg_mean_squared_error")

fig, axes = plt.subplots(1, 2, figsize=(7.2, 2.6), sharey=True)
names = X.columns[::-1]
axes[0].barh(names, forest.feature_importances_[::-1])
axes[0].set(title="impurity, training data", xlabel="share of the decrease")
axes[1].barh(names, perm.importances_mean[::-1], color="C1")
axes[1].set(title="permutation, test data", xlabel="increase of the MSE")
plt.show()
Figure 52.1: Impurity importance on the training data, and permutation importance on the test data (the increase of the MSE, mean of 10 shuffles), of one random forest.

Both put x1 first and x2 second. They differ in two places.

  • The continuous noise gets about a third of the impurity importance of x3, because some of its many thresholds reduce the training loss by chance. The binary noise gets almost nothing, and on test data shuffling either changes nothing.
  • The copy of x1 takes away some of the importance of x1: when x1 is shuffled, the forest still has the copy.
forest_alone = RandomForestRegressor(n_estimators=300, random_state=0).fit(
    Xtrain.drop(columns="x1 copy"), ytrain)
perm_alone = permutation_importance(forest_alone, Xtest.drop(columns="x1 copy"), ytest,
                                    n_repeats=10, random_state=0,
                                    scoring="neg_mean_squared_error")
print("permutation importance of x1, with the copy:   ", round(perm.importances_mean[0], 2))
print("permutation importance of x1, without the copy:", round(perm_alone.importances_mean[0], 2))
permutation importance of x1, with the copy:    7.14
permutation importance of x1, without the copy: 8.57

An important predictor helps to predict \(y\). This does not mean that changing it would change \(y\).