We predict how many miles per gallon a car drives from its number of cylinders, displacement, horsepower, weight, acceleration, model year and origin, and compare linear regression, a random forest, gradient boosting and a neural network.
The data
import numpy as npimport pandas as pdfrom sklearn.datasets import fetch_openmlcars = fetch_openml(data_id=455, as_frame=True, parser="auto").frame.dropna()X = cars.drop(columns="mpg")y = cars["mpg"]cars.head()
1
The cars data of OpenML, 1970 to 1982. Six cars without horsepower are dropped, which leaves 392.
mpg
cylinders
displacement
horsepower
weight
acceleration
model.year
origin
0
18.0
8
307.0
130.0
3504
12.0
70
1
1
15.0
8
350.0
165.0
3693
11.5
70
1
2
18.0
8
318.0
150.0
3436
11.0
70
1
3
16.0
8
304.0
150.0
3433
12.0
70
1
4
17.0
8
302.0
140.0
3449
10.5
70
1
cylinders (3, 4, 5, 6 or 8) and origin (1, 2 or 3 for America, Europe and Japan) are categorical. Only 7 cars have 3 or 5 cylinders.
Figure 53.1: Cross-validated RMSE in miles per gallon, with one standard error. The constant prediction of the mean is left out; its RMSE is 7.8.
The linear model is clearly worse than the others. Among the non-linear models the differences are smaller than the standard errors: with 392 cars we cannot tell them apart. The random forest got there without any tuning. The tuning of the boosting did not help on this small data set.
A linear model is not doomed. Adding all products of two predictors, which is feature engineering, brings it to the level of the trees. The trees find such interactions without being told.
Which predictors matter
Code
from sklearn.inspection import permutation_importancefrom sklearn.model_selection import train_test_splitXtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.25, random_state=0)forest = models["random forest"].fit(Xtrain, ytrain)perm = permutation_importance(forest, Xtest, ytest, n_repeats=20, random_state=0, scoring="neg_mean_squared_error")order = np.argsort(perm.importances_mean)fig, ax = plt.subplots(figsize=(6.0, 2.6))ax.barh(X.columns[order], perm.importances_mean[order], color="C1")ax.set(xlabel="increase of the MSE when shuffled")plt.show()X.drop(columns=["cylinders", "origin"]).corr().round(2)
(a) Permutation importance of the random forest on a held-out quarter of the cars (increase of the MSE, mean of 20 shuffles).
displacement
horsepower
weight
acceleration
model.year
displacement
1.00
0.90
0.93
-0.54
-0.37
horsepower
0.90
1.00
0.86
-0.69
-0.42
weight
0.93
0.86
1.00
-0.42
-0.31
acceleration
-0.54
-0.69
-0.42
1.00
0.29
model.year
-0.37
-0.42
-0.31
0.29
1.00
(b)
Figure 53.2
Model year matters, because cars became more efficient over the 13 years. Weight, displacement and horsepower are strongly correlated with each other, so the forest can replace one by the others, and each of them looks less important than the group is. This is the pattern of the copied predictor on the previous page.