Skip to main content

Case study: fuel consumption

Examples

We predict how many miles per gallon a car drives from its number of cylinders, displacement, horsepower, weight, acceleration, model year and origin, and compare linear regression, a random forest, gradient boosting and a neural network.

The data

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_openml

cars = fetch_openml(data_id=455, as_frame=True, parser="auto").frame.dropna()
X = cars.drop(columns="mpg")
y = cars["mpg"]
cars.head()
1
The cars data of OpenML, 1970 to 1982. Six cars without horsepower are dropped, which leaves 392.
mpg cylinders displacement horsepower weight acceleration model.year origin
0 18.0 8 307.0 130.0 3504 12.0 70 1
1 15.0 8 350.0 165.0 3693 11.5 70 1
2 18.0 8 318.0 150.0 3436 11.0 70 1
3 16.0 8 304.0 150.0 3433 12.0 70 1
4 17.0 8 302.0 140.0 3449 10.5 70 1

cylinders (3, 4, 5, 6 or 8) and origin (1, 2 or 3 for America, Europe and Japan) are categorical. Only 7 cars have 3 or 5 cylinders.

One preprocessing for every model

from sklearn.compose import make_column_transformer
from sklearn.preprocessing import OneHotEncoder, PolynomialFeatures, StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GridSearchCV, KFold
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression, RidgeCV
from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor
from sklearn.neural_network import MLPRegressor

def pre():
    return make_column_transformer(
        (OneHotEncoder(handle_unknown="ignore"), ["cylinders", "origin"]),
        remainder=StandardScaler())

boosting = GridSearchCV(
    make_pipeline(pre(), GradientBoostingRegressor(random_state=0)),
    {"gradientboostingregressor__learning_rate": [0.03, 0.1, 0.3],
     "gradientboostingregressor__n_estimators": [100, 300],
     "gradientboostingregressor__max_depth": [1, 2, 3]},
    cv=KFold(5, shuffle=True, random_state=1), scoring="neg_root_mean_squared_error")

models = {
    "mean": make_pipeline(pre(), DummyRegressor()),
    "linear": make_pipeline(pre(), LinearRegression()),
    "quadratic, ridge": make_pipeline(pre(), PolynomialFeatures(2),
                                      RidgeCV(alphas=np.logspace(-3, 3, 20))),
    "random forest": make_pipeline(pre(), RandomForestRegressor(n_estimators=500,
                                                                random_state=0)),
    "boosting, default": make_pipeline(pre(), GradientBoostingRegressor(random_state=0)),
    "boosting, tuned": boosting,
    "neural network": make_pipeline(pre(), MLPRegressor(hidden_layer_sizes=(64, 64),
                                                        max_iter=3000, random_state=0)),
}
1
A rare number of cylinders can be missing from the training folds; it is then encoded as zeros instead of stopping with an error.
2
The tuning is part of the model, so the outer cross-validation below scores the whole procedure, as in nested cross-validation.
3
All products of two predictors, with the ridge penalty chosen by an inner cross-validation.

The same folds for every model

Code
from sklearn.model_selection import cross_val_score

cv = KFold(10, shuffle=True, random_state=0)
rows = []
for name, model in models.items():
    rmse = -cross_val_score(model, X, y, cv=cv, scoring="neg_root_mean_squared_error")
    rows.append({"model": name, "RMSE": rmse.mean(),
                 "standard error": rmse.std() / np.sqrt(len(rmse))})
table = pd.DataFrame(rows).round(2)
table
model RMSE standard error
0 mean 7.81 0.12
1 linear 3.18 0.12
2 quadratic, ridge 2.62 0.12
3 random forest 2.63 0.15
4 boosting, default 2.67 0.15
5 boosting, tuned 2.68 0.13
6 neural network 2.79 0.16
Code
import matplotlib.pyplot as plt

t = table[table["model"] != "mean"][::-1]
fig, ax = plt.subplots(figsize=(6.0, 2.8))
ax.barh(t["model"], t["RMSE"], xerr=t["standard error"], capsize=3)
ax.set(xlabel="cross-validated RMSE (mpg)", xlim=(2, 3.5))
plt.show()
Figure 53.1: Cross-validated RMSE in miles per gallon, with one standard error. The constant prediction of the mean is left out; its RMSE is 7.8.

The linear model is clearly worse than the others. Among the non-linear models the differences are smaller than the standard errors: with 392 cars we cannot tell them apart. The random forest got there without any tuning. The tuning of the boosting did not help on this small data set.

A linear model is not doomed. Adding all products of two predictors, which is feature engineering, brings it to the level of the trees. The trees find such interactions without being told.

Which predictors matter

Code
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.25, random_state=0)
forest = models["random forest"].fit(Xtrain, ytrain)
perm = permutation_importance(forest, Xtest, ytest, n_repeats=20, random_state=0,
                              scoring="neg_mean_squared_error")
order = np.argsort(perm.importances_mean)
fig, ax = plt.subplots(figsize=(6.0, 2.6))
ax.barh(X.columns[order], perm.importances_mean[order], color="C1")
ax.set(xlabel="increase of the MSE when shuffled")
plt.show()

X.drop(columns=["cylinders", "origin"]).corr().round(2)
(a) Permutation importance of the random forest on a held-out quarter of the cars (increase of the MSE, mean of 20 shuffles).
displacement horsepower weight acceleration model.year
displacement 1.00 0.90 0.93 -0.54 -0.37
horsepower 0.90 1.00 0.86 -0.69 -0.42
weight 0.93 0.86 1.00 -0.42 -0.31
acceleration -0.54 -0.69 -0.42 1.00 0.29
model.year -0.37 -0.42 -0.31 0.29 1.00
(b)
Figure 53.2

Model year matters, because cars became more efficient over the 13 years. Weight, displacement and horsepower are strongly correlated with each other, so the forest can replace one by the others, and each of them looks less important than the group is. This is the pattern of the copied predictor on the previous page.

Further reading

On many tabular data sets, tuned tree ensembles are still as good as or better than neural networks. A careful comparison is Grinsztajn, Oyallon and Varoquaux, Why do tree-based models still outperform deep learning on typical tabular data?, NeurIPS 2022. On small tables, pretrained networks such as TabPFN are now competitive: Hollmann et al., Accurate predictions on small data with a tabular foundation model, Nature 2025.