Always look at the raw data

Examples

Before fitting anything, look at the numbers. Check the units, the ranges and the missing values. A large part of the errors in a project are found this way, and none of them are found by a model.

The weather data

import sys
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

sys.path.insert(0, "scripts")
from figures import density_hist2d, hidden_structure, pairplot

weather = pd.read_csv("https://go.epfl.ch/bio322-weather2015-2018.csv")

y = weather["LUZ_wind_peak"][5:].values
X = weather["LUZ_pressure"][:-5].values

1: The figures on this page are also the figures in the lecture slides, so the two cannot drift apart. They are drawn by scripts/figures.py.
2: The wind peak five hours after the pressure reading on the same row.

fig, ax = plt.subplots()
density_hist2d(ax, X, y)
ax.set(xlabel="LUZ_pressure [hPa]", ylabel="LUZ_wind_peak in 5h [km/h]")
plt.show()

Figure 7.1: Pressure in Luzern against the wind peak five hours later. There is a relationship, but it is not a tight one.

Low pressure goes with high wind, but the spread is large. Even a perfect model cannot predict the wind peak exactly from the pressure alone.

A pair plot shows every predictor against every other one at once: a histogram of each column on the diagonal, a scatter plot with a trend line below it, and a two dimensional histogram above it.

cols = ["LUZ_pressure", "BAS_pressure", "LUZ_sunshine_duration", "LUZ_wind_peak"]
short = ["LUZ press.", "BAS press.", "sunshine", "wind peak"]
sample = weather[cols].sample(3000, random_state=0)

pairplot([sample[c].values for c in cols], short)
plt.show()

Figure 7.2: Four columns of the weather data against each other.

Three things are visible without fitting anything. The wind peak has a long tail: the median is 11 km/h and the largest value in the set is 112. The two pressures are almost the same number twice, with a correlation of 0.995, so one of them carries little information the other does not. And the sunshine duration is zero in 72% of the hours, because it is night for half of them and overcast for many of the rest.

But a pair plot only shows shadows

An empty pair plot is not evidence of an empty data set. Here is an artificial example where \(y\) is an exact function of the three predictors, with no noise at all.

x, y_toy, z = hidden_structure()

print("correlation of each predictor with y:",
      np.round([np.corrcoef(x[:, j], y_toy)[0, 1] for j in range(3)], 3))

1: Four thousand points. Three latent coordinates \(z\) we never get to see, a response \(y = \sin(2\pi \cdot 1.5 (z_1 + z_2 + z_3))\) that oscillates along one direction of that space, and a random rotation, so that what we measure is a mixture of the three coordinates.

correlation of each predictor with y: [-0.02  -0.001 -0.028]

Every correlation is essentially zero, and so is every panel of the pair plot.

pairplot([x[:, 0], x[:, 1], x[:, 2], y_toy],
         ["$x_1$", "$x_2$", "$x_3$", "$y$"])
plt.show()

Figure 7.3: The same pair plot as above, on data whose response is an exact function of the three predictors. Every trend line is flat.

Nor does it help to go looking for pairwise interactions.

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

for degree in (1, 2, 3):
    P = PolynomialFeatures(degree, include_bias=False).fit_transform(x)
    r2 = LinearRegression().fit(P, y_toy).score(P, y_toy)
    print(f"R^2 with every term up to degree {degree}: {r2:.3f}")

R^2 with every term up to degree 1: 0.001
R^2 with every term up to degree 2: 0.002
R^2 with every term up to degree 3: 0.004

And yet the structure is exact. Plot the same 4000 points against the one direction that matters and there it is.

fig, ax = plt.subplots(figsize=(5.2, 2.6))
ax.plot(z.sum(axis=1), y_toy, ".", ms=2.5, alpha=0.5, color="C1")
ax.set(xlabel="$z_1 + z_2 + z_3$", ylabel="$y$")
plt.show()

Figure 7.4: The same points, against the sum of the three latent coordinates.

A pair plot shows one and two dimensional shadows of the data. Structure that needs three predictors at once, or that lives along a direction which is a mixture of the predictors we happened to measure, casts no shadow at all.

Looking at the raw data is necessary. It is not sufficient.