Examples
Real tables arrive with problems that no model can see. This one has four.
import numpy as np
import pandas as pd
raw = pd.DataFrame({
"id": [1, 2, 3, 3, 4, 5],
"weight": ["71", "68.5", "-999", "-999", "80 kg", "75"],
"sex": ["male", "Female", "female ", "female ", "male", "Male"],
})
raw.dtypesid int64
weight str
sex str
dtype: objectweight was read as text, because one entry has a unit in it. -999 is not a weight but a code for “not measured”. sex has four spellings of two categories. And the row with id 3 is there twice.
clean = raw.drop_duplicates()
clean["weight"] = pd.to_numeric(clean["weight"].str.replace(" kg", ""),
errors="coerce")
clean["weight"] = clean["weight"].replace(-999, np.nan)
clean["sex"] = clean["sex"].str.strip().str.lower().astype("category")
cleanerrors="coerce" turns anything that is still not a number into a missing value instead of stopping with an error.
| id | weight | sex | |
|---|---|---|---|
| 0 | 1 | 71.0 | male |
| 1 | 2 | 68.5 | female |
| 2 | 3 | NaN | female |
| 4 | 4 | 80.0 | male |
| 5 | 5 | 75.0 | male |
A fifth kind of problem is one that no code finds: a predictor that is only known after the response. A diagnosis written at discharge predicts perfectly whether a patient was admitted, and it is useless for deciding who to admit. Check for each column when it becomes available.
Two columns, one number and one category, each with a hole in it.
datam = pd.DataFrame({
"age": [12., 41, np.nan, 33, 27, 50],
"gender": pd.Categorical([None, "female", "male", "male", "male", "female"]),
})
datam| age | gender | |
|---|---|---|
| 0 | 12.0 | NaN |
| 1 | 41.0 | female |
| 2 | NaN | male |
| 3 | 33.0 | male |
| 4 | 27.0 | male |
| 5 | 50.0 | female |
The simplest option is to drop the rows. This is fine if only a few rows are affected. It is not fine if the values are missing for a reason, because then the remaining rows are no longer a random sample.
datam.dropna()| age | gender | |
|---|---|---|
| 1 | 41.0 | female |
| 3 | 33.0 | male |
| 4 | 27.0 | male |
| 5 | 50.0 | female |
The other option is to fill the missing values in. Note that the imputer learns the value it fills in, so it belongs inside the fold.
from sklearn.impute import SimpleImputer
imp = SimpleImputer(strategy="most_frequent")
pd.DataFrame(imp.fit_transform(datam), columns=["age", "gender"])| age | gender | |
|---|---|---|
| 0 | 12.0 | male |
| 1 | 41.0 | female |
| 2 | 12.0 | male |
| 3 | 33.0 | male |
| 4 | 27.0 | male |
| 5 | 50.0 | female |
Whether a value is missing can itself carry information: a test that was not ordered, a question that was skipped. add_indicator=True keeps it as an extra column of zeros and ones.
imp = SimpleImputer(strategy="most_frequent", add_indicator=True)
pd.DataFrame(imp.fit_transform(datam), columns=imp.get_feature_names_out())| age | gender | missingindicator_age | missingindicator_gender | |
|---|---|---|---|---|
| 0 | 12.0 | male | False | True |
| 1 | 41.0 | female | False | False |
| 2 | 12.0 | male | True | False |
| 3 | 33.0 | male | False | False |
| 4 | 27.0 | male | False | False |
| 5 | 50.0 | female | False | False |
A predictor with zero variance carries no information.
from sklearn.feature_selection import VarianceThreshold
df = pd.DataFrame({"a": np.ones(5),
"b": np.random.randn(5),
"c": np.linspace(1, 5, 5),
"d": np.linspace(2, 10, 5),
"e": np.zeros(5)})
keep = VarianceThreshold().fit(df)
df.loc[:, keep.get_support()]| b | c | d | |
|---|---|---|---|
| 0 | 0.543815 | 1.0 | 2.0 |
| 1 | -1.107815 | 2.0 | 4.0 |
| 2 | -0.427940 | 3.0 | 6.0 |
| 3 | -0.948993 | 4.0 | 8.0 |
| 4 | -1.071407 | 5.0 | 10.0 |
Columns a and e are constant and are gone. Column d is 2 * c: two predictors with correlation 1 carry the same information twice, which makes the least squares solution non-unique. Without a penalty, drop one of them. With a penalty, as in ridge regression, the solution is unique again and both can stay.
Standardization shifts the data so that the mean is 0 and scales it so that the standard deviation is 1. Some methods need it, for example regularization and \(k\) nearest neighbours. Others do not care.
from sklearn.preprocessing import StandardScaler
height_weight = pd.DataFrame({"height": [165., 175, 183, 152, 171],
"weight": [60., 71, 89, 47, 70]})
scaled = pd.DataFrame(StandardScaler().fit_transform(height_weight),
columns=["height", "weight"])
scaled.round(3)| height | weight | |
|---|---|---|
| 0 | -0.404 | -0.535 |
| 1 | 0.558 | 0.260 |
| 2 | 1.327 | 1.561 |
| 3 | -1.654 | -1.474 |
| 4 | 0.173 | 0.188 |
The scaler learns the mean and the standard deviation from the data. If we fit it on all the data before splitting, the validation fold has already influenced the training set. Use a Pipeline.