Regularization

Examples

Polynomials and \(k\) nearest neighbours made a model more flexible. Regularization goes the other way. It keeps the model but restricts the parameters.

\[ \mathcal L_{\mathrm{L2}}(\theta) = \mathcal L(\theta) + \lambda\|\theta\|_2^2, \qquad \mathcal L_{\mathrm{L1}}(\theta) = \mathcal L(\theta) + \lambda\|\theta\|_1 \]

The first is ridge regression, the second is the lasso. The constant \(\lambda\) is a hyper-parameter. With \(\lambda = 0\) we recover the original loss. The larger \(\lambda\), the less flexible the model.

In sklearn the constant is called alpha.

Ridge

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

rng = np.random.default_rng(9)

def f(x):
    return np.sin(2 * x) + 2 * (x - 0.5) ** 3 - 0.5 * x

n, sigma = 25, 0.15
x = rng.uniform(0, 1, n)
y = f(x) + rng.normal(0, sigma, n)
X = x.reshape(-1, 1)
grid = np.linspace(0, 1, 300).reshape(-1, 1)

A degree 12 polynomial with a growing penalty. The lower panel shows the fitted coefficients. Switch to the lasso and watch them hit exactly zero, which ridge never does.

We standardize before the penalty. See below for why.

Regularization is not invariant under rescaling

The penalty \(\|\theta\|_2^2\) treats every parameter the same. But the parameters are not the same. A predictor measured in meters gets a coefficient a thousand times smaller than the same predictor measured in millimeters, and the penalty therefore hits it a thousand times less.

Xs = np.column_stack([rng.normal(0, 1, 200), rng.normal(0, 1, 200) * 1000])
ys = 2 * Xs[:, 0] + 0.002 * Xs[:, 1] + rng.normal(0, 0.3, 200)

raw = Ridge(alpha=1.0).fit(Xs, ys)
scaled = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(Xs, ys)

print("without standardization:", raw.coef_.round(5))
print("with standardization:   ", scaled[-1].coef_.round(5))

1: The second predictor has a standard deviation 1000 times larger.
2: Both predictors contribute equally to the response.

without standardization: [2.0447e+00 2.0400e-03]
with standardization:    [1.97658 2.08952]

Both predictors contribute the same amount, but the raw coefficients differ by a factor of 1000. Always standardize before regularizing.

The lasso

The lasso sets coefficients exactly to zero. Ridge only shrinks them towards zero.

The first three coefficients are the real ones. The lasso drives the other five to exactly zero, which makes it useful for selecting predictors. Ridge keeps all of them small but non-zero.

An alternative formulation

Both penalties can be written as a constraint instead.

\[ \min_\theta \mathcal L(\theta) \quad\text{subject to}\quad \|\theta\|_2^2 \le s \]

For every \(\lambda\) there is an \(s\) that gives the same solution. This view explains why the lasso produces exact zeros. The L1 constraint region is a diamond with corners on the axes, and the corners are where a coefficient is zero. The L2 region is a ball with no corners.

Choosing lambda

\(\lambda\) is a hyper-parameter, so we cannot read it off the training data. We choose it by cross-validation, which is the subject of week 5.

from sklearn.model_selection import KFold, cross_val_score

lams = np.logspace(-6, 0, 40)
scores = []
for lam in lams:
    model = make_pipeline(PolynomialFeatures(12), StandardScaler(), Ridge(alpha=lam))
    s = cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=0),
                        scoring="neg_mean_squared_error")
    scores.append(-s.mean())

print("best lambda:", round(float(lams[int(np.argmin(scores))]), 6))

best lambda: 0.345511