Supervised learning as likelihood maximization

Examples

Instead of a family of functions and a hand-picked loss, we specify a family of conditional probability distributions \(P(y|x, \theta)\) and maximize the likelihood.

The same twenty points as on the previous page, and three parameters now, because the normal distribution has a width as well as a mean. On the left the mean of the model with bands at one and two standard deviations; on the right the log-likelihood over the two parameters of the mean, light where it is high, with the current position marked. Changing σ alone changes the log-likelihood without moving the line, and it does not move the contours either: σ sets how tall the ridge is, not where it lies. That is the previous page’s loss surface seen from the other side, and its ridge sits on the same ellipses.

Choosing a normal distribution with constant variance is the same as choosing the squared error.

Checking it numerically

import numpy as np
from scipy.optimize import minimize
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(2)
x = rng.uniform(0, 5, 60)
y = 1.3 + 0.8 * x + rng.normal(0, 0.7, 60)


def neg_log_likelihood(params):
    b0, b1, log_sigma = params
    sigma = np.exp(log_sigma)
    resid = y - b0 - b1 * x
    return np.sum(0.5 * np.log(2 * np.pi * sigma**2) + resid**2 / (2 * sigma**2))


opt = minimize(neg_log_likelihood, x0=[0.0, 0.0, 0.0])
lin = LinearRegression().fit(x.reshape(-1, 1), y)

print(f"maximum likelihood: b0 = {opt.x[0]:.4f}, b1 = {opt.x[1]:.4f}, sigma = {np.exp(opt.x[2]):.4f}")
print(f"least squares:      b0 = {lin.intercept_:.4f}, b1 = {lin.coef_[0]:.4f}")
print(f"sigma by hand:      {np.sqrt(np.mean((y - lin.predict(x.reshape(-1,1)))**2)):.4f}")

1

We optimize \(\log\sigma\) instead of \(\sigma\), so that \(\sigma\) stays positive.

maximum likelihood: b0 = 1.3779, b1 = 0.7625, sigma = 0.6765
least squares:      b0 = 1.3779, b1 = 0.7625
sigma by hand:      0.6765

The two agree to four digits. They are the same estimator, which you can prove formally in exercise 1.