Examples
Everything so far assumed a continuous response and a normal distribution. The machine does not care. Change the distribution and the loss changes with it.
Here we do that once, for the smallest possible change: a response with two values.
The response takes one of two values, say A and B. We do not predict the class directly. We predict a probability,
\[ P(Y = \text{A}|x) = p(x), \qquad P(Y = \text{B}|x) = 1 - p(x), \]
and turn that into a decision later.
A probability has to lie in \([0, 1]\), so a linear function will not do. We pass the linear function through the logistic function,
\[ p(x) = s(\eta), \qquad s(\eta) = \frac{1}{1 + e^{-\eta}}, \qquad \eta = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p . \]
The logistic function maps the real line to \((0, 1)\). It is \(0.5\) at \(\eta = 0\) and saturates at both ends. We write \(s\) for it, so that \(\sigma\) always means a standard deviation.
Inverting it gives
\[ \eta = \log\frac{p}{1-p}, \]
the log odds. A coefficient \(\beta_j\) is the change in the log odds for a one unit change in \(x_j\), with the other predictors held fixed.
We do not invent a loss. We write down the log-likelihood,
\[ \ell(\theta) = \sum_{i=1}^{n} \log P(Y = y_i | x_i, \theta), \]
and minimize \(-\ell/n\). For the Bernoulli model that quantity has a name: the cross-entropy loss. It is the negative log-likelihood divided by \(n\), under a new name. Every deep learning library calls it cross_entropy, so it is worth knowing both names.
This model is called logistic regression, or linear binary classification.
import numpy as np
import pandas as pd
rng = np.random.default_rng(24)
def logistic(x):
return 1 / (1 + np.exp(-x))
def data_generator(x, rng=rng):
x = np.asarray(x, dtype=float)
y = np.where(logistic(2 * x - 1) > rng.random(len(x)), "A", "B")
return pd.DataFrame({"x": x, "y": y})
data = data_generator([0., 2., 3.])
data| x | y | |
|---|---|---|
| 0 | 0.0 | B |
| 1 | 2.0 | A |
| 2 | 3.0 | A |
from sklearn.linear_model import LogisticRegression
X = data[["x"]].values
m = LogisticRegression(penalty=None).fit(X, data["y"])
print("intercept:", round(float(m.intercept_[0]), 4))
print("slope: ", round(float(m.coef_[0][0]), 4))intercept: 8.158
slope: -8.5716The model gives a probability for each class.
grid = np.arange(-1, 2.5, 0.5).reshape(-1, 1)
pd.DataFrame({"x": grid[:, 0],
"P(A|x)": m.predict_proba(grid)[:, 0].round(3),
"predicted": m.predict(grid)})| x | P(A|x) | predicted | |
|---|---|---|---|
| 0 | -1.0 | 0.000 | B |
| 1 | -0.5 | 0.000 | B |
| 2 | 0.0 | 0.000 | B |
| 3 | 0.5 | 0.020 | B |
| 4 | 1.0 | 0.602 | A |
| 5 | 1.5 | 0.991 | A |
| 6 | 2.0 | 1.000 | A |
The prediction is the class with the highest probability, which with two classes means a threshold of 0.5. That the threshold is a choice, and what to do about it, is week 7.
from sklearn.metrics import log_loss
def loglik(theta):
b0, b1 = theta
return (np.log(logistic(-b0))
+ np.log(logistic(b0 + 2 * b1))
+ np.log(logistic(-b0 - 3 * b1)))
theta_hat = [float(m.intercept_[0]), float(m.coef_[0][0])]
by_hand = loglik(theta_hat)
by_sklearn = -log_loss(data["y"], m.predict_proba(X), normalize=False)
print(f"by hand: {by_hand:.6f}")
print(f"sklearn: {by_sklearn:.6f}")by hand: -17.143619
sklearn: -0.000412log_loss with normalize=False is the negative log-likelihood. The two agree.
With three data points the fit is far from the generator. This is not a bug. It is what fitting three points looks like.
The cross-entropy is what we fit. It is not always what we report. Once a prediction is a class rather than a probability, the natural summary is the misclassification rate, the average of the zero-one loss
\[ L(y, \hat y) = \begin{cases} 0 & \hat y = y\\ 1 & \hat y \neq y \end{cases} \]
and its complement, the accuracy.
big = data_generator(rng.normal(0, 2, 500))
m2 = LogisticRegression(penalty=None).fit(big[["x"]].values, big["y"])
pred = m2.predict(big[["x"]].values)
print("accuracy: ", round(float((pred == big["y"]).mean()), 3))
print("misclassification: ", round(float((pred != big["y"]).mean()), 3))accuracy: 0.852
misclassification: 0.148We do not fit the zero-one loss directly. It is flat almost everywhere, so its gradient carries no information, and no optimizer can work with it. We fit the cross-entropy, which is smooth, and report the misclassification rate.
That gap between the loss we optimize and the number we care about comes back in week 7, where the misclassification rate turns out to be a poor summary as soon as the classes are imbalanced.