Classification

Theory

In classification the response takes one of \(C\) values. We do not predict the class directly. We predict a probability for each class, and turn that into a decision later.

Two classes

With two classes the response follows a Bernoulli distribution,

\[ P(Y = \text{A}|x) = p(x), \qquad P(Y = \text{B}|x) = 1 - p(x). \]

We need \(p(x) \in [0, 1]\), so a linear function will not do. We pass the linear predictor through the logistic function,

\[ p(x) = s(\eta), \qquad s(\eta) = \frac{1}{1 + e^{-\eta}}, \qquad \eta = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p. \]

The logistic function maps the real line to \((0, 1)\). It is 0.5 at \(\eta = 0\), it goes to 0 for large negative \(\eta\) and to 1 for large positive \(\eta\).

Inverting it gives the link function,

\[ \eta = \log\frac{p}{1-p}. \]

The quantity \(p/(1-p)\) is the odds, so \(\eta\) is the log odds. A coefficient \(\beta_j\) is therefore the change in the log odds for a one unit change in \(x_j\), with the other predictors held fixed.

More than two classes

With \(C\) classes we need \(C\) numbers that are positive and sum to one. We use one linear function per class, \(\eta_1, \dots, \eta_C\), and the softmax,

\[ P(Y = c|x) = \frac{e^{\eta_c}}{\sum_{k=1}^{C} e^{\eta_k}}. \]

Note that adding the same constant to every \(\eta_c\) does not change the result. The parametrization is therefore redundant, and we can always set the parameters of the last class to zero. With \(C = 2\) this gives exactly logistic regression again.

Four classes, three parameters each. Every point of a grid over the two inputs is coloured by the class with the largest score. Note that adding the same amount to all four intercepts changes nothing, which is why one class can always be fixed at zero.

The loss

The log-likelihood of the training data is

\[ \ell(\theta) = \sum_{i=1}^{n} \log P(Y = y_i | x_i, \theta). \]

Minimizing \(-\ell/n\) is called the cross-entropy loss. The name comes from information theory. The cross-entropy between two distributions is

\[ H(P, Q) = -\sum_x P(x)\log Q(x), \]

and minimizing it between the empirical distribution of the data and the model is the same as maximizing the log-likelihood.

This is worth saying plainly. Cross-entropy is the negative log-likelihood of the Bernoulli or categorical model, divided by \(n\). It is a new name and not a new quantity. Every deep learning library calls it cross_entropy, so it is useful to know both names.