Distributions and links
Theory
The recipe has not changed since week 2. Pick a distribution for \(Y\) given \(x\), let one of its parameters depend on the input through \(\eta\), and maximize the likelihood. What changes from row to row of the table is only which parameter, and how \(\eta\) is squeezed into the range that parameter is allowed to take.
The link function
\(\eta\) is a real number. It can be anything from \(-\infty\) to \(\infty\), because it is a linear function of the input, or the output of a network, and neither respects any bound.
The parameter it has to produce usually cannot.
| distribution | its parameter | allowed range | inverse link |
|---|---|---|---|
| normal | mean \(\mu\) | all of \(\mathbb R\) | \(\mu = \eta\) |
| Bernoulli | probability \(p\) | \([0, 1]\) | \(p = s(\eta)\) |
| categorical | \(C\) probabilities | simplex | \(\operatorname{softmax}(\eta)\) |
| Poisson | rate \(\lambda\) | \((0, \infty)\) | \(\lambda = e^\eta\) |
The last column is a squashing function, chosen to land in the right range. Its inverse is the link function \(g\), defined by \(\eta = g(\mathrm{E}[Y|x])\). For the Bernoulli that is the logit \(\log\frac{p}{1-p}\); for the Poisson it is the logarithm; for the normal it is nothing at all.
This is the vocabulary the statistics literature uses, and the one statsmodels uses. It is the same thing we have been doing.
Counts
The one row we have not met is the Poisson. It is what to reach for when the response is a count: the number of bicycles rented in an hour, the number of colonies on a plate, the number of reads at a locus.
\[ Y|x \sim \operatorname{Poisson}(\lambda(x)), \qquad P(Y = k|x) = \frac{\lambda(x)^k e^{-\lambda(x)}}{k!}, \qquad \lambda(x) = e^{\eta} . \]
Two things fall out of that choice, and both are the reason to make it.
The prediction cannot be negative. The exponential sees to it. A linear regression on counts predicts negative counts wherever the fit runs low, which is not a rounding problem but a sign that the model does not know what kind of quantity it is predicting.
The spread is not a free parameter. For a Poisson distribution the variance equals the mean. So the model predicts more scatter where it predicts larger counts, without our asking, and it has no \(\sigma\) to fit. The exponential also makes the effects multiplicative: a coefficient of \(0.1\) means that one unit more of that predictor multiplies the expected count by \(e^{0.1} \approx 1.1\).
If the observed spread is larger than the mean, the data is overdispersed and a Poisson model will understate its own uncertainty. A negative binomial model, which has a separate dispersion parameter, is then the better choice.
The loss, again
For every row, the loss is the negative log-likelihood divided by \(n\), and nothing else,
\[ \mathcal L(\theta) = -\frac1n \sum_{i=1}^{n} \log P(y_i|x_i, \theta). \]
Write that out for each distribution and familiar names appear.
| distribution | \(-\frac1n\sum_i \log P(y_i|x_i,\theta)\) | called |
|---|---|---|
| normal | \(\frac{1}{2n\sigma^2}\sum_i (y_i - \mu_i)^2 + \text{const}\) | mean squared error |
| Bernoulli | \(-\frac1n\sum_i \bigl[y_i\log p_i + (1-y_i)\log(1-p_i)\bigr]\) | cross-entropy |
| categorical | \(-\frac1n\sum_i \log p_{i,y_i}\) | cross-entropy |
| Poisson | \(\frac1n\sum_i \bigl[\lambda_i - y_i\log\lambda_i\bigr] + \text{const}\) | Poisson deviance |
Every one of these is something a library will hand you under a name that sounds like an independent invention. None of them is.
Where the network comes in
Nothing above mentions how \(\eta\) is computed. Replace
\[ \eta = \beta_0 + \beta_1x_1 + \cdots + \beta_px_p \qquad\text{by}\qquad \eta = f_\theta(x) \]
and every row of every table on this page still holds. In a network the squashing function is the output layer, and the loss is the same negative log-likelihood. That is the entire modification.
In practice the two do differ in one respect worth stating now. For a linear \(\eta\) the negative log-likelihood of all four rows is convex, so the fit is unique and any solver finds it. For a network it is not, so two runs give two models. Which of the two you want is a question about the data, not about the table, and the rest of the week is about answering it.