Overview
Classification and counts, linear and learned
Building a supervised model has been two decisions all along, and we have so far made them one at a time.
Which distribution describes \(Y\) given \(x\). We chose a normal in week 2 and a Bernoulli on the two classes page, and the loss followed from the choice both times.
Which family of functions produces the parameter of that distribution. A linear function until week 5, a neural network in week 6.
The two decisions are independent. This week we say that out loud and fill in the grid.
Pages
| Distributions and links | The four rows, the link function, why the loss follows |
| Two ways to classify | Logistic regression and a network, on the same data |
| Evaluating a classifier | Confusion matrix, thresholds, ROC and AUC |
| Counts | Poisson regression, linear and learned |
| Exercises |
Goals
The goal of this week is to
- understand which conditional distribution to choose, given a response variable \(Y\), and which output layer and loss follow from it,
- know how to perform (multiple) logistic regression on a given data set,
- know how to perform Poisson regression on a given data set,
- run the same two problems with a neural network, changing only the output layer,
- understand confusion matrices, thresholds, ROC and AUC, and use them to compare two models honestly, and
- be able to say when the extra flexibility of a network is worth it and when it is not.
The grid
| distribution for \(Y|x\) | with \(\eta = \beta^\top x\) | with \(\eta = f_\theta(x)\) |
|---|---|---|
| normal | linear regression (week 2) | network regression (week 6) |
| Bernoulli | logistic regression (week 2) | this week |
| categorical | multinomial logistic regression (week 5) | this week |
| Poisson | this week | this week |
Reading down a column: the function family is fixed and the response changes. Reading across a row: the response is fixed and the function family changes. Neither move disturbs the other, and that is the whole content of the week.
The table that does not change
A generalized linear model has three parts. A random component, the distribution of \(Y\) given \(x\). A linear predictor \(\eta = \beta_0 + \beta_1x_1 + \cdots + \beta_px_p\). And a link function \(g\) that connects them through \(\eta = g(\mathrm{E}[Y|x])\).
| response | distribution | inverse link, or output layer | loss |
|---|---|---|---|
| continuous | normal | \(\eta\) | squared error |
| two classes | Bernoulli | \(s(\eta) = 1/(1+e^{-\eta})\) | cross-entropy |
| \(C\) classes | categorical | softmax | cross-entropy |
| counts | Poisson | \(e^\eta\) | Poisson deviance |
We write \(s\) for the logistic function, so that \(\sigma\) always means a standard deviation.
The loss in the last column is not a separate choice. It is the negative log-likelihood of the distribution in the second column. Once the distribution is chosen, the loss follows.
And the third column has two readings. For a linear model it is the inverse link function of a generalized linear model. For a network it is the output layer. They are the same object, and that is why the table serves both columns of the grid above.