Overview
Generalized linear regression
Last week we fitted a normal distribution whose mean is a linear function of the input. This week we change the distribution and keep everything else.
Pages
| Classification | Bernoulli and categorical distributions, cross-entropy |
| Logistic regression | Two classes, and the spam data |
| Evaluating a classifier | Confusion matrix, thresholds, ROC and AUC |
| Poisson regression | Count responses |
| Exercises |
Goals
The goal of this week is to
- understand which conditional distribution to choose, given a response variable \(Y\),
- translate the blackboard example of logistic regression into code,
- understand confusion matrices, ROC and AUC,
- know how to perform (multiple) logistic regression on a given data set, and
- know how to perform Poisson regression on a given data set.
The idea
A generalized linear model has three parts.
A random component, the distribution of \(Y\) given \(x\). A linear predictor \(\eta = \beta_0 + \beta_1x_1 + \cdots + \beta_px_p\). And a link function \(g\) that connects them through \(\eta = g(\mathrm{E}[Y|x])\).
| response | distribution | inverse link | loss |
|---|---|---|---|
| continuous | normal | \(\eta\) | squared error |
| two classes | Bernoulli | \(s(\eta) = 1/(1+e^{-\eta})\) | cross-entropy |
| \(C\) classes | categorical | softmax | cross-entropy |
| counts | Poisson | \(e^\eta\) | Poisson deviance |
We write \(s\) for the logistic function, so that \(\sigma\) always means a standard deviation.
The loss in the last column is not a separate choice. It is the negative log-likelihood of the distribution in the second column. Once the distribution is chosen, the loss follows.
In week 7 we will keep this table and replace the linear \(\eta\) by a neural network. Nothing else changes.