Overview
Multilayer perceptrons
Last week we ended with the XOR problem. A linear model cannot solve it, but a linear model on the right features can. The question is where the features come from.
A neural network learns them.
Pages
| Artificial neurons | One neuron, activation functions, matrix notation |
| Building networks | Layers, depth and width, fitting the weather data |
| MNIST | Classification with a network |
| Exercises |
Goals
The goal of this week is to
- understand how neurons and neural networks can be written in code,
- learn how to run regression with neural networks, and
- learn how to run classification with neural networks.
The output layer is a choice of distribution
A network computes a function \(f_\theta(x)\). What we do with it is the same choice we made in week 3.
| response | output layer | distribution | loss |
|---|---|---|---|
| continuous | identity | normal | squared error |
| two classes | \(s(\cdot)\) | Bernoulli | cross-entropy |
| \(C\) classes | softmax | categorical | cross-entropy |
| counts | \(\exp\) | Poisson | Poisson deviance |
This is the same table as in week 3. The only difference is that the linear predictor \(\eta\) is no longer \(\beta^\top x\). The features are learned instead of chosen, and the loss still follows from the distribution.