Artificial neurons
Theory
An artificial neuron takes a weighted sum of its inputs, adds a bias, and passes the result through a non-linear function.
\[ a = g\left(w_0 + \sum_{j=1}^{p} w_j x_j\right) \]
The function \(g\) is called the activation function. Without it, a stack of neurons would collapse into a single linear function, and we would have gained nothing.
Activation functions
| name | definition | notes |
|---|---|---|
| relu | \(\max(0, z)\) | the default, cheap, gradient is 0 or 1 |
| sigmoid | \(s(z) = 1/(1+e^{-z})\) | squashes to \((0, 1)\), gradient vanishes for large \(|z|\) |
| tanh | \(\tanh(z)\) | squashes to \((-1, 1)\), same problem |
| identity | \(z\) | used in the output layer for regression |
Relu is the usual choice for hidden layers. It is fast, and its gradient does not shrink for large inputs. A relu neuron is zero on one side of a hyperplane and linear on the other, so a sum of relu neurons is a piecewise linear function.
Layers
A layer applies many neurons to the same input. In matrix notation, with input \(x \in \mathbb{R}^{p}\) and \(m\) neurons,
\[ a = g\left(W x + b\right), \qquad W \in \mathbb{R}^{m \times p},\ b \in \mathbb{R}^{m} \]
where \(g\) is applied element-wise. A multilayer perceptron stacks these,
\[ h^{(1)} = g\bigl(W^{(1)} x + b^{(1)}\bigr), \quad h^{(2)} = g\bigl(W^{(2)} h^{(1)} + b^{(2)}\bigr), \quad \dots \]
and the last layer produces the output. The parameters \(\theta\) are all the \(W\) and \(b\) together.
How many parameters
A layer from \(m\) inputs to \(k\) outputs has \(k \times m\) weights and \(k\) biases, so \(k(m+1)\) parameters.
A network with 784 inputs, two hidden layers of 100 neurons and 10 outputs has
\[ 100 \cdot 785 + 100 \cdot 101 + 10 \cdot 101 = 89610 \]
parameters. It is easy to build a network with more parameters than data points. This is why week 8 is about regularization.
Depth versus width
A network with one hidden layer can approximate any continuous function, if the layer is wide enough. This is a nice theorem and it is not very useful, because the required width can be enormous.
In practice deeper networks do better than wider ones with the same number of parameters, for most problems. Each layer can reuse what the previous layer computed, so a deep network expresses some functions much more compactly.
Initialization
If we set all weights to zero, every neuron in a layer computes the same thing and receives the same gradient. They stay identical forever. We therefore initialize randomly.
The scale matters. If the weights are too large the activations saturate and the gradients vanish. If they are too small the signal dies out through the layers. torch picks a sensible default, and it is rarely worth changing.
Standardizing the input matters for the same reason. A network with inputs on very different scales starts in a bad place.