Convolution
Theory
A fully connected layer on a 100 by 100 colour image has 30000 inputs. With 100 hidden neurons that is three million parameters in the first layer alone, and none of them know that pixel \((3, 4)\) is next to pixel \((3, 5)\).
A convolutional layer fixes both problems at once.
The idea
We take a small filter, for example 3 by 3, and slide it over the image. At each position we compute a weighted sum of the pixels under the filter, add a bias, and apply an activation function.
The same filter is used at every position. This is where the saving comes from. A 3 by 3 filter has 9 weights and 1 bias, no matter how large the image is.
It also builds in an assumption. A feature that is useful in the top left corner is useful in the bottom right corner too. For images this is usually right.
Volumes
An image has three channels, red, green and blue. So the input is a volume of width, height and depth 3.
A filter has the same depth as its input. A 3 by 3 filter on a 3 channel input has \(3 \times 3 \times 3 = 27\) weights and 1 bias.
A layer has several filters. Each filter produces one output channel, so a layer with \(k\) filters produces an output volume of depth \(k\).
Size arithmetic
Given an input of width \(n\), a filter of size \(f\), stride \(s\) and padding \(p\), the output width is
\[ \left\lfloor \frac{n + 2p - f}{s} \right\rfloor + 1 . \]
Stride is how far the filter moves between positions. Stride 1 moves one pixel, stride 2 moves two and halves the output size.
Padding adds a border of zeros around the input, so that the filter can also be centred on the edge pixels. With \(p = (f-1)/2\) and stride 1 the output has the same size as the input.
Parameter count
A layer with \(k\) filters of size \(f \times f\) on an input of depth \(c\) has
\[ k \left( f^2 c + 1 \right) \]
parameters. Note that this does not depend on the width or the height of the image. This is why convolutional networks can handle large images.
Pooling
A pooling layer reduces the width and the height. Max pooling with a 2 by 2 window takes the largest of every four values, which halves both dimensions.
It has no parameters. It makes the representation smaller and slightly invariant to small shifts.
A typical architecture
A convolutional network alternates convolution and pooling, so that the volume gets thinner and deeper as we go. At the end the volume is flattened and passed to one or two fully connected layers, and then to the output layer.
The early layers learn edges and simple textures. The later layers learn parts and objects. This is not something we programmed. It comes out of the training.
Transfer learning
A network trained on millions of images has learned features that are useful for other image tasks too. Instead of training from scratch we take such a network, throw away its output layer, and train a new one on our data.
This works remarkably well with a few hundred images, which would be far too few to train a network from scratch. The reason is that we are only fitting the last layer, so the number of parameters is small.