Overview

Regularization for networks, and a case study

This week we put everything together on one real data set. We also look at the ways of keeping a network from overfitting.

Pages

The data	Bicycle rentals, and the features we build
A Poisson network	The likelihood from week 3 with learned features
Exercises

Goals

The goal of this week is to go through a case study, in which we follow the recipe of a supervised machine learning project.

Regularization for networks

The L1 and L2 penalties are the same as for linear models,

\[ \mathcal L_{\text{reg}}(\theta) = \mathcal L(\theta) + \lambda\|\theta\|_2^2 \qquad\text{or}\qquad \mathcal L(\theta) + \lambda\|\theta\|_1 . \]

Two things are different. There is no closed form, so \(\lambda\) simply enters the gradient. And there is no unique optimum, so the penalty also decides which of many equally good solutions we end up in.

In deep learning the L2 penalty is usually called weight decay.

Dropout sets a random fraction of the activations to zero during training, and rescales the rest so that the expected sum stays the same. At test time nothing is dropped. It forces the network not to rely on any single neuron. In torch it is nn.Dropout(p), and it is only active in training mode, so remember model.eval() before predicting.

Batch normalization standardizes the activations of a layer over the batch. It was introduced to speed up training and it also has a mild regularizing effect, because the statistics of a batch are noisy.

Early stopping we already saw in week 6. It is often the cheapest of the four.

The recipe

1. Collect data.
2. Look at the raw data and clean it.
3. Choose a representation.
4. Choose a method.
5. Fit and tune the hyper-parameters with cross-validation.
6. If training loss and test loss are both high, take a more flexible method. If training loss is low and test loss is high, take a less flexible one.
7. Repeat 4 to 6.
8. If still unhappy, go back to 2, or collect more data.
9. Fit the best model on all available data.

The rest of this week is that recipe on one data set.