Overview

Flexibility and the bias-variance tradeoff

Linear methods are often not flexible enough. This week we make models more flexible, and then we look at what flexibility costs.

Pages

Error decomposition	Reducible and irreducible error, bias and variance
Polynomial regression	Making a linear model more flexible
k nearest neighbours	A method with no parameters to fit
Regularization	Ridge and lasso, making a model less flexible
Exercises

Goals

The goal of this week is to

1. see a regression and a classification example where linear methods are not flexible enough,
2. understand and run polynomial regression and classification,
3. understand and run k nearest neighbour regression and classification,
4. understand the error decomposition in regression, and
5. understand the bias-variance decomposition in regression.

The tension

A more flexible model fits the training data better. It does not follow that it predicts new data better.

There are two reasons why our prediction can be wrong. The data generating process is noisy, and we cannot do anything about that. And our fitted function differs from the true one, which we can do something about.

The second part splits again. Our function can be systematically wrong, which is bias. Or it can depend too much on which training set we happened to get, which is variance. Flexibility lowers the first and raises the second.