Skip to main content

Exercises

Exercises

Exercise 1

Data is generated with \(p(y|x) = \frac1{\sqrt{2\pi}}e^{-(y - 2x)^2/2}\).

  1. What is the irreducible error.
  2. We fit a linear function and obtain \(\hat f(x) = 0.2 + 1.9x\). What is the reducible error at \(x = 1\).
  3. A method fits all training points exactly, so its training error is zero. Is its reducible error zero. Explain.

Exercise 2

For each situation below, say whether we expect a flexible method to do better or worse than a rigid one, and why.

  1. \(n\) is very large and \(p\) is small.
  2. \(p\) is very large and \(n\) is small.
  3. The relationship between the predictors and the response is strongly non-linear.
  4. The variance of the noise is very large.

Exercise 3

We fit \(k\) nearest neighbours with \(k = 1\).

  1. What is the training error, for regression and for classification.
  2. What does that tell us about the test error.
  3. Sketch how the bias and the variance of \(k\) nearest neighbours change as \(k\) grows from 1 to \(n\).

Exercise 4

Design a concrete example where the variance of a flexible method is larger than the variance of an inflexible method, and the bias goes the other way.

By concrete we mean that the numbers are there to see. If the task were instead “show with a concrete example that predictions under linear regression are invariant under a rescaling of the training inputs but predictions under ridge regression are not”, a valid answer would begin:

Take the data set \((x_1 = 0, y_1 = 1)\), \((x_2 = 1, y_2 = 2)\). Linear regression gives \(\theta_0 = 1\) and \(\theta_1 = (2-1)/(1-0) = 1\), so the prediction at \(x_0 = 2\) is \(\theta_1 x_0 + \theta_0 = 3\). Rescaling the inputs by \(s = 2\) gives \(\tilde\theta_0 = 1\) and \(\tilde\theta_1 = (2-1)/(2-0) = \tfrac12\), so the prediction at \(\tilde x_0 = 4\) is \(\tilde\theta_1 \tilde x_0 + \tilde\theta_0 = 3\), the same as before. For ridge regression with \(\lambda = 1\) the unscaled solution is … etc.

You do not need a simulation. Choose the generator and the two methods so that you can write both the bias and the variance down exactly.

Hint: two training inputs and \(k\) nearest neighbours are enough.

Exercise 5

Consider ridge regression with penalty \(\lambda\|\theta\|_2^2\).

  1. Sketch how the training error, the test error, the bias and the variance change as \(\lambda\) grows from 0 to \(\infty\).
  2. Explain why the intercept is usually not regularized, i.e. for \(Y = \theta_0 + \theta_1 X_1 + \theta_2 X_2 + \cdots \theta_d X_d\) we usually regularize only \(\theta_1, \ldots, \theta_d\) by writing \(\mathcal L_2(\theta) = \mathcal L(\theta) + \lambda \sum_{i=1}^d\theta_i^2\), excluding \(\theta_0\) from regularization. Hint: What would \(\hat Y\) be for infinite regularization strength?
  3. We double the units of one predictor, so its values are twice as large. What happens to its ridge coefficient. What happens if we standardize first, i.e. we replace each predictor \(X_i\) with \(\frac{X_i - \bar X_i}{\sigma_{X_i}}\), where \(\sigma_{X_i}\) is the empirical standard deviation.

Exercise 6

Write a data generator with \(f(x) = 0.3\sin(10x) + 0.7x\) and normal noise with standard deviation 0.1, and \(x\) uniform on \([0, 1]\).

  1. Generate a training set of 30 points and a test set of 5000 points.
  2. Fit polynomials of degree 1 to 15 and plot the training and the test error against the degree.
  3. Repeat the whole thing 20 times with different training sets and plot the average curves. How does the picture change.
  4. Repeat with a training set of 300 points. Where is the minimum now.

Hint: fit the polynomials with the pipeline from polynomial regression, with a StandardScaler() between its two steps as on bias-variance — a degree 15 fit on the raw powers is numerically hopeless.