Skip to main content

Exercises

Exercises

Conceptual

Exercise 1 Conceptual

Data is generated with \(p(y|x) = \frac1{\sqrt{2\pi}}e^{-(y - 2x)^2/2}\).

  1. What is the irreducible error.
  2. We fit a linear function and obtain \(\hat f(x) = 0.2 + 1.9x\). What is the reducible error at \(x = 1\).
  3. A method fits all training points exactly, so its training error is zero. Is its reducible error zero. Explain.

Exercise 2 Conceptual

For each situation below, say whether we expect a flexible method to do better or worse than a rigid one, and why.

  1. \(n\) is very large and \(p\) is small.
  2. \(p\) is very large and \(n\) is small.
  3. The relationship between the predictors and the response is strongly non-linear.
  4. The variance of the noise is very large.

Exercise 3 Conceptual

We fit \(k\) nearest neighbours with \(k = 1\).

  1. What is the training error, for regression and for classification.
  2. What does that tell us about the test error.
  3. Sketch how the bias and the variance of \(k\) nearest neighbours change as \(k\) grows from 1 to \(n\).

Exercise 4 Conceptual

Consider ridge regression with penalty \(\lambda\|\theta\|_2^2\).

  1. Sketch how the training error, the test error, the bias and the variance change as \(\lambda\) grows from 0 to \(\infty\).
  2. Explain why the intercept is usually not regularized.
  3. We double the units of one predictor, so its values are twice as large. What happens to its ridge coefficient. What happens if we standardize first.

Applied

Exercise 5 Applied

Write a data generator with \(f(x) = \sin(2x) + 2(x - 0.5)^3 - 0.5x\) and normal noise with standard deviation 0.15, and \(x\) uniform on \([0, 1]\).

  1. Generate a training set of 30 points and a test set of 5000 points.
  2. Fit polynomials of degree 1 to 15 and plot the training and the test error against the degree.
  3. Repeat the whole thing 20 times with different training sets and plot the average curves. How does the picture change.
  4. Repeat with a training set of 300 points. Where is the minimum now.

Exercise 6 Applied · optional

Compare \(k\) nearest neighbours and polynomial regression on the same generator.

  1. For \(k = 1, \dots, 30\), compute the test error of \(k\) nearest neighbours.
  2. Plot it against \(1/k\) together with the polynomial curve from the previous exercise, plotted against the degree, on the same axes.
  3. Which method does better here, and can you say why from the shape of \(f\).