Skip to main content

Exercises

Exercises

Conceptual

Exercise 1 Conceptual

We fit a regression tree to the data \((x, y) = (1, 2), (2, 3), (3, 9), (4, 10), (5, 11)\).

  1. Compute the reduction in the sum of squares for every possible first split.
  2. Which split does the greedy algorithm take.
  3. Draw the tree of depth 2 and give the predicted value in each leaf.

Exercise 2 Conceptual

Explain in two sentences each.

  1. Why does a random forest not overfit when we add more trees, while gradient boosting does.
  2. Why does a random forest consider only a random subset of the predictors at each split.
  3. Why is a single deep tree a high variance method.

Exercise 3 Conceptual

Consider a support vector machine with a linear kernel.

  1. What happens to the solution if we move a point that is far from the boundary.
  2. What happens if we move a support vector.
  3. Compare this to linear regression, where every point influences the fit. Which behaviour do you want when the data contains outliers.
  4. Describe how the margin and the number of support vectors change as \(C\) goes from very small to very large.

Exercise 4 Conceptual

For each data set below, say which of the four methods you would try first and why.

  1. 500 rows, 10 predictors, mostly categorical, and the client wants to know which predictor matters.
  2. 200 rows, 20000 predictors, two classes.
  3. Two million rows, 30 numeric predictors.
  4. 60000 images of 28 by 28 pixels.

Applied

Exercise 5 Applied

Repeat the comparison from the comparison page on the weather data from week 2.

  1. Cross-validate all five methods on the training set and report the RMSE with the spread across folds.
  2. Which methods are indistinguishable on this data.
  3. Score the winner once on the 2019 to 2020 test set.
  4. Plot the variable importance of the random forest. Does it agree with the coefficients of the linear model.

Exercise 6 Applied

Tune gradient boosting properly.

  1. Cross-validate over the number of trees, the learning rate and the maximum depth, on a grid.
  2. Plot the cross-validated error against the number of trees, one line per learning rate. What is the relationship between the two.
  3. How much better is the tuned model than the default one.
  4. How much of that improvement survives when you score on a held-out test set.

Exercise 7 Applied · optional

Compare kernels for a support vector machine on the MNIST digits, using only the classes 3 and 8 and the first 2000 rows.

  1. Fit with a linear kernel and with an RBF kernel, tuning \(C\) and \(\gamma\).
  2. Report the test accuracy of both.
  3. How many support vectors does each one use, and what does that tell you.