Skip to main content

Exercises

Exercises

Exercise 1

We fit a regression tree to the data \((x, y) = (1, 2), (2, 3), (3, 9), (4, 10), (5, 11)\).

  1. Compute the reduction in the sum of squares for every possible first split.
  2. Which split does the greedy algorithm take.
  3. Draw the tree of depth 2 and give the predicted value in each leaf.

Exercise 2

Explain in two sentences each.

  1. Why does a random forest not overfit when we add more trees, while gradient boosting does.
  2. Why does a random forest consider only a random subset of the predictors at each split.
  3. Why is a single deep tree a high variance method.

Exercise 3

Consider a support vector machine with a linear kernel.

  1. What happens to the solution if we move a point that is far from the boundary.
  2. What happens if we move a support vector.
  3. Compare this to linear regression, where every point influences the fit. Which behaviour do you want when the data contains outliers.
  4. Describe how the margin and the number of support vectors change as \(C\) goes from very small to very large.

Exercise 4

Repeat the comparison from the comparison page on the weather data from week 2.

  1. Cross-validate all five methods on the training set and report the RMSE with the spread across folds.
  2. Which methods are indistinguishable on this data.
  3. Score the winner once on the 2019 to 2020 test set.
  4. Plot the variable importance of the random forest. Does it agree with the coefficients of the linear model.