Skip to main content

Exercises

Exercises

Conceptual

Exercise 1 Conceptual

We cross-validate ten polynomial degrees and report the smallest of the ten scores as our estimate of the test error.

  1. Let \(S_1, \dots, S_{10}\) be the ten scores and assume each one is unbiased, so \(\mathrm{E}[S_d] = \mu_d\). Show that \(\mathrm{E}\bigl[\min_d S_d\bigr] \le \min_d \mu_d\).
  2. When does equality hold. Say what that condition means in words.
  3. The ten scores are not independent, because they come from the same folds of the same data. Does that make the bias larger or smaller than for ten independent scores. Argue informally.

Exercise 2 Conceptual

For a model that is linear in its parameters, leave-one-out has the closed form \(\mathrm{CV}_n = \frac1n\sum_i \bigl((y_i - \hat y_i)/(1 - h_{ii})\bigr)^2\).

  1. Show that \(\sum_i h_{ii} = p + 1\) for a design matrix with \(p\) predictors and an intercept. Hint, \(H\) is a projection, so consider its trace.
  2. What is the average leverage. Explain why \(\mathrm{CV}_n\) blows up when the number of parameters approaches \(n\).
  3. Why is there no similar shortcut for \(K\)-fold with \(K < n\).

Exercise 3 Conceptual

We have daily gene expression measurements from 12 patients, measured every day for 30 days. We want to predict tomorrow’s expression from today’s. Three cross-validation schemes are proposed.

    1. Standard 10-fold cross-validation over all \(12 \times 30\) rows.
    1. Leave one patient out, so 12 folds with one patient each.
    1. Split by time, train on days 1 to 20 of all patients and validate on days 21 to 30.

For each of the two goals below, say which scheme is appropriate and why the others are optimistic.

  1. Predict for a new patient.
  2. Predict tomorrow for a patient who is already in the study.

Observation

Exercise 4 Observation

Open the illustration on the theory page.

  1. Set the strategy to the single 50/50 split and move \(n\) from 20 to 320. Write down the value of “picks within ±1” at each \(n\). It does not improve monotonically. Explain why not, in two sentences, using the shape of the true error curve.
  2. Set \(n = 320\) and step through all four strategies. Rank them. Then look at the width of the band instead of the headline number, and say whether the ranking is the same.
  3. At \(n = 20\) all four strategies score about 80 percent. Is model selection easy when data is scarce. Look at which degree gets selected before you answer.

Critique

Exercise 5 Critique

The code below was written by a language model, asked for cross-validated ridge regression with feature standardization on a gene expression data set. It runs without an error and reports a good score.

It contains three leaks. Each one on its own makes the reported score too good.

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, cross_val_score
from sklearn.preprocessing import StandardScaler

X, y = load_expression_data()          # X: (200 patients, 5000 genes)

# 1. standardise the features
scaler = StandardScaler()
Xs = scaler.fit_transform(X)

# 2. keep the 100 genes most correlated with the outcome
corr = np.array([np.corrcoef(Xs[:, j], y)[0, 1] for j in range(Xs.shape[1])])
top = np.argsort(-np.abs(corr))[:100]
Xs = Xs[:, top]

# 3. tune the ridge penalty
best_alpha, best_score = None, -np.inf
for alpha in [0.01, 0.1, 1, 10, 100, 1000]:
    score = cross_val_score(Ridge(alpha=alpha), Xs, y,
                            cv=KFold(5, shuffle=True, random_state=0),
                            scoring="neg_mean_squared_error").mean()
    if score > best_score:
        best_alpha, best_score = alpha, score

print(f"alpha = {best_alpha},  CV MSE = {-best_score:.3f}")
  1. Find all three leaks. For each one, say exactly which information moves from the held-out data into the fitted model.
  2. Rank them by how much damage they do, given \(n = 200\) and \(p = 5000\). One of them is far worse than the others at these dimensions. Which one, and why.
  3. Rewrite the script so that the number it reports is honest. You will need a Pipeline and a nested loop.
  4. Before you run your version, write down whether you expect the score to get better or worse, and by roughly how much. Then run it.

Exercise 6 Critique

Ask a language model to write a function that evaluates a classifier with cross-validation on an imbalanced data set where only 3 percent of the cases are positive.

  1. Record the answer.
  2. Check three things. Does it stratify the folds. Does it report a metric that survives 97 percent class imbalance. Does any preprocessing sit outside the fold loop.
  3. For each of the three, say whether the answer got it right, got it wrong, or did not mention it. The third case is common, and it is the one that costs points.
  4. In one paragraph, say what you needed to know already in order to find the errors you found.

Applied

Exercise 7 Applied

Reproduce the measurement on the nested CV page, then extend it.

  1. Run it again with a larger search. Use polynomial degree 1 to 10 together with a ridge penalty \(\lambda \in \{10^{-4}, \dots, 10^{2}\}\), so 70 combinations.
  2. Plot the gap between flat cross-validation and the truth as a function of the number of combinations searched, for \(m \in \{2, 5, 10, 25, 70\}\).
  3. The gap grows roughly like the expected minimum of \(m\) noisy draws. What does that mean for a project in which you try 500 configurations.

Exercise 8 Applied · optional

The leave-one-out shortcut assumed a model that is linear in its parameters. Test how far it goes.

  1. Implement leave-one-out by brute force and with the hat matrix formula for polynomial regression. Check that they agree to machine precision.
  2. Now apply the same formula to ridge regression, where the hat matrix is \(H_\lambda = X(X^\top X + \lambda I)^{-1}X^\top\). Does the identity still hold exactly. Check it numerically for three values of \(\lambda\).
  3. Explain why it fails for \(k\) nearest neighbours, and what leverage would even mean there.