Skip to main content

Exercises

Exercises

Exercise 1

We have training data \(((x_1 = 0, y_1 = -1), (x_2 = 2, y_2 = 4), (x_3 = 2, y_3 = 3))\) and a family of densities \(p(y|x) = \frac1{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(y - \theta_0 - \theta_1 x)^2}{2\sigma^2}\right)\).

  1. Write the log-likelihood of \(\theta_0\) and \(\theta_1\) for this data.
  2. Find the \(\hat\theta_0\) and \(\hat\theta_1\) that maximize it. You do not need to solve it analytically. Compare the expression to the loss of linear regression.
  3. Show for general training data that the log-likelihood is maximized by the same \(\hat\theta\) that minimizes the loss of linear regression.

Exercise 2

Data is generated with \(y = 3x + 1 + \epsilon\), where \(\epsilon\) is normal with mean 0 and variance \(\sigma^2\). We have training data \(\mathcal D = ((0, 1), (2, 9))\) and test data \(\mathcal D_\mathrm{test} = ((0, 0), (3, 20))\). We use the family \(f(x) = \theta_0 + \theta_1 x^2\) and the loss \(L(y, \hat y) = |y - \hat y|\).

Which of the following are correct.

  1. The training loss is smallest for \(\hat\theta_0 = 1\) and \(\hat\theta_1 = 2\).
  2. The test loss of \(f(x) = 1 + 2x^2\) at \(x = 0\) for the conditional data generating process is 1. Hint: \(\mathrm{E}|\epsilon| = \sigma\sqrt{2/\pi}\).
  3. The test loss of \(f(x) = 1 + 2x^2\) on the test set is 1.

Exercise 3

In this exercise we build an example where the response looks noisy if we look at only some of the predictors, although it is in fact deterministic.

The time a feather needs to reach the ground when dropped from one meter is determined by the fluffiness \(f \in [1, 4)\), the shape \(x \in \{s, e\}\) for spherical or elongated, and the wind speed \(w \in [0, 5)\) m/s, through

\[ g(f, x, w) = 0.1f^2 + 0.2w + 2 I(x = s) + 0.5 I(x = e). \]

  1. Generate a data set of 500 experiments with different feathers under different conditions.
  2. Make a scatter plot with the fluffiness on the horizontal axis and the time on the vertical axis.
  3. Explain why it looks as if the time depended probabilistically on the fluffiness, although we used a deterministic function.
  4. (optional) If you want a mathematical argument, start from \(p(t|f, x, w) = \delta(t - g(f, x, w))\), marginalize over the other variables and compute \(p(t|f = 2)\) explicitly. Is it a delta distribution?

Exercise 4

The panel under four different losses generates data from

\[ Y = 2x - 1 + \Sigma(x)\,Z, \qquad \Sigma(x) = \Sigma_\mathrm{slope}\,x + \Sigma_0, \qquad X, Z \sim \mathcal N(0, 1), \]

fits \(\hat f(x) = \hat a x + \hat b\) by least squares on the training set, and reports \(\hat a\) and \(\hat b\) next to the four losses.

  1. Write the test loss at \(x_0\) for the conditional process as an expression in \(\hat a\), \(\hat b\), \(x_0\), \(\Sigma_0\) and \(\Sigma_\mathrm{slope}\). Split \(Y\) into its mean and its noise before you square, and remember that \(Z\) has mean zero.
  2. Do the same for the test loss of the joint process. The input is random here as well, so you average over \(X\) too. You will need \(\mathbb E[X] = 0\) and \(\mathbb E[X^2] = 1\).
  3. Check both expressions against the panel. Pick three or four settings of the sliders, read \(\hat a\) and \(\hat b\) off the monitors, evaluate what you derived, and compare with the two loss monitors. They should agree to the three digits shown. If one of them does not, the panel is right and the expression is wrong, so look for the term that is missing.
  4. Your expression for the joint loss should not contain \(x_0\), and the one at \(x_0\) should not contain \(n\) or \(N\). Confirm both on the panel: move \(x_0\) and watch which monitors move, then move \(N\) and the test seed and watch again.
  5. Which term of each expression survives when the fit is perfect, \(\hat a = 2\) and \(\hat b = -1\)? This is the irreducible error. Set \(\Sigma_\mathrm{slope} = 0\), take \(n\) from one end of its range to the other and describe what happens to the training loss and to the joint loss when \(n\) is small and when \(n\) is large. Which value does the joint loss approach, and why can it not go below it?
  6. Put \(n\) back at its largest and move \(N\) from smallest to largest, then change the test seed at small \(N\) and again at large \(N\). Describe how the test set loss behaves relative to the joint loss in each case. Of these two, which is the quantity we actually care about, and which one is the only one we can compute when we do not know the generator?

Exercise 5

Two classes on the weather data. Define the response to be true when the wind peak in five hours exceeds 40 km/h and false otherwise.

  1. What fraction of hours is true? What accuracy does the constant classifier reach?
  2. Fit a logistic regression on the pressure in Luzern. Report the training and test accuracy, and compare to the constant classifier.
  3. Plot the fitted probability against the pressure, with the data as ticks at 0 and 1. At which pressure does the model cross 0.5?
  4. Refit with all predictors. How much does the accuracy improve, and is the improvement of the same size as the one linear regression got from the same change?