Skip to main content

Exercises

Exercises

Conceptual

Exercise 1 Conceptual

We have training data \(((x_1 = 0, y_1 = -1), (x_2 = 2, y_2 = 4), (x_3 = 2, y_3 = 3))\) and a family of densities \(p(y|x) = \frac1{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(y - \theta_0 - \theta_1 x)^2}{2\sigma^2}\right)\).

  1. Write the log-likelihood of \(\theta_0\) and \(\theta_1\) for this data.
  2. Find the \(\hat\theta_0\) and \(\hat\theta_1\) that maximize it. You do not need to solve it analytically. Compare the expression to the loss of linear regression.
  3. Show for general training data that the log-likelihood is maximized by the same \(\hat\theta\) that minimizes the loss of linear regression.

Exercise 2 Conceptual

Data is generated with \(y = 3x + 1 + \epsilon\), where \(\epsilon\) is normal with mean 0 and variance \(\sigma^2\). We have training data \(\mathcal D = ((0, 1), (2, 9))\) and test data \(\mathcal D_\mathrm{test} = ((0, 0), (3, 20))\). We use the family \(f(x) = \theta_0 + \theta_1 x^2\) and the loss \(L(y, \hat y) = |y - \hat y|\).

Which of the following are correct.

  1. The training loss is smallest for \(\hat\theta_0 = 1\) and \(\hat\theta_1 = 2\).
  2. The test loss of \(f(x) = 1 + 2x^2\) at \(x = 0\) for the conditional data generating process is 1.
  3. The test loss of \(f(x) = 1 + 2x^2\) on the test set is 1.

Exercise 3 Conceptual

Take one of the applications of machine learning from the first lecture. Discuss with a colleague

  • the data generating process for \(X\),
  • the data generating process for \(Y|X\),
  • where the noise comes from,
  • a distribution that could be used to model \(Y|X\).

Applied

Exercise 4 Applied

In this exercise we build an example where the response looks noisy if we look at only some of the predictors, although it is in fact deterministic.

The time a feather needs to reach the ground when dropped from one meter is determined by the fluffiness \(f \in [1, 4)\), the shape \(x \in \{s, e\}\) for spherical or elongated, the air density \(\rho \in [1.1, 1.4)\) kg/m³ and the wind speed \(w \in [0, 5)\) m/s, through

\[ g(f, x, \rho, w) = \rho + 0.1f^2 + 0.3w + 2 I(x = s) + 0.5 I(x = e). \]

  1. Generate a data set of 500 experiments with different feathers under different conditions.
  2. Make a scatter plot with the fluffiness on the horizontal axis and the time on the vertical axis.
  3. Explain why it looks as if the time depended probabilistically on the fluffiness, although we used a deterministic function. If you want a mathematical argument, marginalize \(P(t|f, x, \rho, w) = \delta(t - g(f, x, \rho, w))\) over the other variables and show that \(P(t|f)\) is not degenerate.

Exercise 5 Observation

The panel under four different losses generates data from

\[ Y = 2x - 1 + \Sigma(x)\,Z, \qquad \Sigma(x) = \Sigma_\mathrm{slope}\,x + \Sigma_0, \qquad X, Z \sim \mathcal N(0, 1), \]

fits \(\hat f(x) = \hat a x + \hat b\) by least squares on the training set, and reports \(\hat a\) and \(\hat b\) next to the four losses.

  1. Write the test loss at \(x_0\) for the conditional process as an expression in \(\hat a\), \(\hat b\), \(x_0\), \(\Sigma_0\) and \(\Sigma_\mathrm{slope}\). Split \(Y\) into its mean and its noise before you square, and remember that \(Z\) has mean zero.
  2. Do the same for the test loss of the joint process. The input is random here as well, so you average over \(X\) too. You will need \(\mathbb E[X] = 0\) and \(\mathbb E[X^2] = 1\).
  3. Check both expressions against the panel. Pick three or four settings of the sliders, read \(\hat a\) and \(\hat b\) off the monitors, evaluate what you derived, and compare with the two loss monitors. They should agree to the three digits shown. If one of them does not, the panel is right and the expression is wrong, so look for the term that is missing.
  4. Your expression for the joint loss should not contain \(x_0\), and the one at \(x_0\) should not contain \(n\) or \(N\). Confirm both on the panel: move \(x_0\) and watch which monitors move, then move \(N\) and the test seed and watch again.
  5. Which term of each expression survives when the fit is perfect, \(\hat a = 2\) and \(\hat b = -1\)? This is the irreducible error. Set \(\Sigma_\mathrm{slope} = 0\), take \(n\) from one end of its range to the other and describe what happens to the training loss and to the joint loss when \(n\) is small and when \(n\) is large. Which value does the joint loss approach, and why can it not go below it?
  6. Put \(n\) back at its largest and move \(N\) from smallest to largest, then change the test seed at small \(N\) and again at large \(N\). Describe how the test set loss behaves relative to the joint loss in each case. Of these two, which is the quantity we actually care about, and which one is the only one we can compute when we do not know the generator?

Exercise 6 Applied

Run a multiple linear regression on the weather data with two predictors, the pressure in Luzern and the sunshine duration in Luzern. Is the prediction on the test set better than with the pressure alone.

Exercise 7 Applied · optional

Above we used all available predictors. We do not know whether all of them are relevant.

  1. Search systematically for the model with at most 2 predictors that has the lowest test RMSE. Write a function that takes the training data, the test data and a pair of predictors, fits a linear regression and returns the test RMSE.
  2. How much higher is the test error than with all predictors.
  3. How many models did you have to fit.
  4. How many would you have to fit for the best model with at most 5 predictors.