Skip to main content

Exercises

Exercises

All of these use the four data sets from our data sets.

Conceptual

Exercise 1 Conceptual

Fill in this table for the four data sets of the course.

MNIST spam weather bicycle rentals
what is one input \(x_i\)
what is one output \(y_i\)
\(n\)
\(p\)
classification or regression
prediction or interpretation

Read \(n\) and \(p\) off the page where available, or load the data yourself. In some cases the number of predictors may not be obvious. Explain why.

Then answer, in one sentence each.

  1. Who produced the outputs \(y_i\), and how? Your answer should be different for all four.
  2. For which of the four could two careful people disagree about the correct \(y_i\) for the same \(x_i\)? What does that imply about the best accuracy any model could reach?

Exercise 2 Conceptual

The weather data is a table of hourly measurements. The response we predict is the wind peak in Luzern five hours later, which can be build with

y = weather["LUZ_wind_peak"][5:].values
X = weather["LUZ_pressure"][:-5].values
  1. Explain what the two slices do, and why both are needed. What would go wrong if you used weather["LUZ_wind_peak"] and weather["LUZ_pressure"] directly?
  2. Nothing in the measurements says “five hours”. Where does that number come from? Write down what changes in \(X\) and \(y\) for a one hour horizon, and for a 24 hour horizon.
  3. Which of the three horizons do you expect to be easiest to predict? What happens to the achievable accuracy as the horizon grows, and what does it approach in the limit of a very long horizon?

Critique

Exercise 3 Critique

You will use a language model throughout this course. This exercise is about finding out how far you can trust it on a question of fact.

  1. Ask a language model to describe the bicycle rental data set: what the columns are, what values each one takes, how many rows there are, and what the response is. Keep its answer.

  2. Load the data yourself. It is on OpenML with id 42712:

    from sklearn.datasets import fetch_openml
bikes = fetch_openml(data_id=42712, as_frame=True, parser="auto").frame

  3. Check three of its claims against the data. Prefer specific ones, a range or a count rather than “the data contains weather information”. Report which held and which did not, and the line of code that settled each.

  4. If something was wrong, was it wrong in a way you would have noticed without checking? Which kinds of claim did it get right, and which kinds should you always verify?

Applied

Exercise 4 Applied

Before fitting anything, it is worth knowing what “doing nothing” achieves. Two predictions of the Luzern wind peak five hours ahead that use no machine learning at all.

  1. Load the weather data and build y as in exercise 2.
  2. The mean. Predict the same number for every hour, the mean of y. Report the mean squared error.
  3. Persistence. Predict that the wind peak in five hours equals the wind peak now, that is weather["LUZ_wind_peak"][:-5].values. Report the mean squared error.
  4. Which is better, and by how much? Write both numbers down. Every model in the rest of the course has to beat them to be worth anything.
  5. One of the two is the obvious guess, and it loses. Explain why it loses at a five hour horizon, and say at what horizon you would expect it to win.
  6. Compare the error of the mean predictor with y.var(). The two agree exactly. Explain why that is not a coincidence.

Exercise 5 Applied

The same question for the spam data, where the response is a label rather than a number.

  1. Load the spam data and compute the fraction of emails that are spam.
  2. Consider the classifier that answers “ham” for every email, whatever it says. What is its accuracy? This is the number a real filter has to beat.
  3. Someone reports that their filter is 90% accurate on this data. Given what you found in 1, is that impressive? Now suppose the same 90% were reported on a mailbox where only 3 emails in 100 are spam. Is it impressive there?
  4. Two filters have exactly the same accuracy. The first sometimes lets spam through; the second sometimes deletes a real email. Are they equally good? Say what you would want reported instead of a single accuracy, and why.