Skip to main content

Exercises

Exercises

Conceptual

Exercise 1 Conceptual

We fit a logistic regression to predict whether a student passes a course, from \(X_1\) the hours studied and \(X_2\) the grade in a statistics class. We obtain \(\hat\beta_0 = -6\), \(\hat\beta_1 = 0.025\) and \(\hat\beta_2 = 1\).

  1. Estimate the probability that a student who studies 75 hours and had a 4 in the statistics class passes.
  2. How many hours would that student need to study to have a 50 percent chance of passing.
  3. Write \(\hat\beta_1\) as an odds ratio and say in one sentence what it means.

Exercise 2 Conceptual

Here we look at the loss that maximum likelihood gives us for classification with \(C\) classes.

Write the \(i\)-th component of the linear function applied to the \(k\)-th point as \(f_i(x_k) = \theta_{i0} + \theta_{i1}x_{k1} + \cdots + \theta_{ip}x_{kp}\), so that \(\theta\) is a \(C \times (p+1)\) matrix.

  1. Write the log-likelihood for \(n\) points, \(p\) input dimensions and \(C\) classes, and simplify it as much as you can. Use the convention \(s_y(f(x)) = P(y|x)\) for the \(y\)-th component of the softmax.
  2. Take \(C = 3\) and \(p = 2\). Write the log-likelihood explicitly for the training set \(\mathcal D = ((x_1 = (0, 0), y_1 = C), (x_2 = (3, 0), y_2 = A), (x_3 = (0, 2), y_3 = B))\).
  3. With \(C = 2\) the softmax has twice as many parameters as logistic regression. Set \(\theta_{20} = \theta_{21} = 0\) and show that we recover logistic regression. Hint, show that \(s_1(f(x)) = s(f_1(x))\) and \(s_2(f(x)) = 1 - s(f_1(x))\).
  4. Show that we can always set the parameters of the last row to zero, for any \(C\). Hint, show that the softmax with \(\tilde\theta_{ij} = \theta_{ij} - \theta_{Cj}\) gives the same value.

Exercise 3 Conceptual

A test has sensitivity 0.99 and specificity 0.95. The disease affects 1 person in 1000.

  1. Of 100000 people, how many test positive.
  2. Of those, how many actually have the disease.
  3. The AUC of this test is high. Explain why that is not enough to make it useful for screening, and what you would change.

Applied

Exercise 4 Applied

Linear classification of the MNIST digits.

  1. Load MNIST. Use the first 5000 rows so that it does not take too long.
  2. Scale the pixel values to \([0, 1)\) by dividing by 255.
  3. Fit a multinomial logistic regression without penalty. Be patient.
  4. Compute the misclassification rate and the confusion matrix on the training set.
  5. Use rows 60001 to 70000 as a test set and do the same there.
  6. Plot some correctly classified test images and some wrongly classified ones. Are the wrong ones also hard for you.

Exercise 5 Applied

Write a data generator that samples \(x\) from a normal distribution with mean 2 and standard deviation 3, and samples \(y \in \{\text{true}, \text{false}\}\) from a Bernoulli distribution with rate \(s(0.5x - 2.7)\).

  1. Create a training set of size \(n = 20\) and fit a logistic regression.
  2. Look at the fitted parameters and compare them to the generator.
  3. Predict the probability of true on the training input.
  4. Create a test set of size \(n = 10^4\) where the input is always \(x = 4\). Estimate the average test error at \(x = 4\) with the negative log-likelihood.
  5. Compute the same test error directly from the fitted parameters and the known generator, and compare.
  6. Rerun with different training sets of size 20, then with size \(10^4\). Write down what you observe.

Exercise 6 Applied · optional

Poisson regression on the bicycle rental data.

  1. Fit a Poisson regression of count on temp and humidity.
  2. Fit a linear regression on the same inputs and output.
  3. Make a scatter plot with the true counts on the horizontal axis and the predictions of both models on the vertical axis. Add the diagonal.
  4. Comment on the differences you see.