Skip to main content

Exercises

Exercises

Exercise 1

Consider the loss \(\mathcal L(\theta) = \frac1n\sum_i (y_i - \theta_0 - \theta_1 x_i)^2\).

  1. Compute \(\partial \mathcal L / \partial \theta_0\) and \(\partial \mathcal L / \partial \theta_1\).
  2. Write down one step of gradient descent for both parameters.
  3. For which parameter values does gradient descent not make any updates? Compare to the closed form for simple linear regression.

Exercise 2

Take the one-dimensional loss \(\mathcal L(\theta) = \theta^2\).

  1. Write down the gradient descent update with learning rate \(\eta\).
  2. Show that \(\theta_{t} = (1 - 2\eta)^t \theta_0\).
  3. For which \(\eta\) does this converge. For which \(\eta\) does it oscillate. For which \(\eta\) does it diverge?

Exercise 3

Generate 100 points from \(y = 2x_1 - 3x_2 + 1 + \epsilon\) with normal noise. Start from the gradient_descent function in the algorithm. For more on how torch computes gradients, see the PyTorch tutorial on automatic differentiation.

  1. Write the loss function on the training set for a given parameter vector. Use matrix multiplication.
  2. Run gradient descent. Plot the learning curve to see whether it has converged. If you see large fluctuations at the end, lower the learning rate. If the curve is not flat at the end, increase the number of steps. Use a log scale on the vertical axis.
  3. Estimate the coefficients with LinearRegression and compare.
  4. Now run it with a learning rate ten times larger and ten times smaller, and describe what happens.

Exercise 4

  1. In the linear regression panel on gradient descent, start at \(\beta_0 = 0\), \(\beta_1 = -3\) with \(t = 100\), and compare the learning rates 0.01, 0.2 and 1.0. Why does the loss at step 100 stay high for one of them and grow for another?
  2. In the panel on stochastic gradient descent, set the learning rate to 0.05 and \(t = 40\), so that SGD has used 400 data points. Which has the lower loss on all data after these 400 data points, SGD or gradient descent, and why?
  3. In the same panel, set the learning rate to 0.2 and step through the last epochs. Why does SGD not settle at the minimum of the loss on all data?
  4. In the panel comparing gradient descent and Adam, compare how far the two losses are above the minimum at \(t = 50\), with \(x\) as measured and with \(x\) standardized. Why does the order change?

Exercise 5

Explain in two to three sentences each.

  1. Why do we reset the gradient to zero (x.grad.zero_()) in a pytorch implementation of gradient descent.
  2. Why is the learning curve of stochastic gradient descent noisy.
  3. Why does early stopping act like regularization.

Exercise 6

Multinomial logistic regression with \(C\) classes and \(p\) inputs has one linear function per class, \(f_c(x) = \theta_{c0} + \theta_{c1}x_1 + \cdots + \theta_{cp}x_p\), and \[ P(Y = c\,|\,x) = s_c(f(x)) = \frac{e^{f_c(x)}}{\sum_{k=1}^{C} e^{f_k(x)}} . \]

  1. Take \(C = 3\), \(p = 2\), and the parameters \[ \theta = \begin{pmatrix} 0 & 1 & 0\\ 0 & 0 & 1\\ 1 & 0 & 0 \end{pmatrix}, \] whose row \(c\) holds \(\theta_{c0}, \theta_{c1}, \theta_{c2}\). For the three data points \(x_1 = (0, 0)\) with \(y_1 = 3\), \(x_2 = (3, 0)\) with \(y_2 = 1\) and \(x_3 = (0, 2)\) with \(y_3 = 2\), compute all values \(f_c(x_i)\), the probability of each observed label, and the mean negative log-likelihood. Hint: put the three points as rows of a matrix \(\tilde X\) with a first column of ones; then all \(f_c(x_i)\) are the entries of the matrix product \(\tilde X\theta^\top\).
  2. Check your answer with torch.nn.functional.cross_entropy.
  3. Write the log-likelihood \(\sum_{i=1}^n \log P(Y = y_i\,|\,x_i)\) of \(n\) data points with labels \(y_i \in \{1, \ldots, C\}\) and simplify it as much as you can.

Exercise 7 optional

The softmax turns a vector \(z\) of \(C\) numbers into \(C\) probabilities, \(s_c(z) = e^{z_c} / \sum_{k=1}^{C} e^{z_k}\).

  1. Compute \(s(z)\) for \(z = (1, 2, 3)\) and for \(z = (-2, -1, 0)\).
  2. Show that \(s(z + a) = s(z)\) for every number \(a\), where \(z + a\) adds \(a\) to every component of \(z\).
  3. Take \(C = 2\) and \(f_2(x) = 0\). Show that \(s_1(f(x)) = \sigma(f_1(x))\) and \(s_2(f(x)) = 1 - \sigma(f_1(x))\), with \(\sigma(z) = 1/(1 + e^{-z})\).
  4. For the linear functions \(f_c\) of More than two classes, show that the parameters \(\tilde\theta_{cj} = \theta_{cj} - \theta_{Cj}\) give the same probabilities as \(\theta\). How many of the \(C(p + 1)\) parameters are free?
  5. In the softmax panel on More than two classes, set \(a_0\), \(a_1\), \(a_2\) such that all three parameters of class 4 are zero. What are the arrows of the other classes now, and do the probabilities at the black point change?