Skip to main content

Exercises

Exercises

Conceptual

Exercise 1 Conceptual

Consider the loss \(\mathcal L(\theta) = \frac1n\sum_i (y_i - \theta_0 - \theta_1 x_i)^2\).

  1. Compute \(\partial \mathcal L / \partial \theta_0\) and \(\partial \mathcal L / \partial \theta_1\).
  2. Write down one step of gradient descent for both parameters.
  3. Set the two derivatives to zero and solve. Compare to the closed form for simple linear regression.

Exercise 2 Conceptual

Take the one-dimensional loss \(\mathcal L(\theta) = \theta^2\).

  1. Write down the gradient descent update with learning rate \(\eta\).
  2. Show that \(\theta_{t} = (1 - 2\eta)^t \theta_0\).
  3. For which \(\eta\) does this converge. For which \(\eta\) does it oscillate. For which \(\eta\) does it diverge.
  4. What does this tell us about the largest usable learning rate for a general quadratic loss.

Exercise 3 Conceptual

Explain in two sentences each.

  1. Why do we reset the gradient to zero in every step.
  2. Why is the learning curve of stochastic gradient descent noisy.
  3. Why does early stopping act like regularization.

Applied

Exercise 4 Applied

Generate 100 points from \(y = 2x_1 - 3x_2 + 1 + \epsilon\) with normal noise.

  1. Write the loss function on the training set for a given parameter vector. Use matrix multiplication.
  2. Run gradient descent. Plot the learning curve to see whether it has converged. If you see large fluctuations at the end, lower the learning rate. If the curve is not flat at the end, increase the number of steps. Use a log scale on the vertical axis.
  3. Estimate the coefficients with LinearRegression and compare.
  4. Now run it with a learning rate ten times larger and ten times smaller, and describe what happens.

Exercise 5 Applied

Compare batch sizes on the same data.

  1. Run gradient descent with batch sizes 1, 10, 100 and \(n\), with the same learning rate and the same number of epochs.
  2. Plot the four learning curves on one figure.
  3. Plot the same curves against wall clock time instead of epochs. Does the ranking change.

Exercise 6 Applied · optional

The XOR problem. Generate points in two dimensions with class 1 if \(x_1 x_2 > 0\) and class 0 otherwise.

  1. Fit a logistic regression and report the accuracy. Plot the decision boundary.
  2. Add the product \(x_1 x_2\) as a third predictor and fit again.
  3. Explain what the third predictor changed, in terms of the geometry of the problem.