Skip to main content

Exercises

Exercises

Conceptual

Exercise 1 Conceptual

Count the parameters of the following networks. Weights and biases both count.

  1. Input 10, one hidden layer of 50 relu neurons, one output.
  2. Input 784, hidden layers of 100 and 100 relu neurons, 10 outputs.
  3. Input 3, five hidden layers of 20 relu neurons, 2 outputs.

For the second one, how many training images would we need for the number of points to exceed the number of parameters.

Exercise 2 Conceptual

Show that a network with two linear layers and no activation function in between computes the same set of functions as a single linear layer. What does this tell us about the role of the activation function.

Exercise 3 Conceptual

A relu neuron with one input computes \(\max(0, w_0 + w_1 x)\).

  1. Sketch it for \(w_0 = -1, w_1 = 2\).
  2. Sketch the sum of two such neurons with different parameters.
  3. Argue that a network with one hidden layer of \(m\) relu neurons and a linear output is a piecewise linear function with at most \(m\) kinks.

Exercise 4 Conceptual

For each response below, say which output layer and which loss you would use, and why.

  1. The price of a flat.
  2. Whether an email is spam.
  3. Which of 200 species a photograph shows.
  4. The number of visitors to a museum on a given day.
  5. A probability that must lie between 0 and 1, given as a fraction.

Applied

Exercise 5 Applied

Fit a network to the weather data.

  1. Standardize the input and the output.
  2. Fit a network with one hidden layer of 64 relu neurons. Plot the learning curve.
  3. Compute the training and the test RMSE in km/h, and compare to the multiple linear regression from week 2.
  4. Try 2 and 4 hidden layers with the same total number of parameters. Does depth help here.

Exercise 6 Applied

Fit a network to MNIST.

  1. Use the first 60000 images for training and the last 10000 for testing.
  2. Report the test accuracy and the confusion matrix.
  3. Plot ten wrongly classified images with the true label and the prediction. Are they also hard for you.
  4. Now train on only 1000 images. How much does the accuracy drop. What does that tell you about the number of parameters.

Exercise 7 Applied · optional

Fit a polynomial of degree 3 in 1000 input dimensions with a network.

  1. Generate data where only 5 of the 1000 inputs matter.
  2. Fit a network and report the test error.
  3. Fit a lasso on the same data and compare. Which one finds the 5 relevant inputs.