Skip to main content

Exercises

Exercises

Exercise 1

Count the parameters of the following networks. Weights and biases both count.

  1. Input 10, one hidden layer of 50 relu neurons, one output.
  2. Input 784, two hidden layers of 100 relu neurons each, 10 outputs.

The MNIST training set has 60 000 images. What could go wrong when we fit the second network to this data set, and under which conditions is it still fine to do so?

Exercise 2

Show that a network with two linear layers and no activation function in between computes the same set of functions as a single linear layer. What does this tell us about the role of the activation function.

Exercise 3

A relu neuron with one input computes \(\max(0, w_0 + w_1 x)\).

  1. Sketch it for \(w_0 = -1, w_1 = 2\).
  2. Sketch the sum of two such neurons with different parameters.
  3. Argue that a network with one hidden layer of \(m\) relu neurons and a linear output is a piecewise linear function with at most \(m\) kinks.

Exercise 4

Let us assume you want to predict the proportion \(y\) of citizens voting for one of two parties based on 113 different features that depend e.g. on the voting history, income distribution or demographics. Because the proportion is a number between 0 and 1, it is reasonable to model it as a sample from a beta distribution \(p(y|\alpha, \beta) = \frac{y^{\alpha - 1}(1-y)^{\beta - 1}}{B(\alpha, \beta)}\), where \(\alpha>0\) and \(\beta>0\) are shape parameters and \(B\) is the beta function. Let us assume you want to model the dependence of the shape parameters on \(x\) with a neural network.

  1. How many input units should this neural network have?

  2. How many output units should this neural network have?

  3. What kind of activation function would you choose for the output layer?

  4. Which loss function would you use to estimate the network parameters?

  5. What is the advantage of using a beta distribution instead of trying to predict directly the expected proportion with a neural network, which has a single sigmoid output unit?

  6. Optional: fit such a network to simulated data with one input, where the mean and the spread of \(y\) both depend on \(x\):

    rng = np.random.default_rng(1)
x = rng.uniform(-3, 3, 1000)
mu = 0.5 + 0.35 * np.sin(x)                  # the true mean of y
kappa = 5 + 20 * np.exp(-x**2 / 2)           # large kappa, small spread
y = rng.beta(mu * kappa, (1 - mu) * kappa)   # alpha = mu kappa, beta = (1 - mu) kappa

    Plot the data with the fitted and the true mean, each with one standard deviation on either side.

    Hints:

    • Start from the code of a network for the mean and the spread: the same network, AdamW, batches of 32. Only the loss changes, and 200 epochs are needed here.
    • Pass both outputs through torch.nn.functional.softplus to get \(\alpha\) and \(\beta\).
    • torch.distributions.Beta(a, b).log_prob(y) returns \(\log p(y_i\,|\,a_i, b_i)\) for every data point; the loss is minus its mean.
    • The beta distribution has mean \(\frac{\alpha}{\alpha + \beta}\) and variance \(\frac{\alpha\beta}{(\alpha + \beta)^2(\alpha + \beta + 1)}\).
    • Evaluate the network on a grid of \(x\) values inside with torch.no_grad():.

Exercise 5

What is the lowest test error we can hope for on MNIST? Part of the answer lies in the data itself: if some images carry the wrong label, no classifier can get all of them right. Use the 60 000 training and the 10 000 test images.

  1. Are there identical images with different labels?
  2. For every test image, find the most similar other test image and check whether the two have different labels. Look at the closest of these pairs: are the digits ambiguous, or does one of them look mislabelled? Hint: on raw pixels, the Euclidean distance favours thin digits like the 1; scale every image to length one first.
  3. Fit a network with two hidden layers of 100 relu neurons to the training set and plot the training and the test error after every epoch. Look at the 32 misclassified test images the network is most confident about. Which of them look mislabelled?
  4. What is your estimate of the lowest possible test error?

Exercise 6

Fit a network to the weather data.

  1. Standardize the input and the output.
  2. Fit a network with one hidden layer of 64 relu neurons. Plot the learning curve.
  3. Compute the training and the test RMSE in km/h, and compare to the multiple linear regression from week 2.
  4. Try 2 and 4 hidden layers with the same total number of parameters. Does depth help here.