Exercises

Exercises

Conceptual

Exercise 1 Conceptual

Dropout with rate \(p\) sets each activation to zero with probability \(p\) and divides the rest by \(1-p\).

1. Show that the expected value of an activation is unchanged.
2. Why do we divide by \(1-p\) during training instead of multiplying by \(1-p\) at test time.
3. Explain in two sentences why dropout reduces overfitting.

Exercise 2 Conceptual

We model counts with a Poisson distribution and a log link.

1. Write down the negative log-likelihood and drop the terms that do not depend on the parameters.
2. A coefficient in the linear version is 0.2. By what factor does the expected count change for a one unit increase in that predictor.
3. Why does a Poisson model have no separate noise parameter.

Exercise 3 Conceptual

The hour of the day is encoded in three different ways below. For each one, say what the model can and cannot express.

1. As a single number from 0 to 23.
2. As 24 one-hot columns.
3. As \(\sin(2\pi h/24)\) and \(\cos(2\pi h/24)\).

Which one would you choose for the bicycle data, and why.

Critique

Exercise 4 Critique

The code below trains a network on the bicycle data and reports a good test loss. It contains two bugs that make the reported number wrong, and one that makes the model worse than it should be.

scaler = StandardScaler()
X = scaler.fit_transform(features)
X_train, X_test, y_train, y_test = train_test_split(X, counts, test_size=0.25)

model = PoissonNet(X.shape[1], dropout=0.3)
opt = torch.optim.AdamW(model.parameters(), lr=1e-2)

for epoch in range(200):
    for xb, yb in loader:
        opt.zero_grad()
        loss = torch.mean((model(xb) - yb) ** 2)
        loss.backward()
        opt.step()

with torch.no_grad():
    pred = model(torch.tensor(X_test))
print("test loss:", float(torch.mean((pred - y_test) ** 2)))

1. Find all three problems.
2. For each one, say whether it makes the reported number too good, too bad, or simply wrong.
3. Fix the code.

Applied

Exercise 5 Applied

Work through the ladder of models on the real bicycle data from OpenML, id 42712.

1. Fit a Poisson GLM on the raw predictors and report the cross-validated deviance.
2. Add one-hot coding for the hour and the weekday, and report again.
3. Replace the one-hot hour by the cyclic encoding. Which is better here, and by how much.
4. Fit the Poisson network. Does it beat the best GLM.
5. Score the winner once on a test set that you held out before you started.

Exercise 6 Applied

Compare the four ways of regularizing a network on the same data.

1. No regularization.
2. Weight decay, three values.
3. Dropout, three rates.
4. Early stopping.

Report the test deviance for each and say which one gives the best result per unit of effort spent tuning.

Exercise 7 Applied · optional

Replace the Poisson likelihood by a negative binomial one, which has an extra parameter for the dispersion.

1. Write down the loss.
2. Fit it as a second output of the same network.
3. Compare to the Poisson model. Is the data overdispersed.