Exercises

Exercises

Exercise 1

Dropout with rate \(p\) sets each activation to zero with probability \(p\) and divides the rest by \(1-p\).

1. Show that the expected value of an activation is unchanged.
2. Why do we divide by \(1-p\) during training instead of multiplying by \(1-p\) at test time.
3. Explain in two sentences why dropout reduces overfitting.

Exercise 2

The hour of the day is encoded in three different ways below. For each one, say what the model can and cannot express.

1. As a single number from 0 to 23.
2. As 24 one-hot columns.
3. As \(\sin(2\pi h/24)\) and \(\cos(2\pi h/24)\).

Which one would you choose for the bicycle data, and why.

Exercise 3

The code below trains a network on the bicycle data and reports a good test loss. It contains two bugs that make the reported number wrong, and one that makes the model worse than it should be.

scaler = StandardScaler()
X = scaler.fit_transform(features)
X_train, X_test, y_train, y_test = train_test_split(X, counts, test_size=0.25)

model = PoissonNet(X.shape[1], dropout=0.3)
opt = torch.optim.AdamW(model.parameters(), lr=1e-2)

for epoch in range(200):
    for xb, yb in loader:
        opt.zero_grad()
        loss = torch.mean((model(xb) - yb) ** 2)
        loss.backward()
        opt.step()

with torch.no_grad():
    pred = model(torch.tensor(X_test))
print("test loss:", float(torch.mean((pred - y_test) ** 2)))

1. Find all three problems.
2. For each one, say whether it makes the reported number too good, too bad, or simply wrong.
3. Fix the code.

Exercise 4

Work through the ladder of models on the real bicycle data from OpenML, id 42712.

1. Fit a Poisson GLM on the raw predictors and report the cross-validated deviance.
2. Add one-hot coding for the hour and the weekday, and report again.
3. Replace the one-hot hour by the cyclic encoding. Which is better here, and by how much.
4. Fit the Poisson network. Does it beat the best GLM.
5. Score the winner once on a test set that you held out before you started.