A convolutional network

Examples

We fit a convolutional network to the MNIST digits of week 6. The images keep their shape: one channel of 28 by 28 pixels, instead of a vector of 784 numbers.

import numpy as np
import torch
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_openml

mnist = fetch_openml(data_id=554, as_frame=False, parser="auto")
X_all = (mnist.data / 255.0).astype(np.float32).reshape(-1, 1, 28, 28)
y_all = mnist.target.astype(np.int64)
X_train, y_train = torch.tensor(X_all[:60000]), torch.tensor(y_all[:60000])
X_test, y_test = torch.tensor(X_all[60000:]), torch.tensor(y_all[60000:])
print(X_train.shape)

torch.Size([60000, 1, 28, 28])

The first dimension counts the images, the second the channels.

The architecture

torch.manual_seed(1)
cnn = torch.nn.Sequential(
    torch.nn.Conv2d(1, 16, kernel_size=8), torch.nn.ReLU(),
    torch.nn.Conv2d(16, 32, kernel_size=4), torch.nn.ReLU(),
    torch.nn.Flatten(),
    torch.nn.Linear(32 * 18 * 18, 10),
)
x = torch.zeros(1, 1, 28, 28)
for layer in cnn:
    x = layer(x)
    print(f"{layer.__class__.__name__:8s} {tuple(x.shape)}")

1

16 filters of size 8 by 8 on one input channel, stride 1, no padding. The output volume is \(21 \times 21 \times 16\).

2

32 filters of size \(4 \times 4 \times 16\), which give \(18 \times 18 \times 32\).

3

The volume becomes a vector of 10368 numbers, which a dense layer maps to the ten outputs.

Conv2d   (1, 16, 21, 21)
ReLU     (1, 16, 21, 21)
Conv2d   (1, 32, 18, 18)
ReLU     (1, 32, 18, 18)
Flatten  (1, 10368)
Linear   (1, 10)

for layer in cnn:
    n = sum(p.numel() for p in layer.parameters())
    if n:
        print(f"{layer.__class__.__name__:8s} {n:6d} parameters")

Conv2d     1040 parameters
Conv2d     8224 parameters
Linear   103690 parameters

The two convolutional layers have \(16(8 \cdot 8 + 1) = 1040\) and \(32(4 \cdot 4 \cdot 16 + 1) = 8224\) parameters. Most of the parameters are in the dense layer at the end.

Training

The training loop is the one of week 6, with Adam, batches of 32 and five epochs.

opt = torch.optim.Adam(cnn.parameters())
loader = torch.utils.data.DataLoader(
    torch.utils.data.TensorDataset(X_train, y_train), batch_size=32, shuffle=True)

for epoch in range(5):
    for xb, yb in loader:
        opt.zero_grad()
        torch.nn.functional.cross_entropy(cnn(xb), yb).backward()
        opt.step()

with torch.no_grad():
    pred = torch.cat([cnn(xb).argmax(dim=1) for xb in X_test.split(1000)])
print(f"test error {float((pred != y_test).float().mean()):.4f}")

test error 0.0100

The test error is about 1%, without any tuning of the hyper-parameters. The network with one hidden layer of 100 neurons in week 6 had between 2% and 3%.

What the first layer learned

w = cnn[0].weight.detach().numpy()[:, 0]
fig, axes = plt.subplots(2, 8, figsize=(8.8, 2.4))
for ax, f in zip(axes.flat, w):
    ax.imshow(f, cmap="gray")
    ax.set(xticks=[], yticks=[])
plt.show()

Figure 42.1: The 16 filters of the first convolutional layer after training, each 8 by 8. Light is a positive weight, dark a negative one.

Each filter responds to a stroke of some orientation, wherever it is in the image.

with torch.no_grad():
    maps = cnn[1](cnn[0](X_test[:1]))[0].numpy()
fig, axes = plt.subplots(2, 9, figsize=(8.8, 2.2))
axes[0, 0].imshow(X_test[0, 0], cmap="gray_r")
axes[1, 0].axis("off")
for ax, m in zip(list(axes[0, 1:]) + list(axes[1, 1:]), maps):
    ax.imshow(m, cmap="gray_r")
for ax in axes.flat:
    ax.set(xticks=[], yticks=[])
plt.show()

Figure 42.2: One test image and the 16 channels of the first layer’s output for it. Each channel is one filter applied at every position, followed by the relu.