Transfer learning

Examples

The 102 Category Flower Dataset has photographs of 102 kinds of flowers. We take ten photographs of each kind, train on five of them and test on the other five.

import io
import numpy as np
import pandas as pd
import torch
import torchvision
import matplotlib.pyplot as plt
from PIL import Image

flowers = pd.read_parquet("https://huggingface.co/datasets/nkirschi/oxford-flowers/"
                          "resolve/main/data/test-00000-of-00001-96eeec628415add6.parquet")
train = flowers.groupby("label").sample(5, random_state=0)
test = flowers.drop(train.index)
y_train, y_test = train["label"].to_numpy(), test["label"].to_numpy()
print(len(train), "training images,", len(test), "test images")

510 training images, 510 test images

def load(img):
    return Image.open(io.BytesIO(img["bytes"])).convert("RGB")

fig, axes = plt.subplots(1, 8, figsize=(8.8, 1.4))
for ax, (_, row) in zip(axes, train.groupby("label").head(1).head(8).iterrows()):
    ax.imshow(load(row["image"]))
    ax.set(xticks=[], yticks=[], title=f"kind {row['label']}")
plt.show()

Figure 43.1: One training image for each of the first eight kinds of flowers.

Five images per class is far too few to fit a convolutional network from scratch.

weights = torchvision.models.ResNet18_Weights.IMAGENET1K_V1
prep = weights.transforms()
X_train = torch.stack([prep(load(img)) for img in train["image"]])
X_test = torch.stack([prep(load(img)) for img in test["image"]])

torch.manual_seed(0)
small_train = torch.nn.functional.interpolate(X_train, size=64)
small_test = torch.nn.functional.interpolate(X_test, size=64)
cnn = torch.nn.Sequential(
    torch.nn.Conv2d(3, 16, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
    torch.nn.Conv2d(16, 32, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
    torch.nn.Conv2d(32, 64, 3, padding=1), torch.nn.ReLU(), torch.nn.AdaptiveAvgPool2d(1),
    torch.nn.Flatten(), torch.nn.Linear(64, 102))
opt = torch.optim.Adam(cnn.parameters(), lr=1e-3)
yt = torch.tensor(y_train)
for epoch in range(60):
    perm = torch.randperm(len(yt))
    for i in range(0, len(yt), 32):
        idx = perm[i:i + 32]
        opt.zero_grad()
        torch.nn.functional.cross_entropy(cnn(small_train[idx]), yt[idx]).backward()
        opt.step()
with torch.no_grad():
    acc = (cnn(small_test).argmax(dim=1).numpy() == y_test).mean()
print(f"test accuracy from scratch: {acc:.3f}")

test accuracy from scratch: 0.200

Guessing would give \(1/102 \approx 0.01\).

Features learned on another data set

ResNet-18 is a convolutional network with 18 layers, trained on the 1.2 million labelled images of ImageNet, in 1000 classes. Flowers are not among them. We download the trained network, cut off its last layer, and use the 512 numbers that the layer before it computes as the features of each image.

resnet = torchvision.models.resnet18(weights=weights)
resnet.fc = torch.nn.Identity()
resnet.eval()
with torch.no_grad():
    F_train = torch.cat([resnet(x) for x in X_train.split(64)]).numpy()
    F_test = torch.cat([resnet(x) for x in X_test.split(64)]).numpy()
print(F_train.shape)

1

The architecture and the parameters fitted on ImageNet. prep above resizes and normalizes each image the way this network expects.

2

The last layer, a dense layer from 512 features to the 1000 ImageNet classes, is replaced by the identity, so the network returns the 512 features.

3

Evaluation mode, which matters for its batch normalization layers.

4

In batches of 64 images, to keep the memory small.

(510, 512)

On these features we fit a multinomial logistic regression, as in week 7.

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(max_iter=2000).fit(F_train, y_train)
print(f"test accuracy with transferred features: {clf.score(F_test, y_test):.3f}")

test accuracy with transferred features: 0.769

With the same 510 training images the accuracy goes from about 0.2 to about 0.77. None of the 11 million parameters of ResNet-18 were fitted to the flowers. Only the \(102 \cdot 513\) parameters of the logistic regression were.