Recurrent networks

Examples

A convolutional network needs a fixed input size. Sequences do not have one. A sentence can be five words or fifty.

A recurrent network reads a sequence one element at a time and keeps a state.

\[ h_t = g\bigl(W_h h_{t-1} + W_x x_t + b\bigr) \]

The same weights are used at every step. That is the recurrent part, and it is why the network can handle a sequence of any length.

The inductive bias

The assumption is that the same rule applies at every position. This is the same kind of assumption a convolutional network makes about space, applied to time instead.

The state \(h_t\) has to carry everything the network needs to remember. With a plain recurrent network the gradient either shrinks or grows exponentially over many steps, so it cannot learn long dependencies. An LSTM adds gates that decide what to keep and what to forget, which fixes this in practice.

A language detector

We classify a short string as one of a few languages, character by character.

import numpy as np
import torch
import torch.nn as nn

torch.manual_seed(9)
rng = np.random.default_rng(9)

# character frequencies that differ between the three languages
alphabet = "abcdefghijklmnopqrstuvwxyz "
idx = {c: i for i, c in enumerate(alphabet)}

profiles = {
    "en": {"t": 4, "h": 3, "e": 5, " ": 4, "a": 3, "o": 2},
    "de": {"c": 3, "h": 4, "e": 5, "n": 4, " ": 4, "s": 3},
    "fr": {"e": 6, "s": 3, "u": 3, " ": 4, "q": 2, "a": 3},
}


def make_string(lang, length):
    weights = np.ones(len(alphabet))
    for c, w in profiles[lang].items():
        weights[idx[c]] = w
    weights /= weights.sum()
    return "".join(rng.choice(list(alphabet), size=length, p=weights))


langs = list(profiles)
n = 1500
lengths = rng.integers(20, 60, n)
labels = rng.integers(0, 3, n)
strings = [make_string(langs[l], int(k)) for l, k in zip(labels, lengths)]
print(labels[0], strings[0][:40])

2 sueuenhbkqlksppgnuknyznfrb kpuaoehfq

Encoding

Every character becomes an integer, and an embedding layer turns it into a vector. We pad the sequences to the same length so that they fit in one tensor.

maxlen = int(lengths.max())

def encode(s):
    v = [idx[c] for c in s]
    return v + [len(alphabet)] * (maxlen - len(v))

X = torch.tensor([encode(s) for s in strings], dtype=torch.long)
y = torch.tensor(labels, dtype=torch.long)
print("input shape:", tuple(X.shape))

1

Index len(alphabet) is the padding symbol, so the embedding needs one extra row.

input shape: (1500, 59)

The model

class LangNet(nn.Module):
    def __init__(self, n_symbols, hidden=32, n_classes=3):
        super().__init__()
        self.embed = nn.Embedding(n_symbols, hidden)
        self.lstm = nn.LSTM(hidden, hidden, batch_first=True)
        self.out = nn.Linear(hidden, n_classes)

    def forward(self, x):
        h, _ = self.lstm(self.embed(x))
        return self.out(h[:, -1, :])


model = LangNet(len(alphabet) + 1)
print("parameters:", sum(p.numel() for p in model.parameters()))

1

We take the last state of the sequence and classify from it.

parameters: 9443

Xtr, Xte = X[:1200], X[1200:]
ytr, yte = y[:1200], y[1200:]

loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)
loader = torch.utils.data.DataLoader(
    torch.utils.data.TensorDataset(Xtr, ytr), batch_size=64, shuffle=True)

for epoch in range(15):
    model.train()
    for xb, yb in loader:
        opt.zero_grad()
        loss_fn(model(xb), yb).backward()
        opt.step()

model.eval()
with torch.no_grad():
    pred = model(Xte).argmax(dim=1)
print("test accuracy:", round(float((pred == yte).float().mean()), 4))

test accuracy: 0.5933

The network never saw a dictionary. It learned which character combinations are typical for which language.

What replaced recurrent networks

For most tasks with sequences, transformers have replaced recurrent networks. Instead of a state that is passed along, a transformer lets every position look at every other position directly. This removes the long dependency problem and it parallelizes much better on a GPU.

The idea of building the structure of the problem into the architecture is the same. Only the structure is different.