Recurrent networks

Examples

We detect the language of a sentence by reading it one character at a time with a recurrent network, and classifying the last state of that network.

The data

We downloaded a few pages from Wikipedia in five languages, removed accents, lowercased the text and split it into sentences.

import unicodedata
import urllib.request
import numpy as np
import torch
import matplotlib.pyplot as plt

langs = ["es", "en", "da", "it", "fr"]


def normalize(s):
    s = unicodedata.normalize("NFKD", s.casefold())
    return "".join(c for c in s if not unicodedata.combining(c)).strip()


sentences, labels = [], []
for k, lang in enumerate(langs):
    text = urllib.request.urlopen(f"https://go.epfl.ch/bio322-{lang}.txt").read().decode()
    for s in text.split("."):
        if normalize(s):
            sentences.append(normalize(s))
            labels.append(k)
labels = np.array(labels)
for k in range(len(langs)):
    i = np.flatnonzero((labels == k) & (np.array([len(s) for s in sentences]) > 60))[10]
    print(langs[k], "|", sentences[i][:70].replace("\n", " "))

es | [nota 3]​  una buena parte de sus ediciones idiomaticas esta disponibl
en | it has become an element of popular culture, with references in books,
da | [21] i 2017 annoncerede facebook at siden ville hjælpe læsere med at d
it | verso maggio giunsero i primi utenti registrati e, nel corso dell'anno
fr | larry sanger est engage dans cette societe au titre de redacteur en ch

print(dict(zip(langs, np.bincount(labels).tolist())))

{'es': 2887, 'en': 1525, 'da': 1217, 'it': 1447, 'fr': 1533}

Encoding

Each character becomes a one-hot vector over a vocabulary of the letters a to z, the digits, the space, the newline, and _ for every other character. A sentence of \(K\) characters is a matrix of size \(K \times 39\).

vocabulary = list("abcdefghijklmnopqrstuvwxyz0123456789 \n") + ["_"]
index = {c: i for i, c in enumerate(vocabulary)}


def onehot(s):
    x = torch.zeros(len(s), len(vocabulary))
    x[torch.arange(len(s)), [index.get(c, index["_"]) for c in s]] = 1
    return x


print(onehot("wikipedia").shape)

torch.Size([9, 39])

The model

The scanner is a recurrent layer of 64 tanh neurons. At every character it computes its new state from the character and its previous state, \(h_t = \tanh(W_x x_t + W_h h_{t-1} + b)\), with the same weights at every step. A dense classifier maps the last state to the five languages.

class Detector(torch.nn.Module):
    def __init__(self, hidden=64):
        super().__init__()
        self.scanner = torch.nn.RNN(len(vocabulary), hidden, batch_first=True)
        self.classifier = torch.nn.Linear(hidden, len(langs))

    def forward(self, x):
        _, h = self.scanner(x)
        return self.classifier(h[-1])


def batch(idx):
    seqs = [onehot(sentences[i]) for i in idx]
    return torch.nn.utils.rnn.pack_sequence(seqs, enforce_sorted=False)

1

torch.nn.RNN uses tanh neurons by default.

2

The first output is the state after every character, the second only the last state of each sentence.

3

Sentences have different lengths. pack_sequence puts them into one batch, and the scanner then stops at the end of each sentence.

torch.manual_seed(1)
rng = np.random.default_rng(1)
perm = rng.permutation(len(sentences))
test_idx, train_idx = perm[:len(perm) // 5], perm[len(perm) // 5:]
X_test, y_test = batch(test_idx), torch.tensor(labels[test_idx])

model = Detector()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
curve = []
for epoch in range(10):
    model.train()
    order = rng.permutation(train_idx)
    for i in range(0, len(order), 32):
        idx = order[i:i + 32]
        opt.zero_grad()
        loss = torch.nn.functional.cross_entropy(model(batch(idx)), torch.tensor(labels[idx]))
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    model.eval()
    with torch.no_grad():
        out = model(X_test)
    curve.append((float(torch.nn.functional.cross_entropy(out, y_test)),
                  float((out.argmax(dim=1) == y_test).float().mean())))

fig, axes = plt.subplots(1, 2, figsize=(8, 3))
axes[0].plot(range(1, 11), [c[0] for c in curve], "o-")
axes[0].set(xlabel="epoch", ylabel="test loss")
axes[1].plot(range(1, 11), [c[1] for c in curve], "o-")
axes[1].set(xlabel="epoch", ylabel="test accuracy")
plt.tight_layout()
plt.show()

Figure 44.1: Test loss and test accuracy after every epoch, on a fifth of the sentences held out.

The accuracy reaches about 0.7, far above the 0.2 of guessing, and training has not finished. A perfect score is out of reach: some sentences are a single word that is the same in several languages, like “wikipedia”.

What the scanner does

fig, axes = plt.subplots(1, 2, figsize=(8.8, 3.2), width_ratios=[1, 4])
for ax, s in zip(axes, ["wikipedia", sentences[1500][:40]]):
    with torch.no_grad():
        states, _ = model.scanner(onehot(s)[None])
    ax.imshow(states[0].T, aspect="auto", cmap="RdBu", vmin=-1, vmax=1)
    ax.set(xticks=range(len(s)), xticklabels=list(s), yticks=[], ylabel="neuron")
plt.tight_layout()
plt.show()

Figure 44.2: The 64 neurons of the scanner as the characters of two sentences are read in, one column per character. The last column is what the classifier sees.

The state after each character depends on that character and on the state before it, so the last state depends on the whole sentence.

The recipe is the one we used for every model so far:

1. Encode the input in a useful way. Here, characters as one-hot vectors.
2. Parametrize the conditional distribution of the response. Here, a scanner and a classifier with a softmax output.
3. Take the negative log-likelihood as the loss.
4. Minimize it with stochastic gradient descent.