Skip to main content

Two ways to classify

Examples

Both models on this page fit a Bernoulli distribution by maximum likelihood. They differ in one line: what computes \(\eta\).

More than one predictor

With \(p\) predictors the fitted probability is a surface over the input space, and the set of points where it crosses the threshold is the decision boundary. For a linear \(\eta\) that boundary is \(\beta_0 + \beta_1x_1 + \cdots + \beta_px_p = 0\), a hyperplane, whatever the threshold happens to be.

With two predictors the probability is a surface and the decision boundary is a line. Move the threshold and watch the boundary shift without the probability changing at all.

Moving the threshold slides the boundary. It never bends it. That is a property of the function family, not of the data, and it is the thing a network changes.

The same data, twice

We generate a problem no straight line can solve: the two classes sit in alternating quadrants, softened by noise.

import numpy as np
import pandas as pd
import torch
import matplotlib.pyplot as plt

torch.manual_seed(7)
rng = np.random.default_rng(7)

def make(n, rng):
    X = rng.uniform(-3, 3, (n, 2))
    eta = 1.4 * X[:, 0] * X[:, 1]
    y = (rng.random(n) < 1 / (1 + np.exp(-eta))).astype(np.float32)
    return X.astype(np.float32), y

Xtr, ytr = make(600, rng)
Xte, yte = make(4000, rng)
print("class balance:", round(float(ytr.mean()), 3))
1
The product of the two inputs. A linear function of \(x_1\) and \(x_2\) cannot reproduce it, which is the point. 
class balance: 0.488

With a linear \(\eta\)

from sklearn.linear_model import LogisticRegression

lin = LogisticRegression(penalty=None).fit(Xtr, ytr)
print("training accuracy:", round(float(lin.score(Xtr, ytr)), 3))
print("test accuracy:    ", round(float(lin.score(Xte, yte)), 3))
training accuracy: 0.512
test accuracy:     0.502

Fifty percent, give or take. The model is not broken; the family simply has no member that separates these classes.

With a learned \(\eta\)

The training loop is the one from week 6. Two lines are different, and both come out of the table on the overview.

net = torch.nn.Sequential(
    torch.nn.Linear(2, 32), torch.nn.ReLU(),
    torch.nn.Linear(32, 32), torch.nn.ReLU(),
    torch.nn.Linear(32, 1),
)

loss_fn = torch.nn.BCEWithLogitsLoss()
opt = torch.optim.Adam(net.parameters(), lr=1e-2)

Xt = torch.tensor(Xtr)
yt = torch.tensor(ytr).unsqueeze(1)
loader = torch.utils.data.DataLoader(
    torch.utils.data.TensorDataset(Xt, yt), batch_size=64, shuffle=True)

for epoch in range(60):
    for xb, yb in loader:
        opt.zero_grad()
        loss_fn(net(xb), yb).backward()
        opt.step()

with torch.no_grad():
    acc = lambda X, y: float(((net(torch.tensor(X)).squeeze(1) > 0).numpy() == y).mean())
print("training accuracy:", round(acc(Xtr, ytr), 3))
print("test accuracy:    ", round(acc(Xte, yte), 3))
1
One output, and no activation on it. The raw number is \(\eta\).
2
The Bernoulli row of the table. BCEWithLogitsLoss applies \(s(\cdot)\) and takes the negative log-likelihood in one step, which is numerically safer than computing the probability first and then its logarithm. 
training accuracy: 0.83
test accuracy:     0.842

The identical network with torch.nn.Linear(32, 1) and torch.nn.MSELoss() would be the week 6 regression model. One line of the network changed, and it is the loss.

gx, gy = np.meshgrid(np.linspace(-3, 3, 300), np.linspace(-3, 3, 300))
G = np.column_stack([gx.ravel(), gy.ravel()]).astype(np.float32)

with torch.no_grad():
    net_region = (net(torch.tensor(G)).squeeze(1).numpy() > 0).reshape(gx.shape)
lin_region = (lin.decision_function(G) > 0).reshape(gx.shape)

fig, axes = plt.subplots(1, 2, figsize=(6.8, 3.2), sharey=True)
for ax, region, title in [(axes[0], lin_region, "logistic regression"),
                          (axes[1], net_region, "network")]:
    ax.contourf(gx, gy, region, levels=[-0.5, 0.5, 1.5], alpha=0.2)
    ax.plot(Xtr[ytr == 1, 0], Xtr[ytr == 1, 1], ".", ms=3)
    ax.plot(Xtr[ytr == 0, 0], Xtr[ytr == 0, 1], ".", ms=3)
    ax.set(xlabel="$x_1$", title=title)
axes[0].set(ylabel="$x_2$")
plt.show()
Figure 36.1: The decision boundary of each model at the threshold 0.5, over the training data. The linear model has one straight line to work with.
The same likelihood with a widening hidden layer. At zero hidden neurons this is logistic regression and the boundary is a straight line, whatever the data does. Two neurons are already enough to carve the plane into quadrants; the rest buys parameters and a little overfitting. Change the training set and watch how much the wide models move and how little the linear one does.

When the gain is small

The previous example was built so that a network would win, and it won by more than thirty points. That is not the usual size of the effect. Here is a problem where the same substitution buys almost nothing.

The spam data is text, so we need a representation. Bag of words counts how often each word occurs in each email.

from sklearn.feature_extraction.text import CountVectorizer

spam = pd.read_csv("https://go.epfl.ch/bio322-spam.csv", nrows=6000).dropna()
train, test = spam.iloc[:2000], spam.iloc[2000:4000]

vectorizer = CountVectorizer()
counts = vectorizer.fit_transform(train["text"].values).toarray()
freq = counts / np.maximum(counts.sum(axis=1, keepdims=True), 1)

test_counts = vectorizer.transform(test["text"].values).toarray()
test_freq = test_counts / np.maximum(test_counts.sum(axis=1, keepdims=True), 1)

print("emails:", counts.shape[0], " vocabulary:", counts.shape[1])
1
Bag of words. Each column counts how often one word occurs in one email.
2
Divided by the length of the email, so that a long email does not dominate.
3
transform, not fit_transform. The vocabulary has to be the one the model was trained on, or the columns would mean different things. 
emails: 2000  vocabulary: 28128

Each word is one predictor, so \(p\) is in the tens of thousands while \(n\) is 2000. With \(p > n\) the training data can be fitted perfectly, and it will be. This is why week 3 was about regularization.

from sklearn.neural_network import MLPClassifier

y_train = train["label"].values
y_test = test["label"].values

spam_lin = LogisticRegression(penalty=None, max_iter=400).fit(freq, y_train)
spam_net = MLPClassifier(hidden_layer_sizes=(64,), max_iter=60,
                         random_state=7).fit(freq, y_train)

pd.DataFrame({
    "model": ["logistic regression", "network"],
    "training accuracy": [round(spam_lin.score(freq, y_train), 3),
                          round(spam_net.score(freq, y_train), 3)],
    "test accuracy": [round(spam_lin.score(test_freq, y_test), 3),
                      round(spam_net.score(test_freq, y_test), 3)],
})
1
MLPClassifier is sklearn’s shortcut for a network with a softmax output and the cross-entropy loss. Fewer options than torch, which for a problem this size is an advantage.
model training accuracy test accuracy
0 logistic regression 0.996 0.968
1 network 1.000 0.987

The network is ahead, by about two points rather than thirty. Count what it cost: the linear model has one parameter per word plus an intercept, about 28000 of them, and the network has 64 of those per hidden unit, about 1.8 million. The signal in this representation really is close to additive in the word frequencies, so most of that flexibility has nothing to do.

Notice also that both models fit the training set essentially perfectly. Neither is penalized here, and with \(p \gg n\) that is exactly what week 3 warned about. A ridge penalty on the linear model is a far cheaper way to spend effort on this problem than a hidden layer, and it is a good exercise to check how much of the two-point gap survives it.

words = np.array(vectorizer.get_feature_names_out())
order = np.argsort(spam_lin.coef_[0])
pd.DataFrame({"most ham-like": words[order[:10]],
              "most spam-like": words[order[-10:]][::-1]})
most ham-like most spam-like
0 schedul your
1 enron http
2 hourahead more
3 louis click
4 start www
5 hour here
6 messag you
7 date remov
8 if onlin
9 origin net

And this is the other thing the linear model gives you, for free and with no extra machinery. There is no equivalent table for the network: its first layer mixes all 28000 words before anything interpretable happens. Whether that matters is a question about what the model is for.

The gap between training and test accuracy on both models is large. Which of the two is actually better, and how much of that gap is threshold artefact rather than model quality, is the next question but one.