Is an email spam or not? This is binary classification, as in week 2, with one new difficulty: the input is text, not a fixed number of columns. We put together what we have learned so far: a representation of the raw data, a linear model first, a network next, and cross-validation to decide between them.
The data
The emails come from the mailboxes of employees of Enron, an energy company, and from spam collections. They are already lower-cased and stemmed, which is why words like remov or schedul appear below.
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltspam = pd.read_csv("https://go.epfl.ch/bio322-spam.csv").dropna()train, test = spam.iloc[:2000], spam.iloc[2000:4000]X = train["text"]y = (train["label"] =="spam").astype(int).values # spam is the positive classprint("training emails:", len(train), " fraction of spam:", round(y.mean(), 3))train["text"].iloc[1][:300]
training emails: 2000 fraction of spam: 0.52
're swollen ear i final a mor t gag e l o an that work on your term we can lower your mon thli payment by up to vs tradit loa n s get a result in second asupduy paaup bjhnpysv nwfrbo lbntw brklcqr rywxaxmx lkagab kobeokt mmwhnnp fkfhjkni bdndpe lnpkq ruftfewgq mymcnjh kqeztx enocfcvd nmfmqywro fmzwwu'
The 2000 test emails are not looked at until the very end. Every comparison of models below is made by cross-validation on the 2000 training emails.
Bag of words
A logistic regression needs a vector \(x\) of \(p\) numbers for every email. The simplest way to get one is a bag of words: fix a lexicon of \(p\) words and count how often each of them occurs in the email. The order of the words is lost, hence the name.
from sklearn.feature_extraction.text import CountVectorizertoy = ["click here to win money", "the meeting is moved to monday, see you there","win win win"]toy_vectorizer = CountVectorizer().fit(toy)pd.DataFrame(toy_vectorizer.transform(toy).toarray(), columns=toy_vectorizer.get_feature_names_out())
1
fit builds the lexicon, here every word of the three texts. transform counts.
click
here
is
meeting
monday
money
moved
see
the
there
to
win
you
0
1
1
0
0
0
1
0
0
0
0
1
1
0
1
0
0
1
1
1
0
1
1
1
1
1
0
1
2
0
0
0
0
0
0
0
0
0
0
0
3
0
We divide the counts by the number of words in the email, so that a long email does not get large inputs just for being long. These are the word frequencies.
The lexicon is built from the emails the model is fitted on. In cross-validation this has to happen inside every fold, like the standardization in week 4, otherwise the held-out emails would decide which columns exist. A pipeline takes care of it.
from sklearn.pipeline import make_pipelinefrom sklearn.preprocessing import Normalizerdef frequencies():return [CountVectorizer(), Normalizer(norm="l1")]print("p =", len(CountVectorizer().fit(X).vocabulary_), " n =", len(X))
1
Normalizer(norm="l1") divides every row by its sum, which turns the counts into frequencies.
p = 28128 n = 2000
With every word of the training emails in the lexicon, \(p\) is much larger than \(n\).
Linear first
The linear model is the logistic regression of week 2, with \(p\) inputs instead of one, \[
P(Y = \text{spam}\,|\,x) = s(\theta_0 + \theta_1 x_1 + \cdots + \theta_p x_p).
\]
We compare all models with the same function: 5-fold cross-validation, repeated 4 times with different splits, scored by the accuracy and by the log-likelihood per email, the negative of the cross-entropy loss. The log-likelihood also tells us whether the probabilities can be trusted, not only whether they fall on the right side of 0.5.
from sklearn.model_selection import RepeatedKFold, cross_validatecv = RepeatedKFold(n_splits=5, n_repeats=4, random_state=1)def evaluate(model): r = cross_validate(model, X, y, cv=cv, scoring=["accuracy", "neg_log_loss"], return_train_score=True, n_jobs=-1)return {"training accuracy": r["train_accuracy"].mean(),"CV accuracy": r["test_accuracy"].mean(),"CV log-likelihood": r["test_neg_log_loss"].mean()}
from sklearn.linear_model import LogisticRegressionresults = {}results["frequencies, logistic, no regularization"] = evaluate( make_pipeline(*frequencies(), LogisticRegression(penalty=None, max_iter=5000)))pd.DataFrame(results).T.round(3)
training accuracy
CV accuracy
CV log-likelihood
frequencies, logistic, no regularization
0.998
0.968
-0.332
Nearly every training email is classified correctly, as expected with \(p > n\), and the cross-validated accuracy is a few points lower. Like the polynomials of week 3, this model can use an L2 penalty, \(\lambda\|\theta\|_2^2\), with \(\lambda\) chosen by cross-validation.
l2_frequencies = {C: evaluate(make_pipeline(*frequencies(), LogisticRegression(C=C, max_iter=5000)))for C in [100, 1000, 10000]}pd.DataFrame(l2_frequencies).T.rename_axis("C").round(3)
1
sklearn writes the penalty strength as C, which plays the role of \(1/\lambda\): a small C means strong regularization.
training accuracy
CV accuracy
CV log-likelihood
C
100
0.992
0.963
-0.177
1000
0.999
0.973
-0.106
10000
0.998
0.972
-0.127
Precisely, sklearn minimizes \(\frac{1}{2}\|\theta\|_2^2 + C\sum_{i=1}^n
\ell_i\), where \(\ell_i\) is the cross-entropy loss of email \(i\), summed rather than averaged over the \(n\) training emails, and the intercept \(\theta_0\) is not penalized. Dividing by \(Cn\) gives the mean loss plus \(\lambda\|\theta\|_2^2\) with \[
\lambda = \frac{1}{2Cn}.
\] With the \(n = 1600\) emails of a training fold, C=1000 is \(\lambda \approx 3 \cdot 10^{-7}\). Because the loss is summed, the same C regularizes less the more training data there are.
Regularization buys little accuracy, but a lot of log-likelihood. Without it the model is right about as often, but far too sure of itself when it is wrong.
Choosing C on the same folds that then report its score flatters the chosen model a little. Nested cross-validation would avoid that; with three candidate values the effect is small.
A neural network next
The same features, the same Bernoulli likelihood and the same loss, with a network of ten hidden relu neurons instead of a linear function.
MLPClassifier is sklearn’s shortcut for a network with a sigmoid output and the cross-entropy loss, fitted with Adam. It has a small L2 penalty by default (alpha=1e-4).
training accuracy
CV accuracy
CV log-likelihood
frequencies, logistic, no regularization
0.998
0.968
-0.332
frequencies, logistic, L2 (C=1000)
0.999
0.973
-0.106
frequencies, MLP
1.000
0.985
-0.070
The network is better, in accuracy and in log-likelihood. It has ten weights per word, about 280000 parameters against 28000. Is the gain due to the model, or would a linear model do as well with better features?
Feature engineering: TF-IDF
Word frequencies treat every word alike. A word like the occurs in almost every email and says nothing about spam, yet it gets the largest inputs. A word that occurs ten times in one email is not ten times as informative as one that occurs once. TF-IDF (term frequency, inverse document frequency) corrects both, \[
x_{ij} \propto (1 + \log c_{ij}) \cdot \log\frac{n}{n_j},
\] where \(c_{ij}\) is the count of word \(j\) in email \(i\) and \(n_j\) the number of training emails that contain word \(j\). Every email vector is then scaled to length 1.
from sklearn.feature_extraction.text import TfidfVectorizerdef tfidf():return [TfidfVectorizer(sublinear_tf=True)]l2_tfidf = {C: evaluate(make_pipeline(*tfidf(), LogisticRegression(C=C, max_iter=5000)))for C in [10, 100, 1000, 10000]}pd.DataFrame(l2_tfidf).T.rename_axis("C").round(3)
1
sublinear_tf=True replaces the count \(c_{ij}\) by \(1 + \log c_{ij}\). The inverse document frequency in sklearn is \(\log\frac{1+n}{1+n_j} + 1\), a version that never divides by zero.
With TF-IDF features both models improve, and the linear model most. The network still classifies a few emails more correctly, but its gap shrinks from more than a point to a few tenths, and the linear model now has the better log-likelihood. Neither is clearly better. In that situation we keep the linear model: it has ten times fewer parameters, its loss is convex so every fit gives the same answer, and it can be read.
The larger gain came from the representation, not from the function family.
What the linear model has learned
Each word has one coefficient. The largest push towards spam, the smallest towards ham.
The spam words are what one would expect. The ham words are a warning: enron is the company, louis a person there, and hourahead a term of its energy trading. The model has partly learned which mailbox an email comes from. On the test emails, which come from the same mailboxes, this does no harm. On somebody else’s mailbox it might.
Evaluating the final model
Only now do we look at the test emails, with the one model we chose.
sklearn puts the true class in the rows and the predicted class in the columns. Many textbooks, and the slides, do it the other way round. Always check which convention a table uses before reading numbers off it.
test accuracy: 0.986
predicted ham
predicted spam
true ham
940
26
true spam
2
1032
Most errors are ham emails classified as spam. For a spam filter these are the worse errors: a spam email in the inbox is a nuisance, a lost ham email can be expensive. Raising the threshold trades one for the other.
ROC and AUC
The panel below, on simulated data, recalls how the threshold moves the errors.
The confusion matrix at the top is for the current threshold. On the left, each tick is one data point at y = A or y = B, and the curve is the fitted probability P(Y = A | x); the vertical line is where that probability crosses the threshold. On the right is the ROC curve, with the marked point showing where the current threshold sits on it.
The threshold changes nothing about the fitted model: the parameters, the cross-entropy and every predicted probability are fixed before it is chosen. The ROC curve is what we get by sweeping the threshold from 1 to 0, and the area under it, the AUC, is the probability that a randomly chosen spam email gets a higher score than a randomly chosen ham email. Which point on the curve to use is a question about the application, not about the model.
