Evaluating a classifier

Examples

A classifier outputs a probability. A confusion matrix needs a decision. The step between the two is a threshold, and it is a choice.

The confusion matrix

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, roc_curve, roc_auc_score

rng = np.random.default_rng(3)

n = 400
x = rng.normal(0, 2, n)
p = 1 / (1 + np.exp(-(0.9 * x - 0.4)))
y = (rng.random(n) < p).astype(int)

X = x.reshape(-1, 1)
m = LogisticRegression().fit(X, y)
prob = m.predict_proba(X)[:, 1]

cm = confusion_matrix(y, (prob >= 0.5).astype(int))
pd.DataFrame(cm,
             index=["true negative class", "true positive class"],
             columns=["predicted negative", "predicted positive"])

1

Note that sklearn puts the true class in the rows and the predicted class in the columns. Many textbooks do it the other way round. Always check which convention a table uses before reading numbers off it.

	predicted negative	predicted positive
true negative class	156	47
true positive class	51	146

From the four counts we get the usual quantities.

tn, fp, fn, tp = cm.ravel()

pd.DataFrame({
    "quantity": ["accuracy", "sensitivity (recall, TPR)", "specificity (1 - FPR)",
                 "precision", "error rate"],
    "formula": ["(TP+TN)/all", "TP/(TP+FN)", "TN/(TN+FP)", "TP/(TP+FP)", "(FP+FN)/all"],
    "value": [round((tp + tn) / cm.sum(), 3),
              round(tp / (tp + fn), 3),
              round(tn / (tn + fp), 3),
              round(tp / (tp + fp), 3),
              round((fp + fn) / cm.sum(), 3)],
})

	quantity	formula	value
0	accuracy	(TP+TN)/all	0.755
1	sensitivity (recall, TPR)	TP/(TP+FN)	0.741
2	specificity (1 - FPR)	TN/(TN+FP)	0.768
3	precision	TP/(TP+FP)	0.756
4	error rate	(FP+FN)/all	0.245

The threshold is a choice

Lowering the threshold predicts the positive class more often. That raises the true positive rate and the false positive rate at the same time.

The confusion matrix at the top is for the current threshold. On the left, each tick is one data point at y = A or y = B, and the curve is the fitted probability P(Y = A | x); the vertical line is where that probability crosses the threshold. On the right is the ROC curve, with the marked point showing where the current threshold sits on it.

Which error matters is a question about the application, not about the model. For a screening test a false negative may be far worse than a false positive. In that case we accept a lower threshold and more false alarms.

ROC and AUC

The ROC curve is what we get by sweeping the threshold from 1 to 0 and plotting the true positive rate against the false positive rate.

The area under the curve summarizes all thresholds in one number. It is the probability that a randomly chosen positive case gets a higher score than a randomly chosen negative case. A value of 0.5 means the classifier is useless and 1.0 means it separates the classes perfectly.

The AUC does not depend on the threshold, which makes it useful for comparing models. It also does not tell us which threshold to use, so it is not enough on its own.

Imbalanced classes

If 97 percent of cases are negative, a model that always predicts negative has 97 percent accuracy. Accuracy is not a useful number here.

y_rare = np.zeros(1000, dtype=int)
y_rare[:30] = 1
always_negative = np.zeros(1000, dtype=int)

tn, fp, fn, tp = confusion_matrix(y_rare, always_negative).ravel()
print("accuracy:   ", round((tp + tn) / 1000, 3))
print("sensitivity:", round(tp / (tp + fn), 3))

accuracy:    0.97
sensitivity: 0.0

Report sensitivity and precision, or the AUC, whenever the classes are imbalanced.