Overview

Model assessment, model tuning and feature engineering

This week is about one question. Given a model we have fitted, how well will it do on data we have not seen.

The answer is almost never the number our fitting procedure reports.

Pages

Why one split is not enough	How noisy a single held-out split is, and what \(K\)-fold does about it
The fold loop	Cross-validation written out, then the library call
Nested cross-validation	Estimating the test error after we have tuned
Feature engineering	Missing data, standardization, one-hot coding, splines
Exercises

Goals

The goal of this week is to

1. understand the validation-set approach for estimating the test error and tuning hyper-parameters,
2. understand the cross-validation approach for estimating the test error and tuning hyper-parameters,
3. learn how to use different resampling strategies, such as the validation-set approach or cross-validation,
4. learn how to do hyper-parameter tuning,
5. learn how to clean data,
6. understand and apply feature engineering, and
7. understand the implications of transforming the response variable.

Why this week matters

We estimate the test error for two reasons. The first is to report it. The second is to choose between models. Both are estimates from finite data, so both are noisy.

A number we used for choosing cannot also be reported. Most of this week is about that sentence.

The project is scored on data you never see. The most common way to lose points is to report a cross-validation score as if it were a test score.

Notation

\(K\) is the number of folds, and nothing else. We write \(C\) for the number of classes and \(k\) for the number of clusters or neighbours.