Overview

Trees and support vector machines

Neural networks are flexible, but they need tuning and they say little about which predictor matters.

This week we look at two families that are often faster to get right.

Pages

Decision trees	Recursive binary splitting
Ensembles	Random forests and gradient boosting
Support vector machines	The widest margin
Comparison	Four methods on one data set
Exercises

Goals

The goal of this week is to

1. understand tree-based methods,
2. see an example of a support vector classifier, and
3. learn how to use random forests and gradient boosting.

Two different assumptions

A tree splits the input space into rectangles and predicts a constant in each one. It assumes the response is roughly constant on regions that are defined by thresholds on single predictors.

A support vector machine separates the classes with a hyperplane, chosen so that the gap to the nearest points is as wide as possible. It assumes the classes are separable, in the original space or after a transformation.

Both are worth knowing. Tree ensembles are usually the fastest way to a decent result on tabular data, and they come with variable importance. Support vector machines are strong when \(p\) is large and \(n\) is small.