Overview
Dimensionality reduction
Clustering proposed that the data comes from a few groups. Dimensionality reduction proposes something else. The data has many columns, but it really lives on a much smaller space.
Pages
| Principal component analysis | Directions of largest variance, loadings and scores |
| Applications | Compression, denoising, regression |
| Beyond PCA | Limitations, t-SNE and UMAP |
| Exercises |
Goals
The goal of this week is to
- understand PCA from different perspectives,
- learn how to run PCA,
- understand the limitations of PCA, and learn how to use alternatives like t-SNE or UMAP, and
- learn how to use PCA for principal component regression.
Notation
| symbol | ||
|---|---|---|
| centred data | \(X\) | \(n \times p\), column means are zero |
| covariance | \(\hat\Sigma = \frac{1}{n-1}X^\top X\) | \(p \times p\) |
| loadings | \(\Phi\) | \(p \times p\), columns are the principal directions |
| scores | \(Z = X\Phi\) | \(n \times p\), coordinates in the new basis |
| singular values | \(S\) | from \(X = U S V^\top\) |
We write the loadings as \(\Phi\) rather than \(V\), because we will use them as a change of basis. They are the same thing, \(\Phi = V\) and \(Z = US\).
Note that \(\Sigma\) always means the covariance matrix and \(S\) the singular values. These two are easy to confuse.