pca() — Principal Component Analysis

Finds the directions along which the data varies the most.

pca(X)

Key idea

• PC1 is the direction of greatest spread (variance); PC2 is the next-biggest spread, at right angles to it. (e.g. the long axis of the data cloud)
• These directions are eigenvectors of the covariance matrix (how features vary together); their eigenvalues are the spread captured.
• Keep the top few components and you compress the data with little information lost. (e.g. PC1+PC2 explain 95% -> drop the rest)
• It is unsupervised — it uses no labels, only the shape of the data.

Definition

$$\underset{\Vert w\Vert =1}{\max }\mathrm{Var}(Xw)⟺\Sigma w=\lambda w$$

$w$

a principal-component direction (unit vector)

$\Sigma$

the covariance matrix of the data

$\lambda$

the variance along w (an eigenvalue)

$Xw$

the data projected onto w

In plain words

Tilt your head until the data looks as stretched-out as possible. That tilt is the first principal component.

Where you'd use it

• Squashing a hundred features down to two so you can plot and explore them.
• Compressing images or removing noise by keeping only the top components.
• Speeding up other models by feeding them a few components instead of many raw features.

Frequently asked questions

What does PCA do?

Finds new axes (components) capturing the most variance, for compression or visualisation.

Do I scale first?

Yes — otherwise large-variance features dominate the components.