crossval() — Cross-validation

Score a model on data it never trained on to catch overfitting.

crossval(model, data, K)

Key idea

• Split the data into K folds (equal parts); train on K-1 and test on the held-out one, rotating through all K. (e.g. K = 5 folds -> 5 train/test rounds)
• The average error on the held-out folds estimates how the model does on NEW, unseen data.
• Training error always falls as the model gets more complex; CV error is U-shaped, and its bottom is the best model. (e.g. best degree at the minimum of the U)
• It guards against overfitting (memorising noise), which training error alone can never reveal.

Definition

$$\text{CV}=\frac{1}{K}\sum _{k=1}^K\text{error}(\text{model trained without fold }k,\text{fold }k)$$

$K$

how many folds (parts) you split into

$\text{fold }k$

the held-out test part

$\text{CV}$

the average held-out error

In plain words

Do not grade the model on the questions it studied. Hide some data, train, then test on the hidden part, and rotate so every part gets a turn.

Where you'd use it

• Choosing how complex a model should be (e.g. polynomial degree or tree depth).
• Comparing two models fairly before picking one to ship.
• Tuning hyperparameters like the ridge penalty lambda or kNN’s k.

Frequently asked questions

Why cross-validate?

To estimate performance on unseen data and avoid overfitting to one lucky split.

Where should scaling happen?

Inside each fold — fitting it on all the data leaks test information.