biasvar() — Bias–variance trade-off

Why both too-simple and too-complex models predict poorly.

biasvar(model_complexity)

Key idea

• Bias = error from being too rigid (systematic miss); a straight line for a curve has high bias. (e.g. low complexity -> high bias)
• Variance = how much the fit jumps between datasets; complex models chase random noise. (e.g. high complexity -> high variance)
• Total error = noise + bias-squared + variance, and it is U-shaped as complexity grows.
• The best model sits at the bottom of the U, balancing the two.

Definition

$$\mathbb{E}[\text{Err}]={\sigma }^2+\underset{\text{too simple}}{\underbrace{{\text{Bias}}^2}}+\underset{\text{too complex}}{\underbrace{\text{Variance}}}$$

${\sigma }^2$

irreducible noise you can never beat

${\text{Bias}}^2$

error from the model being too rigid

$\text{Variance}$

error from the model reacting to noise

In plain words

Too simple and you always miss the same way; too complex and you wobble all over. The trick is the middle.

Where you'd use it

• Deciding how deep a tree, how many neighbours, or how strong a penalty to use.
• Explaining WHY a model that aced training does badly on new data (overfitting = variance).
• Justifying regularization, which trades a little bias for a lot less variance.

Frequently asked questions

What is the bias–variance tradeoff?

Simple models have high bias/low variance; complex ones the reverse. The lowest test error balances the two.