ridge() — Ridge regression

Linear regression with a penalty that shrinks the coefficients.

ridge(lambda, X, y)

Key idea

• Ridge = least squares (the usual best-fit line) plus a penalty on the size of the coefficients (the term strengths). (e.g. minimise error + lambda * sum(coef^2))
• A bigger lambda (penalty strength) shrinks coefficients more, giving a smoother, less wiggly fit. (e.g. lambda 0 -> OLS; lambda large -> flat)
• It trades a little bias (rigidity) for much less variance — usually better predictions on new data.
• Ridge shrinks but never zeroes coefficients; the lasso can zero them out and pick which features to keep.

Definition

$${\hat{\beta }}^{\text{ridge}}=\arg \underset{\beta }{\min }\sum _i(y_i-x_i^{\top }\beta {)}^2+\lambda \sum _j{\beta }_j^2$$

$\lambda$

the penalty strength (>= 0)

${\beta }_j$

the model coefficients

$\sum {\beta }_j^2$

the size of the coefficients (the squared L2 norm)

$y_i$

the observed value

$$\lambda =0\Rightarrow \text{ordinary least squares};\lambda \to \infty \Rightarrow \beta \to 0$$

In plain words

Least squares can over-react and draw a crazy wiggly line. Ridge tells it to keep the line calm, and a bigger lambda means calmer.

Where you'd use it

• Predicting house prices from dozens of correlated features without the model blowing up.
• Stabilising a model when you have more features than data points, where plain least squares fails.
• Taming a high-degree polynomial so it fits the trend instead of the noise.

Frequently asked questions

Ridge or lasso?

Ridge shrinks all coefficients (keeps every feature); lasso can zero some out (feature selection).