lasso() — Lasso regression

Regression that shrinks some coefficients all the way to zero.

lasso(lambda, X, y)

Key idea

• A coefficient is the weight on an input. Lasso = ordinary fitting + an absolute-value (L1) penalty on those weights. (e.g. minimise error + lambda * sum|coef|)
• Unlike ridge, it can set coefficients to EXACTLY zero — so it automatically drops useless inputs (feature selection). (e.g. keeps 2 of 9 terms)
• Bigger lambda (the penalty strength) zeroes out more inputs, giving a simpler model.
• Among inputs that move together it tends to keep just one; the elastic net blends L1 and L2.

Definition

$${\hat{\beta }}^{\text{lasso}}=\arg \underset{\beta }{\min }\sum _i(y_i-x_i^{\top }\beta {)}^2+\lambda \sum _j|{\beta }_j|$$

$\lambda$

the penalty strength

$\sum |{\beta }_j|$

the L1 size of the coefficients

${\beta }_j$

the coefficients (some become exactly 0)

In plain words

Ridge turns the volume down on every coefficient; the lasso mutes the useless ones completely.

Where you'd use it

• Picking the handful of genes that predict a disease out of thousands.
• Building a simple, explainable model by automatically dropping weak predictors.
• Any problem with more features than data points where you want a sparse answer.

Frequently asked questions

What’s special about lasso?

Its L1 penalty can shrink coefficients exactly to 0, doing automatic feature selection.