The Lasso

• \(\hat{\beta}^L \equiv \underset{\hat{\beta}}{argmin}(RSS + \lambda\sum_{k=1}^p{|\beta_k|})\)
• Shrinks some coefficients to 0 (creates sparse models)

The Lasso, Visually

Uses the 1-norm: \[\|\beta\|_1 = \sum_{j=1}^p{|\beta_j|}\]

How lasso eliminiates predictors.

It can be shown that these shrinkage methods are equivalent to a OLS with a constraint that depends on the type of shrinkage. For two parameters:

• \(|\beta_1|+|\beta_2| \leq s\) for lasso,

• \(\beta_1^2+\beta_2^2 \leq s\) for ridge,

The value of s depends on \(\lambda\). (Larger s corresponds to smaller \(\lambda\)).

Graphically:

“the lasso constraint has corners at each of the axes, and so the ellipse will often intersect the constraint region at an axis”