Ridge & Lasso Regression

Ordinary Least Squares breaks down when features are correlated or when there are more features than samples — the design matrix becomes ill-conditioned or rank-deficient. Ridge and lasso regression add penalty terms that fix the conditioning problem and, in lasso's case, select features automatically. Together they are the canonical examples of how regularisation tames classical regression.

Ridge regression

The ridge objective adds an L2 penalty:

$${\hat{\mathit{\beta }}}_{\text{ridge}}=\arg \underset{\mathit{\beta }}{min}\frac{1}{2N}\parallel \mathbf{y}-X\mathit{\beta }{\parallel }_2^2+\lambda \parallel \mathit{\beta }{\parallel }_2^2.$$

Setting the gradient to zero gives the closed form

$${\hat{\mathit{\beta }}}_{\text{ridge}}=(X^{\mathrm{\top }}X+\lambda NI{)}^{-1}{X}^{\mathrm{\top }}\mathbf{y}.$$

The added $\lambda NI$ shifts every eigenvalue of $X^{\mathrm{\top }}X$ up by $\lambda N$, fixing ill-conditioning when columns are nearly collinear. Geometrically, ridge shrinks all coefficients smoothly toward zero — large coefficients shrink proportionally, none reach exactly zero.

Bayesian interpretation. Ridge is the MAP estimate under a Gaussian prior $\mathit{\beta }\sim \mathcal{N}(\mathbf{0},{\sigma }^2/(\lambda N)\cdot I)$. The penalty strength $\lambda$ encodes the prior's tightness: larger $\lambda$ = stronger belief in small coefficients.

Effective degrees of freedom. For the ridge estimator, $\mathrm{df}(\lambda )=\mathrm{tr}[X(X^{\mathrm{\top }}X+\lambda NI{)}^{-1}{X}^{\mathrm{\top }}]=\sum _i{\sigma }_i^2/({\sigma }_i^2+\lambda N)$, where ${\sigma }_i$ are the singular values of $X$. Ridge spends "effective parameters" smoothly, in contrast to OLS's hard $p$ degrees of freedom.

Lasso regression

The lasso objective uses an L1 penalty:

$${\hat{\mathit{\beta }}}_{\text{lasso}}=\arg \underset{\mathit{\beta }}{min}\frac{1}{2N}\parallel \mathbf{y}-X\mathit{\beta }{\parallel }_2^2+\lambda \parallel \mathit{\beta }{\parallel }_1.$$

There is no closed form — the L1 penalty is non-differentiable at zero — but the problem is convex. Standard solvers: coordinate descent (Friedman et al., 2007), proximal-gradient ISTA / FISTA, and the LARS algorithm.

The geometric distinction from ridge is critical. The L1 ball has corners on the coordinate axes; the squared-error contours touch the ball preferentially at these corners, producing exactly-zero coefficients. Lasso performs automatic feature selection as part of fitting.

For orthonormal $X$ ($X^{\mathrm{\top }}X=I$), the lasso solution is soft thresholding:

$${\hat{\beta }}_j=\mathrm{sign}({\hat{\beta }}_j^{\text{OLS}})\cdot max(|{\hat{\beta }}_j^{\text{OLS}}|-\lambda ,0).$$

Coefficients smaller in magnitude than $\lambda$ are zeroed; larger ones are shrunk by $\lambda$. The general non-orthonormal case is more complex but qualitatively similar.

Elastic net

Lasso has two known weaknesses: (1) when features are highly correlated, lasso picks one and zeros the rest somewhat arbitrarily; (2) when $p>N$, lasso selects at most $N$ features.

Elastic net (Zou & Hastie, JRSS-B 2005) combines both penalties:

$${\hat{\mathit{\beta }}}_{\text{enet}}=\arg \underset{\mathit{\beta }}{min}\frac{1}{2N}\parallel \mathbf{y}-X\mathit{\beta }{\parallel }_2^2+\lambda (\alpha \parallel \mathit{\beta }{\parallel }_1+(1-\alpha )\frac{1}{2}\parallel \mathit{\beta }{\parallel }_2^2),\alpha \in [0,1].$$

The L2 component groups correlated features (their coefficients move together), while L1 drives unimportant ones to zero. Elastic net is the practical default in high-dimensional regression when features are correlated — gene expression, fMRI, NLP feature sets.

When to use which

A quick guide:

• Many predictors, all believed relevant — ridge.
• Suspected sparse truth, uncorrelated features — lasso.
• Suspected sparse truth, correlated features — elastic net.
• $p\gg N$ — lasso or elastic net (ridge can't do feature selection but can still regularise).
• Only goal is generalisation, not interpretability — ridge tends to give slightly lower test error in non-sparse regimes; lasso wins when the truth really is sparse.

Cross-validate $\lambda$ on a log grid; for elastic net, also cross-validate $\alpha$ — typical $\alpha \in \{0.1,0.5,0.9\}$.

What to read next

• Ordinary Least Squares — the unregularised baseline.
• Regularization Theory — the broader framework these are special cases of.
• Logistic Regression — same penalties applied to classification.