Linear & Quadratic Discriminant Analysis

LDA and QDA are generative classifiers: model each class's feature distribution as a multivariate Gaussian, then classify by Bayes' rule. They sit between logistic regression (also linear, but discriminative) and Naive Bayes (also generative, but with a stronger independence assumption). LDA in particular is a workhorse baseline that doubles as a dimensionality reduction technique.

The model

For $K$ classes, assume

$$P(\mathbf{x}\mid y=k)=\mathcal{N}({\mathit{\mu }}_k,{\mathrm{\Sigma }}_k),P(y=k)={\pi }_k.$$

By Bayes' rule, the log-posterior is

$$\log P(y=k\mid \mathbf{x})=-\frac{1}{2}(\mathbf{x}-{\mathit{\mu }}_k{)}^{\mathrm{\top }}{\mathrm{\Sigma }}_k^{-1}(\mathbf{x}-{\mathit{\mu }}_k)-\frac{1}{2}\log |{\mathrm{\Sigma }}_k|+\log {\pi }_k+\text{const}.$$

The classifier picks the class with the highest log-posterior. The shape of the decision boundary depends on what we assume about ${\mathrm{\Sigma }}_k$.

QDA — class-specific covariance

If each class has its own covariance ${\mathrm{\Sigma }}_k$, the decision boundaries are quadratic surfaces — ellipsoids, hyperboloids, paraboloids. QDA fits one mean ${\mathit{\mu }}_k$ and one covariance ${\mathrm{\Sigma }}_k$ per class:

$${\hat{\mathit{\mu }}}_k=\frac{1}{N_k}\sum _{i:y_i=k}{\mathbf{x}}_i,{\hat{\mathrm{\Sigma }}}_k=\frac{1}{N_k}\sum _{i:y_i=k}({\mathbf{x}}_i-{\hat{\mathit{\mu }}}_k)({\mathbf{x}}_i-{\hat{\mathit{\mu }}}_k{)}^{\mathrm{\top }}.$$

QDA needs $K\cdot d(d+1)/2$ parameters for the covariances — quadratically many in feature dimension — which makes it data-hungry.

LDA — shared covariance

If all classes share a single covariance ${\mathrm{\Sigma }}_k=\mathrm{\Sigma }$, the quadratic terms cancel and the decision boundary becomes linear:

$${\delta }_k(\mathbf{x})={\mathbf{x}}^{\mathrm{\top }}{\mathrm{\Sigma }}^{-1}{\mathit{\mu }}_k-\frac{1}{2}{\mathit{\mu }}_k^{\mathrm{\top }}{\mathrm{\Sigma }}^{-1}{\mathit{\mu }}_k+\log {\pi }_k.$$

LDA estimates a single pooled covariance from all classes:

$$\hat{\mathrm{\Sigma }}=\frac{1}{N-K}\sum _k\sum _{i:y_i=k}({\mathbf{x}}_i-{\hat{\mathit{\mu }}}_k)({\mathbf{x}}_i-{\hat{\mathit{\mu }}}_k{)}^{\mathrm{\top }}.$$

This is exactly Gaussian Naive Bayes with non-diagonal covariance — the difference from Naive Bayes is that LDA captures correlations between features.

LDA as dimensionality reduction

Beyond classification, LDA gives a supervised projection. The decision rules depend only on ${\mathrm{\Sigma }}^{-1}{\mathit{\mu }}_k$ — at most $K-1$ independent directions. Fisher's LDA maximises the ratio of between-class to within-class variance:

$$\underset{\mathbf{w}}{max}\frac{{\mathbf{w}}^{\mathrm{\top }}{S}_B\mathbf{w}}{{\mathbf{w}}^{\mathrm{\top }}{S}_W\mathbf{w}},$$

with $S_B$ the between-class scatter and $S_W$ the within-class scatter. The solution is a generalised eigenvalue problem; the top $K-1$ eigenvectors give a $(K-1)$-dimensional projection that maximally separates classes.

This is the supervised counterpart to PCA — PCA finds high-variance directions; LDA finds high class-separation directions. For visualisation of a labelled dataset in 2D, LDA-projected scatter plots are often more informative than PCA.

Regularisation: shrinkage and RDA

When $d$ is large or $N$ small, $\hat{\mathrm{\Sigma }}$ may be singular or near-singular. Two fixes:

• Shrinkage — replace $\hat{\mathrm{\Sigma }}$ with $(1-\alpha )\hat{\mathrm{\Sigma }}+\alpha \cdot \mathrm{diag}(\hat{\mathrm{\Sigma }})$ for some $\alpha \in [0,1]$, blending toward the diagonal-only (Naive Bayes) covariance.
• Regularised Discriminant Analysis (Friedman, 1989) — interpolate between LDA and QDA: ${\hat{\mathrm{\Sigma }}}_k(\gamma )=\gamma {\hat{\mathrm{\Sigma }}}_k+(1-\gamma )\hat{\mathrm{\Sigma }}$, with $\gamma$ chosen by cross-validation.

When LDA / QDA win

• Modest dimension, Gaussian-ish features — LDA matches or beats logistic regression, often with much faster training.
• Multi-class problems with equal effort across classes — LDA naturally handles all $K$ at once.
• As a dimensionality-reduction step before another classifier — Fisher LDA gives strong class-separation directions almost for free.

The Gaussian assumption is rarely exactly true, but LDA is robust to mild deviations. For categorical or extremely skewed data, generalised additive models or tree-based methods are better.

What to read next

• Naive Bayes — same Gaussian likelihood, but with diagonal covariance.
• Logistic Regression — the discriminative analogue with the same linear boundary.
• PCA & SVD — the unsupervised analogue of Fisher LDA.