Logistic Regression

Logistic regression is the canonical model for binary classification — a linear function of features passed through a sigmoid to produce a probability. Despite the name, it is not regression. It is the discrete-output cousin of OLS, the simplest member of the generalised-linear-model family, and the building block from which softmax regression, the perceptron, and the final layer of every classification network all descend.

The model

For binary labels $y\in \{0,1\}$, logistic regression posits

$$P(y=1\mid \mathbf{x})=\sigma ({\mathit{\beta }}^{\mathrm{\top }}\mathbf{x})=\frac{1}{1+e^{-{\mathit{\beta }}^{\mathrm{\top }}\mathbf{x}}}.$$

The sigmoid maps the unbounded linear predictor $\eta ={\mathit{\beta }}^{\mathrm{\top }}\mathbf{x}$ (the log-odds) into a probability in $(0,1)$. Two readings:

• Probabilistic — model the conditional class probability directly.
• Linear log-odds — $\log (P(y=1)/P(y=0))={\mathit{\beta }}^{\mathrm{\top }}\mathbf{x}$. Each unit increase in $x_j$ multiplies the odds of $y=1$ by $e^{{\beta }_j}$.

Maximum likelihood

The log-likelihood of $N$ i.i.d. samples is

$$\log L(\mathit{\beta })=\sum _{i=1}^N{y}_i\log p_i+(1-y_i)\log (1-p_i),p_i=\sigma ({\mathit{\beta }}^{\mathrm{\top }}{\mathbf{x}}_i).$$

Equivalently, the binary cross-entropy loss with a flipped sign. The gradient has a remarkably clean form:

$${\mathrm{\nabla }}_{\mathit{\beta }}\log L=\sum _i(y_i-p_i){\mathbf{x}}_i=X^{\mathrm{\top }}(\mathbf{y}-\mathbf{p}).$$

Just like OLS, the gradient is the design matrix transpose times the residual — but the residual is now in probability space. There is no closed form for $\hat{\mathit{\beta }}$; instead, optimise iteratively.

Optimisation: Newton-Raphson / IRLS

The Hessian is

$$H=-X^{\mathrm{\top }}WX,W=\mathrm{diag}(p_i(1-p_i)).$$

$H$ is negative semi-definite, so the log-likelihood is concave — the unique stationary point is the global maximum. Newton's method updates

$${\mathit{\beta }}^{(t+1)}={\mathit{\beta }}^{(t)}+(X^{\mathrm{\top }}{W}^{(t)}X{)}^{-1}{X}^{\mathrm{\top }}(\mathbf{y}-{\mathbf{p}}^{(t)}).$$

This is Iteratively Reweighted Least Squares (IRLS) — at each step you solve a weighted-least-squares problem. Converges in 5–10 iterations on well-behaved data; a robust default for small-to-medium logistic regression.

For large data, gradient descent / SGD is the practical solver — same approach as deep networks. Convergence is slower but per-step cost is lower.

Regularised logistic regression

L1 / L2 penalties extend directly:

$$\hat{\mathit{\beta }}=\arg \underset{\mathit{\beta }}{min}-\log L(\mathit{\beta })+\lambda \mathrm{\Omega }(\mathit{\beta }).$$

The penalised problem is still strictly convex (assuming non-zero $\lambda$ for L2), with a unique optimum. Glmnet (Friedman et al., 2010) is the canonical R/Python package for fitting elastic-net-penalised logistic regression at scale via coordinate descent.

For complete linear separability, unregularised logistic regression has $\hat{\mathit{\beta }}\to \mathrm{\infty }$ — the likelihood increases unboundedly. Regularisation (or stopping early) is the standard fix.

Multi-class extension: softmax regression

For $K$-class classification, replace the sigmoid with softmax:

$$P(y=k\mid \mathbf{x})=\frac{\exp ({\mathit{\beta }}_k^{\mathrm{\top }}\mathbf{x})}{\sum _{j=1}^K\exp ({\mathit{\beta }}_j^{\mathrm{\top }}\mathbf{x})}.$$

Trained by maximum likelihood with cross-entropy loss. Same convex objective, same gradient form. Softmax regression is what the final layer of every classification deep network computes.

Connection to deep learning

A neural-network classifier is just a stack of logistic regressions whose features are themselves learned. The final layer $\hat{p}=\sigma (W_L{\mathbf{h}}_{L-1}+b_L)$ is binary logistic regression on learned features ${\mathbf{h}}_{L-1}$. Cross-entropy + softmax + linear layer is the universal classification head.

The connection runs deeper: the gradient of cross-entropy + softmax simplifies to (predicted - target) — the same gradient as OLS, applied in probability space. This is one of the reasons cross-entropy + softmax pairs so naturally with backpropagation.

What to read next

• Generalized Linear Models — the unified framework for regression-with-a-link-function.
• Ridge & Lasso Regression — the same penalties applied to OLS.
• Activation Functions — softmax in the deep-learning context.