Adaptive Optimisers — RMSProp, Adam, AdamW

SGD with momentum uses a single learning rate for every parameter. Adaptive optimisers maintain per-parameter step sizes that automatically shrink for high-curvature directions and grow for low-curvature ones, without the user tuning a per-parameter schedule. AdamW is the dominant optimiser for training Transformers; SGD remains competitive on CNN classification.

RMSProp — divide by recent gradient magnitude

Lecture 6e of CSC321 (Hinton, 2012, unpublished but widely cited) proposed a per-parameter step size based on a running estimate of the squared gradient:

$$v_t=\beta v_{t-1}+(1-\beta )({\mathrm{\nabla }}_t{)}^2,{\theta }_{t+1}={\theta }_t-\eta \cdot \frac{{\mathrm{\nabla }}_t}{\sqrt{v_t}+ϵ}.$$

Parameters with large recent gradients get small effective steps; parameters with small recent gradients get larger ones. RMSProp was the first widely used adaptive method and is still the default for training RNNs because it tames their characteristic exploding gradients.

Adam — momentum + RMSProp

Adam: A Method for Stochastic Optimization (Kingma, Ba, ICLR 2015) is RMSProp plus first-moment momentum. Maintain an exponential moving average of both the gradient $m_t$ and its square $v_t$:

$$\begin{array}{rl}{m}_t& ={\beta }_1{m}_{t-1}+(1-{\beta }_1){\mathrm{\nabla }}_t,\\ v_t& ={\beta }_2{v}_{t-1}+(1-{\beta }_2)({\mathrm{\nabla }}_t{)}^2.\end{array}$$

The two EMAs are biased toward zero at the start, so apply bias correction ${\hat{m}}_t=m_t/(1-{\beta }_1^t)$, ${\hat{v}}_t=v_t/(1-{\beta }_2^t)$, then update:

$${\theta }_{t+1}={\theta }_t-\eta \cdot \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}.$$

Default hyperparameters ${\beta }_1=0.9$, ${\beta }_2=0.999$, $ϵ={10}^{-8}$ work across an unreasonably wide range of problems. Adam's empirical robustness — fewer steps to convergence with default settings — is what made it the default optimiser of the deep-learning era.

The Adam–weight-decay trap

In the original Adam, L2 regularisation is implemented as a gradient penalty: ${\mathrm{\nabla }}_t\leftarrow {\mathrm{\nabla }}_t+\lambda {\theta }_t$. After the gradient enters the EMAs and is divided by $\sqrt{{\hat{v}}_t}$, the effective regularisation gets coupled to the per-parameter learning rate — large parameters with small recent gradients get less regularised, the opposite of intent.

Decoupled Weight Decay Regularization (Loshchilov, Hutter, ICLR 2019) introduced AdamW, which applies weight decay separately from the adaptive update:

$${\theta }_{t+1}={\theta }_t-\eta (\frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}+\lambda {\theta }_t).$$

For Transformers and other modern architectures, the difference between Adam and AdamW is large — AdamW is the universal default for LLM and ViT training. The original Adam should generally be considered legacy.

When Adam beats SGD, and when it doesn't

Modern empirical lore:

• Transformers, ViTs, NLP, RL — AdamW dominates. Adaptive step sizes appear necessary for the heterogeneous parameter scales in attention layers.
• CNN image classification — SGD with momentum + cosine schedule often matches or beats AdamW (The Marginal Value of Adaptive Gradient Methods in Machine Learning, Wilson et al., 2017), and produces flatter minima with better OOD generalisation.

The split has resisted unification. The pragmatic answer: AdamW is a stronger default; SGD wins when you can afford a long, careful schedule on a homogeneous architecture.

Variants worth knowing

• Adafactor (Shazeer, Stern, 2018) — Adam without the full second-moment matrix; instead, store row and column statistics. Memory-efficient enough to train large Transformers without optimiser-state offloading.
• Lion (Chen et al., 2023) — sign-of-update momentum, no second moment. Faster than AdamW on some workloads; sensitive to learning-rate tuning.
• LAMB (You et al., 2019) — layerwise-normalised AdamW for very large batch sizes, used to push BERT/ResNet training to enormous batches.

What to read next

• SGD, Momentum, Nesterov — the non-adaptive baseline these methods improve on.
• Learning Rate Schedules — choosing $\eta (t)$ over training.
• PEFT — LoRA + AdamW is the standard fine-tuning stack.