Vanilla RNNs

A recurrent neural network processes a sequence one element at a time, maintaining a hidden state that carries information forward. The vanilla RNN is the simplest version — one tied weight matrix applied at every step. Conceptually clean, painful to train at long horizons, and the gateway concept for everything from LSTMs and seq2seq to the original attention mechanism.

The recurrence

Given an input sequence ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_T$, a vanilla RNN computes hidden states

$${\mathbf{h}}_t=\varphi (W_{xh}{\mathbf{x}}_t+W_{hh}{\mathbf{h}}_{t-1}+{\mathbf{b}}_h),$$

with output ${\mathbf{y}}_t=W_{hy}{\mathbf{h}}_t+{\mathbf{b}}_y$ and $\varphi$ usually tanh. The same weights $W_{xh},W_{hh},W_{hy}$ are used at every step — parameter sharing across time, the structural analogue of weight sharing across space in a CNN.

Backpropagation through time (BPTT)

The gradient with respect to a weight that participates at every step is the sum of contributions from every step:

$$\frac{\partial \mathcal{L}}{\partial W_{hh}}=\sum _{t=1}^T\sum _{k=1}^t\frac{\partial {\mathcal{L}}_t}{\partial {\mathbf{h}}_t}\left(\prod _{j=k+1}^t\frac{\partial {\mathbf{h}}_j}{\partial {\mathbf{h}}_{j-1}}\right)\frac{\partial {\mathbf{h}}_k}{\partial W_{hh}}.$$

The product of Jacobians inside the parentheses is the entire problem. For long sequences, $T$ is large and the product collapses (vanishing gradient) or blows up (exploding gradient).

Vanishing and exploding gradients

The Jacobian $\partial {\mathbf{h}}_j/\partial {\mathbf{h}}_{j-1}=W_{hh}^{\mathrm{\top }}\mathrm{diag}({\varphi }^{\prime }(\cdot ))$. In an iterated product of $t$ such matrices:

• If $\parallel W_{hh}{\parallel }_2\cdot \parallel {\varphi }^{\prime }{\parallel }_{\mathrm{\infty }}<1$, gradients vanish exponentially in $t$. Long-range dependencies become invisible.
• If the product is $>1$, gradients explode. Training diverges.

For tanh, $|{\varphi }^{\prime }|\le 1$ everywhere, so vanishing dominates — vanilla RNNs cannot reliably learn dependencies more than ~10 steps apart. This was the central practical limitation that motivated LSTMs.

Gradient clipping is the standard runtime patch for explosions: clip $\mathrm{\nabla }$ to a maximum norm before each step. It does nothing for vanishing.

Truncated BPTT

Backpropagating through 1000+ steps is prohibitive in memory (must store every hidden state) and gradient quality. The standard hack is truncated BPTT: back-propagate through only the last $K$ steps (typically $K=32$–$200$), even if the forward pass uses much longer histories. This trades a biased gradient for a tractable one and is the workhorse for LM training on RNNs.

What vanilla RNNs are useful for

Despite the training problems, vanilla RNNs:

• Are expressive — Turing-complete given infinite precision (Siegelmann, 1991).
• Are useful for short sequences (text < 20 tokens, audio frames within a phoneme).
• Are conceptually the right starting point for understanding all recurrent architectures.
• Serve as the baseline against which LSTMs/GRUs/Transformers are measured.

In modern practice, almost no production system uses a vanilla RNN — but the recurrence is the foundation for the gated variants and Mamba/SSM that bring recurrence back at scale.

What to read next

• LSTM & GRU — gated recurrences that solve the vanishing-gradient problem.
• Sequence-to-Sequence — encoder–decoder architectures built on RNNs.
• Backpropagation — the underlying gradient algorithm BPTT specialises.