A recurrent neural network (RNN) is a type of neural network designed to process sequences, such as sentences, speech or time series, one step at a time. At each step it combines the new input with a hidden state carried over from the previous step, giving the network a form of memory. RNNs were the leading approach for language and speech before transformers arrived, and their ideas still shape modern sequence models.
Key takeaways
- RNNs handle ordered data where earlier steps affect later ones.
- The hidden state carries information forward through the sequence.
- The same weights are reused at every time step.
- Plain RNNs struggle with long sequences because of vanishing gradients; LSTMs and GRUs were designed to fix this.
- Transformers replaced RNNs for most language tasks after 2017.
An RNN explained in simple terms
When you read a sentence, you do not start fresh at every word. By the time you reach “it” in “The dog chased the ball because it was fast”, you remember what came before and work out what “it” refers to. An RNN reads the same way: word by word, updating a running summary of what it has seen so far. That running summary is the hidden state.
Why sequences need special networks
A regular feedforward neural network takes a fixed-size input and has no sense of order or memory. But in many problems, order matters. “Dog bites man” and “man bites dog” contain the same words with different meanings. Tomorrow’s sales depend on the last few weeks. RNNs were built to capture this kind of dependence on earlier steps, and to handle sequences of any length.
How does an RNN work?
- Start with an initial hidden state, usually zeros.
- At step 1, combine input x₁ with the hidden state to produce a new hidden state h₁.
- Optionally, produce an output from h₁.
- At step 2, combine x₂ with h₁ to produce h₂, and so on.
- Repeat until the end of the sequence.
The core update is: hₜ = tanh(Wₓ·xₜ + Wₕ·hₜ₋₁ + b). The same weights Wₓ and Wₕ are used at every step. Drawn step by step, the network looks like a chain of identical copies, called “unrolling” the RNN.
A worked example: watching memory fade
Take a tiny RNN with a single number as its hidden state, input weight 0.5 and recurrent weight 0.8. Feed it the sequence 1, 0, 0.
- Step 1: h₁ = tanh(0.5 × 1 + 0.8 × 0) = tanh(0.5) ≈ 0.46
- Step 2: h₂ = tanh(0.5 × 0 + 0.8 × 0.46) = tanh(0.37) ≈ 0.35
- Step 3: h₃ = tanh(0.8 × 0.35) = tanh(0.28) ≈ 0.28
The network still “remembers” the first input at step 3, but the signal is shrinking each step. Over dozens or hundreds of steps, the influence of early inputs can fade almost to nothing. That is the heart of the RNN’s biggest weakness.
Types of RNN setups
| Pattern | Input → output | Example |
|---|---|---|
| One-to-many | One input, a sequence out | Generating a caption from an image |
| Many-to-one | A sequence in, one output | Sentiment of a review |
| Many-to-many (aligned) | Output at every step | Tagging each word’s part of speech |
| Many-to-many (encoder-decoder) | Sequence in, different-length sequence out | Translating a sentence |
| Bidirectional | Reads forwards and backwards | Understanding a word using context on both sides |
Training an RNN and the vanishing gradient problem
RNNs are trained with backpropagation through time (BPTT): the network is unrolled across the sequence, the error is calculated, and gradients flow backwards through every step. Because the same weights multiply the signal again and again, gradients tend to shrink towards zero (vanishing) or grow out of control (exploding). Vanishing gradients mean the network cannot learn links between events far apart, such as a subject at the start of a long paragraph and a verb at the end. Researchers including Sepp Hochreiter (1991) and Yoshua Bengio and colleagues (1994) analysed this problem in detail.
How it was solved: LSTM and GRU
- LSTM (long short-term memory), introduced by Hochreiter and Jürgen Schmidhuber in 1997, adds a separate memory cell and gates that decide what to keep, forget and output.
- GRU (gated recurrent unit), introduced in 2014, is a simpler gated design with fewer parameters that often performs similarly.
Exploding gradients are usually handled with gradient clipping, which caps gradient size during training.
RNNs vs transformers
| Aspect | RNN / LSTM | Transformer |
|---|---|---|
| Processing | One step at a time | All positions at once |
| Training speed | Slow; hard to parallelise | Fast on GPUs |
| Long-range links | Weakens over distance | Attention links any two positions directly |
| Memory use per step at inference | Constant | Grows with context length |
After the transformer architecture was introduced in 2017, it replaced RNNs for most language tasks, including large language models. However, recurrence has made a comeback in newer designs such as state space models, which aim to combine transformer-level quality with the RNN’s efficient, constant memory per step.
Where RNNs are still used
RNNs, LSTMs and GRUs remain useful for smaller time-series problems, streaming sensor data, on-device speech and keyboard prediction, and cases where memory and compute are tight.
Limitations
- Struggle with long-range dependencies (plain RNNs especially).
- Slow to train because steps must be processed in order.
- Can suffer from vanishing or exploding gradients.
- Generally outperformed by transformers on large language tasks.