LSTM (long short-term memory) is a type of recurrent neural network that adds a memory cell and three gates, which learn what information to keep, what to forget and what to pass on. This design lets LSTMs learn patterns across long sequences, something plain RNNs struggle with because of vanishing gradients.
Key takeaways
- LSTMs were introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997.
- They add a cell state, a memory lane that carries information across many steps with little change.
- Three gates, forget, input and output, control what enters and leaves memory.
- LSTMs powered many speech and translation systems in the mid-2010s.
- Transformers have replaced them for most large language tasks, but LSTMs remain useful for time series and on-device models.
LSTM explained in simple terms
Think of a note-taker in a long meeting. They cannot write down everything, so for each new point they decide: should I cross out an old note that no longer matters (forget)? Should I write this new point down (input)? Which notes should I mention right now when asked (output)? An LSTM does exactly this at every step of a sequence, using learned rules instead of judgement.
Why LSTMs were needed
A plain RNN passes its memory through the same multiplication at every step. Over long sequences, the training signal shrinks towards zero, so the network cannot learn links between distant events. The LSTM’s cell state is updated mostly by adding and removing information rather than repeatedly multiplying it, which lets gradients flow across many more steps.
The three gates
| Gate | Question it answers | Effect |
|---|---|---|
| Forget gate | What old memory is no longer useful? | Removes information from the cell state |
| Input gate | What new information is worth storing? | Adds selected new information to the cell state |
| Output gate | What part of memory is relevant now? | Decides the hidden state output at this step |
Each gate is a small neural layer with a sigmoid activation that outputs values between 0 and 1. A value near 0 means “block”; near 1 means “let through”. The original 1997 LSTM had input and output gates; the forget gate was added in 2000 by Felix Gers, Schmidhuber and Fred Cummins, and is now standard.
How an LSTM step works
- Look at the new input and the previous hidden state.
- The forget gate scales down parts of the old cell state.
- The input gate chooses which new candidate values to add.
- The cell state is updated: old memory (after forgetting) plus new memory.
- The output gate decides what part of the cell state to expose as the new hidden state.
A worked example: remembering the subject
Consider: “The keys to the old cabinet in the back room are on the table.” To choose “are” rather than “is”, the model must remember that the subject “keys” is plural, across eight words. An LSTM can learn to store “plural subject” in its cell state when it reads “keys”, keep it through “to the old cabinet in the back room” (forget gate stays open), and use it at the verb (output gate opens). When the sentence ends, the forget gate can clear it.
LSTM vs RNN vs GRU vs transformer
| Model | Memory mechanism | Strength | Weakness |
|---|---|---|---|
| Plain RNN | Hidden state only | Simple, small | Forgets over long sequences |
| LSTM | Cell state plus 3 gates | Long-range memory | More parameters, slower |
| GRU | 2 gates, no separate cell | Similar results with fewer parameters | Slightly less expressive in some tasks |
| Transformer | Attention over all positions | Parallel training, very long context | Memory grows with context length |
Where LSTMs have been used
LSTMs were widely used for speech recognition, handwriting recognition, machine translation and text prediction in the mid-2010s. Google’s neural machine translation system, launched in 2016, used deep LSTM networks. Today they remain practical for time-series forecasting, sensor data, anomaly detection and small on-device models.
Limitations
- Sequential processing makes training slow and hard to parallelise.
- More parameters than simple RNNs or GRUs.
- Still weaker than transformers for very long contexts and large-scale language tasks.
- Can overfit small datasets without regularization such as dropout.