A feed-forward network (FFN) is the component of a transformer block that processes each token’s representation on its own, after attention has mixed information between tokens. It typically expands the representation to a larger dimension, applies an activation function, then projects it back. FFN layers hold the majority of a transformer’s parameters.
Key takeaways
- Attention mixes information between tokens; the FFN transforms each token.
- The FFN usually expands to about 4 times the model dimension, then contracts.
- It accounts for roughly two thirds of parameters in many transformer models.
- Modern LLMs often use SwiGLU-style gated FFNs instead of a simple activation.
- Mixture-of-experts designs replace the FFN with many expert FFNs, activating only a few per token.
An FFN explained in simple terms
Think of a transformer block as a meeting followed by private work. In the meeting (attention), everyone shares information. Afterwards, each person goes back to their desk and processes what they heard on their own (the FFN). The private step is where much of the actual thinking happens, and it is identical in structure for everyone, though the input differs.
How an FFN works
- Take the output vector for one token, of size d_model.
- Multiply by the first weight matrix to expand to a larger size, d_ff.
- Apply a non-linear activation, such as ReLU, GELU or SwiGLU.
- Multiply by a second matrix to project back to d_model.
- Add the result to the block’s input through a residual connection.
The same weights are applied to every position, so the FFN is like a small network run separately on each token.
A worked example: counting parameters
Take a model with d_model = 4,096 and d_ff = 16,384, a common 4× ratio.
- First matrix: 4,096 × 16,384 = 67.1 million parameters.
- Second matrix: 16,384 × 4,096 = 67.1 million.
- FFN total per layer: about 134 million.
Attention in the same layer has four matrices of 4,096 × 4,096, about 67 million in total. So the FFN holds roughly twice the attention parameters, which is why FFNs dominate model size and why mixture-of-experts designs target them.
Gated FFNs and SwiGLU
Many recent LLMs use a gated variant, commonly SwiGLU, proposed by Noam Shazeer in 2020. It uses three matrices instead of two: one produces values, one produces a gate that scales them, and one projects back. Gated FFNs tend to perform better at the same parameter count, so d_ff is often reduced to keep the totals comparable.
What FFNs do inside the model
Interpretability research suggests FFN layers act somewhat like a key-value memory: certain input patterns trigger specific stored associations, contributing much of a model’s factual recall. Editing model facts often involves modifying FFN weights.
Mixture of experts
Instead of one FFN per layer, a mixture-of-experts (MoE) layer contains many expert FFNs and a router that sends each token to a small number of them. This increases total parameters while keeping compute per token low, an approach used by several large models.
Limitations
- Dominates parameter count, memory and inference cost.
- Processes tokens independently, so all cross-token work must happen in attention.
- Larger d_ff improves capacity but raises cost quickly.