Mixture of experts (MoE) is an architecture in which each token is routed to a small number of expert sub-networks instead of passing through the whole model. A learned router picks which experts handle each token. The model can therefore hold a very large number of total parameters while using only a fraction of them per token, giving the capacity of a huge model at the compute cost of a much smaller one.
Key takeaways
- MoE layers usually replace the feed-forward network in transformer blocks.
- A router selects the top-k experts per token, commonly one or two.
- Total parameters and active parameters are very different numbers.
- Training needs load-balancing so experts are used evenly.
- All experts must be held in memory, even though only a few run per token.
MoE explained in simple terms
A hospital does not send every patient to every specialist. A triage nurse routes each case to the right one or two doctors. The hospital collectively holds enormous expertise, but any single patient consumes only a small share of it. An MoE model works the same way, with the router as triage.
How an MoE layer works
- A token arrives at the layer.
- The router scores each expert for this token.
- The top-k experts are selected, typically k = 1 or 2.
- Those experts process the token.
- Their outputs are combined, weighted by the router’s scores.
- Remaining experts do nothing for this token.
A worked example
A model has 8 experts per layer and routes each token to 2.
- Total parameters: roughly the size of 8 feed-forward networks per layer.
- Active per token: the equivalent of 2, so about a quarter of the layer’s capacity is used.
Mixtral 8x7B, released in December 2023, illustrates this: around 46.7 billion total parameters with roughly 12.9 billion active per token. Later large models push this further, with hundreds of billions of total parameters and only tens of billions active.
Why MoE is attractive
| Benefit | Explanation |
|---|---|
| More capacity per unit of compute | Total parameters grow without proportional compute |
| Faster inference than a dense model of the same size | Only a few experts run per token |
| Cheaper training for a given quality | Compute is spent where it matters |
| Specialisation | Experts can develop different strengths |
The challenges
- Memory: all experts must be loaded, so serving needs memory sized to the total, not the active, parameters.
- Load balancing: without a balancing loss, the router sends most tokens to a few popular experts, wasting the rest.
- Communication: in distributed serving, tokens must be routed across devices, which adds network traffic.
- Fine-tuning: MoE models can be more sensitive to fine-tuning than dense ones.
- Batching complexity: different tokens take different paths, complicating efficient serving.
A short history
Sparsely-gated mixture of experts was demonstrated for large neural networks by Noam Shazeer and colleagues in 2017. Google’s Switch Transformer (2021) simplified routing to a single expert per token and scaled to a trillion parameters. From late 2023, open models such as Mixtral brought MoE into mainstream use, and several leading models since have used sparse expert designs.
Limitations
- High memory requirements limit local deployment.
- Router decisions are hard to interpret.
- Expert specialisation is often less clean than the metaphor suggests.
- Quality per total parameter is lower than for dense models; the win is per unit of compute.