Model parallelism is the practice of splitting one model across several devices because it is too large to fit on a single GPU. Unlike data parallelism, where each GPU holds a full copy of the model, model parallelism divides the model itself. The two main forms are tensor parallelism, which splits individual weight matrices, and pipeline parallelism, which assigns different layers to different devices.
Key takeaways
- Needed when model weights, gradients and optimiser state exceed one GPU’s memory.
- Tensor parallelism splits within layers; pipeline parallelism splits across layers.
- Large training runs combine several parallelism types, often called 3D parallelism.
- Communication between devices is the main cost.
- Sharding methods such as ZeRO and FSDP blur the line with data parallelism.
Model parallelism explained in simple terms
A book too heavy for one person to carry can be split two ways. Pipeline parallelism gives chapters 1–5 to one person and 6–10 to another, passing work along. Tensor parallelism has two people hold opposite halves of the same page and read together. Both let the group handle a book no individual could manage, and both require coordination.
Why it is needed
A 70-billion-parameter model requires roughly 140 GB just for weights at 16-bit precision, before the attention cache or activations. Training needs several times more for gradients and optimiser state. No single GPU today has that much memory, so the model must be split.
| Requirement | 70B model at 16-bit |
|---|---|
| Weights | about 140 GB |
| Adam optimiser state and gradients during training | several hundred GB more |
| Practical result | Training requires many GPUs working together |
Types of parallelism
| Type | What is split | Communication | Typical scope |
|---|---|---|---|
| Data parallelism | The batch; each device holds a full model copy | Gradient all-reduce per step | Across nodes |
| Tensor parallelism | Individual weight matrices | Frequent, within each layer | Within a node, over fast links |
| Pipeline parallelism | Groups of layers | Activations between stages | Across nodes |
| Expert parallelism | MoE experts across devices | Token routing | Large sparse models |
| Sequence or context parallelism | The sequence itself | Attention-related exchanges | Very long contexts |
Combining them: 3D parallelism
Large training runs typically use tensor parallelism inside a server, where GPUs are connected by high-bandwidth links, pipeline parallelism across servers, and data parallelism across the whole cluster. This layering places the most communication-heavy form where bandwidth is highest.
Sharding: ZeRO and FSDP
DeepSpeed’s ZeRO (2020) and PyTorch’s FSDP take a different approach: keep the data-parallel structure but shard optimiser state, gradients and parameters across devices, gathering them only when needed. This achieves model-scale memory savings while keeping programming simpler, and is now a common default for training large models.
Model parallelism at inference
Serving also uses it. A large model split across several GPUs with tensor parallelism can produce tokens faster than one GPU could, provided the interconnect is fast. Splitting across slow links usually hurts more than it helps, which is why tensor parallelism is kept within a single machine where possible.
Limitations
- Communication overhead reduces efficiency as device counts grow.
- Requires fast interconnects such as NVLink or InfiniBand.
- More complex code, configuration and debugging.
- Failures anywhere in the group can halt the whole job.