Pipeline parallelism splits a model’s layers into consecutive stages, each placed on a different device, so the devices work like an assembly line. Device 1 handles the first group of layers, passes its output to device 2, and so on. To keep every device busy, batches are divided into micro-batches that flow through the pipeline simultaneously. It was introduced at scale by Google’s GPipe work in 2019.
Key takeaways
- Each device holds a contiguous group of layers, called a stage.
- Only activations pass between stages, so communication is lighter than tensor parallelism.
- Idle time at the start and end of each batch is called the pipeline bubble.
- Micro-batching shrinks the bubble by keeping stages busy.
- It works across servers, where interconnects are slower.
Pipeline parallelism explained in simple terms
A car factory does not have one worker build a whole car. Station one fits the chassis, station two the engine, station three the doors. While station three works on car 1, station one is already starting car 2. The line runs fully only once every station has work, which is exactly the pipeline bubble problem.
How it works
- Split the model’s layers into k stages, one per device.
- Split each training batch into m micro-batches.
- Feed micro-batch 1 into stage 1. When it finishes, it moves to stage 2 while stage 1 starts micro-batch 2.
- Backward passes flow in reverse through the same stages.
- Once all micro-batches complete, gradients are applied.
The pipeline bubble
At the start, stages 2 to k wait for work; at the end, stage 1 idles while later stages finish. The wasted fraction is approximately (k − 1) ÷ (m + k − 1).
Worked example with 4 stages:
| Micro-batches (m) | Bubble fraction |
|---|---|
| 1 | 75% |
| 4 | about 43% |
| 16 | about 16% |
| 64 | about 4.5% |
More micro-batches mean less waste, but each is smaller, which can reduce GPU efficiency and affect layers that use batch statistics. Schedules such as 1F1B (one forward, one backward) and interleaved pipelines further reduce the bubble by reordering work.
Pipeline vs tensor vs data parallelism
| Aspect | Pipeline | Tensor | Data |
|---|---|---|---|
| Splits | Layer groups | Inside layers | The batch |
| Communication volume | Moderate, activations only | High, every layer | High, gradients per step |
| Interconnect need | Moderate | Very high | High |
| Main use | Across servers | Within a server | Across the cluster |
| Memory saving | Yes | Yes | Only with sharding |
Large training runs use all three: tensor inside a node, pipeline across nodes, data parallelism over the whole cluster.
Pipeline parallelism at inference
It is less common for serving than tensor parallelism, because it does not speed up a single request; a token still passes through every stage in sequence, adding hop latency. It can, however, raise throughput when many requests flow through the pipeline at once, and it allows a large model to run across machines that lack fast interconnects.
Practical considerations
- Balance stages: an uneven split leaves the slowest stage as the bottleneck.
- Memory differs per stage: early stages hold activations longer during training.
- Failure handling: any stage failing halts the pipeline.
- Combine carefully: stage boundaries interact with tensor-parallel groups.
Limitations
- Bubble overhead never reaches zero.
- Adds latency for single requests at inference.
- Stage balancing requires tuning.
- Debugging distributed pipelines is complex.