Data parallelism is a training strategy where each device holds a complete copy of the model and processes a different slice of the batch, with gradients averaged across devices before the weights are updated. It is the simplest and most widely used form of distributed training, and it scales well as long as the model fits on one device.
Key takeaways
- Every GPU has the full model; only the data differs.
- Gradients are combined with an all-reduce operation each step.
- Effective batch size = per-device batch × number of devices.
- It does not help when the model itself is too large for one GPU; that needs model parallelism.
- Sharding methods such as ZeRO and FSDP extend it to large models.
Data parallelism explained in simple terms
Ten accountants each receive a copy of the same rulebook and a different pile of invoices. They work in parallel, then meet to agree on the updates to the rulebook, so all ten copies stay identical. The work is divided; the knowledge is replicated.
How it works
- Copy the model to every device.
- Split each training batch into equal shards, one per device.
- Each device runs a forward and backward pass on its shard.
- All-reduce: gradients are summed or averaged across all devices.
- Every device applies the same update, so copies stay in sync.
- Repeat.
Because all devices apply identical updates, the models never diverge.
A worked example
Training with 8 GPUs and a per-device batch size of 16:
- Effective batch size: 8 × 16 = 128.
- Each step processes 128 examples.
- Gradients are averaged over all 128 before the update.
Adding gradient accumulation of 4 raises the effective batch to 512 without more hardware. Remember that a larger effective batch usually needs a higher learning rate, applied with warmup.
The communication cost
Every step requires exchanging gradients the size of the model. For a 7-billion-parameter model, that is gigabytes of traffic per step, per device. This is why training clusters use high-bandwidth interconnects, and why communication can become the bottleneck as device counts grow. Techniques such as gradient bucketing and overlapping communication with computation reduce the impact.
Sharded data parallelism
Standard data parallelism wastes memory: every device stores identical optimiser state and gradients. DeepSpeed’s ZeRO (2020) and PyTorch’s FSDP remove this redundancy in stages:
| Stage | What is sharded | Memory saving |
|---|---|---|
| ZeRO-1 | Optimiser state | Large |
| ZeRO-2 | Optimiser state and gradients | Larger |
| ZeRO-3 / FSDP | Optimiser state, gradients and parameters | Largest; enables very big models |
At stage 3, parameters are gathered only when needed, so data parallelism can train models far larger than one GPU’s memory, at the cost of more communication.
Data parallelism at inference
For serving, the same idea appears as replica parallelism: multiple copies of the model behind a load balancer, each serving different users. This scales throughput but does not help a model that is too big for one GPU.
Limitations
- Requires the model, gradients and optimiser state to fit per device, unless sharding is used.
- Communication grows with model size and device count.
- Very large effective batches can require careful hyperparameter tuning.
- Stragglers slow every device, since all must synchronise each step.