FSDP vs DDP: when do you need sharded data parallel?
For full training with AdamW in mixed precision, DDP stops fitting on an 80 GB NVIDIA H100 at roughly 3.75 billion parameters once you keep 20 GB for activations, and above that you need FSDP. Nodus runs either one with the same torchrun command on 1, 2, 4 or 8 GPUs of a machine, with H100s listed…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-11-19 19:28 · DEV Community — Machine Learning
FSDP vs DDP: when do you need sharded data parallel?