AINewsnow

FSDP vs DDP: when do you need sharded data parallel?

For full training with AdamW in mixed precision, DDP stops fitting on an 80 GB NVIDIA H100 at roughly 3.75 billion parameters once you keep 20 GB for activations, and above that you need FSDP. Nodus runs either one with the same torchrun command on 1, 2, 4 or 8 GPUs of a machine, with H100s listed…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-11-19 19:28 · DEV Community — Machine Learning
    FSDP vs DDP: when do you need sharded data parallel?

More stories

  1. Nvidia scandal spurs tighter US AI chip export controls on China — DigiTimes Asia
  2. AWS Adds Managed NVIDIA Dynamo Support to SageMaker HyperPod Inference — AWS Machine Learning Blog
  3. Chinese tech firms brace for AI setbacks as US chip curbs tighten — Washington Post AI
  4. NVIDIA Details How Developers Build Simulations With Frontier AI Agents in Omniverse — NVIDIA Blog
  5. Anthropic Appears Among Market-Moving Names Across Recent Trading Sessions — Barron's
  6. US proposal to curb China's remote access to AI chips rattles Southeast Asia's data center boom — DigiTimes Asia
  7. National security and AI leaders urge DOJ to investigate Nvidia over chip smuggling — NBC News
  8. US congressional commission says China's military seeks Nvidia chips for AI weapons — CNBC Technology

Read the Daily Brief: the day's most important AI stories →