AINewsnow

Scaling to 1T MoE on TorchTitan [P]

Hi everyone! Over the last few months we've been in the trenches with the TorchTitan and PyTorch maintainers, trying to find out how far GH200 (the GPUs inside the JUPITER supercomputer) can be pushed. Short answer: far enough to train a ~1T-parameter MoE at ~1,300 tokens/s per GPU on 384 GPUs. Tha…

Read the full story at r/MachineLearning ↗

Timeline · 1 report

  1. 2026-11-13 16:28 · r/MachineLearning
    Scaling to 1T MoE on TorchTitan [P]

More stories

  1. Jensen Huang resigns as Nvidia CEO amid China spy scandal — TechCrunch AI
  2. Google connects Wallet to Gemini for personal finance insights in the U.S. — Google AI Blog
  3. Anthropic's IPO roadshow proceeds amid market turmoil as Amodei brushes off turbulence — Bloomberg AI
  4. Anthropic CEO Amodei calls compute a 'huge uncertainty' days before record IPO — Business Insider AI
  5. Anthropic’s IPO Is About Six Times Oversubscribed — Bloomberg AI
  6. OpenAI Board Member Warns AI Takeover Could Be Months Away — Gizmodo AI
  7. Hydra-PSIE expands 'Coeziune OM–IA' DOI registry to 60 published works — DEV Community — AI
  8. OpenAI launches ChatGPT Wallet for AI agent purchases — OpenAI News

Get the daily brief of stories like this at 6:30 every morning →