Muon Optimizer Found to Worsen Load Balancing in First Mixture-of-Experts Layer
A new arXiv paper reports that many open mixture-of-experts language models trained with Muon instead of AdamW use per-expert routing-score biases for load balancing, and Muon makes the first MoE layer harder to balance.
Read the full story at arXiv cs.LG ↗
Timeline · 1 report
- 2026-10-30 04:00 · arXiv cs.LG
Muon Makes the First Mixture-of-Experts Layer Harder to Balance