AINewsnow

C = 1.25 is a tolerance, not a constant: the load-tail law behind MoE token dropping

Every sparse Mixture-of-Experts layer has a number in it that nobody derived. You need a static shape for the kernel, so each expert gets a budget c = C · topk · T / E , and C — the capacity factor — arrives as folklore: 1.25 from Switch Transformer, 2.0 from GShard. Those values were chosen at a p…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-16 09:23 · DEV Community — Machine Learning
    C = 1.25 is a tolerance, not a constant: the load-tail law behind MoE token dropping

More stories

  1. Meta releases Muse Spark Open, an 816B-parameter open-weight AI model — AWS Machine Learning Blog
  2. Taiwan to Tighten AI Chip Export Controls to China — Bloomberg AI
  3. Apple Intelligence to enter China beta testing in late October — Techmeme
  4. AI-Generated Ads Flood Midterm Elections — New York Times AI
  5. Stockholm startup Matters raises $5m pre-seed to build AI-native bank — Sifted
  6. Debate Erupts Over Potential AI Bubble and Economic Impact — New York Times AI
  7. Meta and Apple Partner to Run Open-Weight Muse AI Model on Macs — Wall Street Journal Technology
  8. Black Forest Labs releases FLUX.3 Image Klein, a 5.6B open-weight image model — Hugging Face Blog

Get the daily brief of stories like this at 6:30 every morning →