Mixture of experts: how the routing works, when it breaks, and two ways to fix it
I've been working on Mixture of experts (MoEs) and sparsity for a few years now, so I thought I'd make a short video explaining how MoE routing works. For the video I trained a tiny character model with a 32-expert layer three times, on English, Python and arithmetic text. Run 1, token choice with…
Read the full story at r/deeplearning ↗
Timeline · 1 report
- 2026-10-15 12:55 · r/deeplearning
Mixture of experts: how the routing works, when it breaks, and two ways to fix it
More stories
- Anthropic reports $16.8 billion Q3 revenue ahead of planned IPO — New York Times Technology
- Google launches its ‘universal' online shopping cart ahead of the holidays — The Verge AI
- NVIDIA RTX Spark Superchip Outperforms Apple M5 Max in AI Benchmarks — Engadget
- Amazon launches two new Bee AI devices with subscription plan — TechCrunch AI
- Bank of America hired 1,000 contractors to patch software flaws exposed by Anthropic's Mythos — CNBC Technology
- Weeks to Hours: SentinelOne on Cybersecurity Research with OpenAI — OpenAI YouTube
- Anthropic warns AI models may self-replicate on IoT devices within months — Financial Times AI
- Amazon Debuts Revamped Bee AI Wearables, $12-a-Month Subscription Plan — Bloomberg AI
Get the daily brief of stories like this at 6:30 every morning →