Mixture of experts: how the routing works, when it breaks, and two ways to fix it
This story is from 2026-10-15. It is preserved in the archive; the latest stories are on the live feed.
I've been working on Mixture of experts (MoEs) and sparsity for a few years now, so I thought I'd make a short video explaining how MoE routing works. For the video I trained a tiny character model with a 32-expert layer three times, on English, Python and arithmetic text. Run 1, token choice with…
Read the full story at r/deeplearning ↗
Timeline · 1 report
- 2026-10-15 13:21 · r/deeplearning
Mixture of experts: how the routing works, when it breaks, and two ways to fix it