Expert-locality-aware decode routing reduces MoE serving latency
This story is from 2026-08-23. It is preserved in the archive; the latest stories are on the live feed.
Current MoE serving pipelines treat every decode worker as interchangeable, assuming that equalizing request counts suffices to minimize latency. In practice, the weight‑loading cost of each request varies wildly because different inputs activate disjoint expert subsets, and ignoring this fact infl…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-08-23 05:00 · DEV Community — Machine Learning
Expert-locality-aware decode routing reduces MoE serving latency