How Sorting Your Prompts Speeds Up a Mixture-of-Experts Batch Job
Engineer's view. This is a batch job that touches records in arrival order. Each record needs a different index page, so the cache keeps reloading pages. Sort by key and each page loads once per batch. A mixture-of-experts model loads expert weights the same way, so a mixed batch pays for reads nob…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-10-21 10:50 · DEV Community — Machine Learning
How Sorting Your Prompts Speeds Up a Mixture-of-Experts Batch Job