Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff
TL;DR SGLang 0.5.23 (released 2026-11-02) ships an opt-in feature called Paged Experts. It keeps K of the E experts of every MoE layer in GPU memory, holds all E in pinned host RAM, and copies in only the experts a step routes to that are not resident. The expert math still runs on the GPU. What th…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-11-07 00:01 · DEV Community — Machine Learning
Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff