[Paper] Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine…
Read the full story at r/LocalLLaMA ↗
Timeline · 1 report
- 2026-10-21 10:09 · r/LocalLLaMA
[Paper] Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading