83.6 GB MoE on a 16 GB-VRAM / 15 GB-RAM box: ~15 tok/s once warm, host RAM flat at 4.8 GiB, 11 s to first token [measured]
TL;DR — I wrote a runtime that keeps the MoE expert bank on disk and streams it into a fixed VRAM slab with unbuffered I/O, so host RAM never becomes a tier : no page cache for the weights, no page file, RAM flat for the whole run. On my box (RTX 5060 Ti 16 GB, 15.38 GiB RAM, Gen3-class NVMe) a Qwe…
Read the full story at r/LocalLLM ↗