AINewsnow

moe-cache: MoE models bigger than your RAM, 2-12x faster on stock llama.cpp with identical tokens (Qwen3.8-Flash-Next 68 GB @ 24 GB RAM: 5.2 → 12.5 tok/s, Qwen3.6-35B @ 12 GB RAM: 6.9 → 22.6 tok/s)

TL;DR: moe-cache is a plugin for unmodified llama-server (no fork, no patch). When a MoE model doesn't fit in RAM, stock llama.cpp falls off a speed cliff because the OS page cache thrashes. moe-cache manages the CPU-side experts itself, and in the default mode the output tokens are identical to st…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-10-16 07:51 · r/LocalLLM
    moe-cache: MoE models bigger than your RAM, 2-12x faster on stock llama.cpp with identical tokens (Qwen3.8-Flash-Next 68 GB @ 24 GB RAM: 5.2 → 12.5 tok/s, Qwen3.6-35B @ 12 GB RAM: 6.9 → 22.6 tok/s)

More stories

  1. User friendly GUI for configuring and launching llama.cpp — r/LocalLLM
  2. Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming — r/LocalLLM
  3. WHIRL 0.2.0 now runs on the AMD Ryzen AI Max+ 395 (Radeon 8060S): native Windows LLM engine, faster than llama.cpp ROCm and Vulkan from 2k to ~250k tokens (also on the Radeon AI PRO R9700) — r/LocalLLM
  4. Arc Pro B60 runs a 27B coding model at 53 tok/s: full recipe, flags and raw logs — r/LocalLLM
  5. Unsloth's Llama 3.2 Mirror Hits 800K Downloads by Removing Meta's Login Wall — AlphaSignal
  6. V100 16GB: A 2017 GPU running a 35B vision model at 93 tok/s: Tesla V100 16 GB on Windows — r/LocalLLM
  7. 780m and Gemma 4 26b a4b — r/LocalLLM
  8. llama.cpp has come a long way — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →