AINewsnow

llama.cpp's MoE cache nearly doubled 30B decode speed on a free T4

This story is from 2026-10-18. It is preserved in the archive; the latest stories are on the live feed.

If your MoE model is too big for your GPU, llama.cpp fills VRAM with as many whole layers of experts as fit and runs the rest on the CPU, from system RAM. Every token then waits on the slowest part of your machine. Since build b11480 there's a better use for that VRAM: a cache of the experts the mo…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-18 11:02 · DEV Community — AI
    llama.cpp's MoE cache nearly doubled 30B decode speed on a free T4

More stories

  1. Qwen Ships Qwen3-VL 8B Locally, Seeing Images and Video in 5 GB — AlphaSignal
  2. Has anyone used EXO in x86 machines, or am I just better off using llama.cpp or dllama? — r/LocalLLM
  3. Developer Mert Cobanov Runs llama.cpp on a Jailbroken PS5 — AlphaSignal
  4. Qwen3.8-27B at 42 tok/s single-user on one DGX Spark, fastest single-Spark entry on spark-arena, with 99 to 101% of BF16 quality — r/LocalLLM
  5. User friendly GUI for configuring and launching llama.cpp — r/LocalLLM
  6. Academic history research agent on 16GB vRAM, context important — r/LocalLLM
  7. What model do you recommend for coding to this hardware? (LLama.cpp) — r/LocalLLM
  8. Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →