llama : add a GPU cache for MoE experts kept in host memory by am17an · Pull Request #29887 · ggml-org/llama.cpp

Potentially big speedup for MoE models that don’t fully fit in VRAM. Are you GPU Poor? Show your speedups ;) update https://github.com/ggml-org/llama.cpp/pull/30112 MERGED

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

2026-10-07 18:15 · r/LocalLLaMA
llama : add a GPU cache for MoE experts kept in host memory by am17an · Pull Request #29887 · ggml-org/llama.cpp

More stories

Qwen 3.6 35B A3B: 131K context + vision on 6GB VRAM — r/LocalLLaMA
Tested Mellum2.1-12B-A2.5B on PI Coding Agent - surprisingly usable, but not great at one-shot projects — r/LocalLLaMA
Running the uncensored Qwen3.8-27B (HauhauCS) on a 4090 at 262K context and ~130 tok/s — r/LocalLLaMA
Java vllm-like framwork claims 90% of perfomance of llama.cpp on local inference on NVIDIA GPUs by compiling Java to CUDA and cuTile — r/LocalLLM
Success with Qwen3.8 27B GSQ-RCO-IQ3_S on 16GB VRAM — r/LocalLLM
Created laya : Now Introducing a new 800 Million Param physics-based typed decision model with 73k context and image support — r/LocalLLaMA
Found a fix for AMD RX 6600 100% CPU usage — r/LocalLLM
I made a free Mac app that runs local models: chat, pictures, video and voices — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →