AINewsnow

moe-cache : LRFU/MQ-like hybrid cache, ClassLRU by sanmai · Pull Request #31233 · ggml-org/llama.cpp

Replace SLRU with a cache that prioritizes frequently used experts in classes, ranked using decaying usage history. With a warm cache, I measured +6% decode on QFN and +7% on GLM versus SLRU, with 8% fewer uploads. Enables the auto mode for llama_moe_cache with promising benchmarks Waiting for 3096…

Read the full story at r/LocalLLaMA ↗

Timeline · 2 reports

  1. 2026-11-07 16:25 · r/LocalLLaMA
    llama_moe_cache: run missed experts on CPU, add tuning script by am17an · Pull Request #30966 · ggml-org/llama.cpp
  2. 2026-11-06 15:39 · r/LocalLLaMA
    moe-cache : LRFU/MQ-like hybrid cache, ClassLRU by sanmai · Pull Request #31233 · ggml-org/llama.cpp

More stories

  1. Probably old news, but any website you have open can reach your local LLM server while you browse — r/LocalLLM
  2. The Open-Source Paradox: Giving Away AI Makes It Safer (and More Dangerous) — DEV Community — AI
  3. Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU — Hacker News Front Page
  4. CyberTiel35BA3B working on Blender Rigging via MCP. 8GB VRAM, 16GB RAM. llama.cpp. — r/LocalLLaMA
  5. Zero-Downtime LLM Inference on Kubernetes Spot Instances (~70% GPU Cost Savings) — DEV Community — Machine Learning
  6. The inference engine flood.. but where's the quality? — r/LocalLLM
  7. Opus 5.5 wrote its own quantiser for Qwen3.5-0.8B, no llama-quantize, no imatrix. 5.9× better than Unsloth's same-size file. — r/LocalLLM
  8. Does 2-Bit Quantization Break On-Device Safety? What an EMNLP 2026 Study Found, and a GGUF Release Gate — DEV Community — Machine Learning

Get the daily brief of stories like this at 6:30 every morning →