llama_moe_cache: run missed experts on CPU, add tuning script by am17an · Pull Request #30966 · ggml-org/llama.cpp
This is a significant speedup for MoE offloading (low VRAM people). I tried Strata once, and it had some tuning mechanism, so probably this is a similar idea.
Read the full story at r/LocalLLaMA ↗
Timeline · 1 report
- 2026-11-07 16:25 · r/LocalLLaMA
llama_moe_cache: run missed experts on CPU, add tuning script by am17an · Pull Request #30966 · ggml-org/llama.cpp
More stories
- 489.8 tok/s, QWen2.5 7B on RTX3050 for code edit, reproducible decoding — DEV Community — AI
- How to pay for AI API access with crypto — no card, no subscription, no account — DEV Community — AI
- Ollama vs. llama.cpp on the Same GGUF: The 15% Speed Gap Was the Defaults, Not the Engine — DEV Community — AI
- Test if your local inference is safe from the web — r/LocalLLM
- Probably old news, but any website you have open can reach your local LLM server while you browse — r/LocalLLM
- The Open-Source Paradox: Giving Away AI Makes It Safer (and More Dangerous) — DEV Community — AI
- Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU — Hacker News Front Page
- Zero-Downtime LLM Inference on Kubernetes Spot Instances (~70% GPU Cost Savings) — DEV Community — Machine Learning
Get the daily brief of stories like this at 6:30 every morning →