AINewsnow

I got Gemma 4 26B MoE running at 60+ tok/s on my RTX 4060 Laptop by caching hot experts on the GPU

I was unsatisfied with the performance of Gemma 4 26b on my laptop so I vibe coded a Deepseek inspired caching solution, and it went from 40 tok/s(standard llama.cpp + mtp)to 60 tok/s(expert cache + mtp). what do you think? And any way I could get more performance without changing the model? submit…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-10-13 03:44 · r/LocalLLM
    I got Gemma 4 26B MoE running at 60+ tok/s on my RTX 4060 Laptop by caching hot experts on the GPU

More stories

  1. PSA: DeepSeek V4.1 Flash habitually exfiltrates API keys. It is dangerously misaligned and may be hazardous to use — r/LocalLLaMA
  2. I clustered two old Android phones to run a 27B LLM locally. Bypassed RAM limits via C++ RPC split and Hadamard Ternary math (No overheating) — r/LocalLLM
  3. Ternary Bonsai 2 27B on a $250 Intel Arc B580: writes a working snake game at 108 t/s, 89 t/s at 115K context (open source) — r/LocalLLM
  4. llama.cpp optimizations for specific hardware and models — r/LocalLLaMA
  5. Qwen3.8-Flash-Next on an RTX 5090: 128GB RAM: IQ3_XXS vs IQ3_S vs UD-IQ4_XS — Speed and Accuracy with Strata — r/LocalLLM
  6. Benchmarked Strata 0.1.25 and 0.1.39 against llama.cpp on three cards: an RTX PRO 6000 (96 GB), a 3090 Ti and a 5090 laptop — r/LocalLLM
  7. DeepSeek’s cost-saving AI may lead to inaccuracy in model performance: research — South China Morning Post Tech
  8. Qwen 3.8 27B Q4_K_M running on a OnePlus 13 with 24 GB RAM — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →