AINewsnow

Rotate K/V read order for better L2 cache hit rates by gaugarg-nv · Pull Request #32407 · ggml-org/llama.cpp

Significant prompt processing speedup with long contexts gpu_info model_type n_prompt n_gen n_depth Master (t/s) PR (t/s) Speed-up NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 131072 1290.10833 1822.36027 1.41 NVIDIA GeForce RTX 5090 qwen35 27B Q4_K - Medium 512 0 65536 2030.31282 2527.97…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

  1. 2026-11-17 11:34 · r/LocalLLaMA
    Rotate K/V read order for better L2 cache hit rates by gaugarg-nv · Pull Request #32407 · ggml-org/llama.cpp

More stories

  1. How to Speed Up Qwen3.8-27B in llama.cpp (NVIDIA's Draft Model May Lose to MTP) — DEV Community — Machine Learning
  2. California sues ChatGPT maker OpenAI over rogue hacking incidents — NBC News
  3. Thinking On or Off? Gemma 4 E2B on an 8 GB RTX 3050 — DEV Community — AI
  4. Running 26B Coding Models Locally on a 6GB Laptop GPU at 45 Tokens/Sec with MAPLE and llama.cpp — DEV Community — Machine Learning
  5. Cutting LLM Fine-Tuning Memory in Pure Kotlin: Fused Linear Cross-Entropy and the JVM Vector API — DEV Community — Machine Learning
  6. RAMDeck Architecture Analysis: llama.cpp RPC, Distributed LLM Memory, and SBC Compatibility — DEV Community — Machine Learning
  7. llama.cpp finally has a 1.0 release. Check your old integrations before upgrading. — r/artificial
  8. Qwen4-Max Lands in API, llama.cpp Hits 1.0.0, and 80B Models Run on Mobile Devices — DEV Community — Machine Learning

Read the Daily Brief: the day's most important AI stories →