AINewsnow

how much difference does kv cache quants even make?

I currently run qwen flash next 3.8 iq3xxs via strata (like almost everyone lol) and I currently have kv cache at q8 with 262k ctx. Ive never actually tried any other quant and dont know how big the difference even is. Id like to have a bit more context (like 300-400k) but it doesnt fit my ram then…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

  1. 2026-10-25 12:41 · r/LocalLLaMA
    how much difference does kv cache quants even make?

More stories

  1. Qwen 3.8 Flash Next (129 GB) is better than 2.6 bpw DeepSeek V4.1 Flash 1017 (187 GB + 203 GB Engram) — r/LocalLLaMA
  2. Massive B70 spec-decoding improvement - Intel XMX matrix engines refactored by budiony · Pull Request #29864 · ggml-org/llama.cpp — r/LocalLLaMA
  3. r9700 ai pro + 64gb ram, What QFN model to run and what engine? — r/LocalLLM
  4. I ran local AI inference (AMD GPU) for 30 days and measured every watt — here's the real cost vs cloud APIs — r/LocalLLM
  5. "We are Claude" — r/LocalLLM
  6. Come On, even Qwen and GLM uses code to generate random number. Argon why? — r/GeminiAI
  7. Paid $8k for a DGX Spark at Micro Center. Is one Spark useless until you buy a second? — r/LocalLLM
  8. Meet Groot — four smaller models for reasoning, coding and tool use — r/huggingface

Get the daily brief of stories like this at 6:30 every morning →