AINewsnow

Why long context is expensive: the KV cache bill

Your chat history can weigh more than the model that reads it. Llama 3.1 8B is a 16 GB model. Give it a 1M-token prompt and its KV cache alone needs 128 GB. Why the cache exists Transformers have no memory of their own. RNNs kept one small hidden state, constant size, but it forgot things. Transfor…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-15 23:31 · DEV Community — Machine Learning
    Why long context is expensive: the KV cache bill

More stories

  1. QUASAR 4-bit QAT checkpoints for Qwen3.5, Gemma 4 and Muse Glimmer: closer to BF16 than Google’s and NVIDIA’s 4-bit releases — r/LocalLLaMA
  2. P100 (sm_60) patches for llama.cpp: +42% decode and +74% short-prefill on a 2×P100 tensor split — r/LocalLLM
  3. Qwen3.8FN vLLM vs. llama.cpp vs Ninfer6000 — r/LocalLLaMA
  4. I built Harness Studio — an open-source desktop app for running Hugging Face models locally. Looking for feedback and contributors. — r/huggingface
  5. I built Agent Praxis: 6 adversarial environments to test how LLM agents handle traps, metric gaming, and fake authority — r/LocalLLM
  6. Qwen 27B at ~20 tok/s on a jailbroken PS5's GPU. I forked PS5LM and turned it into a native app — r/LocalLLM
  7. Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules — arXiv cs.CL
  8. Qwen3.8 Flash Next vs Qwen3.8 27B - The hype vs the truth. — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →