AINewsnow

KV Cache Math: Why Llama 3.1 8B at 128K Context Won't Fit in 24GB

I quantized Llama 3.1 8B down to Q4 so it would fit comfortably on my 24GB card. The weights came out under 5 GB. Then I set the context to 128K, because the model card says it supports 128K, and the model promptly spilled onto the CPU and generated text at the speed of a fax machine. The weights w…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-09-27 05:35 · DEV Community — Machine Learning
    KV Cache Math: Why Llama 3.1 8B at 128K Context Won't Fit in 24GB

More stories

  1. Accelerating vision-language models with LFM2.5-VL-DSpark — Hugging Face Blog
  2. model : add Ling 3.0 VL support by aetherbird · Pull Request #29151 · ggml-org/llama.cpp — r/LocalLLaMA
  3. Ternary Bonsai 2 27B at up to 532 tok/s on one RTX 4090, native Windows: MTP + n-gram speculative decoding in a from-scratch CUDA engine — r/LocalLLM
  4. Qwen3.8-Flash-Next on 12GB VRAM - 65 tokens per second — r/LocalLLaMA
  5. 42x Faster Prompt Lookup Drafting in llama.cpp — r/LocalLLaMA
  6. Best native alternative to WebUI for remote access to local LLMs? — r/LocalLLM
  7. Qwen3.8 27B on Intel X7 358h + B390, with pi + llama.cpp surprised by its own RAM speed — r/LocalLLM
  8. 2x Radeon AI PRO R9700 + Qwen3.8-27B: 31 t/s on Windows → 113 t/s on Linux/vLLM. The fix was an M.2 riser, because the chipset slot doesn't do PCIe atomics. — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →