AINewsnow

Why a '35GB' Quantized LLM Still OOMs an 80GB GPU: KV Cache, GQA vs. DeepSeek MLA, and the Real VRAM Math

This story is from 2026-10-14. It is preserved in the archive; the latest stories are on the live feed.

Every engineer deploying open-weights models hits the same rite of passage: You take a 70B parameter model, quantize it to 4-bit ( 70B * 0.5 bytes ≈ 35 GB ), provision a single 80GB A100 or H100 (or two 24GB GPUs), and launch vLLM or llama.cpp with a 64K context window and a small concurrent batch…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-14 07:25 · DEV Community — AI
    Why a '35GB' Quantized LLM Still OOMs an 80GB GPU: KV Cache, GQA vs. DeepSeek MLA, and the Real VRAM Math

More stories

  1. I got Gemma 4 26B MoE running at 60+ tok/s on my RTX 4060 Laptop by caching hot experts on the GPU — r/LocalLLM
  2. Minisforum MS-S1 MAX P495 (Ryzen AI Max+ PRO 495, 192GB) llama.cpp Vulkan benchmarks: gpt-oss 120B, MiniMax M2.7, DeepSeek V4 Flash, Qwen3 Coder 30B, Qwen3.8 27B — r/LocalLLM
  3. The algorithm that trains reasoning LLMs can't teach a robot to walk — r/reinforcementlearning
  4. I clustered two old Android phones to run a 27B LLM locally. Bypassed RAM limits via C++ RPC split and Hadamard Ternary math (No overheating) — r/LocalLLM
  5. Research: Gemma 4 E2B on a Pixel 8 Pro (Tensor G3) with llama.cpp — r/LocalLLM
  6. Qwen 27B at ~20 tok/s on a jailbroken PS5's GPU. I forked PS5LM and turned it into a native app — r/LocalLLM
  7. Thank-you post for DeepSeek-V4-Flash-0731 with DSpark — r/LocalLLaMA
  8. Why AnythingLLM is moving from Ollama to Lemonade and llama.cpp — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →