AINewsnow

Why the KV cache, not the weights, decides how many users an LLM server can hold

A 7B model in fp16 takes 13.5 GB of GPU memory for its weights. On an 80 GB card that leaves about 66 GB free, which sounds like plenty. It runs out at around 30 users with 4,096-token contexts, and the weights aren't what fills it. The KV cache is. This post works through the arithmetic. All the n…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-25 11:37 · DEV Community — Machine Learning
    Why the KV cache, not the weights, decides how many users an LLM server can hold

More stories

  1. Kimi K3.1 Pro and Kimi K3.1 Pro Highspeed are available now in AI Gateway — Vercel Blog
  2. NCERT introduces 20-hour AI module for classes 9-12, covers deepfakes and data privacy — Mint AI
  3. Thomas L. Friedman: We Just Set Off the Second Big Bang — New York Times AI
  4. Two outsiders found ways OpenAI’s AI agents could slip past the system built to catch them — r/OpenAI
  5. Qwen 3.8 Flash Next (Q4) got stuck on this loop for hours. I wonder if this is the AI equivalent of having a nightmare and not being able to wake up — r/LocalLLM
  6. Black Forest Labs' FLUX 3 Klein Runs Image Generation on 8GB GPUs — AlphaSignal
  7. Full DeepSeek-V4.1-Flash-1017 (510 GB) on an RTX 3060 12 GB + 64 GB DDR4 + NVMe: 5.8 tok/s cold, 7.4 warm, >97% exact quality. — r/LocalLLM
  8. Securing our frontier model training environment — OpenAI News

Get the daily brief of stories like this at 6:30 every morning →