The KV cache is eating your GPU. MQA and GQA are how models fight back
Quick recap from the last post: the KV cache saves you from recomputing old tokens, and it's the reason generation is fast. But it has a bill. And the bill is memory. I wanted to put an actual number on it, because "the cache is big" doesn't mean much until you see the GB. The formula Per token, yo…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-10-20 17:16 · DEV Community — Machine Learning
The KV cache is eating your GPU. MQA and GQA are how models fight back