Decode Is 300x Memory-Bound. Batching Fixes That Until the Context Gets Long.
I spent last week arguing about KV cache sizes. The useful follow-up is simpler: multiply the cache by the context length and you get the number that actually sets your throughput. Decode does not run out of FLOPs. It runs out of memory bandwidth, by a factor of about 300, and every GPU generation…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-31 14:18 · DEV Community — AI
Decode Is 300x Memory-Bound. Batching Fixes That Until the Context Gets Long.