Why does the KV cache make long-context inference memory-bound rather than compute-bound?

I understand the KV cache avoids recomputing keys and values for past tokens, but I keep reading that it's also why decoding long sequences ends up limited by memory bandwidth rather than FLOPs. Is that mainly because each new token has to read the entire cache from memory while doing relatively li…

Read the full story at r/learnmachinelearning ↗

Timeline · 1 report

1. 2026-10-17 21:05 · r/learnmachinelearning
   Why does the KV cache make long-context inference memory-bound rather than compute-bound?