Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions
This story is from 2026-09-16. It is preserved in the archive; the latest stories are on the live feed.
arXiv:2609.16215v1 Announce Type: new Abstract: GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate state. Systems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore extend GPU memory wi…
Read the full story at arXiv cs.AI ↗
Timeline · 1 report
- 2026-09-16 04:00 · arXiv cs.AI
Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions