[Paper] Virtual Head Attention
The design of attention mechanisms in long-context large language models faces an ``impossible triangle'' among KV cache size, training and prefill FLOPs, and model quality. GQA achieves strong quality but incurs high KV cache overhead; alternative approaches such as MLA and MFA can compress the ca…
Read the full story at r/LocalLLaMA ↗
Timeline · 1 report
- 2026-10-23 15:59 · r/LocalLLaMA
[Paper] Virtual Head Attention