Study Diagnoses Accuracy Loss in Sparse Attention Block Selection
A new arXiv paper reports that sparse attention models reading a fixed budget of key blocks can lose much of their long-context retrieval accuracy, and proposes methods to diagnose and repair block selection.
Read the full story at arXiv cs.LG ↗
Timeline · 1 report
- 2026-11-18 05:00 · arXiv cs.LG
Lost in a Few Heads: Diagnosing and Repairing Block Selection in Sparse Attention