Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval
arXiv:2611.04169v1 Announce Type: new Abstract: Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families), neutral filler costs nothing: accuracy stays at 0…
Read the full story at arXiv cs.CL ↗
Timeline · 1 report
- 2026-11-04 05:00 · arXiv cs.CL
Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval