Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States
arXiv:2610.20637v1 Announce Type: new Abstract: We expand the exponential kernel of attention into a truncated Taylor series of $n$ terms and factorize each term into a tensor-power inner product, which collapses the causal history into exactly $n$ recurrent states, each updated by one rank-one out…
Read the full story at arXiv cs.LG ↗
Timeline · 1 report
- 2026-10-16 04:00 · arXiv cs.LG
Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States