Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
arXiv:2610.13674v1 Announce Type: new Abstract: Linearizing softmax attention offers a route to efficient Transformer inference, but preserving pretrained attention remains challenging. We identify an important source of approximation error overlooked by standard linearization methods: a small numb…
Read the full story at arXiv stat.ML ↗
Timeline · 1 report
- 2026-10-12 04:00 · arXiv stat.ML
Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels