[R] SANATAN ATTENTION: No Growing KV cache (Standard Attention issue), no context decay (Mamba Issue) binary gate hiding inside i² = −1
The idea in one paragraph: over 2x2 real matrices, i² = −1 has two orthogonal solutions, rotation by +90 and −90 degrees: i₊ = [[0, −1], [1, 0]], i₋ = [[0, 1], [−1, 0]] Square either one and you get [[−1, 0], [0, −1]] = −I, so both are genuine square roots of minus one. Their combination is S(θ) =…
Read the full story at r/deeplearning ↗