Gated Segmented State Space — attention replacement that beats a param-matched Transformer on quality, speed AND memory (full code)
One night, six experiments (V1–V6), one Colab T4. I ripped self-attention out of a decoder-only Transformer and replaced it with a gated linear recurrence over a fixed 256-dim state: Dynamic selective gate: g_t = σ(W_g x_t + b_g) — per-token/channel learned filter Hard reset mask: state zeroed at n…
Read the full story at r/learnmachinelearning ↗
Timeline · 1 report
- 2026-10-08 00:44 · r/learnmachinelearning
Gated Segmented State Space — attention replacement that beats a param-matched Transformer on quality, speed AND memory (full code)