[R] Fathom: letting each query choose how many bits of each key channel to read, for sparse decoding over an offloaded KV cache

Sparse attention fetches the top-k keys, but to know which k you first have to rank all n. Every method does that by scanning a cheap compressed copy of every key, and every one of them fixes the bit depth of that copy at design time: SparQ reads 16 or 32 channels at full depth, Double Sparsity rea…

Read the full story at r/learnmachinelearning ↗

Timeline · 1 report

1. 2026-09-29 14:25 · r/learnmachinelearning
   [R] Fathom: letting each query choose how many bits of each key channel to read, for sparse decoding over an offloaded KV cache