Sub‑bit KV cache quantization doubles throughput

LoopVQ compresses attention caches to sub‑bit precision, turning the KV cache memory wall into a speed lever [1] . By encoding multiple loop steps jointly and fine‑tuning the codebooks, the method fits each element into less than one bit while preserving the model’s functional behavior. The impleme…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

1. 2026-11-13 06:00 · DEV Community — Machine Learning
   Sub‑bit KV cache quantization doubles throughput