ISOM-R2: Streaming 1,055,402 Tokens on 3.24 GB Peak VRAM
This story is from 2026-10-02. It is preserved in the archive; the latest stories are on the live feed.
Standard Transformer attention has a memory problem at scale. For a 1.05 million-token context with 28 layers, 2 KV heads, and a head dimension of 128, the FP16 Key-Value cache alone requires: 2 × 28 × 2 × 128 × 1,055,402 × 2 bytes ≈ 28.2 GiB That is before loading a single model weight. On a 40 GB…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-10-02 10:13 · DEV Community — Machine Learning
ISOM-R2: Streaming 1,055,402 Tokens on 3.24 GB Peak VRAM