AINewsnow

ISOM-R2: Streaming 1,055,402 Tokens on 3.24 GB Peak VRAM

Standard Transformer attention has a memory problem at scale. For a 1.05 million-token context with 28 layers, 2 KV heads, and a head dimension of 128, the FP16 Key-Value cache alone requires: 2 × 28 × 2 × 128 × 1,055,402 × 2 bytes ≈ 28.2 GiB That is before loading a single model weight. On a 40 GB…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-02 10:13 · DEV Community — Machine Learning
    ISOM-R2: Streaming 1,055,402 Tokens on 3.24 GB Peak VRAM

More stories

  1. Bring near-Astra intelligence to everyday work with GPT-6.1 Sol on Amazon Bedrock — AWS Machine Learning Blog
  2. Gemini 4 Argon: our next era of frontier intelligence — Google Gemini Blog
  3. Google tests its plan for AI data centers in space with Project Suncatcher — Scientific American
  4. Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs — Allen Institute for AI (Ai2)
  5. Google Releases New Gemini Model With Guardrails Amid A.I. Safety Debate — New York Times Technology
  6. OpenAI announces ‘dots’ agent after scrapping launch of new AI model over safety concerns — The Guardian AI
  7. OpenAI DevDay 2026 Keynote (FULL) — OpenAI YouTube
  8. FTC launches broad investigation into Anthropic, OpenAI — Washington Post AI

Get the daily brief of stories like this at 6:30 every morning →