AINewsnow

Running a 133 GB MoE model on an 8 GB GPU at 11 tokens/s by streaming experts from NVMe

This story is from 2026-10-03. It is preserved in the archive; the latest stories are on the live feed.

I run local models on one home machine: an RTX 5060 with 8 GB , a Core Ultra 5 225F, 31 GiB of RAM and a Gen5 NVMe drive used only for model files. When NVIDIA published Qwen3.8-Flash-Next in NVFP4 (133 GB on disk), the obvious answer was "it doesn't fit". It does now: it runs as a normal chat mode…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-03 11:46 · DEV Community — AI
    Running a 133 GB MoE model on an 8 GB GPU at 11 tokens/s by streaming experts from NVMe

More stories

  1. Breakingviews - COMMENTARY: Nvidia leaves the AI trade hanging in purgatory — Reuters AI
  2. Nvidia scandal spurs tighter US AI chip export controls on China — DigiTimes Asia
  3. AWS Adds Managed NVIDIA Dynamo Support to SageMaker HyperPod Inference — AWS Machine Learning Blog
  4. Chinese tech firms brace for AI setbacks as US chip curbs tighten — Washington Post AI
  5. US proposal to curb China's remote access to AI chips rattles Southeast Asia's data center boom — DigiTimes Asia
  6. NVIDIA Details How Developers Build Simulations With Frontier AI Agents in Omniverse — NVIDIA Blog
  7. Anthropic Appears Among Market-Moving Names Across Recent Trading Sessions — Barron's
  8. Lawmakers Say Nvidia Scandal Highlights Broader Concern Over Chinese Access to US AI — Forbes AI

Read the Daily Brief: the day's most important AI stories →