A 763B MoE on a 12 GB GPU: Our Speed Ceiling Versus What Strata Reports, and What Closes the Gap
TL;DR Two days ago we wrote that a 763B mixture-of-experts model squeezed to 2 bits per weight could, on paper, run on a 12 GB GPU, and we put illustrative speed ceilings of 3.4 to 12 tokens per second on it by assuming every expert had to be fetched cold over PCIe or from an SSD. That post had one…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-11-03 15:08 · DEV Community — Machine Learning
A 763B MoE on a 12 GB GPU: Our Speed Ceiling Versus What Strata Reports, and What Closes the Gap