A hot-expert cache made a 181 GiB model faster on one 96 GiB GPU
The build of GLM-5.3-Flash I run is 181 GiB, and my GPU has 96 GiB. It still writes up to 16.5 tokens a second on one desktop. Offloading more on its own made it slower; keeping each layer's hot experts on the card made it faster. One of the eight bugs on the way gave no error at all. The model jus…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-10-21 19:22 · DEV Community — Machine Learning
A hot-expert cache made a 181 GiB model faster on one 96 GiB GPU