Qwen3.5-9B running on a PS4’s GPU, offline, using its own MTP head for speculative decoding
I made this, so a quick note: it’s my open-source project. Qwen3.5-9B runs on a stock PS4 at 31–39 tok/s. The MTP head that ships with Qwen3.5 made the biggest difference: it drafts up to two tokens and the full model checks them in one pass, which took it from 22 to 31–39 tok/s on this old GPU. We…
Read the full story at r/LocalLLaMA ↗
Timeline · 1 report
- 2026-11-13 16:00 · r/LocalLLaMA
Qwen3.5-9B running on a PS4’s GPU, offline, using its own MTP head for speculative decoding