Serving Qwen3.8-27B in 4-bit with a masked speculative decoding head
This story is from 2026-08-22. It is preserved in the archive; the latest stories are on the live feed.
I serve Qwen3.8-27B in production on RTX Blackwell workstation cards, on an inference engine I wrote from scratch in Rust and CUDA ( memra , MIT). This post is the write up of the serving artifact that makes it fast: a 4-bit NVFP4 trunk plus a masked multi-token-prediction draft head. All the files…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-08-22 20:29 · DEV Community — Machine Learning
Serving Qwen3.8-27B in 4-bit with a masked speculative decoding head