Speculative decoding: block drafter accepts ~7 tokens/step on code but only ~2.5 on prose. Ceiling or fixable?
Measured on Qwen3.8-27B (EXL3 4.0bpw target, DFlash2 block drafter quantized to EXL3 4bpw, 7-token drafts, greedy, output identical to plain decode), 2x T4: code: ~7.4 accepted tokens per verify step, 154 tok/s prose: ~2.5 accepted tokens per step, 35-37 tok/s plain decode: ~18-19 tok/s Someone on…
Read the full story at r/deeplearning ↗
Timeline · 1 report
- 2026-10-25 13:04 · r/deeplearning
Speculative decoding: block drafter accepts ~7 tokens/step on code but only ~2.5 on prose. Ceiling or fixable?