Using Q38FN with dflash2? Try lower draft model quants.
Back with Qwen 3.8 27B i heard the advice that a Q4 draft model would produce roughly the same acceptance rate as a Q8 or higher. I have had my agent create a Q38FN Draft model in F16, Q8 and Q4, and the Q4 seemed to deliver the biggest speedup. I thus wanted to make it known that if you use Q38FN…
Read the full story at r/LocalLLaMA ↗
Timeline · 1 report
- 2026-10-27 10:35 · r/LocalLLaMA
Using Q38FN with dflash2? Try lower draft model quants.