AINewsnow

Speculative decoding: block drafter accepts ~7 tokens/step on code but only ~2.5 on prose. Ceiling or fixable?

Measured on Qwen3.8-27B (EXL3 4.0bpw target, DFlash2 block drafter quantized to EXL3 4bpw, 7-token drafts, greedy, output identical to plain decode), 2x T4: code: ~7.4 accepted tokens per verify step, 154 tok/s prose: ~2.5 accepted tokens per step, 35-37 tok/s plain decode: ~18-19 tok/s Someone on…

Read the full story at r/deeplearning ↗

Timeline · 1 report

  1. 2026-10-25 13:04 · r/deeplearning
    Speculative decoding: block drafter accepts ~7 tokens/step on code but only ~2.5 on prose. Ceiling or fixable?

More stories

  1. Kimi K3.1 Pro and Kimi K3.1 Pro Highspeed are available now in AI Gateway — Vercel Blog
  2. Two outsiders found ways OpenAI’s AI agents could slip past the system built to catch them — r/OpenAI
  3. Thomas L. Friedman: We Just Set Off the Second Big Bang — New York Times AI
  4. Qwen 3.8 Flash Next (Q4) got stuck on this loop for hours. I wonder if this is the AI equivalent of having a nightmare and not being able to wake up — r/LocalLLM
  5. GitHub - vincalkr/DS41Engine: DeepSeek-V4.1-Flash 1017 on 1x-2x RX 7900 XTX (gfx1100, ROCm/HIP): max optimizations — r/LocalLLM
  6. Securing our frontier model training environment — OpenAI News
  7. NCERT introduces 20-hour AI module for classes 9-12, covers deepfakes and data privacy — Mint AI
  8. Block unveils Dessa, a "true speech-to-speech" AI model trained on "nothing but human conversation" that it says runs locally on a MacBook Pro M4 Max or newer — Techmeme

Get the daily brief of stories like this at 6:30 every morning →