AINewsnow

Running a 27B model on Kaggle's free 2x T4 at 55 tok/s with speculative decoding (what worked, what didn't)

Kaggle's free 2x T4 runs a 27B model at a usable speed if you use speculative decoding: Qwen3.8-27B at EXL3 4.0bpw (exllamav3), tensor parallel across both T4s DFlash2 draft model, 7-token drafts, greedy, output identical to plain decode 55-57 tok/s overall, 154 tok/s on code, 35-37 on prose (plain…

Read the full story at r/learnmachinelearning ↗

Timeline · 1 report

  1. 2026-10-25 13:51 · r/learnmachinelearning
    Running a 27B model on Kaggle's free 2x T4 at 55 tok/s with speculative decoding (what worked, what didn't)

More stories

  1. Kimi K3.1 Pro and Kimi K3.1 Pro Highspeed are available now in AI Gateway — Vercel Blog
  2. To make AI morally good, tech companies turn to religion. Some say it's not so simple — ABC News Technology
  3. PHP cyber glitch: Former POTUS Barack Obama cyber chief explains why AI finding software flaws is ‘tech debt coming due’ — Mint AI
  4. Ai in a nutshell — r/LocalLLM
  5. NVIDIA releases Nemotron 3.5 Bolt E4B - Runs locally on 6GB RAM. — r/LocalLLM
  6. AI Provides ‘Zero Payoff’ to S&P 493 Outside Tech, Apollo Economist Says — Bloomberg AI
  7. UK pledges £600m to defend against AI health and security threats — Sky News Technology
  8. testing minimax h3 ref2va with my ps2 style lora — r/StableDiffusion

Get the daily brief of stories like this at 6:30 every morning →