AINewsnow

Qwen3.8-Flash-Next on 12GB VRAM - 65 tokens per second

A while ago I posted 15 tok/s output and 100-120 tok/s prompt processing with the IQ3_XXS quant on a 12GB RTX 5070 using llama.cpp. Since then I built my own inference engine for this one model and this kind of PC. The same IQ3_XXS now runs at ~65 tok/s output and ~430 tok/s prompt processing , and…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

  1. 2026-09-24 17:30 · r/LocalLLaMA
    Qwen3.8-Flash-Next on 12GB VRAM - 65 tokens per second

More stories

  1. Transformers now runs llama.cpp quants — Hugging Face Blog
  2. Performance tune for gemma4-26b-a4b flash attention shape. by frobnitzem · Pull Request #28450 · ggml-org/llama.cpp · GitHub — r/LocalLLaMA
  3. I trained a 360M-param Python model from scratch on two workstation GPUs and wrote up every step, including the bugs — r/learnmachinelearning
  4. PSA: llama.cpp -cram should be increased for agentic workflows (default is 8192) — r/LocalLLaMA
  5. Gufo: the all-in-one strix halo inference engine — r/LocalLLM
  6. My foray into local ai. Two BC-250 ex mining apus running Qwen3.6-35B-A3B Q4_K_M at 60 tok/s with 64k context — r/LocalLLaMA
  7. Stanford's MAttr Tops AI Interpretability Benchmark by Nearly 3x — AlphaSignal
  8. I turned Qwen3.8-27B Q2_64 + llama.cpp into a fully TypeSafe AI-compatible Jev-like system. OpenAI API still intact! World’s first Vision-enabled Jev-like model! <10 GB VRAM, 170 ms on an RTX 3090 and ~140 tok/s in chat. 76% vs. 88% Jev-1.13 Acc. on a diverse 22,000-request typed-decision benchmark — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →