AINewsnow

Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included.

Write-up, serve command and raw data: https://github.com/lukolszewski/vllm This shows vLLM fixes and setup that let you put PLE on SSD and save 108GB of system RAM, use int8 kv and squeeze more users in your VRAM without the performace hit of fp8. Speeds in comparison to my earlier llama-multigpu p…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

  1. 2026-10-12 08:37 · r/LocalLLaMA
    Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included.

More stories

  1. Running AI Models in the Browser: Practical WebGPU/WASM Inference for Production Web Apps — DEV Community — Machine Learning
  2. Qwen 27B at ~20 tok/s on a jailbroken PS5's GPU. I forked PS5LM and turned it into a native app — r/LocalLLM
  3. Qwen3.8 Flash Next vs Qwen3.8 27B - The hype vs the truth. — r/LocalLLM
  4. Dual Rx 580 32GB ram 2600x — r/LocalLLM
  5. common : add --spec-draft-n-ubatch by ggerganov · Pull Request #30459 · ggml-org/llama.cpp — r/LocalLLaMA
  6. Running Qwen3.8-27B at 128K Context on an RTX 5060 Ti 16GB + GTX 1080 8GB — 21.9 tok/s — r/LocalLLM
  7. A couple of hours of work - serious speed improvements — r/LocalLLM
  8. I clustered two old Android phones to run a 27B LLM locally. Bypassed RAM limits via C++ RPC split and Hadamard Ternary math (No overheating) — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →