AINewsnow

Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming

I have been working on implementing adaptive KV streaming and RDNA 4 / ROCm optimizations for llama.cpp to enable running Qwen3.8-27B (IQ4_XS weights, Q8_0/Q4_0 KV cache) with its full 262,144-token context window on a single 16 GB RX 9070 XT GPU at 80+ t/s decode. Qwen 27B IQ4_XS is ~14.3 GB, leav…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-10-17 00:15 · r/LocalLLM
    Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming

More stories

  1. Ternary Bonsai 2 (2B, 4B, and 9B) just released on Hugging Face. The 2B can run entirely locally in the browser at up to 500 tok/s on an M4 Max with custom WebGPU kernels. — r/LocalLLaMA
  2. I built Agent Praxis: 6 adversarial environments to test how LLM agents handle traps, metric gaming, and fake authority — r/LocalLLM
  3. Community debates Qwen Flash Next performance and quantization options — r/LocalLLaMA
  4. Community tests reveal mixed results for Flux 3 Klein image model — Hugging Face Blog
  5. SpaceSlop Elysium GTA: The Reckoning — r/LocalLLM
  6. Looking for a harness to replace Copilot + VScode — r/LocalLLM
  7. Best harness/framework for local LLMs to run long, autonomous tasks without babysitting? — r/LocalLLM
  8. llama.cpp has come a long way — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →