You can offload most of Qwen3.8-Flash-Next's KV cache to RAM with little decode slowdown
This story is from 2026-09-16. It is preserved in the archive; the latest stories are on the live feed.
I'm pretty sure it can be done with any model based on qwen4exp, which Qwen's next local models will be based on. You can use a quant that barely fits in VRAM and still run at the model's maximum context length without kv cache quantization, since most of the KV cache can live in system RAM. I actu…
Read the full story at r/LocalLLaMA ↗
Timeline · 1 report
- 2026-09-16 13:24 · r/LocalLLaMA
You can offload most of Qwen3.8-Flash-Next's KV cache to RAM with little decode slowdown