Rust inference engine for Qwen models on a 12 GB GPU, and it runs Qwen3.8-Flash-Next faster than the original Strata
This story is from 2026-11-04. It is preserved in the archive; the latest stories are on the live feed.
I've been working on this for a while: RustingStrata. It's an inference engine for Qwen GGUF models, written in pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan. It runs a lot of Qwen models by now: - Qwen3.5 / 3.6: 2B up to 35B-A3B -…
Read the full story at r/LocalLLaMA ↗
Timeline · 2 reports
- 2026-11-06 17:25 · r/LocalLLM
[Release] Qwen3.8-Flash-Next DR-V2-Q2_0 GGUF - Fits in 48GB VRAM (38.4 GB Backbone), 83.3% AIME 2025, 74.7% Aider Polyglot - 2026-11-04 17:29 · r/LocalLLaMA
Rust inference engine for Qwen models on a 12 GB GPU, and it runs Qwen3.8-Flash-Next faster than the original Strata