AINewsnow

I wrote a Rust inference engine for Qwen models on a 12 GB GPU, decode is ~1.7x llama.cpp on my RTX3060

I've been working on this for a while: RustingStrata. It runs Qwen GGUF models (Qwen3.5/3.6 from 2B up to 35B-A3B, Qwen3 4B and 30B-A3B, Qwen3-Next 80B). It's pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan. The main trick is for MoE…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-10-31 17:55 · r/LocalLLM
    I wrote a Rust inference engine for Qwen models on a 12 GB GPU, decode is ~1.7x llama.cpp on my RTX3060

More stories

  1. Which coding agent gets the most out of a local Qwen 3.8 27B? I lined up 10 and logged tokens, cache and sandbox blocks — r/AI_Agents
  2. Apple Intelligence to launch in China with iOS 27.3 in December — The Verge AI
  3. Community tooling pushes Qwen 3.8 Flash Next to usable speeds on consumer hardware — r/LocalLLaMA
  4. Run a 35B to 180B LLM Without a GPU: VIDRAFT POCKET Passes 3.15 Million Downloads — DEV Community — AI
  5. 4 Strix Halo boxes! Finally joined local AI — r/LocalLLM
  6. Spooky Machine: A Local Qwen Ghost Story Generator — DEV Community — AI
  7. How to Run LFM3-Omni-2B GGUF Locally (Only llama.cpp Makes It Talk Back) — DEV Community — Machine Learning
  8. A game scripting language that runs speech-to-text 5x faster than llama.cpp on CPU — Lobsters AI

Get the daily brief of stories like this at 6:30 every morning →