AINewsnow

Looking for a card to run Qwen 125B IQ4 at >70tok/s and 256k context

I've recently discovered that strata can make local inference not boring. I'm getting 2100tok/s 69tok/s speeds with the default IQ3 125B model with 128k context. Speed goes down to 1400tok/s 40tok/s for Unsloth IQ4_K_XL (also Qwen 125B with 128 context). Unfortunately my card is only 12GB and conte…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-10-21 15:46 · r/LocalLLM
    Looking for a card to run Qwen 125B IQ4 at >70tok/s and 256k context

More stories

  1. Qwen3.8-Flash-Next 125B Q2_0 on a 24GB M4 Mac mini: ~8.1 tok/s with Strata — r/LocalLLM
  2. Together Post-Training: frontier-grade model shaping, the way that fits you — Together AI Blog
  3. Qwen 3.8 Flash Next optimized for two CMP 170hx GPUs — r/LocalLLaMA
  4. Why do people keep referring to 128 GB RAM? — r/LocalLLM
  5. The Flux Klein model had a node called "reference latent"—and I used it to achieve very good style transfers (for example, transforming an image into a painting). Does anyone know if it’s possible to use "reference latent" with Qwen 2.1 ? — r/comfyui
  6. PUBG's creators just trained a Qwen competitor from scratch — r/LocalLLM
  7. LoRA Training for Fibo 2.0 Added — r/StableDiffusion
  8. Image editing recommendations for old hardware — r/comfyui

Get the daily brief of stories like this at 6:30 every morning →