AINewsnow

Qwen3.8-27B UD-Q4_K_XL@262k context(66 tok/s at that depth) on a single 3090: 138 tok/s sampled decode (157 greedy, 233 code, 382 parallel),KL divergence @160k 0.00472 (q8_0: 0.00467)

Same card, same model file (unsloth's UD-Q4_K_XL), same client and prompts, one session, llama.cpp tuned at its fastest setting (4 MTP drafts, q8_0 KV): Short prompts: 1.5x to 2.1x, depending on the prompt. Writing code 233 vs 111 tok/s, a migration plan 159 vs 90, explaining a hash map 145 vs 85,…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-11-02 15:35 · r/LocalLLM
    Qwen3.8-27B UD-Q4_K_XL@262k context(66 tok/s at that depth) on a single 3090: 138 tok/s sampled decode (157 greedy, 233 code, 382 parallel),KL divergence @160k 0.00472 (q8_0: 0.00467)

More stories

  1. Running an LLM fully offline on Android with Kotlin coroutines — DEV Community — AI
  2. For those of you running both ComfyUI and local LLMs, how are you managing GPU resources? — r/StableDiffusion
  3. LFM3-1.7b seems impressive — r/LocalLLaMA
  4. 769 tok/s on a Single RTX 4090: Spark-X2.6-1.7B DSpark — DEV Community — Machine Learning
  5. Halving the KV Cache Doubles Capacity and Buys 1% of Speed — DEV Community — Machine Learning
  6. Are bespoke inference engines overhyped? My Strata experience on 2× R9700 — r/LocalLLaMA
  7. How we fit a 27B model on a 12 GB laptop GPU — DEV Community — Machine Learning
  8. Qwen3.6-35B-A3B on a laptop RTX 5050 8GB: ~1000 tok/s prefill and ~40-46 tok/s decode (config + steps) — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →