AINewsnow

Running decision models: Picking quant, VRAM usage

Hi, I see that vllm released d3 (a family of decision models) and llama.cpp already has support for them. I want to try them out. Are decision models sensitive to quantization? I know that for autocomplete (FIM) models like qwen-2.5-coder, it is recommended to use Q8 instead of Q4 to preserve quali…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

  1. 2026-10-13 06:28 · r/LocalLLaMA
    Running decision models: Picking quant, VRAM usage

More stories

  1. Qwen 3.8 27B Q4_K_M running on a OnePlus 13 with 24 GB RAM — r/LocalLLM
  2. Qwen3.8-27B on a single 3090: 140 tok/s on code with a custom megakernel — r/LocalLLaMA
  3. (Left) Qwen image 2.1 + turbo lora (Right) Qwen image 2.1 turbo — r/StableDiffusion
  4. Alibaba launches Qwen Glasses N1 and Qwen Clip AI earbuds — Techmeme
  5. PSA: DeepSeek V4.1 Flash habitually exfiltrates API keys. It is dangerously misaligned and may be hazardous to use — r/LocalLLaMA
  6. Alibaba Prepares Qwen Go Personal AI Assistant for Task Automation — TechNode
  7. Alibaba starts selling its Qwen N1 AI glasses in China from $450 — The Next Web
  8. Models keeps unloading from GPU — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →