AINewsnow

Qwen3-4B in 1.24 GB at perplexity 16.6 (IQ2_XXS at the same size: 28.6) - a tiny 8-state trellis quant for llama.cpp

Hey r/LocalLLM ! We made a 2-bit weight format for llama.cpp that fits Qwen3-4B in 1.24 GB at WikiText-2 perplexity 16.6 . At the same file size, IQ2_XXS gets 28.6 and ik_llama.cpp's IQ2_KT gets 18.1 . The trick is a tiny 8-state trellis code (Ungerboeck, 1982) over groups of 4 weights, after a Had…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-10-19 19:44 · r/LocalLLM
    Qwen3-4B in 1.24 GB at perplexity 16.6 (IQ2_XXS at the same size: 28.6) - a tiny 8-state trellis quant for llama.cpp

More stories

  1. DeepSeek V4.1 Flash 1017 now available on AI Gateway — Vercel Blog
  2. I don't get the qwen 3.8 flash next hype — r/LocalLLaMA
  3. Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming — r/LocalLLM
  4. WHIRL 0.2.0 now runs on the AMD Ryzen AI Max+ 395 (Radeon 8060S): native Windows LLM engine, faster than llama.cpp ROCm and Vulkan from 2k to ~250k tokens (also on the Radeon AI PRO R9700) — r/LocalLLM
  5. llama.cpp support for Cohere models — r/LocalLLaMA
  6. Budget rig for local LLM inference + distillation: 2x RTX 3060 12GB vs 2x Tesla P100 16GB? — r/LocalLLM
  7. Is there an alternative to base llama.cpp for RDNA2 gpus in general with local LLM usage ? — r/LocalLLaMA
  8. llama server's default port changed from 8080 to 9931 — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →