AINewsnow

How to Compare Quantized LLMs Fairly in llama.cpp: Same Bytes, Same Windows, Same GPU

Here are two GGUF files of Qwen3-4B, both sold as "2-bit". One scores a WikiText-2 perplexity of 28.61, the other 16.64. Is the second one better? Maybe. Before you believe a gap like that, you need three facts: how many bytes each file really takes, which text windows were scored and how, and whet…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-25 12:46 · DEV Community — Machine Learning
    How to Compare Quantized LLMs Fairly in llama.cpp: Same Bytes, Same Windows, Same GPU

More stories

  1. Qwen 3.8 Flash Next (Q4) got stuck on this loop for hours. I wonder if this is the AI equivalent of having a nightmare and not being able to wake up — r/LocalLLM
  2. GitHub - vincalkr/DS41Engine: DeepSeek-V4.1-Flash 1017 on 1x-2x RX 7900 XTX (gfx1100, ROCm/HIP): max optimizations — r/LocalLLM
  3. Massive B70 spec-decoding improvement - Intel XMX matrix engines refactored by budiony · Pull Request #29864 · ggml-org/llama.cpp — r/LocalLLaMA
  4. I ran local AI inference (AMD GPU) for 30 days and measured every watt — here's the real cost vs cloud APIs — r/LocalLLM
  5. LLM on Android Phone updates — r/LocalLLM
  6. I ran a blind test on local AI agents. Here are the results, guess who's who — r/AI_Agents
  7. What's the best front-end for LLM assisted studying? — r/LocalLLM
  8. How to Run Underdog Saluki 27B: 2-Bit Qwen3.8-27B Tool-Calling Guide — MarkTechPost

Get the daily brief of stories like this at 6:30 every morning →