AINewsnow

New quant types suggested: IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K

Looks quite interesting. Lots of numbers presented, but for a single model only. Highlights: File sizes are the same as Q-quants. KLD improves substantially over Q-quants. Performance suffers slightly Creating IQ quants is somewhat slow. Asterisks apply... https://github.com/ggml-org/llama.cpp/pull…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

  1. 2026-10-18 10:06 · r/LocalLLaMA
    New quant types suggested: IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K

More stories

  1. NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning — NVIDIA Technical Blog
  2. Notes from a 2 AM On-Call: When llama-swap's Model Reload Hit a 30s Timeout Wall — DEV Community — Machine Learning
  3. Round 3: Tested 128K context on a 3090 with real code (Django) and a novel. Finding one thing is solid, counting is hopeless — r/LocalLLM
  4. Qwen Ships Qwen3-VL 8B Locally, Seeing Images and Video in 5 GB — AlphaSignal
  5. Efforts to upstream adaptive MTP depth to llama.cpp abandoned — r/LocalLLM
  6. Has anyone used EXO in x86 machines, or am I just better off using llama.cpp or dllama? — r/LocalLLM
  7. Developer Mert Cobanov Runs llama.cpp on a Jailbroken PS5 — AlphaSignal
  8. Qwen3.8-27B at 42 tok/s single-user on one DGX Spark, fastest single-Spark entry on spark-arena, with 99 to 101% of BF16 quality — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →