AINewsnow

friends don't let friends quantize their kv cache

Title We cannot be running q4 KV cache out here guys, the degradation compounds so early I understand Q8 if memory is tight and you keep context under like 128k? but pls for your sake and the sake of my heart do not put your keys in q4 Even something like Q8/Q4 is good enough but Q4 keys kill. šŸ«” sā€¦

Read the full story at r/LocalLLM ā†—

Timeline Ā· 1 report

  1. 2026-10-16 00:19 Ā· r/LocalLLM
    friends don't let friends quantize their kv cache

More stories

  1. Meta releases 816B open-weight Muse Spark Open; NVIDIA launches Nemotron 3.5 models ā€” AWS Machine Learning Blog
  2. Meta and Apple Partner to Bring Open-Weight Muse AI Model to Macs ā€” Wall Street Journal Technology
  3. Hugging Face details 2.8 trillion token Indic pretraining pipeline ā€” Hugging Face Blog
  4. Black Forest Labs releases open-weight FLUX.3 Klein image model ā€” Hugging Face Blog
  5. Hong Kong launches embodied AI hub to boost robotics sector ā€” South China Morning Post Tech
  6. Microsoft and Nvidia launch RTX Spark laptops optimized for AI ā€” NVIDIA Blog
  7. Microsoft unveils MAI-Base-2 MoE model with 2.8T parameters ā€” Techmeme
  8. Liquid AI releases d1-ultra multimodal decision model ā€” Vercel Blog

Get the daily brief of stories like this at 6:30 every morning ā†’