AINewsnow

Quantizing the KV cache in Ollama on RTX Spark: half the memory, a third slower

This story is from 2026-10-27. It is preserved in the archive; the latest stories are on the live feed.

I predicted that an 8-bit KV cache would take qwen3-coder:30b from 20 to 34 tokens a second at a 100K-token prompt, and a 4-bit cache to 52. On my Surface Laptop Ultra with RTX Spark, both made it slower: 13.8 tokens a second with q8_0 and 13.1 with q4_0, against 19.9 with the default f16 cache. Th…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-27 16:09 · DEV Community — AI
    Quantizing the KV cache in Ollama on RTX Spark: half the memory, a third slower

More stories

  1. DeepMind unveils Genie 4 world model promising more realistic physics and humans — Ars Technica AI
  2. Groundhog Day for AI agents: Prompt caching patterns for using OpenAI models on Amazon Bedrock — AWS Machine Learning Blog
  3. Microsoft to reportedly offer buyouts to 60,000 employees in latest cuts — The Guardian AI
  4. Introducing GPT-6.1 Terra — OpenAI News
  5. Meta announces $95B offering to fund AI — Semafor Technology
  6. Can we trust LLMs to judge whether a research idea is novel? — Allen Institute for AI (Ai2)
  7. Announcing SOCI-indexed images for faster SageMaker AI jobs — AWS Machine Learning Blog
  8. Apple unveils a 14" MacBook Pro with an M6, which has a 12-core CPU and a 12-core GPU, 16GB or 32GB of unified memory, and Wi-Fi 7, for $1,999+ from November 6 (Apple) — Techmeme

Get the daily brief of stories like this at 6:30 every morning →