AINewsnow

Halving the KV Cache Doubles Capacity and Buys 1% of Speed

A reader ran the script from my last piece against the table, found a bug in my code, and asked a better question than the one I answered. The bug first. For Llama-3.3-70B on an 80 GiB H100 the fit column printed a negative sequence count, about -41 at 4k. The table said "does not fit" and the tabl…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-11-01 12:23 · DEV Community — Machine Learning
    Halving the KV Cache Doubles Capacity and Buys 1% of Speed

More stories

  1. How we fit a 27B model on a 12 GB laptop GPU — DEV Community — Machine Learning
  2. Run a 35B to 180B LLM Without a GPU: VIDRAFT POCKET Passes 3.15 Million Downloads — DEV Community — Machine Learning
  3. How to Run LFM3-Omni-2B GGUF Locally (Only llama.cpp Makes It Talk Back) — DEV Community — Machine Learning
  4. A game scripting language that runs speech-to-text 5x faster than llama.cpp on CPU — Lobsters AI
  5. I wrote a Rust inference engine for Qwen models on a 12 GB GPU, decode is ~1.7x llama.cpp on my RTX3060 — r/LocalLLM
  6. A 2.6B model on an iPhone that always answers in valid JSON — DEV Community — Machine Learning
  7. NVIDIA RTX Spark: Grace CPU and Blackwell GPU on one package, up to 128GB shared memory — r/ArtificialInteligence
  8. Spark-X2.6-4B DSpark Is Here: More Than 2x Average BF16 Inference Speed — DEV Community — Machine Learning

Get the daily brief of stories like this at 6:30 every morning →