AINewsnow

I made llama.cpp's slot saves write only the changed chunks: a FUSE filesystem that dedups KV-cache snapshots

I run an agent on a local model behind a KV-cache proxy. After every answer the proxy has llama-server save the slot state (`/slots/0?action=save`), so the next request doesn't recompute the ~7K-token system prompt and tools. Each save is the full state, rewritten from scratch: 300 MB to 1.2 GB per…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-10-16 19:58 · r/LocalLLM
    I made llama.cpp's slot saves write only the changed chunks: a FUSE filesystem that dedups KV-cache snapshots

More stories

  1. What training actually does to an LLM — DEV Community — Machine Learning
  2. Llama.cpp on a jailbroken PS5: run any GGUF model natively on the console — Hacker News Front Page
  3. Fully local, private voice + chat AI assistant (zero cloud) — Qwen3.8-27B on 2× RTX PRO — r/LocalLLM
  4. Qwen3.8 Flash Next (Low Quant) vs 27B Q5: Is Strata Worth It on a 32GB VRAM + 32GB RAM System? — r/LocalLLM
  5. MeddeX: A Retrieval-Augmented LLaMA-3 System with Gemini Vision for Evidence-Backed Medical Report Analysis — arXiv cs.CL
  6. QUASAR 4-bit QAT checkpoints for Qwen3.5, Gemma 4 and Muse Glimmer: closer to BF16 than Google’s and NVIDIA’s 4-bit releases — r/LocalLLaMA
  7. Gemini leaked its thinking tokens by just asking what tag it uses to end thinking — r/ArtificialInteligence
  8. Ternary Bonsai 2 (2B, 4B, and 9B) just released on Hugging Face. The 2B can run entirely locally in the browser at up to 500 tok/s on an M4 Max with custom WebGPU kernels. — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →