AINewsnow

PSA: llama.cpp -cram should be increased for agentic workflows (default is 8192)

Just a quick PSA. llama.cpp does have prompt caching. if you are running large context lengths and have long multiturn projects, increasing -cram can provide you with massive speedups. There is a point where context lengths can get so large that 8192mb is not enough and the whole context needs to b…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

  1. 2026-09-24 15:44 · r/LocalLLaMA
    PSA: llama.cpp -cram should be increased for agentic workflows (default is 8192)

More stories

  1. Transformers now runs llama.cpp quants — Hugging Face Blog
  2. Performance tune for gemma4-26b-a4b flash attention shape. by frobnitzem · Pull Request #28450 · ggml-org/llama.cpp · GitHub — r/LocalLLaMA
  3. I trained a 360M-param Python model from scratch on two workstation GPUs and wrote up every step, including the bugs — r/learnmachinelearning
  4. Qwen3.8-Flash-Next on 12GB VRAM - 65 tokens per second — r/LocalLLaMA
  5. Gufo: the all-in-one strix halo inference engine — r/LocalLLM
  6. My foray into local ai. Two BC-250 ex mining apus running Qwen3.6-35B-A3B Q4_K_M at 60 tok/s with 64k context — r/LocalLLaMA
  7. Stanford's MAttr Tops AI Interpretability Benchmark by Nearly 3x — AlphaSignal
  8. I turned Qwen3.8-27B Q2_64 + llama.cpp into a fully TypeSafe AI-compatible Jev-like system. OpenAI API still intact! World’s first Vision-enabled Jev-like model! <10 GB VRAM, 170 ms on an RTX 3090 and ~140 tok/s in chat. 76% vs. 88% Jev-1.13 Acc. on a diverse 22,000-request typed-decision benchmark — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →