llama.cpp was copying 643 MB to the CPU per request. Disabling state saving nearly doubled throughput
We were running Qwen3.8-27B on an H100 with llama.cpp and 16 concurrent clients. Throughput was 58 tokens/sec, time to first token was almost two seconds, and nvidia-smi showed around 35% GPU utilization. I build Tensward, an open-source inference analysis and tuning tool. While investigating this,…
Read the full story at r/LocalLLM ↗
Timeline · 1 report
- 2026-11-15 14:58 · r/LocalLLM
llama.cpp was copying 643 MB to the CPU per request. Disabling state saving nearly doubled throughput