Why LLM latency grows with conversation length, and the measured fix: a flat 0.1605 s to 0.1651 s step over 20,000 steps
LLM latency grows with conversation length because the prompt the model must read grows with it. If the host holds the state and shows the model the exact state instead of the history, the prompt stays the size of the state and the step time stays flat. Measured: a step took 0.1605 s early and 0.16…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-18 05:09 · DEV Community — AI
Why LLM latency grows with conversation length, and the measured fix: a flat 0.1605 s to 0.1651 s step over 20,000 steps