LLM agent token cost: why a kept chat grows with the square of the steps, and the measured fix (prompt caching counted)
An agent that keeps its whole chat resends everything on every step, so the tokens it sends grow with the square of the number of steps. Holding the state on the host and showing the model the exact state keeps the prompt the size of the state. Measured on Gemini 3.8 Flash: 85 times fewer prompt to…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-18 05:08 · DEV Community — AI
LLM agent token cost: why a kept chat grows with the square of the steps, and the measured fix (prompt caching counted)