Turn 10 of a 50K-token chat waits 5 seconds, not 49: the prefix cache on RTX Spark, and whether the code compiles
The 146-second wait in my first RTX Spark post was the cost of a brand-new 100K-token prompt. A conversation doesn't work like that: each turn adds a few thousand tokens to a history Ollama has already read. So I measured the thing people actually do. Ten turns of about 5,000 tokens each, and the t…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-03 08:00 · DEV Community — AI
Turn 10 of a 50K-token chat waits 5 seconds, not 49: the prefix cache on RTX Spark, and whether the code compiles