3.6x faster local LLM decoding under load, and the six rows where we lose
We run a small crew of AI agents on one mini PC. Two requests are in flight most of the day, so the number we care about is not how fast a model decodes alone. It is how fast it decodes while another request is loading a long prompt. That is where we built our own inference engine, unum-rt, to do b…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-15 21:33 · DEV Community — AI
3.6x faster local LLM decoding under load, and the six rows where we lose