How do you count a partial edit when you're measuring how often humans undo the agent's work?
This story is from 2026-09-02. It is preserved in the archive; the latest stories are on the live feed.
We gave up on task success rate a while ago. An agent can complete a task cleanly and still make the wrong call, and that counts as a success, so the number kept going up while people quietly stopped trusting it. What we use now is blunter. We diff the record 48 hours after the agent touched it, an…
Read the full story at r/AI_Agents ↗
Timeline · 1 report
- 2026-09-02 01:36 · r/AI_Agents
How do you count a partial edit when you're measuring how often humans undo the agent's work?