Same Model, Same Scaffold, 10 Runs: Agent Benchmark Scores Still Move by Up to 6 Points
This story is from 2026-10-27. It is preserved in the archive; the latest stories are on the live feed.
Two teams run the same model, scaffold, and temperature on SWE-bench Verified, once each. Team A reports 32.4%; Team B reports 28.8%. Both post their logs. Neither is lying. The gap comes from one distribution. Table 1 of On Randomness in Agentic Evals has this row: DeepSWE-preview on nano-agent, t…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-10-27 20:56 · DEV Community — Machine Learning
Same Model, Same Scaffold, 10 Runs: Agent Benchmark Scores Still Move by Up to 6 Points