48-Hour Field Notes: My Homemade Eval Set Was Scoring My Prompt, Not the Model
This story is from 2026-10-27. It is preserved in the archive; the latest stories are on the live feed.
I spent two days comparing models on my own little eval set, and the winner changed every time I reordered the examples. That is not a model difference, that is my eval set leaking. Here is the setup. Free model access means I can afford three repeats per item instead of one, and a free server opti…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-27 13:55 · DEV Community — AI
48-Hour Field Notes: My Homemade Eval Set Was Scoring My Prompt, Not the Model