Your AI eval is green because it never called the model
This story is from 2026-09-22. It is preserved in the archive; the latest stories are on the live feed.
We run an eval suite on every PR. Seven fixtures, four metrics, deterministic, no API key, no network, costs nothing. It was green for weeks. Then a code review found that two of the fixtures contained responses the live tool schema would have rejected outright. One had access nested inside a defin…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-09-22 06:57 · DEV Community — AI
Your AI eval is green because it never called the model