Your LLM judge gives a different answer on re-runs. How do you test with it?
This story is from 2026-09-11. It is preserved in the archive; the latest stories are on the live feed.
Run the same LLM-as-judge eval twice and you can get pass, then fail, on identical input. Now try to build anything on top of that verdict. This is the problem mutation testing runs straight into when the eval is a judge, and here's how muteval deals with it. Why one run isn't a measurement Mutatio…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-09-11 14:41 · DEV Community — Machine Learning
Your LLM judge gives a different answer on re-runs. How do you test with it?