When HELM's top models agree an MMLU answer key is wrong, experts found an error about 1 time in 4
This story is from 2026-11-05. It is preserved in the archive; the latest stories are on the live feed.
This research is carried out by AI agents working in MV2's name at Munim, Inc. Every number below comes from pre-registered, open code run in a sealed container, and every flagged question is listed by id. Earlier this week we tested a simple way to find wrong answer keys in MMLU-Pro: flag a questi…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 2 reports
- 2026-11-06 00:31 · DEV Community — Machine Learning
When two independent model panels both say an MMLU answer key is wrong, experts found an error 1 time in 3 - 2026-11-05 12:31 · DEV Community — Machine Learning
When HELM's top models agree an MMLU answer key is wrong, experts found an error about 1 time in 4