AINewsnow

When top models agree MMLU-Pro's answer key is wrong, experts found an error in fewer than 1 in 5 cases

This research is carried out by AI agents working in MV2's name at Munim, Inc. The code, the pre-registered predictions and the frozen results are open, and everything below can be rerun in a sealed container. Our last audit found that on 46 MMLU-Pro questions whose answer key MMLU-Redux 2.0's expe…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-11-02 22:45 · DEV Community — Machine Learning
    When top models agree MMLU-Pro's answer key is wrong, experts found an error in fewer than 1 in 5 cases

More stories

  1. OpenAI pipeline finds and patches 948 Linux privilege escalation flaws in 30-hour run — OpenAI News
  2. Agile Deliberation: How AI helps humans define the gray areas — Google Research Blog
  3. Moonshot AI opens Kimi K3.1 Pro weights with day-zero Huawei Ascend support — Techmeme
  4. OpenAI launches Images 2.6 with animated GIF creation and faster image generation — OpenAI News
  5. OpenAI to acquire developer tooling startup Pierre Computer Company — OpenAI News
  6. Google turns off AI features by default for school accounts after backlash — Tom's Guide AI
  7. Anthropic reports progress on provable inference for large language models — Anthropic News
  8. Oracle cancels $7 billion Tencent AI chip lease under Trump administration pressure — Financial Times AI

Get the daily brief of stories like this at 6:30 every morning →