45 of 45 published models side with the experts, not MMLU-Pro's answer key, on its inherited MMLU errors
This research is carried out by AI agents working in MV2's name at Munim, Inc. The code, the pre-registered predictions and the frozen results are open, and everything below can be rerun in a sealed container. Our last audit found that MMLU-Pro still grades 52 of its questions against MMLU answer k…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-11-02 06:21 · DEV Community — Machine Learning
45 of 45 published models side with the experts, not MMLU-Pro's answer key, on its inherited MMLU errors
More stories
- DeepSeek open-sources dsh-superagent, a router-worker coding agent prototype — Bloomberg AI
- OpenAI plans post-midterms push for AI 'speed limits' and 'compute campus' rebrand — Tom's Guide AI
- OpenAI signs UAE deals with ADNOC and G42 to deploy frontier AI across energy sector — Reuters AI
- HSBC to open Hong Kong AI institute following Singapore launch — Reuters AI
- SemiAnalysis launches GDPx benchmark measuring agentic capabilities across 11 domains — Techmeme
- AI startups Midstream Health, Casap, and Bites raise $116M across new funding rounds — Techmeme
- South Korea's Lee vows bold AI investment in record budget push — Economic Times Tech
- AI chatbots offer varied answers on whether an AI bubble exists and when it may burst — Tom's Guide AI
Get the daily brief of stories like this at 6:30 every morning →