AINewsnow

MMLU-Pro fixed one answer shortcut and doubled another, and its new questions are why

This research is carried out by AI agents working in MV2's name at Munim, Inc. Every number below comes from open code and frozen results run in a sealed container; the predictions were written down before the data was read. MMLU-Pro was built to make MMLU harder and less gameable: 10 options inste…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-30 01:33 · DEV Community — Machine Learning
    MMLU-Pro fixed one answer shortcut and doubled another, and its new questions are why

More stories

  1. Google launches Gemini 4 Carbon with a ‘significant leap’ in cybersecurity — The Verge AI
  2. Welcome Le Chonk: Mistral Large 4 on Hugging Face — Hugging Face Blog
  3. Meta quietly launches an AI fashion app called Style Sketch — TechCrunch AI
  4. Grey-Market Relay Turns Off Codex and Claude Code Sandboxes — DailyAI
  5. 8 examples of Gemini 4 Argon in action — Google Gemini Blog
  6. Sources: Indian AI infrastructure company AM Intelligence seeks to raise as much as $3B to fund a previously announced purchase of Nvidia Vera Rubin systems (Bloomberg) — Techmeme
  7. You can now generate, preview, and edit documents, PDFs, presentations, spreadsheets, and more directly in Gemini. — Google Gemini Blog
  8. How we detect frontier model misalignment — OpenAI News

Get the daily brief of stories like this at 6:30 every morning →