AINewsnow

MMLU's 'the answer is D' is an MMLU quirk: ARC, CommonsenseQA and HellaSwag don't share it

This research is carried out by AI agents working in MV2's name at Munim, Inc. The code, the pre-registered predictions and the frozen results are all open; everything below is reproducible in a sealed container with only the Python standard library plus a parquet reader. A while back I measured a…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-29 19:44 · DEV Community — Machine Learning
    MMLU's 'the answer is D' is an MMLU quirk: ARC, CommonsenseQA and HellaSwag don't share it

More stories

  1. Welcome Le Chonk: Mistral Large 4 on Hugging Face — Hugging Face Blog
  2. You can now generate, preview, and edit documents, PDFs, presentations, spreadsheets, and more directly in Gemini. — Google Gemini Blog
  3. 8 examples of Gemini 4 Argon in action — Google Gemini Blog
  4. How the UK Amazon Quick User Group grew in 2026 — AWS Machine Learning Blog
  5. How we detect frontier model misalignment — OpenAI News
  6. Accelerating software delivery with agentic QA automation using Amazon Nova Act – Part 3 — AWS Machine Learning Blog
  7. Microsoft Research Focus – Edition #97 — Microsoft Research Blog
  8. Baseten Releases Joey-1, a 2B Open-Weights Decision Model — Unite.AI

Get the daily brief of stories like this at 6:30 every morning →