AINewsnow

A benchmark you cannot train on, now a pay-per-call API

This story is from 2026-09-16. It is preserved in the archive; the latest stories are on the live feed.

Every few months a model "scores 92% on MMLU" and then fails in production on the exact things the benchmark said it could do. The score was real. The evaluation wasn't — the questions had been on GitHub and arXiv for years, and the model had memorized them. The usual answer is to rotate questions…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-09-16 03:02 · DEV Community — AI
    A benchmark you cannot train on, now a pay-per-call API

More stories

  1. OpenAI launches Images 2.6 in ChatGPT with animated GIF creation and faster generation — OpenAI News
  2. Anthropic unveils FlashProve prototype to cryptographically verify which AI model wrote a response — Runway Research
  3. OpenAI to Acquire Developer Tools Startup Pierre Computer — OpenAI News
  4. Cat Wu, product lead for Claude Code, leaves Anthropic — Business Insider AI
  5. Musk says Grok Bot costs are 'unsustainably high' as usage limits tighten — Forbes AI
  6. Exit Poll Finds AI and Data Centers Weighed Heavily in Midterm Votes — Barron's
  7. AI Data-Center Stocks Slide After Democratic Midterm Sweep, Renewables Rally — Barron's
  8. Apple opens Siri AI Developer Preview to all developers in iOS 27.3 beta — Engadget

Get the daily brief of stories like this at 6:30 every morning →