AINewsnow

A "decontaminated" dataset still had 52 of HumanEval's 164 problems in it

We trained a 360M parameter code model from scratch and published 26.8% pass@1 on HumanEval. Two days later a reader asked whether we had checked our instruction data for benchmark overlap. We had not. So we did, and the honest number is 22.6% . Here is what was in the data. The scan Two checks aga…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-12 23:21 · DEV Community — Machine Learning
    A "decontaminated" dataset still had 52 of HumanEval's 164 problems in it

More stories

  1. Meta Launches Muse Wonder AI Models to Compete at Frontier — Meta Newsroom
  2. Microsoft President Says AI Is Less Regulated Than Milk — BBC Technology
  3. Epic Games delays game, mandates AI-first production model — Techmeme
  4. Anthropic AI Model Submits False Homicide Tip to Philadelphia Police — Mint AI
  5. NVIDIA Opens Vera Rubin Architecture to Scale AI Infrastructure — NVIDIA Blog
  6. Evidence on AI's climate impact remains mixed — The Conversation AI (US)
  7. Chinese mathematicians form alliance to develop specialized AI models — Techmeme
  8. Meta Unveils Liquid-Cooled Network Rack and 102.4T Switch at OCP Summit — Engineering at Meta

Get the daily brief of stories like this at 6:30 every morning →