AINewsnow

SWE-bench Is Not Contaminated — But Its Test Files Were Public Years Before the Fixes

This story is from 2026-09-02. It is preserved in the archive; the latest stories are on the live feed.

Every AI coding benchmark gets the same accusation eventually: "the model has seen the answers." For SWE-bench — the 500-instance benchmark that became the default yardstick for coding agents — the accusation is unusually hard to dismiss, because the data comes from public GitHub issues. If a model…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-09-02 02:02 · DEV Community — AI
    SWE-bench Is Not Contaminated — But Its Test Files Were Public Years Before the Fixes

More stories

  1. Anthropic says Claude 'leads' 26 percent of its AI R&D work — Engadget
  2. Google announces new experimental "CC" AI agent for families — Ars Technica AI
  3. Novo Nordisk Will Use Anthropic’s Claude for Drug Research — Wall Street Journal Technology
  4. Introducing Kimi K3 on Amazon Bedrock — AWS Machine Learning Blog
  5. Optimizing agent system prompts with Amazon Bedrock AgentCore — AWS Machine Learning Blog
  6. Introducing Amazon SageMaker HyperPod Inference Gateway — AWS Machine Learning Blog
  7. Introducing Astra for Law — OpenAI News
  8. Gemini Hacked Three Companies in First Known Breakout by Google’s AI — Wall Street Journal Technology

Get the daily brief of stories like this at 6:30 every morning →