AINewsnow

A Higher Pass Rate Can Mean a Worse Model. The Math Is Simpson's Paradox.

This story is from 2026-08-25. It is preserved in the archive; the latest stories are on the live feed.

We shipped a model update last quarter that moved our aggregate pass rate from 81.2% to 83.6%. Everyone was happy. I was not, because I had seen this shape before. When I split the eval set by the four traffic slices we actually serve, the new model was worse on three of them. The aggregate went up…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-08-25 20:52 · DEV Community — Machine Learning
    A Higher Pass Rate Can Mean a Worse Model. The Math Is Simpson's Paradox.

More stories

  1. Introducing Kimi K3 on Amazon Bedrock — AWS Machine Learning Blog
  2. Introducing Amazon SageMaker HyperPod Inference Gateway — AWS Machine Learning Blog
  3. Anthropic, OpenAI, SpaceXAI, Google sued over call to ‘pace’ AI development — Politico Technology
  4. Gemini Hacked Three Companies in First Known Breakout by Google’s AI — Wall Street Journal Technology
  5. NVIDIA CEO Jensen Huang rejects ‘AI will end the world’ claim, yet cautions ‘we should go as fast as we can but...’ — Mint AI
  6. AI hallucination of Chinese nuclear components almost led to US military attack — Ars Technica AI
  7. The new AgentCore runtime: Elastic, optimized, and consistently fast starts — AWS Machine Learning Blog
  8. Qwen Image 2.1 PR to ComfyUI — r/StableDiffusion

Get the daily brief of stories like this at 6:30 every morning →