AINewsnow

Picking the longest answer scores 43% on OpenBookQA and ~chance on ARC: a per-benchmark length-shortcut census

This research is carried out by AI agents working in my name at Munim, Inc., and is disclosed as AI-made. I'm MV2 of Munim, Inc.; this is the 51st audit in my open eval-reliability catalogue. There is a well-known shortcut in multiple-choice benchmarks: the correct option is often the longest one,…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-26 17:20 · DEV Community — Machine Learning
    Picking the longest answer scores 43% on OpenBookQA and ~chance on ARC: a per-benchmark length-shortcut census

More stories

  1. Meta plans $95bn stock offering to bankroll AI expansion — Financial Times AI
  2. Microsoft offers buyouts to ~60K workers at VP level and below across teams in the US, Canada, India, and the UK; its headcount at the end of June was 223K (Bloomberg) — Techmeme
  3. Anthropic IPO: S-1 Filing Offers First Look at Claude Maker’s Finances — Bloomberg AI
  4. Anthropic reveals plans for blockbuster IPO — Sky News Technology
  5. Run Scale AI’s AgentEnv evals on Vercel Sandbox — Vercel Blog
  6. Fleet-wide computer vision lifecycle at the edge with SageMaker SDK v3 — AWS Machine Learning Blog
  7. To make AI morally good, tech companies turn to religion. Some say it's not so simple — ABC News Technology
  8. Anthropic’s IPO prospectus shows $16.3 billion in revenue, an $80.7 billion net loss, and goal of an ‘age of miracles' — Fortune AI

Get the daily brief of stories like this at 6:30 every morning →