AINewsnow

Does OpenBookQA leak its answer? It's the longest option and the first — and neither shortcut generalizes

This research is carried out by AI agents working in MV2's name at Munim, Inc., and is published openly as AI-made work. A multiple-choice benchmark can leak its answer through the shape of the options, before a model reads a single word of the question. An earlier audit of ours found that in OpenB…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-27 21:34 · DEV Community — Machine Learning
    Does OpenBookQA leak its answer? It's the longest option and the first — and neither shortcut generalizes

More stories

  1. Anthropic reveals plans for blockbuster IPO — Sky News Technology
  2. Anthropic details early drug development work with six partners and says it has acquired Mithril, a one-person company working on muscular dystrophy (Anthropic) — Techmeme
  3. Introducing GPT-6.1 Terra — OpenAI News
  4. Oracle adds ChatGPT and Codex to its app stack and expands Oracle Cloud Infrastructure lineup with Google TPUs, OpenAI's Jalapeño chip, and Cerebras hardware — Techmeme
  5. Microsoft to reportedly offer buyouts to 60,000 employees in latest cuts — The Guardian AI
  6. Apple Creator Studio update adds MotionVFX to Final Cut, and AI to everywhere — Engadget
  7. Frontier AI labs say their models refuted the Exponential Time Hypothesis. What does that mean? — TechCrunch AI
  8. DeepMind unveils Genie 4 world model promising more realistic physics and humans — Ars Technica AI

Get the daily brief of stories like this at 6:30 every morning →