AINewsnow

How much can we actually trust open base model benchmarks anymore? New study ("Benchleaking") finds ERNIE-5.5-Flash-Base reciting unseen test sets in exact order.

This story is from 2026-11-16. It is preserved in the archive; the latest stories are on the live feed.

Link to the report & repo: https://github.com/GAIR-NLP/Benchleaking Saw this making the rounds today and thought it was worth sharing here. Basically, they removed stop tokens during few-shot evals and found that ERNIE-5.5-Flash-Base just keeps going after answering—reciting 96% of HumanEval, 85% o…

Read the full story at r/machinelearningnews ↗

Timeline · 1 report

  1. 2026-11-16 16:07 · r/machinelearningnews
    How much can we actually trust open base model benchmarks anymore? New study ("Benchleaking") finds ERNIE-5.5-Flash-Base reciting unseen test sets in exact order.

More stories

  1. Baidu's Robin Li warns AI bubble risk if investment fails to deliver profits — Bloomberg AI
  2. Baidu and Xiaomi CEOs warn of widening gap between AI hype and returns — South China Morning Post Tech
  3. Baidu Q3 revenue slips 1% as AI cloud nearly doubles and search ads shrink — DigiTimes Asia
  4. OpenAI launches GPT-6.2 Astra and Sol as Google ships Gemini 4 Neon and Microsoft unveils MAI Code 2 — OpenAI News
  5. Google launches Gemini 4 Neon mid-tier model and opens Genie 4 to Gemini API developers — Google DeepMind YouTube
  6. Anthropic shares soar in record-breaking $107.3 billion IPO debut — Anthropic (X)
  7. Google books Blue Origin's New Glenn for 2029 launch of AI server satellites — Google: The Keyword
  8. AWS expands SageMaker AI with data preparation tools and cross-Region inference routing — AWS Machine Learning Blog

Read the Daily Brief: the day's most important AI stories →