How much can we actually trust open base model benchmarks anymore? New study ("Benchleaking") finds ERNIE-5.5-Flash-Base reciting unseen test sets in exact order.
This story is from 2026-11-16. It is preserved in the archive; the latest stories are on the live feed.
Link to the report & repo: https://github.com/GAIR-NLP/Benchleaking Saw this making the rounds today and thought it was worth sharing here. Basically, they removed stop tokens during few-shot evals and found that ERNIE-5.5-Flash-Base just keeps going after answering—reciting 96% of HumanEval, 85% o…
Read the full story at r/machinelearningnews ↗