AINewsnow

LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts

In a head-to-head eval, my new system prompt beat the old one in 66% of comparisons. I already had the "we're shipping v2" message typed. Then I swapped which answer went first and reran the exact same 300 pairs. The new prompt won 43%. The judge, the answers, the rubric and temperature 0 were all…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-11-01 12:15 · DEV Community — Machine Learning
    LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts

More stories

  1. DeepSeek open-sources dsh-superagent, a proactive coding agent prototype — Bloomberg AI
  2. Apple CEO John Ternus says company won't ship AI for AI's sake — Techmeme
  3. DeepSeek-V4.1-Flash-1017 release sparks quantization debate and local deployment benchmarks — r/LocalLLaMA
  4. Mistral releases weights for trillion-parameter Le Chonk model with 1M-token context — Wired AI
  5. Big Tech sheds nearly $900bn as AI spending fears trigger market rout — Financial Times AI
  6. Brazil courts data centers with tax breaks and cheap power as US opposition grows — Fortune AI
  7. Foxconn hints at new OpenAI tie-up for AI hardware, says no AI bubble — Economic Times Tech
  8. Nace AI open-sources Drex-Max v1, a 27B multimodal decision model for local use — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →