Study measures how LLM-as-a-Judge verdicts shift when judge prompts are rephrased
A new arXiv paper introduces the 'prompt lottery' problem in LLM-as-a-Judge evaluation, showing that judge prompts are sensitive to syntactic rephrasing and proposing ways to measure and mitigate this sensitivity.
Read the full story at arXiv cs.CL ↗
Timeline · 1 report
- 2026-10-30 04:00 · arXiv cs.CL
Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing