AINewsnow

Blind review exposed how much our rubric depended on the first score

We had 480 model outputs reviewed by 3 people, which gave us 1440 total reviews. With prior scores and comments visible on the 2nd and 3rd passes, the report counted 1022 matching judgments across those 1440 reviews. We treated that as solid inter rater agreement and decided the rubric was stable.…

Read the full story at r/deeplearning ↗

Timeline · 1 report

  1. 2026-10-18 20:35 · r/deeplearning
    Blind review exposed how much our rubric depended on the first score

More stories

  1. DeepSeek V4.1 Flash 1017 now available on AI Gateway — Vercel Blog
  2. Anthropic commits billions to a 'public benefit fund' — Business Insider AI
  3. Anthropic proposes AI safety bill with high-risk thresholds and federal preemption — Politico Technology
  4. Microsoft cancels its deal to be the anchor tenant and rent 23K Nvidia AI chips at Nscale's Essex, UK, data center, which may not get grid power until the 2030s — Techmeme
  5. AI Gateway logs now show session affinity — Vercel Blog
  6. Consumer apprehension grows as US retailers adopt AI-assisted pricing tools — ABC News Technology
  7. Anthropic gives third-party auditors ’employee-like access’ in AI industry first — ZDNET AI
  8. NVIDIA RTX Spark Laptops Launch Globally with Local AI Support — NVIDIA Technical Blog

Get the daily brief of stories like this at 6:30 every morning →