How I Test an LLM Feature: Field Notes on Golden Sets, LLM-as-Judge, and Eval Gates in CI

This story is from 2026-10-04. It is preserved in the archive; the latest stories are on the live feed.

Headline: An LLM feature cannot be unit-tested with string equality, because the same prompt returns different text on every call, so I assert on properties of the output instead of the output itself. The three changes that made my AI features safe to refactor were a version-controlled golden set o…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

1. 2026-10-04 06:00 · DEV Community — AI
   How I Test an LLM Feature: Field Notes on Golden Sets, LLM-as-Judge, and Eval Gates in CI