AINewsnow

How to Test an LLM App: Evals That Run in CI

This story is from 2026-10-22. It is preserved in the archive; the latest stories are on the live feed.

Collect about 50 real inputs, label each with the behaviour you expect instead of an exact answer, check what you can with plain code, use an LLM judge only for the rest and calibrate it against your own labels, then run the whole set on every pull request. That is enough to catch most regressions…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-22 17:10 · DEV Community — AI
    How to Test an LLM App: Evals That Run in CI

More stories

  1. OlmoEarth v1.3: State-of-the-art embeddings for the whole planet — Allen Institute for AI (Ai2)
  2. Uber brings ChatGPT Work to all employees with OpenAI — OpenAI News
  3. Claude Duet is Anthropic's answer to Meta Muse and OpenAI Dots — Engadget
  4. Write your own training loop — without managing GPUs — with Amazon SageMaker training session APIs — AWS Machine Learning Blog
  5. Stop guessing auto scaling targets for TTS on Amazon SageMaker AI — AWS Machine Learning Blog
  6. Cognition Becomes First Customer for NVIDIA Vera Rubin NVL72 on CoreWeave Cloud — CoreWeave Blog
  7. Ex-OpenAI Science Head Kevin Weil Launches Startup With $215 Million — Bloomberg AI
  8. Meta agrees deal to keep Zuckerberg in California despite wealth tax proposal — Financial Times AI

Get the daily brief of stories like this at 6:30 every morning →