I gave an LLM a pull request and asked what would break. GitHub history had the answer key.
This story is from 2026-10-16. It is preserved in the archive; the latest stories are on the live feed.
Every "AI reviews your tickets" demo has the same hole in it: the output reads well whether or not it is true. Plausibility is the one thing a language model is reliably good at, so grading on it measures nothing. I wanted a number instead. Not "do these risks sound smart" but did it name the thing…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-16 11:04 · DEV Community — AI
I gave an LLM a pull request and asked what would break. GitHub history had the answer key.