robots.txt makes two decisions, and most sites only make one
This story is from 2026-09-18. It is preserved in the archive; the latest stories are on the live feed.
A crawler arriving at your site is doing one of two jobs, and they have nothing in common: Answering a question right now — retrieving your page so an assistant can summarise or cite it. Filling a training corpus — taking the text away to shape a future model. Most robots.txt files on the web treat…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-09-18 02:34 · DEV Community — AI
robots.txt makes two decisions, and most sites only make one