Your robots.txt probably does not say what you think it says about AI crawlers
I read 700 of the highest-ranked sites' robots.txt files on 9 November 2026, matched each one against the AI crawler tokens their operators document, and published the whole reading — domain by domain, with the exact line that decided each verdict — as open data. 407 of the 700 files could actually…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-16 00:01 · DEV Community — AI
Your robots.txt probably does not say what you think it says about AI crawlers