I checked robots.txt on the top 5,000 sites. 238 of them block ChatGPT citations by accident.
This story is from 2026-09-08. It is preserved in the archive; the latest stories are on the live feed.
On 7 September 2026 I fetched the robots.txt file of the 5,000 highest ranked sites in the Tranco list. 2,771 of them served one. This is what those files say about 19 AI crawlers, and the raw JSON is linked at the bottom. Method: each robots.txt was fetched once with an identified user agent, no i…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-09-08 22:27 · DEV Community — AI
I checked robots.txt on the top 5,000 sites. 238 of them block ChatGPT citations by accident.