Which AI crawlers do 261 top websites block? Study data
This story is from 2026-10-15. It is preserved in the archive; the latest stories are on the live feed.
We checked the robots.txt files of 285 of the most visited websites to see how they treat AI crawlers. 261 of them returned a readable file. This post sums up what we found. The full report and the raw data are linked at the end. The headline numbers 59 of the 261 sites (22.6%) disallow GPTBot, the…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-15 13:35 · DEV Community — AI
Which AI crawlers do 261 top websites block? Study data