Requesting /robots.txt as GPTBot reads a clean file on 330 of 351 sites that refuse GPTBot at the homepage
This story is from 2026-09-26. It is preserved in the archive; the latest stories are on the live feed.
A common way to check whether a site blocks GPTBot is to request its robots.txt with GPTBot's user agent: curl -A "GPTBot" https://yoursite.com/robots.txt . I tried that check at scale on September 25, and it reads clean on most of the sites in my panel that turn GPTBot away. What I ran The panel i…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-09-26 07:31 · DEV Community — AI
Requesting /robots.txt as GPTBot reads a clean file on 330 of 351 sites that refuse GPTBot at the homepage