Your robots.txt might be hiding your site from ChatGPT. Here's how to check in 10 minutes
This story is from 2026-10-08. It is preserved in the archive; the latest stories are on the live feed.
A lot of sites added this to robots.txt some time in the last two years: User -agent: GPTBot Disallow: / User -agent: ClaudeBot Disallow: / User -agent: * Disallow: /api/ That's a reasonable choice if you don't want your content used to train models. But many teams also have a CDN rule, a WAF setti…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-08 11:37 · DEV Community — AI
Your robots.txt might be hiding your site from ChatGPT. Here's how to check in 10 minutes