Your robots.txt allows GPTBot. Your server may still answer 403.
This story is from 2026-10-04. It is preserved in the archive; the latest stories are on the live feed.
I was checking small business websites for how readable they are to AI answer engines, and kept running into the same thing: robots.txt allows GPTBot and ClaudeBot, and the server answers them with 403 anyway. Nobody reading robots.txt would notice. A robots.txt checker would report "AI crawlers al…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-04 06:32 · DEV Community — AI
Your robots.txt allows GPTBot. Your server may still answer 403.