robots.txt for AI Crawlers: Which Bots to Allow and Which to Block
This story is from 2026-11-14. It is preserved in the archive; the latest stories are on the live feed.
TL;DR: OpenAI, Anthropic and Perplexity each run separate bots for training, search and user-triggered fetches. Block the training bot and you can still show up in AI search. Block the search bot and you disappear from that product's answers. User-triggered fetchers may ignore robots.txt entirely.…
Read the full story at DEV Community — AI ↗
Timeline · 4 reports
- 2026-11-16 07:09 · DEV Community — AI
Is robots.txt enough to let AI crawlers in? (Usually not) - 2026-11-15 14:13 · DEV Community — AI
Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it - 2026-11-14 14:32 · DEV Community — AI
How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More - 2026-11-14 10:37 · DEV Community — AI
robots.txt for AI Crawlers: Which Bots to Allow and Which to Block