Publishers reassess robots.txt rules as AI crawlers split into training, search and fetch bots
A series of guides explains that OpenAI, Anthropic and Perplexity each run separate crawlers for training, search and user-triggered fetches, so blocking one bot in robots.txt does not block the others, and host or firewall settings ultimately decide access.
Read the full story at DEV Community — AI ↗
Timeline · 6 reports
- 2026-11-15 14:13 · DEV Community — AI
Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it - 2026-11-14 14:32 · DEV Community — AI
How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More - 2026-11-14 10:37 · DEV Community — AI
robots.txt for AI Crawlers: Which Bots to Allow and Which to Block - 2026-11-14 08:26 · DEV Community — AI
robots.txt for AI crawlers: let search bots in, keep training bots out - 2026-11-13 09:35 · DEV Community — AI
Can AI Crawlers Read Your Website? robots.txt Is Only Half the Answer - 2026-11-13 02:55 · DEV Community — AI
How to Check Which AI Crawlers Your robots.txt Actually Blocks