robots.txt for AI crawlers: let search bots in, keep training bots out
This story is from 2026-11-14. It is preserved in the archive; the latest stories are on the live feed.
You can block the bots that train AI models without disappearing from ChatGPT, Perplexity or Claude answers. The trick is to treat AI bots as two different groups and to name each bot explicitly in robots.txt. Two kinds of AI bots Search and assistant bots fetch your pages to answer a user's questi…
Read the full story at DEV Community — AI ↗
Timeline · 5 reports
- 2026-11-16 07:09 · DEV Community — AI
Is robots.txt enough to let AI crawlers in? (Usually not) - 2026-11-15 14:13 · DEV Community — AI
Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it - 2026-11-14 14:32 · DEV Community — AI
How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More - 2026-11-14 10:37 · DEV Community — AI
robots.txt for AI Crawlers: Which Bots to Allow and Which to Block - 2026-11-14 08:26 · DEV Community — AI
robots.txt for AI crawlers: let search bots in, keep training bots out