Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it
This story is from 2026-11-15. It is preserved in the archive; the latest stories are on the live feed.
Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com A couple of years ago, robots.txt was mostly about search engines. Now a typical site's logs include a whole list of AI-related crawlers: OpenAI's, Anthropic's, Common Crawl's, Perplexity's, Apple's, ByteD…
Read the full story at DEV Community — AI ↗
Timeline · 2 reports
- 2026-11-16 07:09 · DEV Community — AI
Is robots.txt enough to let AI crawlers in? (Usually not) - 2026-11-15 14:13 · DEV Community — AI
Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it