AINewsnow

Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it

This story is from 2026-11-15. It is preserved in the archive; the latest stories are on the live feed.

Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com A couple of years ago, robots.txt was mostly about search engines. Now a typical site's logs include a whole list of AI-related crawlers: OpenAI's, Anthropic's, Common Crawl's, Perplexity's, Apple's, ByteD…

Read the full story at DEV Community — AI ↗

Timeline · 2 reports

  1. 2026-11-16 07:09 · DEV Community — AI
    Is robots.txt enough to let AI crawlers in? (Usually not)
  2. 2026-11-15 14:13 · DEV Community — AI
    Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it

More stories

  1. Is that really GPTBot? How to see which AI agents visit your site and which ones are fake — DEV Community — AI
  2. I checked 10 SEO agency websites for AI-search readiness. The real bugs were the boring ones. — DEV Community — AI
  3. I'm not a developer, but I keep getting burned by AI answers in my work. How do you decide what to trust? — DEV Community — AI
  4. The AI Search Revolution: How Websites Must Prepare for ChatGPT, Claude & AI Search Engines — DEV Community — AI
  5. AI Visibility in 2026: The Human–Agent Playbook for SEO, AEO & GEO — DEV Community — AI
  6. We tracked 287 AI answers about AI-visibility tools. Here are the sources ChatGPT, Gemini, Perplexity and Claude actually cite — DEV Community — AI
  7. What is the best tool to track my brand in ChatGPT, Claude, Perplexity and Gemini? — DEV Community — AI
  8. Is your website visible to ChatGPT and Perplexity? A practical GEO checklist (and how to automate it) — DEV Community — AI

Read the Daily Brief: the day's most important AI stories →