AINewsnow

How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More

This story is from 2026-11-14. It is preserved in the archive; the latest stories are on the live feed.

Originally published on ChorusWire . Each AI company documents separate robots.txt agents for training, search and user-triggered fetches. Blocking one does not block the others, and robots.txt is voluntary. To communicate that you do not want certain crawlers collecting your site's content for AI…

Read the full story at DEV Community — AI ↗

Timeline · 3 reports

  1. 2026-11-16 07:09 · DEV Community — AI
    Is robots.txt enough to let AI crawlers in? (Usually not)
  2. 2026-11-15 14:13 · DEV Community — AI
    Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it
  3. 2026-11-14 14:32 · DEV Community — AI
    How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More

More stories

  1. Is that really GPTBot? How to see which AI agents visit your site and which ones are fake — DEV Community — AI
  2. I'm not a developer, but I keep getting burned by AI answers in my work. How do you decide what to trust? — DEV Community — AI
  3. The AI Search Revolution: How Websites Must Prepare for ChatGPT, Claude & AI Search Engines — DEV Community — AI
  4. AI Visibility in 2026: The Human–Agent Playbook for SEO, AEO & GEO — DEV Community — AI
  5. We tracked 287 AI answers about AI-visibility tools. Here are the sources ChatGPT, Gemini, Perplexity and Claude actually cite — DEV Community — AI
  6. What is the best tool to track my brand in ChatGPT, Claude, Perplexity and Gemini? — DEV Community — AI
  7. Do AI chat engines block scrapers, and how do providers get around it? — DEV Community — AI
  8. ChatGPT, Claude, Gemini and Grok helped plan terror attacks in US government test — The Independent Tech

Read the Daily Brief: the day's most important AI stories →