AINewsnow

robots.txt for AI Crawlers: Which Bots to Allow and Which to Block

This story is from 2026-11-14. It is preserved in the archive; the latest stories are on the live feed.

TL;DR: OpenAI, Anthropic and Perplexity each run separate bots for training, search and user-triggered fetches. Block the training bot and you can still show up in AI search. Block the search bot and you disappear from that product's answers. User-triggered fetchers may ignore robots.txt entirely.…

Read the full story at DEV Community — AI ↗

Timeline · 4 reports

  1. 2026-11-16 07:09 · DEV Community — AI
    Is robots.txt enough to let AI crawlers in? (Usually not)
  2. 2026-11-15 14:13 · DEV Community — AI
    Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it
  3. 2026-11-14 14:32 · DEV Community — AI
    How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More
  4. 2026-11-14 10:37 · DEV Community — AI
    robots.txt for AI Crawlers: Which Bots to Allow and Which to Block

More stories

  1. Is that really GPTBot? How to see which AI agents visit your site and which ones are fake — DEV Community — AI
  2. I'm not a developer, but I keep getting burned by AI answers in my work. How do you decide what to trust? — DEV Community — AI
  3. The AI Search Revolution: How Websites Must Prepare for ChatGPT, Claude & AI Search Engines — DEV Community — AI
  4. AI Visibility in 2026: The Human–Agent Playbook for SEO, AEO & GEO — DEV Community — AI
  5. We tracked 287 AI answers about AI-visibility tools. Here are the sources ChatGPT, Gemini, Perplexity and Claude actually cite — DEV Community — AI
  6. What is the best tool to track my brand in ChatGPT, Claude, Perplexity and Gemini? — DEV Community — AI
  7. Do AI chat engines block scrapers, and how do providers get around it? — DEV Community — AI
  8. ChatGPT, Claude, Gemini and Grok helped plan terror attacks in US government test — The Independent Tech

Read the Daily Brief: the day's most important AI stories →