AINewsnow

Publishers reassess robots.txt rules as AI crawlers split into training, search and fetch bots

A series of guides explains that OpenAI, Anthropic and Perplexity each run separate crawlers for training, search and user-triggered fetches, so blocking one bot in robots.txt does not block the others, and host or firewall settings ultimately decide access.

Read the full story at DEV Community — AI ↗

Timeline · 6 reports

  1. 2026-11-15 14:13 · DEV Community — AI
    Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it
  2. 2026-11-14 14:32 · DEV Community — AI
    How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More
  3. 2026-11-14 10:37 · DEV Community — AI
    robots.txt for AI Crawlers: Which Bots to Allow and Which to Block
  4. 2026-11-14 08:26 · DEV Community — AI
    robots.txt for AI crawlers: let search bots in, keep training bots out
  5. 2026-11-13 09:35 · DEV Community — AI
    Can AI Crawlers Read Your Website? robots.txt Is Only Half the Answer
  6. 2026-11-13 02:55 · DEV Community — AI
    How to Check Which AI Crawlers Your robots.txt Actually Blocks

More stories

  1. Google isn't the only Front Door anymore — DEV Community — AI
  2. We Scanned 370k+ Websites for AI Readiness: Here’s Why Most Fail (and How to Fix It) — DEV Community — AI
  3. Best AI Research Tools in 2027: 7 Compared (Perplexity, ChatGPT, Gemini, Claude, NotebookLM, Consensus, Elicit) — DEV Community — AI
  4. I built a Next.js testbed to see what AI crawlers can actually read — DEV Community — AI
  5. Your website might be invisible to ChatGPT (and you'd never know) — DEV Community — AI
  6. Can ChatGPT, Claude and Perplexity actually crawl your site? A 10-minute check — DEV Community — AI
  7. 500B tokens on one project: what an AI-agent decompilation story says about managing work, not just code — DEV Community — AI
  8. How does the RemotePC MCP server work? — DEV Community — AI

Get the daily brief of stories like this at 6:30 every morning →