AINewsnow

robots.txt alone may not control AI crawler access to websites

This story is from 2026-11-13. It is preserved in the archive; the latest stories are on the live feed.

Publishers are learning that robots.txt only governs crawler permissions, while hosts, CDNs and firewalls decide actual access. AI firms run separate bots for training, search and user-triggered fetches, so blocking one does not block the others.

Read the full story at DEV Community — AI ↗

Timeline · 6 reports

  1. 2026-11-16 07:09 · DEV Community — AI
    Is robots.txt enough to let AI crawlers in? (Usually not)
  2. 2026-11-15 14:13 · DEV Community — AI
    Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it
  3. 2026-11-14 14:32 · DEV Community — AI
    How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More
  4. 2026-11-14 10:37 · DEV Community — AI
    robots.txt for AI Crawlers: Which Bots to Allow and Which to Block
  5. 2026-11-14 08:26 · DEV Community — AI
    robots.txt for AI crawlers: let search bots in, keep training bots out
  6. 2026-11-13 09:35 · DEV Community — AI
    Can AI Crawlers Read Your Website? robots.txt Is Only Half the Answer

More stories

  1. Perplexity Launches Orchestrator API With Same-Model Failover — Unite.AI
  2. Is that really GPTBot? How to see which AI agents visit your site and which ones are fake — DEV Community — AI
  3. I'm not a developer, but I keep getting burned by AI answers in my work. How do you decide what to trust? — DEV Community — AI
  4. The AI Search Revolution: How Websites Must Prepare for ChatGPT, Claude & AI Search Engines — DEV Community — AI
  5. AI Visibility in 2026: The Human–Agent Playbook for SEO, AEO & GEO — DEV Community — AI
  6. We tracked 287 AI answers about AI-visibility tools. Here are the sources ChatGPT, Gemini, Perplexity and Claude actually cite — DEV Community — AI
  7. What is the best tool to track my brand in ChatGPT, Claude, Perplexity and Gemini? — DEV Community — AI
  8. Do AI chat engines block scrapers, and how do providers get around it? — DEV Community — AI

Read the Daily Brief: the day's most important AI stories →