AINewsnow

robots.txt for AI crawlers: let search bots in, keep training bots out

This story is from 2026-11-14. It is preserved in the archive; the latest stories are on the live feed.

You can block the bots that train AI models without disappearing from ChatGPT, Perplexity or Claude answers. The trick is to treat AI bots as two different groups and to name each bot explicitly in robots.txt. Two kinds of AI bots Search and assistant bots fetch your pages to answer a user's questi…

Read the full story at DEV Community — AI ↗

Timeline · 5 reports

  1. 2026-11-16 07:09 · DEV Community — AI
    Is robots.txt enough to let AI crawlers in? (Usually not)
  2. 2026-11-15 14:13 · DEV Community — AI
    Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it
  3. 2026-11-14 14:32 · DEV Community — AI
    How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More
  4. 2026-11-14 10:37 · DEV Community — AI
    robots.txt for AI Crawlers: Which Bots to Allow and Which to Block
  5. 2026-11-14 08:26 · DEV Community — AI
    robots.txt for AI crawlers: let search bots in, keep training bots out

More stories

  1. Is that really GPTBot? How to see which AI agents visit your site and which ones are fake — DEV Community — AI
  2. I'm not a developer, but I keep getting burned by AI answers in my work. How do you decide what to trust? — DEV Community — AI
  3. The AI Search Revolution: How Websites Must Prepare for ChatGPT, Claude & AI Search Engines — DEV Community — AI
  4. AI Visibility in 2026: The Human–Agent Playbook for SEO, AEO & GEO — DEV Community — AI
  5. We tracked 287 AI answers about AI-visibility tools. Here are the sources ChatGPT, Gemini, Perplexity and Claude actually cite — DEV Community — AI
  6. What is the best tool to track my brand in ChatGPT, Claude, Perplexity and Gemini? — DEV Community — AI
  7. Do AI chat engines block scrapers, and how do providers get around it? — DEV Community — AI
  8. ChatGPT, Claude, Gemini and Grok helped plan terror attacks in US government test — The Independent Tech

Read the Daily Brief: the day's most important AI stories →