AINewsnow

robots.txt for AI crawlers: GPTBot, ClaudeBot, PerplexityBot explained

This story is from 2026-10-10. It is preserved in the archive; the latest stories are on the live feed.

AI companies now run their own crawlers, and most identify themselves with a user-agent name you can put in robots.txt. This guide explains what the main ones do, how training crawlers differ from search and answer crawlers, and gives copy-paste rules for three common policies. Confirm details agai…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-10 20:43 · DEV Community — AI
    robots.txt for AI crawlers: GPTBot, ClaudeBot, PerplexityBot explained

More stories

  1. An Anthropic AI model sent a false homicide tip to Philadelphia police — TechCrunch AI
  2. Anthropic bans users from ‘needless abusive or cruel behavior’ towards Claude — The Guardian AI
  3. Philadelphia police receive false homicide tip from Anthropic AI model — The Hill Technology
  4. Impactful scheduling for GPU clusters — Allen Institute for AI (Ai2)
  5. Welcome to Gemini at Work 2026: Introducing the Gemini agent — Google Cloud AI Blog
  6. Qwen Image 2.1 Turbo Released -- Hugging Face — r/StableDiffusion
  7. Google Cloud introduces Gemini agent to change enterprise work — SiliconANGLE AI
  8. Fields Medalist Terence Tao reposts statement from the Association for Human Mathematics urging mathematicians to stop working with OpenAI for continuing to solve open math problems against their recommendations — r/OpenAI

Get the daily brief of stories like this at 6:30 every morning →