AINewsnow

Which websites block GPTBot and ClaudeBot? 30 sites checked (November 2026)

Short answer: most big sites block AI training crawlers, and they treat Anthropic more harshly than OpenAI. On 1 November 2026 we read the robots.txt of 30 well-known sites. 28 answered. Of those 28, 25 block ClaudeBot and 17 block GPTBot . Eight let GPTBot in and keep ClaudeBot out, including The…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-01 16:02 · DEV Community — AI
    Which websites block GPTBot and ClaudeBot? 30 sites checked (November 2026)

More stories

  1. DeepSeek open-sources dsh-superagent, a proactive coding agent prototype — Bloomberg AI
  2. Google, Microsoft, OpenAI and Anthropic to honor shared robots.txt AI training opt-out — OpenAI News
  3. Claude Code and Codex ecosystem grows as developers run multiple AI coding agents together — Microsoft Research Blog
  4. AnyModelBot: an open-source local AI bot that connects to 70 mainstream models with one key — DEV Community — AI
  5. We Benchmarked TTFT Across OpenAI, Anthropic, and Groq Over 10,000 Requests — Here Is the Data — DEV Community — Machine Learning
  6. Making sure the AI publishes the form I actually looked at — DEV Community — AI
  7. Your AI's memory should be yours, not the vendor's: how SKV works — DEV Community — AI
  8. ChatGPT, Claude, and Gemini experience simultaneous outages — Mashable AI

Get the daily brief of stories like this at 6:30 every morning →