AINewsnow

Who blocks AI crawlers? I checked the robots.txt of the top 1,000 sites

This story is from 2026-10-02. It is preserved in the archive; the latest stories are on the live feed.

Every big AI company runs a crawler. OpenAI has GPTBot, Anthropic has ClaudeBot, Google runs Google-Extended, Perplexity has PerplexityBot. A website can wave them through or turn them away with a few lines in a file called robots.txt. I wanted to know how many actually turn them away, so I pulled…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-02 15:04 · DEV Community — AI
    Who blocks AI crawlers? I checked the robots.txt of the top 1,000 sites

More stories

  1. Perplexity Launches Orchestrator API With Same-Model Failover — Unite.AI
  2. Is that really GPTBot? How to see which AI agents visit your site and which ones are fake — DEV Community — AI
  3. I'm not a developer, but I keep getting burned by AI answers in my work. How do you decide what to trust? — DEV Community — AI
  4. The AI Search Revolution: How Websites Must Prepare for ChatGPT, Claude & AI Search Engines — DEV Community — AI
  5. AI Visibility in 2026: The Human–Agent Playbook for SEO, AEO & GEO — DEV Community — AI
  6. We tracked 287 AI answers about AI-visibility tools. Here are the sources ChatGPT, Gemini, Perplexity and Claude actually cite — DEV Community — AI
  7. What is the best tool to track my brand in ChatGPT, Claude, Perplexity and Gemini? — DEV Community — AI
  8. Do AI chat engines block scrapers, and how do providers get around it? — DEV Community — AI

Read the Daily Brief: the day's most important AI stories →