AINewsnow

Who blocks AI crawlers? I checked the robots.txt of the top 1,000 sites

This story is from 2026-10-02. It is preserved in the archive; the latest stories are on the live feed.

Every big AI company runs a crawler. OpenAI has GPTBot, Anthropic has ClaudeBot, Google runs Google-Extended, Perplexity has PerplexityBot. A website can wave them through or turn them away with a few lines in a file called robots.txt. I wanted to know how many actually turn them away, so I pulled…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-02 15:04 · DEV Community — AI
    Who blocks AI crawlers? I checked the robots.txt of the top 1,000 sites

More stories

  1. OpenAI announces ‘dots’ agent after scrapping launch of new AI model over safety concerns — The Guardian AI
  2. Google announces Gemini 4 Argon AI model, but you can't use it yet — Ars Technica AI
  3. Google rolls out Gemini 4 Argon, its most advanced AI model — CNBC Technology
  4. Top AI and tech firms sign 'morally binding' accord to 'self-police' development after meeting at White House — Euronews Next
  5. Can Google's new model really catch up to OpenAI and Anthropic at the frontier? — CNBC Technology
  6. Holy smoke! Google cooked OpenAI and Anthropic — r/GeminiAI
  7. Trump gets Big Tech to sign AI safety pledge: What OpenAI, Google, Meta and Anthropic have agreed to do — Mint AI
  8. My Mom is So Reliant on AI that I’m Worried Her Mental Function is Declining — r/antiai

Get the daily brief of stories like this at 6:30 every morning →