AINewsnow

750 sites allow ChatGPT's or Perplexity's crawler in robots.txt and refuse it at the server. Half are not on Cloudflare.

This story is from 2026-09-30. It is preserved in the archive; the latest stories are on the live feed.

A few posts back I measured 336 of 422 sites behind Cloudflare's managed bot rules refusing GPTBot and ClaudeBot at the edge with nothing in robots.txt to explain it ( post here ). Then I widened the test to 8,000 mid-ranked sites and found 178 that allow OAI-SearchBot or PerplexityBot in robots.tx…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-09-30 03:26 · DEV Community — AI
    750 sites allow ChatGPT's or Perplexity's crawler in robots.txt and refuse it at the server. Half are not on Cloudflare.

More stories

  1. OpenAI launches Dots, its Muse competitor — The Verge AI
  2. OpenAI pauses AI training, launches ‘extensive’ review after multiple rogue agent incidents — Mint AI
  3. OpenAI Scraps Release of New AI Model Over Safety Concerns — Wall Street Journal Technology
  4. Meta Muse AI shares user's address on marketplace - Here is what went wrong and why it raises privacy concerns — Mint AI
  5. Bring near-Astra intelligence to everyday work with GPT-6.1 Sol on Amazon Bedrock — AWS Machine Learning Blog
  6. Introducing GPT-6.1 Sol — OpenAI News
  7. OpenAI DevDay: You Can Now Use GPT-6.1 Sol and Dots, Plus Big Subscription Changes — CNET AI
  8. OpenAI pulls the plug on GPT 6.1 Astra as agents keep crossing lines — InfoWorld AI

Get the daily brief of stories like this at 6:30 every morning →