AINewsnow

750 sites allow ChatGPT's or Perplexity's crawler in robots.txt and refuse it at the server. Half are not on Cloudflare.

This story is from 2026-09-30. It is preserved in the archive; the latest stories are on the live feed.

A few posts back I measured 336 of 422 sites behind Cloudflare's managed bot rules refusing GPTBot and ClaudeBot at the edge with nothing in robots.txt to explain it ( post here ). Then I widened the test to 8,000 mid-ranked sites and found 178 that allow OAI-SearchBot or PerplexityBot in robots.tx…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-09-30 03:26 · DEV Community — AI
    750 sites allow ChatGPT's or Perplexity's crawler in robots.txt and refuse it at the server. Half are not on Cloudflare.

More stories

  1. AI chatbots predict how AI companions could reshape relationships by 2030 — Tom's Guide AI
  2. How to Check if ChatGPT Recommends Your Business (Manual Method + Free Tool) — DEV Community — AI
  3. 2026 AI tool roundups compare SEO, social, image, music, comic and security options — DEV Community — AI
  4. AI Does More Every Month. Skill Still Wins. Why We're All In on AI SEO — DEV Community — AI
  5. Free AI Discoverability Audit by AISONX for AI Search Presence and Performance — DEV Community — AI
  6. The SSAP Protocol: Why We Are Teaching AI to Stop "Talking" and Start "Saving" Travelers in China — DEV Community — AI
  7. How to tell if ChatGPT, Claude or Perplexity send visitors to your site (in any analytics tool) — DEV Community — AI
  8. 16 days of my own analytics: Google sent 1,791 visitors, ChatGPT 108, Perplexity 0. On one site ChatGPT beat Google, 47 to 28, all on narrow tool pages like a DKIM checker. Answer-shaped pages get the AI clicks. Anyone seeing Perplexity traffic? — DEV Community — AI

Get the daily brief of stories like this at 6:30 every morning →