750 sites allow ChatGPT's or Perplexity's crawler in robots.txt and refuse it at the server. Half are not on Cloudflare.
This story is from 2026-09-30. It is preserved in the archive; the latest stories are on the live feed.
A few posts back I measured 336 of 422 sites behind Cloudflare's managed bot rules refusing GPTBot and ClaudeBot at the edge with nothing in robots.txt to explain it ( post here ). Then I widened the test to 8,000 mid-ranked sites and found 178 that allow OAI-SearchBot or PerplexityBot in robots.tx…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-09-30 03:26 · DEV Community — AI
750 sites allow ChatGPT's or Perplexity's crawler in robots.txt and refuse it at the server. Half are not on Cloudflare.
More stories
- AI chatbots predict how AI companions could reshape relationships by 2030 — Tom's Guide AI
- How to Check if ChatGPT Recommends Your Business (Manual Method + Free Tool) — DEV Community — AI
- 2026 AI tool roundups compare SEO, social, image, music, comic and security options — DEV Community — AI
- AI Does More Every Month. Skill Still Wins. Why We're All In on AI SEO — DEV Community — AI
- Free AI Discoverability Audit by AISONX for AI Search Presence and Performance — DEV Community — AI
- The SSAP Protocol: Why We Are Teaching AI to Stop "Talking" and Start "Saving" Travelers in China — DEV Community — AI
- How to tell if ChatGPT, Claude or Perplexity send visitors to your site (in any analytics tool) — DEV Community — AI
- 16 days of my own analytics: Google sent 1,791 visitors, ChatGPT 108, Perplexity 0. On one site ChatGPT beat Google, 47 to 28, all on narrow tool pages like a DKIM checker. Answer-shaped pages get the AI clicks. Anyone seeing Perplexity traffic? — DEV Community — AI
Get the daily brief of stories like this at 6:30 every morning →