Is GPTBot Silently Blocked by Your Cloudflare WAF? How to Test and Fix AI Crawler Access
This story is from 2026-10-04. It is preserved in the archive; the latest stories are on the live feed.
You launch a site, set up your robots.txt to welcome AI search bots, and move on: User-agent: GPTBot Allow: / A few weeks later, you check your server access logs or wonder why your content never gets cited in ChatGPT Search or Perplexity. You find zero crawler hits. The culprit is almost never you…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-04 14:08 · DEV Community — AI
Is GPTBot Silently Blocked by Your Cloudflare WAF? How to Test and Fix AI Crawler Access