Is GPTBot Silently Blocked by Your Cloudflare WAF? How to Test and Fix AI Crawler Access
This story is from 2026-10-04. It is preserved in the archive; the latest stories are on the live feed.
You launch a site, set up your `robots.txt` to welcome AI search bots, and move on: txt User-agent: GPTBot Allow: / A few weeks later, you check your server access logs or wonder why your content never gets cited in ChatGPT Search or Perplexity. You find zero crawler hits. The culprit is almost nev…
Read the full story at DEV Community — AI ↗
Timeline · 2 reports
- 2026-10-04 14:08 · DEV Community — AI
Is GPTBot Silently Blocked by Your Cloudflare WAF? How to Test and Fix AI Crawler Access - 2026-10-04 13:58 · DEV Community — AI
Is GPTBot Silently Blocked by Your Cloudflare WAF? How to Test and Fix AI Crawler Access