How sites answer GPTBot, classified in Python
This story is from 2026-10-15. It is preserved in the archive; the latest stories are on the live feed.
TLDR We sent the same homepage request to 2,096 sites twice, once as Chrome and once as GPTBot, from one US datacenter IP. A "block" turned out to be a 403, a 402 payment wall, a challenge page or a 429, and which one a crawler gets depends on the CDN in front of the site. The classifier and the pa…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-15 13:20 · DEV Community — AI
How sites answer GPTBot, classified in Python