I measured 528 sites' robots.txt: two thirds blocking GPTBot also block the crawler that cites them
Two thirds of the sites blocking GPTBot are also blocking the crawler that sends them traffic. I measured it. Blocking GPTBot is a reasonable decision. OpenAI documents it as the crawler that gathers training data, you get nothing back for it, and plenty of publishers have decided that trade isn't…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-08 12:30 · DEV Community — AI
I measured 528 sites' robots.txt: two thirds blocking GPTBot also block the crawler that cites them