Two thirds of sites blocking GPTBot are also blocking the crawler that brings them traffic
Blocking GPTBot and blocking OAI-SearchBot are not the same decision. Most robots.txt files treat them as if they were, and the cost of that confusion is invisible: nothing in your analytics will ever tell you about the visitor who never arrived. GPTBot collects pages to train a model. Deciding you…
Read the full story at DEV Community — AI ↗
Timeline · 2 reports
- 2026-11-08 12:30 · DEV Community — AI
I measured 528 sites' robots.txt: two thirds blocking GPTBot also block the crawler that cites them - 2026-11-07 12:03 · DEV Community — AI
Two thirds of sites blocking GPTBot are also blocking the crawler that brings them traffic