Is that really GPTBot? Checking AI crawlers against their official IP lists
This autumn we ran a six-week test with Giovanni, who builds a WordPress plugin that generates llms.txt and similar files for AI engines. The question was simple: do AI crawlers actually read these files? To answer it we had to count AI crawler requests in a server log, and that turned out to be th…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-17 13:20 · DEV Community — AI
Is that really GPTBot? Checking AI crawlers against their official IP lists