Is that really GPTBot? Verify AI crawler traffic in your logs with 50 lines of Python
This story is from 2026-10-26. It is preserved in the archive; the latest stories are on the live feed.
Open any access log today and you'll see a parade of AI crawlers: GPTBot , OAI-SearchBot , ChatGPT-User , PerplexityBot . Before you decide to block them, allow them or celebrate them, it's worth asking a boring question: are they real? A user agent is just a string. Any scraper can send OAI-Search…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-26 10:57 · DEV Community — AI
Is that really GPTBot? Verify AI crawler traffic in your logs with 50 lines of Python