What 4,700 bot visits taught me about AI crawlers (and a tool to check any site's robots.txt
A few weeks ago I wanted to answer a simple question: how many of the "visitors" on my small side projects are actually people? The answer turned out to be: not many. Most of the traffic was crawlers, a lot of them from AI companies, and I had no easy way to see which ones, how often, or where they…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-05 03:03 · DEV Community — AI
What 4,700 bot visits taught me about AI crawlers (and a tool to check any site's robots.txt