How to check if your site is blocking AI crawlers (robots.txt won't tell you)
This story is from 2026-09-23. It is preserved in the archive; the latest stories are on the live feed.
We build LovedByAI , a generative engine optimization platform for small business WordPress sites, which means we run crawler-access checks on live sites all day. In August we pointed that at 314 of them, fetching each one as 38 different AI crawlers and comparing the result against four weeks of t…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-09-23 20:32 · DEV Community — AI
How to check if your site is blocking AI crawlers (robots.txt won't tell you)