I read all 28 AI crawlers' docs looking for crawl-delay. Only 4 say anything
This story is from 2026-09-13. It is preserved in the archive; the latest stories are on the live feed.
The received wisdom about crawl-delay is that it is dead. It was never in the original robots.txt specification, Google has said publicly that it ignores the directive, and most crawler parsers are modelled on Google's. So the standard advice is: do not bother, rate-limit at the CDN instead. I main…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-09-13 15:36 · DEV Community — AI
I read all 28 AI crawlers' docs looking for crawl-delay. Only 4 say anything