robots.txt alone may not control AI crawler access to websites
This story is from 2026-11-13. It is preserved in the archive; the latest stories are on the live feed.
Publishers are learning that robots.txt only governs crawler permissions, while hosts, CDNs and firewalls decide actual access. AI firms run separate bots for training, search and user-triggered fetches, so blocking one does not block the others.
Read the full story at DEV Community — AI ↗
Timeline · 6 reports
- 2026-11-16 07:09 · DEV Community — AI
Is robots.txt enough to let AI crawlers in? (Usually not) - 2026-11-15 14:13 · DEV Community — AI
Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it - 2026-11-14 14:32 · DEV Community — AI
How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More - 2026-11-14 10:37 · DEV Community — AI
robots.txt for AI Crawlers: Which Bots to Allow and Which to Block - 2026-11-14 08:26 · DEV Community — AI
robots.txt for AI crawlers: let search bots in, keep training bots out - 2026-11-13 09:35 · DEV Community — AI
Can AI Crawlers Read Your Website? robots.txt Is Only Half the Answer