Is robots.txt enough to let AI crawlers in? (Usually not)
This story is from 2026-11-16. It is preserved in the archive; the latest stories are on the live feed.
More sites now ask a simple question: can ChatGPT, Claude and Perplexity actually read my pages? Most people open robots.txt , see nothing scary, and stop there. That check is not enough. Two layers can block an AI crawler robots.txt. A Disallow rule for a user agent like GPTBot or PerplexityBot .…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-16 07:09 · DEV Community — AI
Is robots.txt enough to let AI crawlers in? (Usually not)