AINewsnow

Is robots.txt enough to let AI crawlers in? (Usually not)

This story is from 2026-11-16. It is preserved in the archive; the latest stories are on the live feed.

More sites now ask a simple question: can ChatGPT, Claude and Perplexity actually read my pages? Most people open robots.txt , see nothing scary, and stop there. That check is not enough. Two layers can block an AI crawler robots.txt. A Disallow rule for a user agent like GPTBot or PerplexityBot .…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-16 07:09 · DEV Community — AI
    Is robots.txt enough to let AI crawlers in? (Usually not)

More stories

  1. How AI Answer Engines Read Your WordPress Site (and How to Make Them Read It Right) — DEV Community — AI
  2. Is that really GPTBot? How to see which AI agents visit your site and which ones are fake — DEV Community — AI
  3. I checked 10 SEO agency websites for AI-search readiness. The real bugs were the boring ones. — DEV Community — AI
  4. I'm not a developer, but I keep getting burned by AI answers in my work. How do you decide what to trust? — DEV Community — AI
  5. The AI Search Revolution: How Websites Must Prepare for ChatGPT, Claude & AI Search Engines — DEV Community — AI
  6. AI Visibility in 2026: The Human–Agent Playbook for SEO, AEO & GEO — DEV Community — AI
  7. We tracked 287 AI answers about AI-visibility tools. Here are the sources ChatGPT, Gemini, Perplexity and Claude actually cite — DEV Community — AI
  8. What is the best tool to track my brand in ChatGPT, Claude, Perplexity and Gemini? — DEV Community — AI

Read the Daily Brief: the day's most important AI stories →