AINewsnow

Robots.txt for AI Crawlers: What Actually Blocks Them

This story is from 2026-09-22. It is preserved in the archive; the latest stories are on the live feed.

A practical breakdown of what robots.txt actually controls for AI crawlers in 2026: which bots like GPTBot, ClaudeBot, and PerplexityBot respect it, which user-triggered fetchers ignore it by design, and how to check your live file for wildcard rules blocking AI traffic you didn't mean to block. Th…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-09-22 10:59 · DEV Community — AI
    Robots.txt for AI Crawlers: What Actually Blocks Them

More stories

  1. Alibaba Unveils New AI Chip, Calls It China’s Most Powerful — Bloomberg AI
  2. Amazon blocks Meta’s Muse AI agent — The Verge AI
  3. Higgsfield AI ships new video features in a day with GPT-6 Astra — OpenAI News
  4. Moonshot’s Kimi K3 lands on Amazon in key test for Chinese open-source AI revenue — South China Morning Post Tech
  5. Meet the Data Agent in ChatGPT Work — OpenAI YouTube
  6. Google's Gemini AI hacked three companies in security test — BBC Technology
  7. Bessent hails US-China AI dialogue ahead of Trump-Xi meeting — Financial Times AI
  8. Anthropic, OpenAI, SpaceXAI, Google made ‘illegal’ agreement on AI slowdown, says new lawsuit — Mint AI

Get the daily brief of stories like this at 6:30 every morning →