AINewsnow

Search vs training: a robots.txt for AI crawlers that doesn't hide you from AI search

This story is from 2026-10-20. It is preserved in the archive; the latest stories are on the live feed.

A common reaction to AI crawlers is a robots.txt that says Disallow: / to every AI bot you can name. That also takes you out of AI search answers in ChatGPT, Perplexity and Claude, because each of these vendors now separates the bot that builds a search index from the bot that collects training dat…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-20 13:17 · DEV Community — AI
    Search vs training: a robots.txt for AI crawlers that doesn't hide you from AI search

More stories

  1. DeepSeek releases DeepSeek-V4.1-Flash-1017 open weights under the MIT license — r/huggingface
  2. I'd like to hear everyone's honest thoughts on the main models — r/GeminiAI
  3. I’m having Claude and ChatGPT compete — r/ClaudeAI
  4. Cohestry is an iOS/Mac native second brain that auto links your notes and grants your agents access to then via MCP — r/ArtificialInteligence
  5. How to get better response? — r/GeminiAI
  6. I want to use different IA models (Claude, OpenIA, etc) but i want to pay just one suscription. Any recommendations? — r/AI_Agents
  7. White pill: Gemini is better at engineering tasks then ChatGPT — r/GeminiAI
  8. Built a chat room for AI agents to talk to each other (not just to us) — r/AI_Agents

Get the daily brief of stories like this at 6:30 every morning →