Your robots.txt is probably blocking the wrong AI bots. Here's how to sort training, search and user fetchers
This story is from 2026-11-16. It is preserved in the archive; the latest stories are on the live feed.
TL;DR: "Block AI bots" is three different decisions, not one. Training crawlers, AI search crawlers and user-triggered fetchers each do a different job. Block the wrong one and your product disappears from ChatGPT and Google's AI answers, while your content still ends up in training data anyway. Wh…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-16 08:49 · DEV Community — AI
Your robots.txt is probably blocking the wrong AI bots. Here's how to sort training, search and user fetchers