AINewsnow

Your robots.txt is probably blocking the wrong AI bots. Here's how to sort training, search and user fetchers

This story is from 2026-11-16. It is preserved in the archive; the latest stories are on the live feed.

TL;DR: "Block AI bots" is three different decisions, not one. Training crawlers, AI search crawlers and user-triggered fetchers each do a different job. Block the wrong one and your product disappears from ChatGPT and Google's AI answers, while your content still ends up in training data anyway. Wh…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-16 08:49 · DEV Community — AI
    Your robots.txt is probably blocking the wrong AI bots. Here's how to sort training, search and user fetchers

More stories

  1. OpenAI launches GPT-6.2 Astra and Sol as Google ships Gemini 4 Neon and Microsoft unveils MAI Code 2 — OpenAI News
  2. ChatGPT, Claude, Gemini and Grok helped plan terror attacks in US government test — The Independent Tech
  3. ChatGPT and Gemini tested on holiday gift recommendations for a drummer who loves vinyl — Tom's Guide AI
  4. California Sues OpenAI For Allegedly Misleading and Endangering the Public — Forbes AI
  5. Harvard Study Finds US Hyperscale Data Centers Emit Carbon on Par With Denmark — CNET AI
  6. AI roundup: Microsoft agent cloud GA, NVIDIA $115.8B quarter, Vera Rubin tops MLPerf — DigiTimes Asia
  7. How Large Language Models Work: A Plain-English Guide to AI Chatbots — DEV Community — Machine Learning
  8. GAO test finds Claude, ChatGPT, Gemini, and Grok produced viable terror plans — FedScoop

Read the Daily Brief: the day's most important AI stories →