AINewsnow

AI training: one robots.txt line to say no

First published in GetPack Magazine , sourced guides to use AI well as a student. Until last week, telling an AI company not to train on your site meant one line per crawler: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, CCBot for Common Crawl, and another one every time a…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-03 17:30 · DEV Community — AI
    AI training: one robots.txt line to say no

More stories

  1. Is your Lovable or Bolt app invisible to ChatGPT? Check what AI crawlers actually see — DEV Community — AI
  2. How We Built an Automated /llms.txt + 955-Node 301 Shield Pipeline to Rank #1 in ChatGPT, Perplexity & Google (2026) [Nov 04] — DEV Community — AI
  3. How to use MCP in the Web Browser (ChatGPT, Claude, DeepSeek & Gemini) with Web MCP Bridge — DEV Community — AI
  4. I think I finally figured it out — r/GeminiAI
  5. Can AI Assistants Actually Read Your Website? A 15-Minute Access Check — DEV Community — AI
  6. Claude vs. ChatGPT vs. Gemini for business: How to choose the right AI model — DEV Community — AI
  7. Comparing Batch and Real-Time LLM Inference Economics — DEV Community — AI
  8. Which AI chatbot does each industry depend on? Referral shares for 2,429 top websites (September 2026) — DEV Community — AI

Get the daily brief of stories like this at 6:30 every morning →