AINewsnow

We Read the robots.txt of the Top 10,000 Sites. Half That Block AI Training Also Block AI Search

Blocking GPTBot became a common move in 2023 and 2024. A site adds two lines to robots.txt, and OpenAI stops using its pages to train models. Plenty of teams did it, and it's a reasonable call. Some sites went further than GPTBot and blocked AI crawlers in bulk, including the ones that put a site i…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-05 19:42 · DEV Community — AI
    We Read the robots.txt of the Top 10,000 Sites. Half That Block AI Training Also Block AI Search

More stories

  1. OpenAI to keep expanding AI data centers despite election results favoring restrictions — DigiTimes Asia
  2. Claude Code drives wave of community-built apps, tools and workflows — DEV Community — AI
  3. How VistaPrint helps small businesses get the word out — OpenAI News
  4. OpenAI raises $11 billion at $1.25 trillion valuation — TechCrunch AI
  5. Man sues OpenAI, alleging ChatGPT helped his wife plan to poison him — Bloomberg AI
  6. OpenAI Introduces BioBasicsBench to Test AI Agents on Wet-Lab Science — OpenAI News
  7. OpenAI launches GPT-6 Cyber, a security model that found 81 Firefox flaws and 948 Linux bugs — VentureBeat AI
  8. Perplexity launches Teammates, always-on AI agents for ongoing work — The Verge AI

Get the daily brief of stories like this at 6:30 every morning →