AINewsnow

robots.txt for AI crawlers: block training, keep AI search, and test that it works

This story is from 2026-10-22. It is preserved in the archive; the latest stories are on the live feed.

At some point someone on your team will ask you to "block the AI bots". Before you add a blanket rule, it is worth knowing that the major AI providers run different crawlers for different purposes, and that blocking the wrong one removes your site from AI search answers you probably want to appear…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-22 15:00 · DEV Community — AI
    robots.txt for AI crawlers: block training, keep AI search, and test that it works

More stories

  1. OlmoEarth v1.3: State-of-the-art embeddings for the whole planet — Allen Institute for AI (Ai2)
  2. Anthropic takes on Muse with Claude Duet, its new AI agent — The Verge AI
  3. Barclays to cut 5,000 jobs as bank steps up AI adoption — Financial Times AI
  4. NVIDIA Unveils Minerva to Bring End-to-End Observability for AI Factories — NVIDIA Blog
  5. Enact, which develops full-stack tech for general-purpose robots, emerges from stealth with a $30M pre-seed from Mistral CEO Arthur Mensch, Yann LeCun, and more — Techmeme
  6. Running Frontier-Class AI Models Across 8 NVIDIA DGX Spark Nodes — NVIDIA Technical Blog
  7. Agents on Live Hardware: Reason Where You Must, Engineer Everywhere Else — CoreWeave Blog
  8. From Training to Serving: Reflection’s Beam on CoreWeave — CoreWeave Blog

Get the daily brief of stories like this at 6:30 every morning →