AINewsnow

Say no to AI training, not to AI search: robots.txt per bot

This story is from 2026-10-20. It is preserved in the archive; the latest stories are on the live feed.

This note has an interactive demo. Try it in the original "Block the AI bots" sounds like one switch, but it isn't. Anthropic uses three different crawlers, one for training, one for search and one for requests a person makes in Claude, and robots.txt can treat each one differently. That means you…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-20 07:30 · DEV Community — AI
    Say no to AI training, not to AI search: robots.txt per bot

More stories

  1. Inception debuts Mercury 3, a diffusion LLM that it says tops GPT-6 Luna, Claude Haiku 5.5, and GLM-5.3 Flash on agentic benchmarks while being 4x to 8x faster — Techmeme
  2. Structured outputs for Anthropic Claude models on Amazon Bedrock — AWS Machine Learning Blog
  3. Anthropic says it’s committing 5% of its shares to a ‘public benefit fund’ — TechCrunch AI
  4. People are using AI to make their smart TVs dumb again — TechCrunch AI
  5. Unsloth Qwen 3.8 flash next is insane, can be compared with Opus 5.5 xhigh — r/LocalLLM
  6. Arc Pro B60 runs a 27B coding model at 53 tok/s: full recipe, flags and raw logs — r/LocalLLM
  7. Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming — r/LocalLLaMA
  8. How do you guys manage the 5-hour limit on Pro without wasting so much usage? — r/ClaudeAI

Get the daily brief of stories like this at 6:30 every morning →