AINewsnow

Our store's robots.txt says no to AI training and yes to AI search. Here's what that actually means

I opened our own robots.txt this week expecting a few lines. It's mostly a list of bots we turned away, and I didn't write any of it by hand. The store runs behind Cloudflare, and Cloudflare's managed robots.txt is switched on. It prepends a block of rules and keeps it updated as new crawlers show…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-12 17:02 · DEV Community — AI
    Our store's robots.txt says no to AI training and yes to AI search. Here's what that actually means

More stories

  1. Nvidia CEO Jensen Huang resigns amid affair reports; Colette Kress seen as successor — CSET Georgetown
  2. Anthropic launches Claude Fable 5.2, topping benchmarks and drawing developer comparisons — Anthropic (X)
  3. AWS releases local emulator of 200+ services written entirely by AI agents — AWS Machine Learning Blog
  4. Google DeepMind launches AlphaGenome 2 to predict genetic variant effects by cell type — Google DeepMind Blog
  5. Google DeepMind's AlphaProof Code formally verifies Signal's cryptographic code in four weeks — Google DeepMind Blog
  6. Scientists use Google AI to help restore giant kelp forests — Google AI Blog
  7. AI data center boom accelerates globally, from China's state-backed buildout to Europe — DigiTimes Asia
  8. Google Opens Pre-Orders for Gemini-Powered Intelligent Eyewear Starting at $395 — Google AI Blog

Get the daily brief of stories like this at 6:30 every morning →