AINewsnow

Your robots.txt has an AI policy whether you wrote one or not

Most robots.txt files were written years ago to keep Googlebot out of /wp-admin/ or a staging folder. Since then, a dozen AI crawlers have arrived. Each has its own user-agent token, and each one controls something different. If your file never mentions them, you still have a policy: whatever each…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-04 17:28 · DEV Community — AI
    Your robots.txt has an AI policy whether you wrote one or not

More stories

  1. OpenAI to Acquire Developer Tools Startup Pierre Computer — OpenAI News
  2. Tooling ecosystem grows around running Claude Code and Codex autonomously and remotely — Runway Research
  3. Apple opens Siri AI to all developers with iOS 27.3 beta and new language support — Engadget
  4. Moonshot AI releases Kimi K3.1 Pro open weights as MIT reports weight-space programming gains — Techmeme
  5. Google to cut off Search for schools that reject AI, drawing backlash — Forbes AI
  6. Tencent launches Hy4 open-weight AI model with vision and computer-use capabilities — SCMP China Tech
  7. Democratic Midterm Wins Cloud Outlook for US AI Data Center Expansion — DigiTimes Asia
  8. Voters in 10 States Decide on AI Data Center Restrictions in Midterm Elections — Barron's

Get the daily brief of stories like this at 6:30 every morning →