AINewsnow

Your robots.txt Allows AI Crawlers. Your Server May Not.

This story is from 2026-10-28. It is preserved in the archive; the latest stories are on the live feed.

Originally published on Contento Solutions , where I write about SEO and AI visibility for service businesses. Nine of 18 agency sites turned an AI crawler away. None of the nine said so in robots.txt. I tested 12 AI crawlers against 18 marketing agencies that sell search visibility on 16 September…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-28 13:00 · DEV Community — AI
    Your robots.txt Allows AI Crawlers. Your Server May Not.

More stories

  1. Anthropic reveals plans for blockbuster IPO — Sky News Technology
  2. Meta's Waddle World is a Tamagotchi-like AI pet app — Engadget
  3. Anthropic's IPO filing shows soaring sales and steep losses — France 24 — Artificial Intelligence
  4. DeepMind unveils Genie 4 world model promising more realistic physics and humans — Ars Technica AI
  5. ChatGPT’s Advice to Cut Her Seizure Drug Led to Devastating Consequences — New York Times AI
  6. Artificial Analysis launches the Intelligence Index v5, with new coding and science benchmarks, built on private datasets; Claude Opus 5.5 (max) leads at 58 (@artificialanlys) — Techmeme
  7. Why Merge CEO Shensi Ding stays hands-on with Codex — OpenAI YouTube
  8. People are using AI to make expensive computer programs free. This could change everything — The Conversation AI (US)

Get the daily brief of stories like this at 6:30 every morning →