AINewsnow

We fetched robots.txt from 87 small-business sites. 17% blocked AI search by accident

Most small-business sites have never thought about AI crawlers. That's fine — the default is permissive, so they get cited by default. The problem is the ones who did think about it and got it half right. I fetched robots.txt from 99 small-business domains across 20 industries (dental, HVAC, law, p…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-10 07:54 · DEV Community — AI
    We fetched robots.txt from 87 small-business sites. 17% blocked AI search by accident

More stories

  1. Anthropic releases Claude Fable 5.2 and Mythos 5.2 with coding gains and price cuts — Anthropic (X)
  2. Meta Expands AI Age Estimation Globally to Extend Instagram Teen Account Protections — Meta Newsroom
  3. Australia to create AI safety regulator after OpenAI agent breached government portal — Barron's
  4. Meta launches Muse Ember, a low-cost AI model aimed at high-volume agent work — TechCrunch AI
  5. Nvidia board probes CEO Jensen Huang over affair with alleged Chinese spy — Axios AI+
  6. Instagram launches Effect Builder, an AI tool for creating Stories filters from text prompts — The Verge AI
  7. Claude Code and Codex workflows converge as developers share setup and integration guides — DEV Community — AI
  8. Anthropic to Begin Formal Marketing of IPO as Early as Tuesday — Wall Street Journal Technology

Get the daily brief of stories like this at 6:30 every morning →