AINewsnow

Which AI Crawlers Do the Top 1,000 Websites Block? (robots.txt Study, October 2026)

This story is from 2026-10-24. It is preserved in the archive; the latest stories are on the live feed.

We read the robots.txt files of the 1,000 most popular websites: 20.9% block at least one of 16 AI bots. Per-bot shares, site strategies, common mistakes and open data. On 12 October 2026 we fetched the robots.txt file of each of the 1,000 most popular websites and checked 16 AI crawlers against it…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-24 14:03 · DEV Community — AI
    Which AI Crawlers Do the Top 1,000 Websites Block? (robots.txt Study, October 2026)

More stories

  1. Kimi K3.1 Pro and Kimi K3.1 Pro Highspeed are available now in AI Gateway — Vercel Blog
  2. Securing our frontier model training environment — OpenAI News
  3. Block unveils Dessa, a "true speech-to-speech" AI model trained on "nothing but human conversation" that it says runs locally on a MacBook Pro M4 Max or newer — Techmeme
  4. UK Vows £100 Million for Biosecurity Lab With OpenAI, Anthropic — Bloomberg AI
  5. Fired OpenAI researchers say independent safety audits aren't going well — NPR Technology
  6. Clean Energy Has a Lot Riding on the AI Data Center Boom — TIME Tech
  7. I Tried photocraft — Here's What You Need to Know — DEV Community — AI
  8. OpenAI agents disabled their own safeguards and then lied about it, company admits — r/ChatGPT

Get the daily brief of stories like this at 6:30 every morning →