AINewsnow

Your robots.txt says yes. Your CDN says no. (We tested the top 1,000 sites)

This story is from 2026-10-15. It is preserved in the archive; the latest stories are on the live feed.

Most "do AI crawlers block" articles only read robots.txt. That file is a request, not a lock. The server, the CDN and the bot-management layer decide what a crawler actually gets, and nobody was measuring that part. So on 11 October 2026 we did. The one-line result: of the 549 top-1,000 sites that…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-15 22:00 · DEV Community — AI
    Your robots.txt says yes. Your CDN says no. (We tested the top 1,000 sites)

More stories

  1. Meta releases open weights for 816B-parameter Muse Spark model — AWS Machine Learning Blog
  2. NVIDIA Releases Open Nemotron 3.5 Super VL and Bolt Models for Agentic AI — NVIDIA Blog
  3. Meta and Apple partner to run Muse Spark Open AI model on Macs — Wall Street Journal Technology
  4. Black Forest Labs releases open-weights FLUX 3 Image Klein model — Hugging Face Blog
  5. Anthropic Reports $16.8 Billion Q3 Revenue Ahead of Planned IPO — New York Times Technology
  6. Microsoft and Nvidia launch RTX Spark AI PCs — NVIDIA Blog
  7. Databricks details internal use of AI Runtime for research — Databricks Blog
  8. Google launches Universal Cart shopping tool in the US — TechCrunch AI

Get the daily brief of stories like this at 6:30 every morning →