AINewsnow

Who's actually blocking AI crawlers in Europe? A 185-site census (first dated point)

This story is from 2026-10-01. It is preserved in the archive; the latest stories are on the live feed.

Dated: 2026-10-01, 17:15–17:22 UTC — single crawl pass, fixed panel, fixed method. Pennyforge research artifact. $0 data (public robots.txt + /.well-known/ai-crawler , plain HTTP with a browser user-agent). Companion data: .tmp-sweep/r141/tdm-census-v2-2026-10-01.json (per-site results, full 12-UA…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-01 23:55 · DEV Community — AI
    Who's actually blocking AI crawlers in Europe? A 185-site census (first dated point)

More stories

  1. Bring near-Astra intelligence to everyday work with GPT-6.1 Sol on Amazon Bedrock — AWS Machine Learning Blog
  2. Gemini 4 Argon: our next era of frontier intelligence — Google Gemini Blog
  3. Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs — Allen Institute for AI (Ai2)
  4. Google Releases New Gemini Model With Guardrails Amid A.I. Safety Debate — New York Times Technology
  5. OpenAI cancels release of new artificial intelligence model over safety concerns — France 24 — Artificial Intelligence
  6. OpenAI DevDay 2026 Keynote (FULL) — OpenAI YouTube
  7. Introducing GPT-6.1 Sol — OpenAI News
  8. OpenAI’s Dots Are Always-On AI Agents—and Its Answer to Meta’s Muse — Wired AI

Get the daily brief of stories like this at 6:30 every morning →