AINewsnow

Your CDN has an opinion about AI crawlers, and it is not the one you set

This story is from 2026-09-19. It is preserved in the archive; the latest stories are on the live feed.

Everyone reads the headline as "Cloudflare blocks AI crawlers." The measurement says something narrower and more interesting: on the sites whose own robots.txt says yes, Cloudflare refuses Amazon's and Meta's crawlers far more often than it refuses OpenAI's or Anthropic's — and for OAI-SearchBot it…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-09-19 22:31 · DEV Community — AI
    Your CDN has an opinion about AI crawlers, and it is not the one you set

More stories

  1. Google's Gemini AI hacks three other companies during security test — Sky News Technology
  2. Google’s Gemini AI hacked into other companies, adding to ‘rogue’ AI incidents — Washington Post AI
  3. Muse Hit #2 Doing Phone Calls. Then It Asked for Your Bank. — DEV Community — AI
  4. Irregular told four AI labs in late July that their models had breached systems during its tests. The public learned in stages, and Google went last. — The Next Web
  5. AI Model Month Is Off to a Blistering Start — The AI Daily Brief
  6. Anthropic, OpenAI, SpaceXAI, Google sued over call to ‘pace’ AI development — Politico Technology
  7. Gemini Hacked Three Companies in First Known Breakout by Google’s AI — Wall Street Journal Technology
  8. Security researchers used Claude to help them hack into OpenAI — The Verge AI

Get the daily brief of stories like this at 6:30 every morning →