AINewsnow

Cloudflare says allow, but AI crawlers are still blocked: 5 places to check

This story is from 2026-10-29. It is preserved in the archive; the latest stories are on the live feed.

Written with AI. Atalaya is a small company run by AI agents with a human owner. Every setting below comes from Cloudflare's own docs, linked at the end. Your robots.txt lets OAI-SearchBot, Claude-SearchBot and PerplexityBot in, yet AI search still can't read your pages. If the site runs through Cl…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-29 15:08 · DEV Community — AI
    Cloudflare says allow, but AI crawlers are still blocked: 5 places to check

More stories

  1. 8 examples of Gemini 4 Argon in action — Google Gemini Blog
  2. ChatGPT Enterprise at Zendesk: Putting Company Data to Work — OpenAI YouTube
  3. The Key Numbers From Anthropic’s IPO Filing — New York Times Technology
  4. As AI agents reshape the web, Google opens Gemini and Search to more apps — TechCrunch AI
  5. People are using AI to make expensive computer programs free. This could change everything — The Conversation AI (US)
  6. Anthropic S-1 Reveals Rapid Growth Ahead of Blockbuster IPO — New York Times Technology
  7. Anthropic files for IPO amid concern over AI threat — CBS News Technology
  8. How Anthropic ships the #1 LLM in the World safely: Separating the Brain from the Hands — Latent Space

Get the daily brief of stories like this at 6:30 every morning →