AINewsnow

I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.

Everyone "knows" that the web is locking out AI crawlers. Cloudflare flips the default to block, robots.txt blocklists go viral, every SEO newsletter screams about GPTBot. So I ran the actual test. I pulled robots.txt , llms.txt , sitemap.xml , and the homepage JSON-LD from 30 well-known sites — an…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-12 02:57 · DEV Community — AI
    I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.

More stories

  1. OpenAI's quarterly revenue crosses $10 billion as operating losses widen — The Information AI
  2. OpenAI launches Businesses in ChatGPT beta with verified profiles and Business Agents — OpenAI News
  3. OpenAI Acquires AI Travel Startup Odessia — The Information AI
  4. Over 20,000 veterans claim free year of ChatGPT Plus — OpenAI News
  5. Lifetime AI bundle offering access to Claude, Gemini and 20+ models discounted ahead of Black Friday — CNET AI
  6. ChatGPT walks through the seven main AI doomer arguments and rebuts each one — Tom's Guide AI
  7. OpenAI to retire Advanced and Standard voice modes from ChatGPT Voice on December 11, 2026 — OpenAI (X)
  8. Paul Weiss to open Silicon Valley office, hiring lawyer behind OpenAI's Microsoft and Alphabet deals — Reuters AI

Get the daily brief of stories like this at 6:30 every morning →