AINewsnow

We checked robots.txt on 1,069 top websites: who blocks AI crawlers, and who still allows ChatGPT search

Everyone has an opinion on whether websites are blocking AI crawlers. We wanted the numbers, so on 4 November 2026 we downloaded the Tranco ranking of popular domains, took the top 2,000, fetched each site's robots.txt, and parsed the 1,069 that gave us a usable file for rules about 13 AI crawlers.…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-09 17:48 · DEV Community — AI
    We checked robots.txt on 1,069 top websites: who blocks AI crawlers, and who still allows ChatGPT search

More stories

  1. Job seekers turn to ChatGPT to research what employees really think of employers — Fortune AI
  2. Man sues OpenAI, alleging ChatGPT helped his wife plot his murder — Tom's Guide AI
  3. Guides show how to create Diwali 2026 images and greetings with Gemini AI and ChatGPT — Mint AI
  4. OpenAI raises $11 billion at $1.25 trillion valuation, below initial $1.4 trillion target — Bloomberg AI
  5. OpenAI adds Showcase tab to Developers plugin, offers API credits to subscribers — OpenAI (X)
  6. Introducing ChatGPT Images 2.6 — OpenAI YouTube
  7. Analysis | Already using ChatGPT or Claude? These simple power-ups let you do a lot more. — Washington Post AI
  8. Google makes Gemini Omni 1.2 Flash generally available with 15-second video and lower pricing — DEV Community — AI

Get the daily brief of stories like this at 6:30 every morning →