AINewsnow

We read the robots.txt of 151 big websites. The syntax was fine. The problems were elsewhere.

This story is from 2026-10-29. It is preserved in the archive; the latest stories are on the live feed.

On 29 October 2026 we fetched the robots.txt of 151 large websites (news, shops, SaaS, docs, social networks, education, SEO) and ran every file through a linter. 135 files could be read. The short version 44% block at least one of the six best-known AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Ex…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-29 21:46 · DEV Community — AI
    We read the robots.txt of 151 big websites. The syntax was fine. The problems were elsewhere.

More stories

  1. 8 examples of Gemini 4 Argon in action — Google Gemini Blog
  2. You can now generate, preview, and edit documents, PDFs, presentations, spreadsheets, and more directly in Gemini. — Google Gemini Blog
  3. Google launches Gemini 4 Carbon with a ‘significant leap’ in cybersecurity — The Verge AI
  4. Finnish police launch criminal probe over forest cleared for Google data center — The Hill Technology
  5. Google expands AI Search ads globally as core ad growth slows — TechCrunch AI
  6. How to use Gemini to update your resume — Google Gemini Blog
  7. Google AI gets cagey when asked who won the 2020 elections — Ars Technica AI
  8. Dive into new topics with immersive explorations in the Gemini app. — Google Gemini Blog

Get the daily brief of stories like this at 6:30 every morning →