AINewsnow

111 of the top 1,000 sites block GPTBot. 35 never mention the bot behind ChatGPT search.

This story is from 2026-10-06. It is preserved in the archive; the latest stories are on the live feed.

111 of the top 1,000 sites fully block GPTBot in robots.txt. 51 of those 111 don't fully block OAI-SearchBot, and 35 of the 51 never mention OAI-SearchBot at all. GPTBot is OpenAI's training crawler. OAI-SearchBot is the one that decides whether a page can show up in ChatGPT search. So a third of t…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-06 10:06 · DEV Community — AI
    111 of the top 1,000 sites block GPTBot. 35 never mention the bot behind ChatGPT search.

More stories

  1. OpenAI DevDay 2026 Keynote (FULL) — OpenAI YouTube
  2. GPT-6 SOL AND LUNA ARE OUT!!! — Matthew Berman
  3. OpenAI explains how it will watermark ChatGPT text to comply with EU provenance rules — r/OpenAI
  4. OpenAI launches visual ads that appear alongside image generation results — TechCrunch AI
  5. Building advertising for the way people use AI — OpenAI News
  6. We built a computer-use API that cuts tokens by up to 90% on repeat tasks. Plugs into Claude Code, Codex, Cursor or your own code — r/AI_Agents
  7. Can I use Work to use 6.1 Sol for my regular chat searches? — r/OpenAI
  8. People are asking ChatGPT to help them decide how to vote in the midterms — r/ChatGPT

Get the daily brief of stories like this at 6:30 every morning →