AINewsnow

GPTBot in robots.txt: the hosting toggle developers need to check

This story is from 2026-09-07. It is preserved in the archive; the latest stories are on the live feed.

Your robots.txt may express an AI policy you did not write. We checked the homepage and robots.txt of 9,037 live AI tools listed on directree on 6 and 7 September 2026. Of those, 945 explicitly disallow OpenAI’s GPTBot in its own user-agent group: 10.5% of the sample. Treat AI crawler rules as depl…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-09-07 09:35 · DEV Community — AI
    GPTBot in robots.txt: the hosting toggle developers need to check

More stories

  1. Anthropic, OpenAI, SpaceXAI, Google sued over call to ‘pace’ AI development — Politico Technology
  2. Gemini Hacked Three Companies in First Known Breakout by Google’s AI — Wall Street Journal Technology
  3. Introducing the Australian Youth Safety Blueprint — OpenAI News
  4. Anthropic selects Accenture as first embedded evaluator to help implement Amodei's slowdown proposal — CNBC Technology
  5. Anthropic mulls new AI model ahead of IPO to counter OpenAI's GPT-6 Astra, says report: What we know — Mint AI
  6. OpenAI ‘ethically hacked’ with help of Anthropic’s Claude chatbot — The Guardian AI
  7. OpenAI researchers be like — r/agi
  8. Mathematician Terence Tao: “we have to slow down AI. the pace is insane, and there's no reason to be this fast — no reason at all" — r/ArtificialInteligence

Get the daily brief of stories like this at 6:30 every morning →