AINewsnow

ClaudeBot, GPTBot, ChatGPT-User: which AI crawler does what

Open your server log and you find names like ClaudeBot, GPTBot, ChatGPT-User, OAI-SearchBot and meta-externalagent. They are AI crawlers, and they are not all doing the same thing. Most AI companies send up to three of them, each for a different job. One collects pages to train future models. One k…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-03 10:40 · DEV Community — AI
    ClaudeBot, GPTBot, ChatGPT-User: which AI crawler does what

More stories

  1. Altman courts Samsung and SK as OpenAI pushes data center agenda amid chip shortage — Economic Times Tech
  2. OpenAI signs four Abu Dhabi agreements spanning energy, government and investment — Reuters AI
  3. OpenAI to merge ChatGPT's Chat and Work tabs into a single desktop experience — TechCrunch AI
  4. Heimstaden adopts ChatGPT Enterprise and Codex to streamline property management — OpenAI News
  5. OpenAI weighs rebranding data centers as 'compute campuses' amid growing backlash — Techmeme
  6. Beginner's 15-minute lesson plan aims to teach ChatGPT basics to new users — Tom's Guide AI
  7. OpenAI pays $5,000 bounty for ChatGPT share-link bug that could run code in viewers' sessions — Techmeme
  8. Google, OpenAI, Microsoft and Anthropic to honor shared robots.txt AI-training opt-out — r/OpenAI

Get the daily brief of stories like this at 6:30 every morning →