AINewsnow

Blocked GPTBot? It Does Not Hide You From ChatGPT Search

TL;DR: AI crawlers come in two kinds. Training crawlers (GPTBot, ClaudeBot) collect pages to train models. Search and citation crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User) fetch your page to answer a user right now and name you as the source. Blocking a training crawler do…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-06 13:00 · DEV Community — AI
    Blocked GPTBot? It Does Not Hide You From ChatGPT Search

More stories

  1. Generative Engine Optimization (GEO): Getting Cited on ChatGPT Search and Perplexity — DEV Community — AI
  2. How AI Search Engines Rank Brands: GEO & LLM Trust Distribution — DEV Community — AI
  3. I built a free tool to see if ChatGPT & Perplexity recommend your website — DEV Community — AI
  4. AI search can reach almost every site we audited. Most give it little worth quoting — DEV Community — AI
  5. How to Add /llms.txt and Schema.org JSON-LD to Any Website in 60 Seconds (Get Cited by ChatGPT & Perplexity) — DEV Community — AI
  6. AI Search Readiness Checker: Can AI Crawlers Read Your Site? — DEV Community — AI
  7. Agentic shopping, and how AI shopping agents choose — DEV Community — AI
  8. GEO Generative Engine Optimization Guide: The Complete 2026 Strategy — DEV Community — AI

Get the daily brief of stories like this at 6:30 every morning →