AINewsnow

A site can rank in Google while GPTBot cannot read it

This story is from 2026-09-11. It is preserved in the archive; the latest stories are on the live feed.

A page that ranks in Google is not necessarily readable by GPTBot or ClaudeBot. Your robots.txt can allow one crawler and reject another, and JavaScript-only copy can leave a fetcher staring at an empty shell. That means an assistant can name a competitor with a worse product simply because it has…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-09-11 05:36 · DEV Community — AI
    A site can rank in Google while GPTBot cannot read it

More stories

  1. Google turns off Gemini and Search by default for K-12 school accounts after backlash — Techmeme
  2. Google to cut off Search for schools that ban AI, drawing backlash — Forbes AI
  3. TeraWulf secures Google credit backstop for Anthropic data-center project in Kentucky — Reuters AI
  4. DuckDuckGo pitches itself to schools amid backlash over Google's AI policy — TechCrunch AI
  5. Google extends voluntary buyout offers to employees in several countries — Economic Times Tech
  6. Google Cloud outlines its latest monthly AI announcements and updates — Google Cloud AI Blog
  7. Ai2 and Google partner on NeuralGCM+, an open-source AI climate model — Allen Institute for AI (Ai2)
  8. OpenAI says ChatGPT reaches 1.2B weekly users, search volume near 20% of Google's — Techmeme

Get the daily brief of stories like this at 6:30 every morning →