AINewsnow

Your robots.txt allows GPTBot. Your server may still answer 403.

This story is from 2026-10-04. It is preserved in the archive; the latest stories are on the live feed.

I was checking small business websites for how readable they are to AI answer engines, and kept running into the same thing: robots.txt allows GPTBot and ClaudeBot, and the server answers them with 403 anyway. Nobody reading robots.txt would notice. A robots.txt checker would report "AI crawlers al…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-04 06:32 · DEV Community — AI
    Your robots.txt allows GPTBot. Your server may still answer 403.

More stories

  1. NVIDIA DGX Spark 64GB Gives Developers More Ways to Build and Scale Local AI — NVIDIA Blog
  2. Guided Vision in Gemini Live: built for accessibility — Google Gemini Blog
  3. Google tests its plan for AI data centers in space with Project Suncatcher — Scientific American
  4. OpenAI DevDay 2026 Keynote (FULL) — OpenAI YouTube
  5. A model guide for the GPT-6 family — OpenAI News
  6. The latest AI news we announced in September 2026 — Google AI Blog
  7. Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs — Allen Institute for AI (Ai2)
  8. OpenAI Fires Researchers for Allegedly Sharing Information with AI Safety Group — Wall Street Journal Technology

Get the daily brief of stories like this at 6:30 every morning →