AINewsnow

Trying to get site accessible to ChatGPT again - "restricted URL"

This story is from 2026-09-17. It is preserved in the archive; the latest stories are on the live feed.

Hi all, We used to block AI agents from accessing our site using robots.txt. I remember that, at the time, I asked ChatGPT to fetch a page and it did tell me that it couldn't because of the robots.txt. Fast forward a few months, we now have updated the robots.txt file to allow certain URLs to be ac…

Read the full story at r/ChatGPTPro ↗

Timeline · 1 report

  1. 2026-09-17 23:24 · r/ChatGPTPro
    Trying to get site accessible to ChatGPT again - "restricted URL"

More stories

  1. Meet the Data Agent in ChatGPT Work — OpenAI YouTube
  2. Microsoft and OpenAI Workers Worry About ‘Largest Theft of Labor’ in History — New York Times Technology
  3. Anthropic mulls new AI model ahead of IPO to counter OpenAI's GPT-6 Astra, says report: What we know — Mint AI
  4. I ran Claude code and Codex in parallel for 15 days. Here's what I found. — r/AI_Agents
  5. ChatGPT for Word is now available — OpenAI YouTube
  6. Reimagining IT with ChatGPT — OpenAI YouTube
  7. Meet ChatGPT: Ask Your First Question | OpenAI Academy — OpenAI YouTube
  8. How to Ask ChatGPT Better Questions | OpenAI Academy — OpenAI YouTube

Get the daily brief of stories like this at 6:30 every morning →