AINewsnow

Is your site invisible to ChatGPT? Check your robots.txt

This story is from 2026-11-04. It is preserved in the archive; the latest stories are on the live feed.

A lot of people now ask ChatGPT or Perplexity "what's a good tool for X" instead of Googling it. If your product never comes up, there might be a dull reason: your robots.txt is telling AI crawlers to stay out. Some starter templates do this. So do some CDN "block AI bots" toggles. You might not ev…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-04 20:19 · DEV Community — AI
    Is your site invisible to ChatGPT? Check your robots.txt

More stories

  1. AI Daily Digest November 6, 2026: OpenAI Ships GPT-6 Cyber, Meta Buys Conductor, Anthropic Locks Down Claude Enterprise — DEV Community — AI
  2. Generative Engine Optimization (GEO): Getting Cited on ChatGPT Search and Perplexity — DEV Community — AI
  3. How AI Search Engines Rank Brands: GEO & LLM Trust Distribution — DEV Community — AI
  4. I built a free tool to see if ChatGPT & Perplexity recommend your website — DEV Community — AI
  5. AI search can reach almost every site we audited. Most give it little worth quoting — DEV Community — AI
  6. How to Add /llms.txt and Schema.org JSON-LD to Any Website in 60 Seconds (Get Cited by ChatGPT & Perplexity) — DEV Community — AI
  7. AI Search Readiness Checker: Can AI Crawlers Read Your Site? — DEV Community — AI
  8. Agentic shopping, and how AI shopping agents choose — DEV Community — AI

Get the daily brief of stories like this at 6:30 every morning →