AINewsnow

I checked what 31 big websites tell AI crawlers in robots.txt, and found a bug in my own tool

Can ChatGPT, Claude or Perplexity read your website? The first place to look is a small text file: robots.txt . I wanted to see what big websites actually say in it, so I picked 40 well-known sites and audited their home pages for 12 AI crawlers. The results split much more cleanly by industry than…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-05 22:59 · DEV Community — AI
    I checked what 31 big websites tell AI crawlers in robots.txt, and found a bug in my own tool

More stories

  1. How to Add /llms.txt and Schema.org JSON-LD to Any Website in 60 Seconds (Get Cited by ChatGPT & Perplexity) — DEV Community — AI
  2. AI Search Readiness Checker: Can AI Crawlers Read Your Site? — DEV Community — AI
  3. How to expose your SaaS data to Claude and ChatGPT with MCP (a real 44-tool server) — DEV Community — AI
  4. How We Built an Automated /llms.txt + 955-Node 301 Shield Pipeline to Rank #1 in ChatGPT, Perplexity & Google (2026) [Nov 05] — DEV Community — AI
  5. Perplexity launches Teammates, always-on AI agents for ongoing work — The Verge AI
  6. I Tried ChatGPT, Claude, and Gemini Without Giving Them My Real Email — DEV Community — AI
  7. שָׁרְט is identity and شَرْط is the condition. شָׁרְט renders on both Claude and ChatGPT — r/artificial
  8. How does the RustRover debugger MCP server work? — DEV Community — AI

Get the daily brief of stories like this at 6:30 every morning →