AI Crawlers Ignore llms.txt 97% of the Time So Stop Making One
This story is from 2026-09-26. It is preserved in the archive; the latest stories are on the live feed.
A 137,000-domain Ahrefs study published in June 2026 found that 97% of llms.txt files received zero requests from any bot or human. GPTBot, ClaudeBot, and their peers crawl HTML directly and ignore the file entirely. This article breaks down what the data actually shows, why llms.txt became a specu…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-09-26 10:59 · DEV Community — AI
AI Crawlers Ignore llms.txt 97% of the Time So Stop Making One