The robots.txt rule almost every tester gets wrong
This story is from 2026-09-03. It is preserved in the archive; the latest stories are on the live feed.
Here is a robots.txt. Is /private/report.pdf crawlable? User - agent : * Disallow : / private / Allow : / private / report . pdf If you answered "no, it's under /private/ ", you have company — including, in my experience, a few online testers. The correct answer is yes , and the reason is a rule th…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-09-03 21:30 · DEV Community — AI
The robots.txt rule almost every tester gets wrong