20 of 172 major websites won't show a script their robots.txt
robots.txt is the one file on a website written for robots. So it surprised me that when a plain script asked 172 big websites for theirs, 20 of them said no. Redfin, The Home Depot, Lowe's, Sephora, DoorDash, Udemy, Apartments.com and 13 others answered GET /robots.txt with an error: 19 with a 403…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-17 15:00 · DEV Community — AI
20 of 172 major websites won't show a script their robots.txt