What a crawler can retrieve from nine small product sites: robots, sitemaps, JSON-LD and llms.txt
This story is from 2026-10-26. It is preserved in the archive; the latest stories are on the live feed.
When you run several small product sites, it is easy to assume that robots rules, sitemaps, structured data and language annotations are all in place. We checked instead, once, on all nine public sites of the IVRYN portfolio, and published the result as a dated snapshot. What was collected One auto…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-26 00:01 · DEV Community — AI
What a crawler can retrieve from nine small product sites: robots, sitemaps, JSON-LD and llms.txt