What 9,483 small business websites actually serve to an AI crawler
This story is from 2026-10-25. It is preserved in the archive; the latest stories are on the live feed.
Over two weeks I pointed a crawler at 11,060 small business websites with JavaScript off, a self-identifying user agent and three pages per site, and stored what came back. This is what the schema, the robots files and the served HTML look like across 9,483 sites that responded, and why an assistan…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-25 11:21 · DEV Community — AI
What 9,483 small business websites actually serve to an AI crawler