Stopping a crawl when the data stops looking real
This story is from 2026-09-22. It is preserved in the archive; the latest stories are on the live feed.
TL;DR I built scrapy-jev , a Scrapy pipeline that asks a fast, cheap AI model whether each scraped field still looks like a real product name, price, SKU, or category, and stops the crawl when too many items fail the check. It works, and it caught the kind of breakage that normally goes unnoticed u…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-09-22 16:46 · DEV Community — AI
Stopping a crawl when the data stops looking real