Dumping your whole site into the agent is making it worse and everyone keeps doing it anyway
I watched someone point a website crawler API at an entire domain last week, dump 3000 pages into a vector store, and then ask me why retrieval had got worse than when they only had 40 pages in there. It got worse because 3000 pages of website is not 3000 pages of content, it is nav and legal and 4…
Read the full story at r/AI_Agents ↗
Timeline · 1 report
- 2026-10-09 16:54 · r/AI_Agents
Dumping your whole site into the agent is making it worse and everyone keeps doing it anyway