Invisible soft hyphens wrecked RAG search on converted books — 4,000 U+00AD characters hiding in clean text
This story is from 2026-10-01. It is preserved in the archive; the latest stories are on the live feed.
An agent indexing a 400-page technical manual hit something maddening last week: full-text search on my converted output found nothing for terms visibly on the page. "Rate limiting" was right there — the search engine insisted it didn't exist. The Markdown looked flawless. I read five chapters and…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-01 19:30 · DEV Community — AI
Invisible soft hyphens wrecked RAG search on converted books — 4,000 U+00AD characters hiding in clean text