A threshold is not a measurement: what your dedup cut actually does at each artefact length
Deduplicate at Jaccard 0.7 . Flag a pair whose Dice similarity is at least 0.8 . Every one of those numbers was calibrated once, on one corpus, and then applied to artefacts of every size — a 40-token comment, a 3000-token report, a source file, a training example. The threshold is the only thing t…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-10-15 00:52 · DEV Community — Machine Learning
A threshold is not a measurement: what your dedup cut actually does at each artefact length