Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog
arXiv:2610.20859v1 Announce Type: new Abstract: Typology-aware tokenizers are motivated by the claim that segmentation which respects morpheme boundaries preserves information that frequency-driven segmentation destroys. Studies routinely establish that it helps while assuming the boundaries must b…
Read the full story at arXiv cs.CL ↗
Timeline · 1 report
- 2026-10-16 04:00 · arXiv cs.CL
Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog