Why words are split into several tokens?
This completely contradicts logical thinking. It would make sense if one word would be at most 2 tokens (1 for the word and 1 that signals meaning in case of homonyms), but this is never the case. In fact, I read it degrades performance. Why?
Read the full story at r/LanguageTechnology ↗
Timeline · 1 report
- 2026-09-24 18:32 · r/LanguageTechnology
Why words are split into several tokens?