New Multi-Word Tokenizer for 204 Languages Uses Fewer Tokens Than GPT-4o's, Even on English
Researchers introduce a multi-word tokenizer covering 204 languages that produces fewer tokens than GPT-4o's tokenizer, including on English. It addresses token inflation for languages like those of India, which need 1.6 to 14 times as many tokens as English on GPT-4o.
Read the full story at arXiv cs.CL ↗
Timeline · 1 report
- 2026-11-11 05:00 · arXiv cs.CL
A Multi-Word Tokenizer for 204 Languages That Needs Fewer Tokens Than GPT-4o's, Including on English