Quiz: characters, words and tokens are different measures of text length, and LLM limits use tokens
Short true-or-false quiz on text length in LLMs. Details vary by model and tokenizer, so treat the answers as general behavior and check the one you use. The same sentence in English and in Portuguese or Japanese uses about the same number of tokens. One word is one token. A model’s context window…
Read the full story at r/artificial ↗
Timeline · 1 report
- 2026-10-24 17:01 · r/artificial
Quiz: characters, words and tokens are different measures of text length, and LLM limits use tokens