AINewsnow

I animated what GPT-4o's tokenizer does to "strawberry": real token ids, a tiny BPE run, and why the space belongs to the word (6 min video) [D]

I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong. It follows one word the whole way: "How many r's are in strawberry?" is 8 tokens in o200k_base (the GPT-4o / GPT-5 tokenizer). " strawberry" is a single token, id 101830, s…

Read the full story at r/MachineLearning ↗

Timeline · 1 report

  1. 2026-11-01 18:01 · r/MachineLearning
    I animated what GPT-4o's tokenizer does to "strawberry": real token ids, a tiny BPE run, and why the space belongs to the word (6 min video) [D]

More stories

  1. DeepSeek-V4.1-Flash draws wave of local benchmarks, quants and hardware tests — r/LocalLLM
  2. OpenAI lowers funding target for new round to $20 billion — Financial Times AI
  3. Beginner-friendly 15-minute lesson plan introduced for learning ChatGPT — Tom's Guide AI
  4. Mistral releases Large 4 publicly as Le Chonk ranks among top open coding models — ZDNET AI
  5. OpenAI tightens ChatGPT limits on weapons information for teen users — Reuters AI
  6. Malicious ChatGPT Share Link Could Run Code in Users' Sessions, Researchers Report — Techmeme
  7. New AI benchmarks and audits spark rivalry among leading frontier models — TheAIGRID
  8. Cognizant forms OpenAI Frontier Capability Group with over 25,000 professionals — The Globe and Mail

Get the daily brief of stories like this at 6:30 every morning →