AINewsnow

111, 133, 172: The Same 192 Characters in Three Tokenizers

I keep a bad habit: when a prompt looks short, I assume it is cheap. Then I pasted one Chinese code-review request into three tokenizers. Same 192 characters: 111 tokens in DeepSeek V4, 133 in o200k_base , 172 in cl100k_base . A 55% spread. The rule I had been using, four characters per token, pred…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-03 22:54 · DEV Community — AI
    111, 133, 172: The Same 192 Characters in Three Tokenizers

More stories

  1. NVIDIA Shrinks DeepSeek V4.1 Flash by 75 GiB With Almost No Quality Loss — AlphaSignal
  2. DeepSeek launches Creator Mode for building custom plugins via natural language — Techmeme
  3. DeepSeek Drops Enhanced Coding Agent With Aim to Rival Anthropic — Bloomberg AI
  4. Fireworks Announces Faster Agent Inference and Sovereign Deployment — Unite.AI
  5. Global Venture Funding Jumps 45% to $59B in October on Record DeepSeek Round — Crunchbase News AI
  6. DeepSeek Launches Creator Mode for Custom AI Plugins — Sam Altman (X)
  7. E2E Networks Launches Blaze for Pay-Per-Token AI Inference — Analytics India Magazine
  8. I tried to make Opus 5.5 cheaper with a cheap coding subagent, but It got worse and more expensive — r/ClaudeAI

Get the daily brief of stories like this at 6:30 every morning →