AINewsnow

Bengali costs 6.3x the tokens English does — I counted

Bengali costs 6.3x the tokens English does — I counted The same paragraph costs 278 tokens in Bengali and 44 in English on Llama 3.1's tokenizer — a 6.3x markup for saying the same thing. On Gemma 3, the same paragraph costs 51 tokens in Bengali and 44 in English, a 1.16x markup. The gap is a token…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-14 15:08 · DEV Community — AI
    Bengali costs 6.3x the tokens English does — I counted

More stories

  1. llama.cpp ships v1.0.0 and promises API breaks only in majors — DEV Community — AI
  2. ggml v1.0.0: What the New Stable API Means for Local ML Developers — DEV Community — Machine Learning
  3. Clover-35B-A3B on llama.cpp: what "AR mode only" means, and a GGUF that answers in Gs — DEV Community — AI
  4. RTX 5090 32GB + 64GB RAM — What local LLM would should I run? — r/LocalLLM
  5. A 120B Model Ran About 4x Faster Than a 27B One: 8 Local LLMs on a Surface Laptop Ultra — DEV Community — AI
  6. DeepSeek API Expansions, Local 284B MoE Runs, and New Developer Tooling — DEV Community — Machine Learning
  7. Can First-Person Word Frequency Track Models’ Stance Maintenance Under Pressure? — DEV Community — AI
  8. MACaQ 🐒: Open-source quantizer with a "refinement" mode for existing GGUFs + custom llama.cpp & vLLM kernels — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →