AINewsnow

Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining

arXiv:2610.33016v1 Announce Type: new Abstract: Llama-3, Qwen3 and many other open LLMs pre-tokenize text with a regular expression whose word class admits only Unicode letters. Devanagari vowel signs are combining marks, so this regex splits Nepali words at every vowel sign, and models that use it…

Read the full story at arXiv cs.CL ↗

Timeline · 1 report

  1. 2026-10-27 04:00 · arXiv cs.CL
    Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining

More stories

  1. 8 GB GPU, 262K context 3.5 BPW Qwen-Flash-Next Coder 70 T/s using llama.cpp — r/LocalLLaMA
  2. Nanbeige4.3-33B Q4 on a MacBook Pro M5 Max, mlx-lm vs llama.cpp vs Kapitsa, Prefill/Decode 512 to 32K range. — r/LocalLLaMA
  3. Quad 3090 vs 256 M5 Ultra For Agentic Coding — r/LocalLLM
  4. Update: MAFDET now runs on Apple Silicon. Qwen3-14B answers in 12 s instead of 33 on an M2 Pro — r/LocalLLM
  5. Qwen3.8-27B Q6 on one RTX 5090: 69 → 100 tok/s short, 35 → 64 tok/s at 120K context, just by leaving LM Studio for a native llama.cpp build (full benchmarks + recipe) — r/LocalLLM
  6. I created a benchmark to test local LLMs on your Snapdragon NPU (open source) — r/LocalLLM
  7. boro64/Ternary-Nemotron-3.5-Lightning-30B-A3B-GGUF · Hugging Face — r/LocalLLaMA
  8. I built my first RAG project using Whisper, BGE-M3 and Ollama — what should I improve? — r/learnmachinelearning

Get the daily brief of stories like this at 6:30 every morning →