AINewsnow

tokenizers v1 (rust)

Hey all! I am Aritra from Hugging Face. I wanted to share an update on the `tokenizers` library that we have at Hugging Face. It has gone under major changes and we have finally released version 1 of it. Here are what we are most excited about: > multiple language support > multi-thread scaling > m…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

  1. 2026-09-21 15:10 · r/LocalLLaMA
    tokenizers v1 (rust)

More stories

  1. Qwen Image 2.1 PR to ComfyUI — r/StableDiffusion
  2. we made a 27b model for creative writing. performs as good as claude fable 5, at a 40x cheaper price, open weights. — r/GeminiAI
  3. Bessent Targets OpenAI Managers for Hugging Face Incident — Bloomberg AI
  4. A quick Minimax H3 news round-up - 18th September 2026 — r/comfyui
  5. Change the camera movement/angle for your existing video clip - Minimax H3 V2V CrossView-Warp LoRA — r/StableDiffusion
  6. US Treasury Secretary Scott Bessent says OpenAI's Hugging Face hacking incident in July "is the responsibility of the OpenAI management, not a bunch of agents" (Yash Roy/Bloomberg) — Techmeme
  7. UN says AI safeguards can’t wait for certainty — The Verge AI
  8. Qwen/Qwen-Image-2.1 · Hugging Face — r/StableDiffusion

Get the daily brief of stories like this at 6:30 every morning →