AINewsnow

Tokenizing a 12K-token prompt in 0.03 s instead of 25.7 s, and other fixes from running real prompts

New versions of NeuroBrix , our open-source inference engine, and of nbxtensor , its tensor library, are on PyPI. Everything below came from running real prompts on our own engine. Long prompts are tokenized in linear time, with the same tokens A 12,265-token prompt to a Llama-tokenizer model took…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-11-12 18:53 · DEV Community — Machine Learning
    Tokenizing a 12K-token prompt in 0.03 s instead of 25.7 s, and other fixes from running real prompts

More stories

  1. Llama App - Windows’ta ücretsiz indir ve yükle — Microsoft Research Blog
  2. Strix Halo? Do NOT use Llama.cpp — r/LocalLLM
  3. RustingStrata vs llama.cpp runs DeepSeek-V4.1-Flash Q2 (249 GB) on a single RTX 3060 12 GB with 30 GB RAM — r/LocalLLM
  4. llama.cpp Gets Major CUDA Improvements: FP8 Support and Faster GPU Inference — r/LocalLLM
  5. KV Cache Math: Llama 3.1 8B Needs 16 GiB Just to Hold 128K Tokens — DEV Community — Machine Learning
  6. Gemma4-31b latent space discovery: Caring and Contempt for user controlled by the same 3 neurons! Plus, how to find your own steering recipes and export to GGUF steering vector for llama.cpp — r/LocalLLaMA
  7. Ten engines for one box: which local inference engine fits a 100 GB rig with two 16 GB cards? — DEV Community — Machine Learning
  8. Open-sourcing TokenShark: Step through local LLMs block-by-block, live-edit the flow in your browser, and trace where model thinks — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →