AINewsnow

I made a language model that lives in your CPU's L3 cache

In January I wrote a spec for myself: ">50 tokens/second on one CPU core." That's a deliberately boring number — llama.cpp does that on a laptop. So I kept pulling the thread, and nine months later the number is 3,405 tok/s on a single core of a $60 CPU , with a 1.9 MB model I trained from scratch.…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-12 13:01 · DEV Community — Machine Learning
    I made a language model that lives in your CPU's L3 cache

More stories

  1. Qwen3.8-Flash-Next (125B) at ~100 tok/s on an M5 Ultra Mac Studio with llama.cpp — r/LocalLLM
  2. Qwen3.6 35B-A3B optimized for a 4 GB GTX 1050 Ti Mobile: up to 2× prefill and 3× decode vs llama.cpp — r/LocalLLM
  3. strix-llama new branch: prompt processing +21–54% over Vulkan and up to 2.7× over stock ROCm kernel at 128k, same outputs quality — r/LocalLLaMA
  4. Do you actually test open-model inference compatibility? — r/LocalLLaMA
  5. PSA: DeepSeek V4.1 Flash habitually exfiltrates API keys. It is dangerously misaligned and may be hazardous to use — r/LocalLLaMA
  6. Benchmarked 4 Local Models on 3 Harnesses — r/LocalLLM
  7. What local AI and software do you use? Which is best? — r/LocalLLM
  8. Gemma4 E4B EXL3 - Alternative to Llama.cpp on Jetson Orin — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →