VelaVec: A 9.8M-Param Retrieval Encoder That Runs 53 Faster Than Its Teacher — on the CPU Alone
This story is from 2026-09-12. It is preserved in the archive; the latest stories are on the live feed.
Distilling a 33M model to 9.8M, then making the small model brutally fast with a pure-Rust inference engine (AMX + BNNS + NEON). No GPU, no Python, no warm-up. 5.4 ms cold start. Embedding models face a quiet but brutal constraint: the smaller they get, the less they know — and the bigger they get,…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-09-12 13:47 · DEV Community — Machine Learning
VelaVec: A 9.8M-Param Retrieval Encoder That Runs 53 Faster Than Its Teacher — on the CPU Alone