NyoomFloat12: How a 12-Bit Fixed-Length Format Doubles LLM Inference Throughput Without Losing a Single Bit
NyoomFloat12: How a 12-Bit Fixed-Length Format Doubles LLM Inference Throughput Without Losing a Single Bit Serving large language models at scale is fundamentally a memory problem. During autoregressive decoding, every generated token requires reading the entire weight matrix and the accumulated K…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-10-30 16:04 · DEV Community — Machine Learning
NyoomFloat12: How a 12-Bit Fixed-Length Format Doubles LLM Inference Throughput Without Losing a Single Bit