FP16 Quantization Quality Loss: We Blamed the Architecture, the Bug Was One Overflowed Tensor
Our fp16 quantization post-mortem was wrong in the expensive direction: it told anyone with a mask-gated model to give up on half-precision, and that verdict would have kept a 52%-smaller download off the internet. The real cause was sixteen BatchNorm variance tensors larger than fp16's maximum rep…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-10-17 01:10 · DEV Community — Machine Learning
FP16 Quantization Quality Loss: We Blamed the Architecture, the Bug Was One Overflowed Tensor