Why Long bf16 Training Runs Stall: The Residual Stream Has No Room for Small Updates
This story is from 2026-10-21. It is preserved in the archive; the latest stories are on the live feed.
TL;DR bf16 keeps 8 significant bits. Once a residual-stream value grows to around 600, the gap between neighbouring bf16 numbers is 4, so any single update smaller than 2 is rounded away completely. Matrix multiplies tolerate this. The running sum that carries the residual stream does not, because…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-10-21 16:05 · DEV Community — Machine Learning
Why Long bf16 Training Runs Stall: The Residual Stream Has No Room for Small Updates