Why softmax returns NaN at a logit of 710, and the one-line fix
If a training run has ever turned its loss into nan with no warning worth reading, there's a fair chance softmax did it. The formula on the whiteboard is fine. The floating-point version of it isn't, and the fix is one line. Where it breaks A float64 holds numbers up to about 1.8 × 1 0 308 . Since…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-11-03 09:05 · DEV Community — Machine Learning
Why softmax returns NaN at a logit of 710, and the one-line fix