Z-Loss Backward Geometry in Dense Output Heads and Sparse Routers
This story is from 2026-09-16. It is preserved in the archive; the latest stories are on the live feed.
arXiv:2609.16179v1 Announce Type: new Abstract: Z-loss has been widely applied to the logits of language-model output heads and sparse mixture-of-experts routers. Z-loss constrains the softmax log-normalizers of these output heads and routers, thereby limiting large-logit excursions, reducing finit…
Read the full story at arXiv cs.LG ↗
Timeline · 1 report
- 2026-09-16 04:00 · arXiv cs.LG
Z-Loss Backward Geometry in Dense Output Heads and Sparse Routers