How Far is Adam from Natural Gradient Descent?
This story is from 2026-10-02. It is preserved in the archive; the latest stories are on the live feed.
arXiv:2610.00004v1 Announce Type: new Abstract: Adam is the standard optimizer in deep learning, yet its geometric relationship to natural gradient descent (NGD) contains unresolved questions. We study Adam's full update rule, including momentum, as a diagonal empirical Fisher approximation subject…
Read the full story at arXiv cs.LG ↗
Timeline · 1 report
- 2026-10-02 04:00 · arXiv cs.LG
How Far is Adam from Natural Gradient Descent?