What optimizer.step() Actually Does: One Step of SGD, Momentum and Adam by Hand
This story is from 2026-08-28. It is preserved in the archive; the latest stories are on the live feed.
last time i took a two-layer network and computed all four gradients by hand, and then ran the same thing in pytorch and autograd landed on the same four numbers, 30, 30, 20, 10 . and then i stopped there, which is exactly one line too early, because in a real training loop the next thing that happ…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-08-28 08:13 · DEV Community — Machine Learning
What optimizer.step() Actually Does: One Step of SGD, Momentum and Adam by Hand