Gradient descent, momentum and Adam racing down the same narrow valley, 80 steps each

The loss is f(x, y) = 0.01x² + y², so the valley curves 100 times more sharply across than along. All three start at (-4.6, 2.0) and get 80 steps. Gradient descent (lr 0.9) overshoots the steep direction on every step, so it zigzags across the valley while creeping along it, and ends at a loss of 0…

Read the full story at r/learnmachinelearning ↗

Timeline · 1 report

1. 2026-10-28 17:29 · r/learnmachinelearning
   Gradient descent, momentum and Adam racing down the same narrow valley, 80 steps each