The One Knob I Never Swept Was the Optimizer. Adam Beat My Tuned SGD by 0.7
When I ranked training knobs by effect size , I swept the learning rate, the weight decay, the batch size and the augmentation — and never once questioned the torch.optim.SGD at the top of the file. That wasn't a decision; it was a reflex backed by the most durable folklore in deep learning: SGD ge…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-10-16 00:02 · DEV Community — Machine Learning
The One Knob I Never Swept Was the Optimizer. Adam Beat My Tuned SGD by 0.7