Warmup Did Nothing for My Learning Rate — Then It Fully Rescued the Wrong One
Learning-rate warmup has the strangest status of any training-recipe fragment: universally shipped, rarely justified. It earned its reputation in transformer training, where skipping it genuinely detonates runs, and from there it diffused into every config file on the internet — including CNN recip…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-11-04 23:45 · DEV Community — Machine Learning
Warmup Did Nothing for My Learning Rate — Then It Fully Rescued the Wrong One