AINewsnow

Weight Decay 0.05: Catastrophe in Adam, Optimum in AdamW

Every recipe on this blog since the optimizer face-off has carried weight_decay=5e-4 — a number I originally picked for SGD because every CIFAR repo on the internet uses it, then ported to Adam when Adam won, with exactly zero measurement of whether it belongs there. That's the precise shape of inh…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-11-03 01:25 · DEV Community — Machine Learning
    Weight Decay 0.05: Catastrophe in Adam, Optimum in AdamW

More stories

  1. OpenAI appoints former US Army Secretary Dan Driscoll to national security role — OpenAI News
  2. Anthropic reports $17B quarterly revenue, mounting losses and spending commitments ahead of IPO — Reuters AI
  3. Marvell to acquire Lumentum in $115 billion all-stock AI data-centre deal — DigiTimes Asia
  4. OpenAI to merge ChatGPT's Chat and Work tabs into a single desktop experience — TechCrunch AI
  5. Apple CEO John Ternus says company won't ship AI for AI's sake — Techmeme
  6. Anthropic confirms new Claude Code limits to prevent recreating existing video games — Engadget
  7. OpenAI launches Grid Protectors Program to cut data center power use during grid strain — OpenAI News
  8. Israeli startup 42AI exits stealth with $62.5M seed for behavioral world model — TechCrunch AI

Get the daily brief of stories like this at 6:30 every morning →