AINewsnow

Where should gradient clipping go in a mixed-precision LoRA training step?

I work on CloudGPU. When you use PyTorch gradient scaling, clip after scaler.unscale_(optimizer) and before scaler.step(optimizer) . Clipping scaled gradients changes the meaning of the threshold. During gradient accumulation, wait until the effective batch is complete before unscaling. 1. Identify…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-11-11 06:57 · DEV Community — Machine Learning
    Where should gradient clipping go in a mixed-precision LoRA training step?

More stories

  1. Anthropic IPO expected to begin trading Nov. 19 at $1.4 trillion valuation — CNBC Technology
  2. Nvidia CEO Jensen Huang faces spy scandal and scrutiny over China AI chip access — New York Times AI
  3. DeepSeek's Harness agent app reaches 1.54 million daily users — Bloomberg AI
  4. OpenAI Acquires AI Travel Startup Odessia — The Information AI
  5. STATERA Library expands robotics drive and connector engineering documentation — DEV Community — AI
  6. Google's Gemini 4 Carbon model rolls out across Copilot, API, and agent tools — Google DeepMind Blog
  7. Cognizant Launches Claude Managed Services With 25,000-Strong Frontier Capability Group — Economic Times Tech
  8. OpenAI launches Businesses in ChatGPT in beta with claimable profiles and Business Agents — OpenAI News

Get the daily brief of stories like this at 6:30 every morning →