Clipping gradients to 5 can still leave a norm of 7.07
This story is from 2026-09-12. It is preserved in the archive; the latest stories are on the live feed.
For a gradient of (6, 8), clipping each component to 5 gives (5, 5), with an L2 norm of 7.07. Clipping the norm to 5 gives (3, 4), preserving the direction. In PyTorch, clip_grad_norm_ also treats the parameters you pass in as one combined gradient, not a separate limit per tensor. I’m building Gla…
Read the full story at r/deeplearning ↗
Timeline · 1 report
- 2026-09-12 13:51 · r/deeplearning
Clipping gradients to 5 can still leave a norm of 7.07