Raising the Batch Size Makes Batch Norm's Inference Statistics 13x Worse
This story is from 2026-09-03. It is preserved in the archive; the latest stories are on the live feed.
In training, batch norm normalises each row with the statistics of the batch it happens to be in. At inference there is no batch, so it uses a running average collected during training. Those are two different functions, and the layer swaps one for the other silently. See it: https://dev48.infy.uk/…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-09-03 05:46 · DEV Community — Machine Learning
Raising the Batch Size Makes Batch Norm's Inference Statistics 13x Worse