124M parameters, 17,054 MiB to train: I traced where GPT-2 training memory actually goes
This story is from 2026-10-28. It is preserved in the archive; the latest stories are on the live feed.
My new blog post started with a very wrong intuition: GPT-2 small has 124M parameters, so training it should be pretty cheap on memory. I traced step 2 on an A40, with fp32 AdamW and 8 sequences of 1,024 tokens. The peak was 17,054 MiB. At that moment, weights + Adam + masks and persistent overhead…
Read the full story at r/deeplearning ↗
Timeline · 1 report
- 2026-10-28 04:49 · r/deeplearning
124M parameters, 17,054 MiB to train: I traced where GPT-2 training memory actually goes