AINewsnow

I thought GPT-2 was small. Then I measured a training step: 17,054 MiB for 124M parameters.

I went into this thinking: GPT-2 is only 124M parameters. How much memory could training it possibly need? Turns out, quite a lot lol. I traced the second training step of my from-scratch GPT-2 in PyTorch. Setup: one A40, fp32 AdamW, batch 8, 1,024 tokens per sequence. Peak allocated memory: 17,054…

Read the full story at r/learnmachinelearning ↗

Timeline · 2 reports

  1. 2026-10-28 04:49 · r/deeplearning
    124M parameters, 17,054 MiB to train: I traced where GPT-2 training memory actually goes
  2. 2026-10-28 04:47 · r/learnmachinelearning
    I thought GPT-2 was small. Then I measured a training step: 17,054 MiB for 124M parameters.

More stories

  1. Anthropic reveals plans for blockbuster IPO — Sky News Technology
  2. Introducing GPT-6.1 Terra — OpenAI News
  3. ChatGPT can work with your iCloud Mail and Calendar thanks to new plugin support — Engadget
  4. Retell uses OpenAI models to help power 100M monthly calls — OpenAI News
  5. OpenAI expands partnership with Oracle to bring its apps to ChatGPT — OpenAI News
  6. Greptile takes bug detection beyond code review with GPT-6 Astra — OpenAI News
  7. Stack Overflow for Agents triples signups with ChatGPT plugin — OpenAI YouTube
  8. GPT-6.1 Sol helps CodeRabbit strengthen security across complex codebases — OpenAI News

Get the daily brief of stories like this at 6:30 every morning →