[P] Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT

Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc. We can mitigate this with L…

Read the full story at r/deeplearning ↗

Timeline · 2 reports

1. 2026-10-27 18:16 · r/MachineLearning
   Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT [P]
2. 2026-10-26 23:15 · r/deeplearning
   [P] Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT