Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT [P]
Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc. We can mitigate this with L…
Read the full story at r/MachineLearning ↗
Timeline · 1 report
- 2026-10-27 18:16 · r/MachineLearning
Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT [P]
More stories
- Welcome Le Chonk: Mistral Large 4 on Hugging Face — Hugging Face Blog
- 8 examples of Gemini 4 Argon in action — Google Gemini Blog
- You can now generate, preview, and edit documents, PDFs, presentations, spreadsheets, and more directly in Gemini. — Google Gemini Blog
- Canada to Take Equity Stake in AI Firm Cohere — Wall Street Journal Technology
- How we detect frontier model misalignment — OpenAI News
- Accelerating software delivery with agentic QA automation using Amazon Nova Act – Part 3 — AWS Machine Learning Blog
- Baseten Releases Joey-1, a 2B Open-Weights Decision Model — Unite.AI
- How the UK Amazon Quick User Group grew in 2026 — AWS Machine Learning Blog
Get the daily brief of stories like this at 6:30 every morning →