AINewsnow

[P] Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT

Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc. We can mitigate this with L…

Read the full story at r/huggingface ↗

Timeline · 1 report

  1. 2026-10-26 15:07 · r/huggingface
    [P] Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT

More stories

  1. Meta plans $95bn stock offering to bankroll AI expansion — Financial Times AI
  2. Anthropic IPO: S-1 Filing Offers First Look at Claude Maker’s Finances — Bloomberg AI
  3. Microsoft offers buyouts to ~60K workers at VP level and below across teams in the US, Canada, India, and the UK; its headcount at the end of June was 223K (Bloomberg) — Techmeme
  4. To make AI morally good, tech companies turn to religion. Some say it's not so simple — ABC News Technology
  5. Groundhog Day for AI agents: Prompt caching patterns for using OpenAI models on Amazon Bedrock — AWS Machine Learning Blog
  6. Build agentic retrieval on a Managed Amazon Bedrock Knowledge Base with CDK — AWS Machine Learning Blog
  7. Anthropic's Claude bypassed a paywall to access Georgia voter data without authorization — Axios AI+
  8. Can AI automate Epoch? — Epoch AI

Get the daily brief of stories like this at 6:30 every morning →