STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules
arXiv:2610.12599v1 Announce Type: new Abstract: Sparse computation can scale the parametric capacity of LLMs without proportionally increasing active computation, but pretraining sparse architectures remains expensive and data-hungry. Existing sparse upcycling methods mainly target Mixture-of-Exper…
Read the full story at arXiv cs.LG ↗
Timeline · 1 report
- 2026-10-12 04:00 · arXiv cs.LG
STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules