Nested Sparsity for Scalable Pretraining and Continual Learning
arXiv:2610.31818v1 Announce Type: new Abstract: Mixture-of-experts (MoE) transformers, the standard backbone for modern large-scale deep learning, employ conditional computation: only a sparse subset of model weights is used in each forward pass. In MoE transformers, this sparsity arises in the MoE…
Read the full story at arXiv cs.LG ↗
Timeline · 1 report
- 2026-10-26 04:00 · arXiv cs.LG
Nested Sparsity for Scalable Pretraining and Continual Learning