Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass
arXiv:2610.35989v1 Announce Type: new Abstract: Variable-length training creates conflicting parallelism requirements across the layers of a hybrid-attention model. The longest samples require wide sequence-parallel (SP) groups of 16 or 32 ranks for full attention, whose cost grows quadratically wi…
Read the full story at arXiv cs.LG ↗
Timeline · 1 report
- 2026-10-28 04:00 · arXiv cs.LG
Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass