SpecRelay paper proposes edge-cloud speculative decoding that amortizes or avoids round trips
A new arXiv paper, SpecRelay, describes edge-cloud speculative decoding where an edge device drafts tokens and a larger cloud model verifies them in parallel, aiming to amortize or avoid the round trip.
Read the full story at arXiv cs.AI ↗
Timeline · 1 report
- 2026-11-17 05:00 · arXiv cs.AI
SpecRelay: Amortize or Avoid the Round Trip in Edge-Cloud Speculative Decoding