Sharding vs replicas across multiple GPUs, just trying to plan this right [R]
Hey all, I’m planning a small multi-GPU setup across two boxes, with a mix of long-context and short-chat workloads. Should a bigger model be sharded across all the cards, or is it better to run smaller independent instances per card and load-balance? My gut says sharding helps single-request laten…
Read the full story at r/MachineLearning ↗
Timeline · 1 report
- 2026-11-12 13:01 · r/MachineLearning
Sharding vs replicas across multiple GPUs, just trying to plan this right [R]