24 seconds per iteration instead of 0.4. I paid for six hours of GPU compute and trained on CPU the entire time.
This story is from 2026-09-06. It is preserved in the archive; the latest stories are on the live feed.
Failure 1 — CUDA silently fell back to CPU My training job launched on Vast.ai and ran to completion. Iteration time was 24 seconds instead of 0.4 seconds. CUDA had fallen back to CPU silently. PyTorch logged nothing. I had been billed for six hours of GPU compute while training on an unaccelerated…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-09-06 10:22 · DEV Community — Machine Learning
24 seconds per iteration instead of 0.4. I paid for six hours of GPU compute and trained on CPU the entire time.