AWS details detecting and recovering from silent data corruption on SageMaker HyperPod

An AWS Machine Learning Blog post describes silent data corruption, a GPU fault that returns wrong but plausible numbers without errors, and adds three checks to a PyTorch DDP training loop to detect it.

Read the full story at AWS Machine Learning Blog ↗

Timeline · 1 report

2026-11-11 23:30 · AWS Machine Learning Blog
Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

More stories

AWS launches standalone Web Search API for Amazon Bedrock AgentCore — AWS Machine Learning Blog
AWS launches local emulator simulating 200+ services, written entirely by AI agents — InfoWorld AI
AWS adds usage-based pricing for SageMaker AI bidirectional streaming — AWS Machine Learning Blog
Moonshot AI's Kimi K3.1 Pro expands availability on Amazon Bedrock amid pricing push — AWS Machine Learning Blog
Amazon and Google set for major windfalls from Anthropic IPO — Reuters AI
Ring launches home security drone and enterprise patrol robot under new Ring Robotics subbrand — TechCrunch AI
AWS outlines centralized data lake for Amazon Bedrock tokenomics observability — AWS Machine Learning Blog
ChatGPT Enterprise Now Available Through AWS Marketplace — OpenAI News

Get the daily brief of stories like this at 6:30 every morning →