SageMaker AI adds KV cache-aware routing and cross-Region inference routing
This story is from 2026-11-17. It is preserved in the archive; the latest stories are on the live feed.
Amazon SageMaker AI real-time endpoints now support KV cache-aware routing, which sends requests to instances with the longest cached prompt match. AWS also detailed cross-Region inference routing, used by Salesforce to serve models across 30+ Regions and hundreds of accounts.
Read the full story at AWS Machine Learning Blog ↗
Timeline · 4 reports
- 2026-11-19 18:20 · AWS Machine Learning Blog
Keep inference running through regional capacity shortfalls with cross-Region inference routing in Amazon SageMaker AI - 2026-11-19 09:58 · DEV Community — Machine Learning
How Salesforce uses cross-Region inference routing to address model-serving capacity gaps - 2026-11-18 20:13 · AWS Machine Learning Blog
How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI - 2026-11-17 21:11 · AWS Machine Learning Blog
Amazon SageMaker AI real-time endpoints now support KV cache aware routing