Google, Red Hat and IBM release llm-d 1.0, an open-source Kubernetes stack for distributed LLM inference (Gemini prefix cache hit 35% to 70%)
llm-d 1.0 is the first stable release of the CNCF Sandbox inference stack founded by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA. It sits above vLLM and SGLang and turns a cluster of model servers into one pool with cache-aware routing, tiered KV cache and prefill/decode disaggregation. 1.0 fo…
Read the full story at r/machinelearningnews ↗
Timeline · 1 report
- 2026-10-21 04:48 · r/machinelearningnews
Google, Red Hat and IBM release llm-d 1.0, an open-source Kubernetes stack for distributed LLM inference (Gemini prefix cache hit 35% to 70%)