Amazon· Bedrock / NovaProduct launch

Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine

Amazon SageMaker HyperPod introduces a tiered KV cache architecture using Curvine, a distributed cache filesystem, to extend KV cache across GPU, CPU, and shared NVMe, enabling cross-replica cache reuse and reducing time-to-first-token.

From the source

On a test deployment, this achieved up to a 100 percent cross-Pod cache hit rate, up to a 2.7x TTFT improvement, and cross-node L2 read latency of about 56 ms for a approximately 1,900-token prompt.

aws.amazon.com

Announced

12 Aug 2026 13:42 UTC