AWS launched PREFIX_AWARE routing on SageMaker real-time inference endpoints, sending requests with the same prompt prefix to the same instance so KV caches stay warm. Benchmarks with Llama 3.1 70B on vLLM show up to 77% lower P50 time-to-first-token and KV cache hit rates rising from ~25% to over 80%. The routing requires no model container changes and adds only 1.3–1.9 ms of routing overhead per request.
Amazon published a review of 13 SageMaker AI inference capabilities launched in 2026 across managed endpoints and HyperPod Inference. The launches target the hardest parts of generative AI serving: cold starts, GPU capacity, OpenAI compatibility, observability, and token-level latency. Practitioners get concrete options for both fully managed and Kubernetes-native deployment paths.
Amazon published a review of 13 SageMaker AI inference capabilities launched in 2026 across managed endpoints and HyperPod Inference. The launches target the hardest parts of generative AI serving: cold starts, GPU capacity, OpenAI compatibility, observability, and token-level latency. Practitioners get concrete options for both fully managed and Kubernetes-native deployment paths.
AWS announced a tiered KV cache feature for large language model inference on SageMaker HyperPod, developed with Curvine. It reduces infrastructure costs and improves time-to-first-token by avoiding recomputation of identical prompts, benefiting teams running models like Qwen, Llama, and DeepSeek.
AWS announced a tiered KV cache feature for large language model inference on SageMaker HyperPod, developed with Curvine. It reduces infrastructure costs and improves time-to-first-token by avoiding recomputation of identical prompts, benefiting teams running models like Qwen, Llama, and DeepSeek.
A practitioner running a 10-call agent workflow through vLLM asks how to best process batches of prompts sharing a common prefix. Their setup routes traffic via an llm-d/vllm-router with KV-cache-aware routing into a pool of vLLM workers. They propose sending one call first to prime the cache block before dispatching the rest, and ask whether this is a sound strategy.
A practitioner running a 10-call agent workflow through vLLM asks how to best process batches of prompts sharing a common prefix. Their setup routes traffic via an llm-d/vllm-router with KV-cache-aware routing into a pool of vLLM workers. They propose sending one call first to prime the cache block before dispatching the rest, and ask whether this is a sound strategy.
Amazon announced SageMaker HyperPod Inference Gateway, a Kubernetes-native addon that routes LLM inference requests using real-time GPU signals instead of round-robin load balancing. Benchmarks show P95 first-token latency cuts up to 98% and throughput gains up to 50% under mixed or bursty workloads. It deploys as a single EKS addon with no application or SDK changes.
Amazon announced SageMaker HyperPod Inference Gateway, a Kubernetes-native addon that routes LLM inference requests using real-time GPU signals instead of round-robin load balancing. Benchmarks show P95 first-token latency cuts up to 98% and throughput gains up to 50% under mixed or bursty workloads. It deploys as a single EKS addon with no application or SDK changes.
A new open-source project called inference-cache provides a Kubernetes-native control plane for cache-aware LLM routing, aiming to reduce redundant computation of shared prefixes. It addresses a common inefficiency in serving repeated LLM prompts by leveraging cached prefixes.
A new open-source project called inference-cache provides a Kubernetes-native control plane for cache-aware LLM routing, aiming to reduce redundant computation of shared prefixes. It addresses a common inefficiency in serving repeated LLM prompts by leveraging cached prefixes.