Amazon announced SageMaker HyperPod Inference Gateway, a Kubernetes-native addon that routes LLM inference requests using real-time GPU signals instead of round-robin load balancing. Benchmarks show P95 first-token latency cuts up to 98% and throughput gains up to 50% under mixed or bursty workloads. It deploys as a single EKS addon with no application or SDK changes.
Amazon published a review of 13 SageMaker AI inference capabilities launched in 2026 across managed endpoints and HyperPod Inference. The launches target the hardest parts of generative AI serving: cold starts, GPU capacity, OpenAI compatibility, observability, and token-level latency. Practitioners get concrete options for both fully managed and Kubernetes-native deployment paths.
Amazon published a review of 13 SageMaker AI inference capabilities launched in 2026 across managed endpoints and HyperPod Inference. The launches target the hardest parts of generative AI serving: cold starts, GPU capacity, OpenAI compatibility, observability, and token-level latency. Practitioners get concrete options for both fully managed and Kubernetes-native deployment paths.
AWS announced Disaggregated Prefill and Decode (DPD) for LLM inference on SageMaker HyperPod, separating compute-bound prefill and memory-bound decode onto distinct GPU pools connected via EFA with RDMA. This eliminates interference where long prompts stall token generation for concurrent requests.
AWS announced Disaggregated Prefill and Decode (DPD) for LLM inference on SageMaker HyperPod, separating compute-bound prefill and memory-bound decode onto distinct GPU pools connected via EFA with RDMA. This eliminates interference where long prompts stall token generation for concurrent requests.
AWS launched PREFIX_AWARE routing on SageMaker real-time inference endpoints, sending requests with the same prompt prefix to the same instance so KV caches stay warm. Benchmarks with Llama 3.1 70B on vLLM show up to 77% lower P50 time-to-first-token and KV cache hit rates rising from ~25% to over 80%. The routing requires no model container changes and adds only 1.3–1.9 ms of routing overhead per request.
AWS launched PREFIX_AWARE routing on SageMaker real-time inference endpoints, sending requests with the same prompt prefix to the same instance so KV caches stay warm. Benchmarks with Llama 3.1 70B on vLLM show up to 77% lower P50 time-to-first-token and KV cache hit rates rising from ~25% to over 80%. The routing requires no model container changes and adds only 1.3–1.9 ms of routing overhead per request.
AWS announced new Ray capabilities on SageMaker HyperPod, integrating the open-source Ray framework with HyperPod's purpose-built infrastructure for foundation model training and serving. The integration uses KubeRay to manage Ray clusters on Kubernetes, enabling distributed Python workloads such as Ray Train and Ray Serve.
AWS announced new Ray capabilities on SageMaker HyperPod, integrating the open-source Ray framework with HyperPod's purpose-built infrastructure for foundation model training and serving. The integration uses KubeRay to manage Ray clusters on Kubernetes, enabling distributed Python workloads such as Ray Train and Ray Serve.
An AWS blog post presents a validated architecture pairing Amazon SageMaker HyperPod with Qumulo's Cloud Native Qumulo and Cloud Data Fabric to train models in one AWS Region while data remains in another. A 1.02B-parameter LLaMA v3 run across us-east-2 to us-west-2 (60 ms latency) matched co-located throughput after NeuralCache warmup. This lets teams avoid petabyte-scale replication or cross-Region latency penalties.
An AWS blog post presents a validated architecture pairing Amazon SageMaker HyperPod with Qumulo's Cloud Native Qumulo and Cloud Data Fabric to train models in one AWS Region while data remains in another. A 1.02B-parameter LLaMA v3 run across us-east-2 to us-west-2 (60 ms latency) matched co-located throughput after NeuralCache warmup. This lets teams avoid petabyte-scale replication or cross-Region latency penalties.