AWS launched model caching for Amazon SageMaker Inference on HyperPod, pre-loading model weights and container images onto cluster nodes' local NVMe storage before pods request them. Pods can then serve traffic in seconds instead of the 25-30+ minutes cold downloads take for large models like DeepSeek-R1. The feature is generally available in all HyperPod regions and enabled via a modelCacheConfig section in existing deployment specs.
A developer tested SIE, Superlinked's open-source inference engine, running locally, and found model loading dominated end-to-end time. The server exposes model states via /v1/models, making dynamic loading observable. Cold-start versus warm requests differed by nearly an order of magnitude, underscoring the gap between inference latency and model availability latency.
A developer tested SIE, Superlinked's open-source inference engine, running locally, and found model loading dominated end-to-end time. The server exposes model states via /v1/models, making dynamic loading observable. Cold-start versus warm requests differed by nearly an order of magnitude, underscoring the gap between inference latency and model availability latency.
AWS announced a tiered KV cache feature for large language model inference on SageMaker HyperPod, developed with Curvine. It reduces infrastructure costs and improves time-to-first-token by avoiding recomputation of identical prompts, benefiting teams running models like Qwen, Llama, and DeepSeek.
AWS announced a tiered KV cache feature for large language model inference on SageMaker HyperPod, developed with Curvine. It reduces infrastructure costs and improves time-to-first-token by avoiding recomputation of identical prompts, benefiting teams running models like Qwen, Llama, and DeepSeek.
AWS launched PREFIX_AWARE routing on SageMaker real-time inference endpoints, sending requests with the same prompt prefix to the same instance so KV caches stay warm. Benchmarks with Llama 3.1 70B on vLLM show up to 77% lower P50 time-to-first-token and KV cache hit rates rising from ~25% to over 80%. The routing requires no model container changes and adds only 1.3–1.9 ms of routing overhead per request.
AWS launched PREFIX_AWARE routing on SageMaker real-time inference endpoints, sending requests with the same prompt prefix to the same instance so KV caches stay warm. Benchmarks with Llama 3.1 70B on vLLM show up to 77% lower P50 time-to-first-token and KV cache hit rates rising from ~25% to over 80%. The routing requires no model container changes and adds only 1.3–1.9 ms of routing overhead per request.
An AWS blog describing agent-driven workflows for running SageMaker HyperPod FM infrastructure tasks. LLM agents chain dependent operations—network setup, capacity attachment, storage, fault recovery—that each previously required separate APIs and manual handoffs. Aimed at infrastructure teams operating foundation model training clusters.
An AWS blog describing agent-driven workflows for running SageMaker HyperPod FM infrastructure tasks. LLM agents chain dependent operations—network setup, capacity attachment, storage, fault recovery—that each previously required separate APIs and manual handoffs. Aimed at infrastructure teams operating foundation model training clusters.