Releases·Tool Release
AWS launches Ray Serve Deep Learning Containers as TorchServe maintenance ends, offering managed inference migration path
With TorchServe no longer actively maintained — no planned bug fixes, features, or security patches — AWS announced Ray Serve Deep Learning Containers to help teams run and migrate TorchServe model inference workloads. The containers give practitioners a supported serving path as TorchServe compatibility with newer PyTorch and CUDA versions lapses.
0 engagement1 sourceWed, Sep 9, 2026, 03:51 PM
AI Overview
- AWS launched Ray Serve Deep Learning Containers after TorchServe's official notice ended maintenance, with no future bug fixes, features, or security patches.
- The GPU Ray Serve DLC bundles PyTorch, Ray Serve with FastAPI and Uvicorn, and FFmpeg with NVIDIA hardware acceleration on the NVIDIA Amazon Linux 2023 base image.
- AWS demonstrated serving the Qwen3-VL-2B vision-language model on a single g5.xlarge (A10G, 24 GB VRAM) EKS pod, using float16 and exposing HTTP on port 8000.
- Ray Serve replaces TorchServe's handler classes, torch-model-archiver, and config.properties with a @serve.deployment Python class and .bind() registration; KubeRay extends it to multi-node serving.
- Separate DLC images ship for Amazon EKS, EC2, and SageMaker, each with environment-specific entrypoints and shared dependencies, with security patches applied at build time.
Entities
Related
Releases·Tool Release
AWS launches multi-node Ray LLM Deep Learning Container for disaggregated LLM inference on Amazon EKS
AWS published a tutorial deploying a single LLM across two GPU nodes on Amazon EKS using the multi-node Ray LLM Deep Learning Container, with separate prefill and decode nodes behind one OpenAI-compatible endpoint. The container bundles Ray Serve LLM, vLLM, NIXL, and the EFA stack, letting practitioners configure PD disaggregation in YAML without hand-building the GPU dependency stack. This matters for teams running RAG, summarization, and agentic workloads where compute-heavy prefill stalls latency-sensitive decode.
AWS published a tutorial deploying a single LLM across two GPU nodes on Amazon EKS using the multi-node Ray LLM Deep Learning Container, with separate prefill and decode nodes behind one OpenAI-compatible endpoint. The container bundles Ray Serve LLM, vLLM, NIXL, and the EFA stack, letting practitioners configure PD disaggregation in YAML without hand-building the GPU dependency stack. This matters for teams running RAG, summarization, and agentic workloads where compute-heavy prefill stalls latency-sensitive decode.
0 engagement1 sourcerssMon, Oct 12, 2026, 03:50 PM
Releases·Tool Release
AWS adds Ray integration to SageMaker HyperPod for distributed training and serving
AWS announced new Ray capabilities on SageMaker HyperPod, integrating the open-source Ray framework with HyperPod's purpose-built infrastructure for foundation model training and serving. The integration uses KubeRay to manage Ray clusters on Kubernetes, enabling distributed Python workloads such as Ray Train and Ray Serve.
AWS announced new Ray capabilities on SageMaker HyperPod, integrating the open-source Ray framework with HyperPod's purpose-built infrastructure for foundation model training and serving. The integration uses KubeRay to manage Ray clusters on Kubernetes, enabling distributed Python workloads such as Ray Train and Ray Serve.
0 engagement1 sourcerssMon, Aug 24, 2026, 07:32 PM
Releases·Tool Release
AWS and Hugging Face launch one-click model deployment to SageMaker Studio
AWS and Hugging Face announced a new integration that lets developers deploy models from Hugging Face to Amazon SageMaker Studio with a single click, eliminating the previous multi-step process of setting up domains, IAM permissions, and GPU quotas. The move aims to reduce friction for developers moving from model discovery to enterprise deployment.
AWS and Hugging Face announced a new integration that lets developers deploy models from Hugging Face to Amazon SageMaker Studio with a single click, eliminating the previous multi-step process of setting up domains, IAM permissions, and GPU quotas. The move aims to reduce friction for developers moving from model discovery to enterprise deployment.
0 engagement1 sourcerssTue, Jul 7, 2026, 09:15 PM
Products·launched
InferCrane: open-source platform for self-hosted AI model deployment and rollback
InferCrane is an open-source infrastructure platform for operating self-hosted, open-weight, and custom-model inference in production. It provides a stable OpenAI-compatible endpoint that manages the full lifecycle of model deployments, including revision deployment, traffic routing, scaling, monitoring, and rollback. It solves the operational complexity of running AI models in production for developers and teams.
InferCrane is an open-source infrastructure platform for operating self-hosted, open-weight, and custom-model inference in production. It provides a stable OpenAI-compatible endpoint that manages the full lifecycle of model deployments, including revision deployment, traffic routing, scaling, monitoring, and rollback. It solves the operational complexity of running AI models in production for developers and teams.
1 engagement1 sourceredditThu, Aug 27, 2026, 10:12 PM
Releases·Tool Release
axem open sources Shaide, a K8s-native platform for multi-model LLM inference
axem co-founders open sourced Shaide, a Kubernetes-native platform for running multiple LLMs on self-hosted infrastructure. It targets the pain of scaling several models independently across GPU nodes and routing requests between replicas without an external cloud.
axem co-founders open sourced Shaide, a Kubernetes-native platform for running multiple LLMs on self-hosted infrastructure. It targets the pain of scaling several models independently across GPU nodes and routing requests between replicas without an external cloud.
5 engagement2 sourceshackernews, redditTue, Sep 1, 2026, 01:40 PM