Releases·Tool Release
AWS integrates NVIDIA NVRx into PyTorch FSDP on Amazon EKS for fault-tolerant multi-node H100 training
An AWS engineering post details integrating the NVIDIA Resiliency Extension (NVRx) with PyTorch FSDP training on Amazon EKS, combining async checkpointing with two fault-recovery layers: in-process restart and ft_launcher in-job restart. Benchmarks on H100 clusters show async checkpointing sustains 99%+ training efficiency versus 57-61% for synchronous saves, while recovery layers restore training in seconds instead of full job restarts.
0 engagement1 sourceWed, Sep 16, 2026, 06:59 PM
AI Overview
- AWS shows NVRx async checkpointing hits 99%+ training efficiency vs 57-61% sync on 16-64 H100 GPUs, cutting ~40% I/O wall-time loss
- NVRx adds pip-installable recovery layers to FSDP: in-process restart for soft faults and ft_launcher for hard faults like SIGKILL and OOM
- EKS setup pairs p5.48xlarge H100 nodes with EFA networking and FSx Lustre checkpoint storage, tested on Llama-3.1-8B
Entities
Related
Developments·Paper
Simulating fault tolerance with stage skipping in pipeline-parallel training [R]
Templar published a blog on fault tolerance in Crucible, its distributed pre-training platform combining data-parallel replicas with pipeline parallelism. Stage skipping lets activations and gradients bypass an offline pipeline stage for multiple steps, so healthy workers keep training. Simulations with a 178M model showed validation loss stayed close to no-failure baselines at 1% per-replica failure probability, pointing toward training on unreliable workers and spot instances.
Templar published a blog on fault tolerance in Crucible, its distributed pre-training platform combining data-parallel replicas with pipeline parallelism. Stage skipping lets activations and gradients bypass an offline pipeline stage for multiple steps, so healthy workers keep training. Simulations with a 178M model showed validation loss stayed close to no-failure baselines at 1% per-replica failure probability, pointing toward training on unreliable workers and spot instances.
2 engagement1 sourceredditTue, Sep 22, 2026, 03:47 PM
Releases·Tool Release
LM Resiliency open-sources fault localization and fast checkpoint recovery for LLM pre-training
LM Resiliency released an open-source tool for distributed LLM pre-training that localizes faults and accelerates checkpoint recovery. It includes 16 GPU feature tests and a GPU qualification workflow, targeting reliability in large-scale training.
LM Resiliency released an open-source tool for distributed LLM pre-training that localizes faults and accelerates checkpoint recovery. It includes 16 GPU feature tests and a GPU qualification workflow, targeting reliability in large-scale training.
29 engagement1 sourcegithubMon, Aug 10, 2026, 03:10 AM
Releases·Tool Release
GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe
Crusoe engineers Connor Guerrero, Young Jeong, and Nikhil Gupta presented Managed Slurm on Kubernetes at the AI Engineer World's Fair 2026, showing automatic remediation of GPU failures during distributed training. The stack combines Slurm's scheduling with Kubernetes-driven node health and observability, resuming training from checkpoint in under 15 minutes with no human action.
Crusoe engineers Connor Guerrero, Young Jeong, and Nikhil Gupta presented Managed Slurm on Kubernetes at the AI Engineer World's Fair 2026, showing automatic remediation of GPU failures during distributed training. The stack combines Slurm's scheduling with Kubernetes-driven node health and observability, resuming training from checkpoint in under 15 minutes with no human action.
7 engagement1 sourceyoutubeSat, Oct 3, 2026, 09:00 PM
Developments·Paper
Field report details full fine-tuning Qwen3-32B on 16 NVIDIA B300s
A new arXiv paper reports operational experience full-fine-tuning a 32.76B-parameter dense model (Qwen3-32B) on 16 NVIDIA B300 GPUs across two nodes using FSDP/ZeRO-3. It is among the first published field accounts on this accelerator, offering calibrated measurements and a power-draw triage table for diagnosing bottlenecks. The authors claim no new algorithm, focusing instead on integrated field experience and operational hardening.
A new arXiv paper reports operational experience full-fine-tuning a 32.76B-parameter dense model (Qwen3-32B) on 16 NVIDIA B300 GPUs across two nodes using FSDP/ZeRO-3. It is among the first published field accounts on this accelerator, offering calibrated measurements and a power-draw triage table for diagnosing bottlenecks. The authors claim no new algorithm, focusing instead on integrated field experience and operational hardening.
0 engagement1 sourcearxivThu, Aug 6, 2026, 12:11 PM
Releases·Tool Release
Scaling MoE reinforcement learning on Amazon EKS with EFA and DeepEP with 40% more throughput
AWS published an engineering guide for scaling MoE reinforcement-learning post-training (RLHF/GRPO) on Amazon EKS, combining EFA networking, DeepEP expert-parallel communication, and EC2 Spot instances for rollout generation. Amazon's libfabric contributions to DeepEP enable native EFA support, delivering 40% higher aggregate rollout throughput on 48 P5en instances. The work targets the communication bottleneck that increasingly dominates sparse MoE training as expert parallelism spans more nodes.
AWS published an engineering guide for scaling MoE reinforcement-learning post-training (RLHF/GRPO) on Amazon EKS, combining EFA networking, DeepEP expert-parallel communication, and EC2 Spot instances for rollout generation. Amazon's libfabric contributions to DeepEP enable native EFA support, delivering 40% higher aggregate rollout throughput on 48 P5en instances. The work targets the communication bottleneck that increasingly dominates sparse MoE training as expert parallelism spans more nodes.
0 engagement1 sourcerssFri, Sep 25, 2026, 04:29 PM