DeadPool: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint

2026-07-01

AI Summary

Training large language models across thousands of GPUs is highly susceptible to hardware and software failures, yet existing fault-tolerance approaches either incur substantial runtime overhead or suffer from prolonged recovery latency. This work proposes a highly resilient training framework that leverages memory checkpoints on non-critical paths to provide spatial redundancy, combined with runtime communicator reconstruction and an overlapping schedule of computation and checkpointing. The approach achieves zero-overhead execution under fault-free conditions and enables sub-second hot-swapping of failed nodes. For the first time, it unifies fault-tolerance efficiency with training performance: when training a 65-billion-parameter model on 512 A100 GPUs, the system incurs no checkpointing overhead during normal operation and recovers from permanent failures within 40 seconds.

Abstract

State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware stack. Existing fault-tolerance mechanisms either impose non-trivial overhead during failure-free execution or suffer from prolonged recovery latency, particularly under scenarios where a small subset of compute nodes experience permanent failures. %The tradeoff between failure-free overhead and recovery latency forms a space forms a Pareto frontier We present DeadPool to simultaneously address both optimization objectives. DeadPool incorporates a fault-tolerance mechanism that restores LLM training via hot-swapping, namely by replacing failed nodes with spare nodes without terminating the complete job. The hot-swapping of DeadPool is enabled by two ideas: First, it exploits an off-critical-path in-memory checkpointing mechanism for spatial redundancy. Second, it introduces a communicator reconstruction protocol that replaces failed nodes with spare nodes at runtime. DeadPool efficiently overlaps the in-memory checkpointing with computation, thus introducing zero overhead during error-free execution. Upon permanent node failures, DeadPool can rebuild memory states with minimal recomputation by leveraging in-memory checkpoints. We evaluate DeadPool across scales (up to 512 NVIDIA A100 GPUs) and LLMs (up to 65B parameters), and observe zero checkpoint overhead with hot-swapping recovery completing in under 40 seconds. These results show that DeadPool simultaneously achieves both zero-overhead error-free execution and extremely low recovery cost.

Problem

• LLM training
• fault tolerance
• node failure
• checkpointing
• recovery latency

Innovation

• hot-swapping
• zero-overhead checkpointing
• fault tolerance
• in-memory checkpointing
• communicator reconstruction