GoCkpt: Gradient-Assisted Multi-Step overlapped Checkpointing for Efficient LLM Training

2025-11-10

AI Summary

Traditional checkpointing in large language model (LLM) training incurs frequent GPU-CPU data transfers, causing computation stalls and reduced throughput. This paper proposes GoCkpt, a low-interference fault-tolerant checkpointing system that distributes I/O overhead across multiple training steps via multi-step overlapped checkpoint transmission, gradient-assisted state reconstruction, and CPU-side consistency recovery. It further employs bandwidth-aware transfer scheduling and dual-layer SSD I/O optimization. Crucially, GoCkpt decouples checkpoint persistence from computation and leverages gradient information to ensure recoverability of CPU-resident model states. Experiments demonstrate that, compared to state-of-the-art asynchronous checkpointing methods, GoCkpt improves training throughput by 38.4%, reduces interruption time by 86.7%, and delivers an additional 4.8% throughput gain.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Safety and Robustness
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The accuracy of large language models (LLMs) improves with increasing model size, but increasing model complexity also poses significant challenges to training stability. Periodic checkpointing is a key mechanism for fault recovery and is widely used in LLM training. However, traditional checkpointing strategies often pause or delay GPU computation during checkpoint saving for checkpoint GPU-CPU transfer, resulting in significant training interruptions and reduced training throughput. To address this issue, we propose GoCkpt, a method to overlap checkpoint saving with multiple training steps and restore the final checkpoint on the CPU. We transfer the checkpoint across multiple steps, each step transfers part of the checkpoint state, and we transfer some of the gradient data used for parameter updates. After the transfer is complete, each partial checkpoint state is updated to a consistent version on the CPU, thus avoiding the checkpoint state inconsistency problem caused by transferring checkpoints across multiple steps. Furthermore, we introduce a transfer optimization strategy to maximize GPU-CPU bandwidth utilization and SSD persistence throughput. This dual optimization overlapping saves across steps and maximizing I/O efficiency significantly reduces invalid training time. Experimental results show that GoCkpt can increase training throughput by up to 38.4% compared to traditional asynchronous checkpoint solutions in the industry. We also find that GoCkpt can reduce training interruption time by 86.7% compared to the state-of-the-art checkpoint transfer methods, which results in a 4.8% throughput improvement.

Problem

• Reduces training interruptions from GPU-CPU checkpoint transfers
• Solves checkpoint state inconsistency across multiple training steps
• Maximizes I/O efficiency during checkpoint saving and restoration

Innovation

• Overlaps checkpoint saving with multiple training steps
• Transfers partial states and gradient data progressively
• Optimizes GPU-CPU bandwidth and SSD persistence throughput