checkpoint management

In large-scale DNN distributed training, checkpointing is tightly coupled with model parallelism strategies and hardware topology, severely limiting fault tolerance and elastic scalability. To address this, we propose the “distributed storage, unified loading” paradigm: during saving, model parameters are stored in a distributed representation aligned with the current parallel configuration; during restoration, they are uniformly reconstructed into a logically consistent parameter view. We design a universal checkpoint format—incorporating merged parameter representations and mapping metadata—a Universal Checkpoint Language (UCL), and an on-demand state reconstruction mechanism, achieving, for the first time, full decoupling of checkpointing from parallel configurations. Evaluated on LLaMA, Bloom, and other mainstream large models under diverse parallelism paradigms—including tensor parallelism (TP), pipeline parallelism (PP), data parallelism (DP), and context parallelism (CP)—our approach reduces post-failure recovery time by 12–28% on average, significantly enhancing cross-configuration portability and system robustness.

Training large foundation models (LFMs) faces significant challenges in checkpoint management, including poor cross-framework compatibility, tight coupling with parallelization strategies, heterogeneous storage backends, and severe I/O bottlenecks. To address these, this work proposes an industrial-grade unified archival system. Its core contributions are: (1) a novel parallelism-agnostic checkpoint serialization format; (2) a full-stack I/O optimization framework integrating a dynamic resharding engine, multi-framework abstraction interfaces (PyTorch/Megatron/DeepSpeed), asynchronous high-throughput storage adapters, and a distributed I/O monitoring toolchain; and (3) runtime support for cross-parallelism resharding, multi-backend adaptivity, and rapid failure recovery. Experiments demonstrate an average 54.20× reduction in checkpoint blocking time, with peak checkpoint save and load speedups of 9.96× and 8.80×, respectively. The system has been stably deployed in production environments scaling to over one thousand GPUs.

This work addresses the prevalent issue in large-model (~100B) training where numerous suboptimal checkpoints—exhibiting significant capability trade-offs—are discarded. We propose a scalable weighted fusion method to transform them into Pareto-optimal models. Methodologically, we introduce a continuous weight optimization framework grounded in linear model merging, integrated with multi-task performance evaluation and gradient-driven multi-objective search. Empirically, we demonstrate for the first time that nearly all checkpoints—including those clearly suboptimal in isolation—contribute meaningfully to high-quality ensembles, challenging the conventional paradigm of selecting only top-performing checkpoints for merging. Experiments show that our fused models systematically outperform individual checkpoints and state-of-the-art merging baselines across diverse tasks—including instruction following and code generation—thereby substantially expanding the overall capability frontier.

To address the high checkpointing overhead and low fault tolerance efficiency in distributed training of ultra-large-scale sparse Mixture-of-Experts (MoE) models, this paper proposes MoC-System, a hybrid checkpointing system. Methodologically, it introduces (1) Partial Expert Checkpointing (PEC), a novel mechanism that selects expert subsets to achieve algorithm-system co-optimization; and (2) a two-level asynchronous checkpointing manager that decouples in-memory snapshotting from persistent storage, integrating fully sharded storage with ZeRO-2 and expert parallelism. Evaluated within the Megatron-DeepSpeed framework, MoC-System reduces per-checkpoint overhead by up to 98.9% while improving average downstream task accuracy by 1.08%—without any accuracy loss. This work constitutes the first systematic solution to efficient fault tolerance for highly scalable MoE model training.

Intermediate checkpoints in diffusion models (DMs) and consistency models (CMs) are often underutilized, despite evidence that optimal weights frequently reside in non-convex “basins” where SGD fails to converge. Method: We propose LCSC—a learning-based checkpoint selection and combination framework—that employs evolutionary search to automatically learn linear weighting coefficients over trajectory checkpoints, integrates multi-stage weights, and synergistically combines consistency distillation with diffusion sampling optimization. Contribution/Results: LCSC establishes a generalizable checkpoint-weighted averaging paradigm that improves both generation quality and inference efficiency without increasing computational cost at deployment. On CIFAR-10 and ImageNet-64, LCSC achieves up to 23× training speedup; reduces DM sampling NFE from 15 to 9; and enables CM single-step inference to outperform the two-step baseline—demonstrating for the first time that trajectory-weighted averaging can transcend SGD’s convergence limitations, thereby introducing a novel training paradigm for generative models.

This work addresses the challenge of resource dynamics in shared clusters during large language model training, where conventional checkpoint-based state migration incurs substantial I/O overhead and GPU stalls. To overcome this, the authors propose ETC, a novel framework that eliminates checkpointing entirely by exploiting state locality and leveraging peer-to-peer GPU communication with communication aggregation to enable storage-free, low-fragmentation elastic state migration. Seamlessly integrated into the Megatron-LM hybrid parallel training system, ETC reduces migration overhead by 2.33–6.37× compared to checkpoint-based approaches across diverse parallelism configurations, significantly enhancing the practicality and efficiency of elastic training for large models.

This work addresses the challenge of frequent training interruptions in large language models caused by diverse system failures, which existing single-level checkpointing schemes struggle to mitigate without incurring high overhead or slow recovery. The authors propose a cluster-aware, three-tier hierarchical checkpointing architecture that aligns checkpoint placement with fault heterogeneity: lightweight differential checkpoints are stored in local or peer-node memory for rapid access, while heavyweight base checkpoints are asynchronously migrated to remote persistent storage. This design achieves, for the first time, cross-layer global consistency guarantees matched to fault types, enabling fast, cluster-informed recovery without blocking training progress. Experiments on a 40-billion-parameter model demonstrate total checkpointing times under 10 seconds, significantly reducing training overhead and supporting high-frequency checkpointing.

This work addresses the “scientific amnesia” problem in continual DPO training, where models retain prior behaviors yet fail to accumulate reusable methodological knowledge. We formalize this phenomenon as a measurable, industrial-scale challenge and introduce a comprehensive diagnostic framework comprising a procedural pipeline, a 30-round HumanEval subdomain benchmark, and five categories of strategy proposers. Leveraging chain-wise training of Qwen2.5-7B-Instruct with FSDP-sharded DPO checkpoints, we evaluate strategies including rule-based scheduling, retrieval-augmented memory, Bayesian optimization, and a meta-scientific reasoning model (MSCL). Experiments reveal that only rule-based scheduling improves performance in homogeneous single-seed chains, while MSCL uniquely enhances outcomes in heterogeneous chains. Across multi-seed settings, retrieval-augmented memory yields the highest average gains, though differences among methods do not reach statistical significance.

This work addresses the inefficiency in notebook-based distributed workflows, where minor modifications often trigger full re-execution, severely hindering iterative development and reproducibility. To overcome this limitation, the authors propose NBRewind, a system that, for the first time, enables fine-grained incremental execution and cross-platform portability while preserving reproducibility. NBRewind integrates a dual-kernel architecture—comprising auditing and replay components—with cell-level incremental checkpoints and inter-cell dataflow analysis. It further leverages standardized notebook packaging to facilitate efficient partial re-execution. Evaluation in real-world high-performance computing (HPC) scenarios demonstrates that NBRewind incurs minimal overhead for incremental checkpointing and substantially improves both execution efficiency and cross-site reproducibility.

This work addresses the lack of resource-centric computational efficiency metrics—specifically in terms of node-hours—for existing supercomputers and large-scale AI training platforms operating under high failure rates. It proposes the first efficiency evaluation framework grounded in resource consumption rather than execution time, unifying failure rate, mean time between failures, and checkpoint/restart overhead into a cohesive resource-based model. The framework extends Daly’s (2006) model to accommodate heterogeneous scientific workloads. Validated on one year of production data from the Frontier supercomputer, the approach leverages runtime log analysis, joint modeling of failures and checkpointing, and optimization algorithms to accurately quantify the expected fraction of resources usable for scientific computation and to determine optimal checkpoint intervals that minimize resource loss.