checkpointing

To address the high storage overhead and performance degradation caused by frequent checkpointing in large-scale distributed deep learning, this paper proposes LowDiff, a general-purpose differential checkpointing framework. LowDiff pioneers the extension of differential checkpointing from recommendation systems to generic distributed training scenarios. It innovatively integrates layer-granularity gradient reuse, batched compressed gradient writes, dynamic checkpoint frequency adjustment, CPU-asynchronous persistence, and a lightweight snapshot mechanism—enabling lossless, compression-free per-iteration checkpointing. Experimental results across diverse representative workloads demonstrate that LowDiff incurs less than 3.1% runtime overhead, significantly improving fault tolerance efficiency and training throughput. By offering low-overhead, high-frequency, and broadly applicable fault tolerance, LowDiff establishes a new paradigm for high-performance distributed training.

In large-scale DNN distributed training, checkpointing is tightly coupled with model parallelism strategies and hardware topology, severely limiting fault tolerance and elastic scalability. To address this, we propose the “distributed storage, unified loading” paradigm: during saving, model parameters are stored in a distributed representation aligned with the current parallel configuration; during restoration, they are uniformly reconstructed into a logically consistent parameter view. We design a universal checkpoint format—incorporating merged parameter representations and mapping metadata—a Universal Checkpoint Language (UCL), and an on-demand state reconstruction mechanism, achieving, for the first time, full decoupling of checkpointing from parallel configurations. Evaluated on LLaMA, Bloom, and other mainstream large models under diverse parallelism paradigms—including tensor parallelism (TP), pipeline parallelism (PP), data parallelism (DP), and context parallelism (CP)—our approach reduces post-failure recovery time by 12–28% on average, significantly enhancing cross-configuration portability and system robustness.

Training large foundation models (LFMs) faces significant challenges in checkpoint management, including poor cross-framework compatibility, tight coupling with parallelization strategies, heterogeneous storage backends, and severe I/O bottlenecks. To address these, this work proposes an industrial-grade unified archival system. Its core contributions are: (1) a novel parallelism-agnostic checkpoint serialization format; (2) a full-stack I/O optimization framework integrating a dynamic resharding engine, multi-framework abstraction interfaces (PyTorch/Megatron/DeepSpeed), asynchronous high-throughput storage adapters, and a distributed I/O monitoring toolchain; and (3) runtime support for cross-parallelism resharding, multi-backend adaptivity, and rapid failure recovery. Experiments demonstrate an average 54.20× reduction in checkpoint blocking time, with peak checkpoint save and load speedups of 9.96× and 8.80×, respectively. The system has been stably deployed in production environments scaling to over one thousand GPUs.

Neural ODE training suffers from high computational cost, excessive memory consumption, and numerical instability during backpropagation. To address these challenges, this paper introduces the Algebraically Invertible ODE Solver family, grounded in algebraically invertible numerical integration. The method integrates high-order implicit/explicit reversible schemes, adjoint-state techniques, and memory–computation co-optimization to achieve, for the first time, high-order accuracy, strict numerical stability, and exact gradient computation in backpropagation. Unlike recursive checkpointing, our approach achieves strictly superior time and memory complexity bounds. Extensive evaluation on multiple benchmark ODE tasks demonstrates a 2.1× reduction in training latency and a 68% decrease in GPU memory usage, while preserving gradient precision and numerical robustness.

Intermediate checkpoints in diffusion models (DMs) and consistency models (CMs) are often underutilized, despite evidence that optimal weights frequently reside in non-convex “basins” where SGD fails to converge. Method: We propose LCSC—a learning-based checkpoint selection and combination framework—that employs evolutionary search to automatically learn linear weighting coefficients over trajectory checkpoints, integrates multi-stage weights, and synergistically combines consistency distillation with diffusion sampling optimization. Contribution/Results: LCSC establishes a generalizable checkpoint-weighted averaging paradigm that improves both generation quality and inference efficiency without increasing computational cost at deployment. On CIFAR-10 and ImageNet-64, LCSC achieves up to 23× training speedup; reduces DM sampling NFE from 15 to 9; and enables CM single-step inference to outperform the two-step baseline—demonstrating for the first time that trajectory-weighted averaging can transcend SGD’s convergence limitations, thereby introducing a novel training paradigm for generative models.

Traditional checkpointing in large language model (LLM) training incurs frequent GPU-CPU data transfers, causing computation stalls and reduced throughput. This paper proposes GoCkpt, a low-interference fault-tolerant checkpointing system that distributes I/O overhead across multiple training steps via multi-step overlapped checkpoint transmission, gradient-assisted state reconstruction, and CPU-side consistency recovery. It further employs bandwidth-aware transfer scheduling and dual-layer SSD I/O optimization. Crucially, GoCkpt decouples checkpoint persistence from computation and leverages gradient information to ensure recoverability of CPU-resident model states. Experiments demonstrate that, compared to state-of-the-art asynchronous checkpointing methods, GoCkpt improves training throughput by 38.4%, reduces interruption time by 86.7%, and delivers an additional 4.8% throughput gain.

This work addresses the high storage overhead and resource contention caused by periodic full checkpointing in large language model training. The authors propose a layer-wise pruned checkpoint fusion framework that leverages the non-uniform update patterns across model layers, retaining only significantly updated layers to construct composite checkpoints. This approach enables, for the first time, fine-grained, layer-level control over both model weights and optimizer states, supports flexible integration of diverse selection strategies, and incorporates a delta-aware merging mechanism. Evaluated on Llama3.1-8B and Qwen2.5-7B, the method reduces checkpoint size by 4.3× and decreases saving time by 2.8× while preserving training convergence and final model quality.

Large language models (LLMs) suffer from prohibitive memory overhead during full fine-tuning, primarily due to storing complete activations for exact gradient computation. This work introduces the first optimizer-centric lightweight training framework that eliminates the need for exact gradients. Our method employs efficient low-rank Jacobian approximation to estimate gradients and incorporates an error-feedback correction mechanism to ensure unbiased estimation with bounded variance. Crucially, it is architecture-agnostic—requiring no model structural modifications—and compatible with standard LLMs. Theoretical analysis guarantees convergence properties, while empirical evaluation across diverse downstream tasks demonstrates stable convergence and performance on par with or exceeding conventional baselines. Memory consumption for optimizer states and activations is reduced by up to 50%. Our core contribution lies in decoupling optimization from exact gradient computation, establishing a novel paradigm for memory-efficient full fine-tuning of LLMs.

This work addresses the challenge of excessive peak memory consumption during LoRA fine-tuning of large language models on edge devices, which often exceeds the capacity of consumer-grade hardware. The authors propose a synergistic optimization framework that integrates dynamic dequantization, selective activation caching, and offloading to disk, complemented by a novel semantic-correlation-based token subset approximation of softmax, logits masking, and an efficient checkpointing mechanism. This approach substantially reduces memory footprint with negligible degradation in model performance. Experimental results demonstrate up to 26× and 28× reductions in peak memory usage on Llama-3.2-3B and Qwen-2.5-3B, respectively, enabling billion-parameter-scale models to undergo full LoRA fine-tuning for the first time on resource-constrained devices.

This work addresses the substantial accelerator memory consumption of model parameters, gradients, and optimizer states in standard mixed-precision training, which hinders the scalability of large models. The authors propose a memory-efficient training method that significantly reduces quantization error in 8-bit optimizer states through compact master weight partitioning and a novel compression-expansion function. By integrating 16-bit gradients, an improved weight splitting strategy, and a gradient checkpointing mechanism, the approach remains compatible with mainstream optimizers such as SGD, AdamW, and Lion. The method reduces AdamW’s per-parameter memory footprint from 16 bytes to 7 bytes (or 5 bytes when gradients are released) and halves model checkpoint size, achieving lossless training quality across multiple vision and language benchmark tasks.