TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training

2026-05-17

AI Summary

This work addresses the challenge of frequent training interruptions in large language models caused by diverse system failures, which existing single-level checkpointing schemes struggle to mitigate without incurring high overhead or slow recovery. The authors propose a cluster-aware, three-tier hierarchical checkpointing architecture that aligns checkpoint placement with fault heterogeneity: lightweight differential checkpoints are stored in local or peer-node memory for rapid access, while heavyweight base checkpoints are asynchronously migrated to remote persistent storage. This design achieves, for the first time, cross-layer global consistency guarantees matched to fault types, enabling fast, cluster-informed recovery without blocking training progress. Experiments on a 40-billion-parameter model demonstrate total checkpointing times under 10 seconds, significantly reducing training overhead and supporting high-frequency checkpointing.

Abstract

Large Language Model (LLM) training is frequently interrupted by a heterogeneous spectrum of failures, from common GPU crashes to catastrophic cluster-wide outages. Existing checkpointing systems rely on monolithic, single-tier storage backend, forcing a trade-off between state-saving overhead and recovery speed. We propose TierCheck, a cluster-aware tiered checkpointing system that aligns storage placement with failure heterogeneity. TierCheck adopts a three-tier design that maintains lightweight differential checkpoints in local and peer memory for fast localized recovery, while asynchronously migrating heavyweight base checkpoints to remote persistent storage. It also ensures strict global consistency across tiers without stalling training, and achieves fast cluster-aware checkpoint restoration during recovery. Evaluations on models up to 40 billion parameters show that TierCheck achieves low training overhead, reduces end-to-end checkpointing time to under 10s, and supports high-frequency checkpointing, ultimately striking an optimal balance between low-overhead persistence and fast recovery.

Problem

• fault tolerance
• checkpointing
• large language model training
• failure heterogeneity
• storage tiering

Innovation

• tiered checkpointing
• fault tolerance
• large language model training
• cluster-aware recovery
• global consistency