Direct Model State Migration for Elastic Training of Large Language Models

2026-07-06

AI Summary

This work addresses the challenge of resource dynamics in shared clusters during large language model training, where conventional checkpoint-based state migration incurs substantial I/O overhead and GPU stalls. To overcome this, the authors propose ETC, a novel framework that eliminates checkpointing entirely by exploiting state locality and leveraging peer-to-peer GPU communication with communication aggregation to enable storage-free, low-fragmentation elastic state migration. Seamlessly integrated into the Megatron-LM hybrid parallel training system, ETC reduces migration overhead by 2.33–6.37× compared to checkpoint-based approaches across diverse parallelism configurations, significantly enhancing the practicality and efficiency of elastic training for large models.

Abstract

Large language model (LLM) training shall adapt to dynamic resources in shared clusters to tackle the elasticity, including passive preemption and optimistic scaling. State migration across device sets is required when altering the hybrid-parallel configuration due to dynamic resources. Existing solutions rely on checkpoint-based mechanisms, which persist complete states to storage for resuming with re-assigned resources, forcing all GPUs to stall when transferring model states. Despite optimization efforts, checkpoint-based solutions incur prohibitive latency due to data movement across memory hierarchies. We propose ETC, a checkpoint-free state migration framework for elastic hybrid-parallel LLM training. We exploits the state locality to minimize inter-GPU data movement, replacing storage persistence with direct peer-to-peer communication. Besides, we eliminate node fragmentation through communication coalescing. Integrated with Megatron-LM, ETC reduces migration overhead by 2.33$\times$ to 6.37$\times$ compared to checkpoint-based solutions across diverse parallel configurations. By enabling efficient migration, ETC unlocks practical elastic training in production environments.

Problem

• elastic training
• large language models
• state migration
• hybrid-parallel
• checkpoint overhead

Innovation

• elastic training
• state migration
• checkpoint-free
• hybrid-parallel
• peer-to-peer communication