Parallelism Strategy Chaining for Fast Training Convergence

2026-09-07

AI Summary

本文提出CONA方法，通过在线策略链解决大型语言模型训练效率问题，使用计算吞吐量和梯度统计构建的代理指标动态调整并行策略，加快达到目标验证困惑度的速度。

Abstract

Selecting a parallelism strategy - the configuration of data, tensor, and pipeline parallelism degrees together with micro- and global-batch sizes - largely determines the training efficiency of large language models. State-of-the-art methods search for a parallelism strategy offline and select the single strategy that minimizes per-iteration time. But we find that they neglect the target validation perplexity and time-to-perplexity (TTP). In particular, our analysis reveals that the best strategy yielding the fastest perplexity improvement changes multiple times during training. As a result, state-of-the-art methods are 1.8-11.4x slower in TTP than the strategy sequence that selects the best strategy at each iteration. This paper proposes CONA, a new training method that introduces online strategy chaining. Instead of a single strategy selected offline, CONA ranks candidate strategies during training using a surrogate metric built from compute throughput and gradient statistics, and switches the current strategy to a new strategy with a higher metric. In our evaluation with GPT-3 1.3B, BERT-Large, and Llama-3.2-1B, CONA reaches the target validation perplexity 1.4-9.6x faster than state-of-the-art methods. Moreover, CONA closely tracks the perplexity achieved by the sequence that selects the best strategy at each iteration, within 2.6%.

Problem

• parallelism strategy
• validation perplexity
• time-to-perplexity
• training efficiency
• large language models

Innovation

• Online Strategy Chaining
• Compute Throughput
• Gradient Statistics
• Time-to-Perplexity (TTP)
• Parallelism Strategy