Poseidon: DAG-Guided Parallelism Search for LLM Pre-Training on Heterogeneous Clusters

2026-09-05

AI Summary

为解决异构集群上大语言模型预训练时硬件利用率低及配置搜索空间过大问题，Poseidon采用基于有向无环图的时间模型和两种高效策略来优化并行化，提高训练效率。

Abstract

With the rapid advancement of accelerator technologies, pre-training large language models (LLMs) on heterogeneous accelerator clusters has become increasingly crucial for maximizing hardware utilization. Existing systems, however, suffer from inaccurate training time modeling, which undermines the parallelization optimizations built upon it. Moreover, for current approaches, the vast configuration search space makes exhaustive exploration infeasible, forcing a trade-off between search time and training efficiency. To overcome these limitations, we introduce Poseidon, an efficient and scalable LLM training framework designed with heterogeneity awareness. Its core is an explicit training time model based on a directed acyclic graph. Building on this graph, Poseidon employs two efficient, theoretically grounded strategies: stage-level pruning via early stopping with partial estimation, and layer-to-stage mapping exploiting a ridge-like distribution pattern. These strategies reduce the search space without sacrificing optimal training efficiency. Experiments on heterogeneous clusters show that Poseidon improves training throughput by up to $2.76\times$ over state-of-the-art systems.

Problem

• heterogeneous clusters
• large language models
• training time modeling
• parallelization optimizations

Innovation

• DAG-Guided
• heterogeneous clusters
• parallelism search
• training time model
• stage-level pruning