HAPT: Heterogeneity-Aware Automated Parallel Training on Heterogeneous Clusters

2025-09-29

AI Summary

To address low resource utilization in distributed training on heterogeneous GPU clusters, this paper proposes an automated parallel training framework. Methodologically, it introduces: (1) a fine-grained cross-operator parallelism planner that enables hardware-aware, customized operator partitioning for heterogeneous devices; (2) a heterogeneity-aware 1F1B (one-forward-one-backward) scheduling mechanism that dynamically reorders micro-batch execution to maximize compute-communication overlap; and (3) an integrated optimization combining load balancing, communication cost modeling, and cross-cluster memory/bandwidth adaptation. Experimental evaluation on real-world heterogeneous GPU clusters demonstrates that the framework achieves 1.3–1.6× speedup over state-of-the-art systems—including PyTorch DDP and DeepSpeed—while significantly alleviating communication bottlenecks and improving end-to-end training throughput and hardware resource utilization.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

With the rapid evolution of GPU architectures, the heterogeneity of model training infrastructures is steadily increasing. In such environments, effectively utilizing all available heterogeneous accelerators becomes critical for distributed model training. However, existing frameworks, which are primarily designed for homogeneous clusters, often exhibit significant resource underutilization when deployed on heterogeneous accelerators and networks. In this paper, we present Hapt, an automated parallel training framework designed specifically for heterogeneous clusters. Hapt introduces a fine-grained planner that efficiently searches a wide space for the inter-operator parallel strategy, enabling Hapt to alleviate communication overheads while maintaining balanced loads across heterogeneous accelerators. In addition, Hapt implements a heterogeneity-aware 1F1B scheduler that adaptively adjusts the execution timing and ordering of microbatches based on network characteristics, maximizing computation-communication overlap under cross-cluster interconnects while incurring only minimal memory overhead. Our evaluation results show that Hapt can deliver 1.3x-1.6x higher performance on heterogeneous clusters than state-of-the-art training frameworks.

Problem

• Optimizing distributed training on heterogeneous GPU clusters
• Reducing communication overheads across varied network interconnects
• Balancing computational loads among diverse accelerator architectures

Innovation

• Fine-grained planner searches inter-operator parallel strategies
• Heterogeneity-aware scheduler adapts microbatch execution timing
• Maximizes computation-communication overlap with minimal memory overhead