UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming

2023-07-31

arXiv.org

AI Summary

Existing automatic parallelization methods separately optimize inter-layer and intra-layer parallelism strategies, leading to suboptimal parallel configurations in distributed training. Method: This paper proposes the first unified framework that jointly optimizes both dimensions by modeling parallel strategy search as a Mixed-Integer Quadratic Programming (MIQP) problem, enabling globally optimal device allocation. It departs from conventional divide-and-conquer optimization paradigms, supporting end-to-end automatic search with architecture-aware adaptation tailored specifically for Transformer models. Contribution/Results: Evaluated on five mainstream Transformer models, our approach achieves up to a 3.80× throughput improvement and reduces parallel strategy search time by up to 107× compared to prior methods. The framework significantly enhances both the efficiency and scalability of distributed training while maintaining rigorous mathematical formulation and practical deployability.

Abstract

Distributed learning is commonly used for training deep learning models, especially large models. In distributed learning, manual parallelism (MP) methods demand considerable human effort and have limited flexibility. Hence, automatic parallelism (AP) methods have recently been proposed for automating the parallel strategy optimization process. Existing AP methods suffer from sub-optimal solutions because they do not jointly optimize the two categories of parallel strategies (i.e., inter-layer parallelism and intra-layer parallelism). In this paper, we propose a novel AP method called UniAP, which unifies inter- and intra-layer automatic parallelism by mixed integer quadratic programming. To the best of our knowledge, UniAP is the first parallel method that can jointly optimize the two categories of parallel strategies to find an optimal solution. Experimental results show that UniAP outperforms state-of-the-art methods by up to 3.80$ imes$ in throughput and reduces strategy optimization time by up to 107$ imes$ across five Transformer-based models.

Problem

• Unify inter-layer and intra-layer parallelism
• Optimize parallel strategy automatically
• Enhance throughput and reduce optimization time

Innovation

• Unifies inter-intra layer parallelism
• Utilizes mixed integer quadratic programming
• Jointly optimizes parallel strategies efficiently