WallFacer: Harnessing Multi-dimensional Ring Parallelism for Efficient Long Sequence Model Training

2024-06-30

AI Summary

To address high communication overhead and poor scalability in distributed training of long-sequence Transformers, this paper proposes a multidimensional ring-parallel architecture. We introduce a novel tensor-parallel paradigm that decouples communication topology from model structure, incorporating an additional parallel dimension to reduce per-iteration communication volume, and design a dynamic ring-based scheduling mechanism to alleviate bandwidth bottlenecks. The architecture supports cross-modal training for both NLP and CV workloads and is compatible with heterogeneous hardware. Experiments demonstrate 77.12% training speedup on GPT-style models and 114.33% on DiT models—substantially outperforming existing long-sequence training methods—while maintaining seamless scalability to arbitrary sequence lengths.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Training Transformer models on long sequences in a distributed setting poses significant challenges in terms of efficiency and scalability. Current methods are either constrained by the number of attention heads or excessive communication overheads. To address this problem, we propose WallFacer, a multi-dimensional distributed training system for long sequences, fostering an efficient communication paradigm and providing additional tuning flexibility for communication arrangements. Specifically, WallFacer introduces an extra parallel dimension to substantially reduce communication volume and avoid bandwidth bottlenecks. Through comprehensive experiments across diverse hardware environments and on both Natural Language Processing (NLP) and Computer Vision (CV) tasks, we demonstrate that our approach significantly surpasses state-of-the-art methods that support near-infinite sequence lengths, achieving performance improvements of up to 77.12% on GPT-style models and up to 114.33% on DiT (Diffusion Transformer) models.

Problem

• Efficiently trains Transformers on long sequences
• Reduces communication overhead in distributed training
• Enables scalable near-infinite-context model training

Innovation

• Introduces multi-dimensional concentric distributed training system
• Divides peer-to-peer communication into sub-rings
• Adds extra parallel dimension to reduce communication volume