Collective Communication for Distributed LLM Systems: Planning, Runtime Adaptation, and Computation Coordination

2026-08-15

AI Summary

This study addresses collective communication bottlenecks in distributed large language models operating across heterogeneous hardware and hybrid parallelism. We propose a three-tier optimization framework centered on collective communication that integrates a communication taxonomy with topology-aware scheduling, dynamic GPU runtime mapping, and computation-communication overlap. This approach achieves system-level optimization spanning planning, execution, and end-to-end coordination. Furthermore, this work establishes a generalized optimization paradigm tailored for multi-NIC heterogeneous interconnects, significantly enhancing both training and inference efficiency. By systematically addressing these challenges, the proposed framework not only improves performance in complex distributed environments but also delineates critical future research directions for communication optimization in large-scale distributed systems.

Abstract

Distributed large language model (LLM) systems increasingly rely on collective communication primitives such as AllReduce (AR), ReduceScatter (RS), AllGather (AG), and AlltoAll (A2A). In modern LLM training and serving clusters, heterogeneous GPU interconnects, multi-NIC networking, mixed parallelism strategies, low-latency inference requests, and high-throughput training pipelines have motivated increasingly diverse ways to plan, execute, and overlap collective communication. This paper presents a tutorial-style, collective-centric taxonomy for collective communication. We organize recent advances into three layers: communication planning, which generates topology-aware collective schedules; communication execution and adaptation, which maps these schedules onto GPU runtimes and hardware in real clusters; and computation-communication coordination, which turns collective optimization into end-to-end training and inference benefits. We further discuss open challenges and future opportunities for collective communication in distributed LLM systems.

Problem

• Collective Communication
• Distributed LLM Systems
• Communication Planning
• Runtime Adaptation
• Computation Coordination

Innovation

• Collective Communication
• Communication Planning
• Runtime Adaptation
• Computation Coordination
• Distributed LLM