collective communication optimization

In large-scale distributed ML training, collective communication primitives (e.g., AllReduce, AllGather, Broadcast) generate high-bandwidth, bursty traffic, causing network congestion and packet loss. Method: This work systematically characterizes the communication behavior of mainstream LLMs—including DeepSeek-V3, GPT, and Llama—under diverse parallelism strategies, scales, and network topologies. Leveraging fine-grained empirical analysis of NVIDIA NCCL logs, we quantitatively identify how operation type, message size, and request distribution affect network anomalies. We propose a resource-coordinated optimization framework tailored to LLM communication patterns, jointly optimizing collective primitive scheduling and network topology adaptation. Contribution/Results: Experiments demonstrate that our approach significantly mitigates congestion, improving both communication efficiency and stability in distributed training and inference—without modifying model architecture or training algorithms.

Existing direct-connect network topologies exhibit inadequate adaptability to diverse scales, node degrees, and latency–bandwidth trade-offs in high-performance computing (HPC) collective communication. Method: This paper proposes an iterative expansion framework grounded in small-scale optimal base topologies. It introduces, for the first time, a graph-synthesis-driven automatic topology generation mechanism and designs the first polynomial-time collective communication scheduling algorithm for canonical large-scale topologies—including Dragonfly and Fat-Tree. Contribution/Results: The work unifies topology synthesis and scheduling optimization within a single modeling framework, enabling cross-platform deployment and large-scale simulation validation. Experimental evaluation demonstrates that the proposed approach reduces average communication latency by 23.6% and improves bandwidth utilization by 31.4% compared to conventional topologies, while significantly enhancing scalability and practical applicability in real-world HPC systems.

To address throughput bottlenecks in collective communication (e.g., all-reduce) during large language model training over heterogeneous interconnects—such as hybrid switch fabrics and direct-attached accelerators—this paper proposes the first theoretically optimal broadcast/aggregation tree scheduling framework supporting arbitrary topologies with strongly polynomial-time solvability. Our method constructs throughput-optimal spanning trees via graph-theoretic modeling, establishes a topology-agnostic universal scheduling formulation, and introduces a lightweight runtime adaptation layer. Evaluated on AMD MI250 and NVIDIA DGX A100 platforms, the framework achieves significant throughput improvements over RCCL/NCCL, yielding measurable LLM training acceleration. Moreover, its scheduling generation is both faster and higher-quality than current state-of-the-art methods, uniquely combining provable optimality with practical deployability.

This work addresses the significant communication bottleneck in multi-GPU training caused by the serial execution of computation and communication. The authors propose a portable runtime mechanism that requires no modifications to vendor libraries or kernels. By dynamically controlling on-chip resource occupancy of compute kernels, elevating the scheduling priority of communication streams, and leveraging shared memory for compute footprint management and cross-GPU resource coordination, the approach effectively enables concurrent execution of computation and collective communication. Evaluated on NVIDIA A40, A100, H100, and AMD MI250X GPUs, the method reduces end-to-end training time by up to 25.5%.

Existing schedulers for distributed machine learning jobs in multi-tenant torus-topology clusters struggle to simultaneously optimize communication efficiency—constrained by task shape—and cluster utilization, often facing an inherent trade-off between the two. This paper proposes the first joint dynamic adaptation framework that co-optimizes task shape and optical circuit-switched topology. By identifying isomorphic task shapes and enabling real-time topology reconfiguration, our approach jointly satisfies job placement and communication requirements while minimizing network contention and maximizing resource utilization. Evaluations on a 4,096-node torus cluster simulator demonstrate that our method improves absolute cluster utilization by 57% and reduces job completion time by up to 11×, significantly surpassing the performance limits of conventional schedulers.

This work addresses the performance bottleneck in large-scale AI training and inference caused by inefficient overlap between computation and communication, as well as high communication overhead. To this end, the authors develop a customized collective communication library for the Meta MTIA 300 accelerator, integrating a backend network within the chip package for the first time. By combining near-memory computing (NMC) with a dedicated message engine (ME), the design enables full communication offload. The paper introduces a compiler-driven communication model, topology-aware algorithms, and one-sided communication primitives tailored for inference, optimizing collective operations across heterogeneous scale-up and scale-out networks. Experiments demonstrate that, in training scenarios, intra-rack collective bandwidth reaches 940 GB/s with less than 0.5% impact on concurrent compute throughput; in inference, communication latency is significantly reduced, greatly enhancing compute-communication pipeline efficiency.

This study systematically investigates the impact of network topology on collective communication performance in large-scale machine learning training. Focusing on Clos (fat-tree) and torus topologies, the work proposes an analytical model to quantitatively evaluate their completion times for operations such as AllReduce, while jointly accounting for network failures and task placement strategies. It establishes, for the first time, a quantitative relationship between topology structure and collective communication efficiency, demonstrating that Clos topologies consistently outperform torus networks across most scenarios—particularly in terms of communication latency, fault tolerance, and scheduling flexibility. These findings provide a rigorous theoretical foundation for designing network architectures tailored to ML training clusters.

Collective communication in distributed machine learning often becomes a performance bottleneck due to the neglect of physical network topology and process group structure. This work proposes a scalable and general framework for synthesizing collective communication algorithms that, for the first time, incorporates process-group awareness into algorithm generation, supporting arbitrary communication patterns. By integrating topology-aware modeling with optimized search strategies, the framework automatically generates high-performance communication algorithms tailored to the actual process groups and underlying network topology. Experimental results demonstrate that the framework can synthesize an All-to-All algorithm for a 512-NPU system within 11.68 minutes, achieving performance close to the theoretical optimum.

This work addresses the challenge of efficiently exploring the vast and physically constrained design space of cross-layer heterogeneous systems to support mixed AI and high-performance computing (HPC) workloads. To this end, the authors propose CHASE, a novel framework that decouples hardware architecture design from task mapping. CHASE leverages hierarchical type graphs for system modeling, a topology-aware mapper, and a telemetry-guided optimizer to enable application-driven architecture search under deployment constraints. Experimental results demonstrate that CHASE achieves geometric mean speedups of 6.20× and 2.12× on sparse computing and large language model workloads, respectively, while reducing mapping time by 60.5% on average and converging to near-global-optimal solutions within 64 iterations.

This work addresses the performance bottlenecks caused by communication complexity in large-scale heterogeneous chiplet systems by proposing a topology-agnostic dynamic computation migration framework. Instead of merely relocating data, the framework innovatively migrates entire computational contexts—including both code and associated data—to more favorable locations. It integrates a multi-bandwidth-domain chiplet architecture, a hierarchical routing mechanism, and a lightweight machine learning–assisted traffic prediction and scheduling strategy to enable communication-aware load placement and adaptive routing optimization. Experimental results demonstrate migration success rates of 75.2%–97.9%, average latency reductions of 16.4%–62.5%, and up to a 12.5× improvement in throughput. Under large language model (LLM) workloads, the system achieves average improvements of 4.9× in execution time, 5.9× in throughput, and 1.8× in energy efficiency.