Trivance: Latency-Optimal AllReduce by Shortcutting Multiport Networks

2026-02-19

AI Summary

This work proposes Trivance, an algorithm designed to address the high latency and communication congestion inherent in AllReduce operations over direct-connect topologies such as rings and tori. Trivance leverages three-way hopping communication combined with a fused reduction mechanism on bidirectional multi-port rings, advancing synchronously in both directions at each step. It achieves latency-optimal AllReduce in log₃n steps while reducing network congestion to one-third that of the Bruck algorithm. The approach naturally extends to multidimensional tori and demonstrates significant performance gains in high-bandwidth scenarios: it accelerates messages up to 8 MiB by 5–30% and maintains optimal latency and throughput even for messages as large as 128 MiB in 3D tori. Trivance thus represents the current state-of-the-art latency-optimal AllReduce solution.

Technology Category

• Search and Optimization: Distributed Search
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Multiagent Systems: Agent Communication

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

AllReduce is a fundamental collective operation in distributed computing and a key performance bottleneck for large-scale training and inference. Its completion time is determined by the number of communication steps, which dominates latency-sensitive workloads, and the communication distance affecting both latency- and bandwidth-bound regimes. Direct-connect topologies, such as torus networks used in Google's TPUv4, are particularly prone to large communication distances due to limited bisection bandwidth. Latency-optimal algorithms such as Bruck's complete AllReduce in $\log_3 n$ steps on a bidirectional ring, but incur large communication distances that result in substantial congestion. In contrast, recent approaches such as Swing reduce communication distance and congestion, but are inherently required to perform $\log_2 n$ steps to complete AllReduce, sacrificing latency-optimality. In this paper, we present Trivance, a novel AllReduce algorithm that completes within $\log_3 n$ steps, while reducing congestion compared to Bruck's algorithm by a factor of three and preserving bandwidth-optimality. Trivance exploits both transmission ports of a bidirectional ring within each step to triple the communication distance along both directions simultaneously. Furthermore, by performing joint reductions, Trivance improves both the number of steps and network congestion. We further show that Trivance extends naturally to multidimensional torus networks, retaining its latency advantage while achieving performance comparable to bandwidth-optimal algorithms for large messages. Our empirical evaluation shows that Trivance improves state-of-the-art approaches by 5-30% for message sizes up to 8\,MiB, in high-bandwidth settings up to 32MiB and for 3D tori up to 128MiB. Throughout the evaluation, Trivance remains the best-performing latency-optimal algorithm.

Problem

• AllReduce
• latency-optimal
• network congestion
• communication distance
• distributed computing

Innovation

• AllReduce
• latency-optimal
• congestion reduction
• multiport networks
• torus topology