On the Computation Rate of All-Reduce

2026-02-25

AI Summary

This study investigates the fundamental computation rate limit of the All-Reduce operation in communication networks with arbitrary bandwidth, defined as the maximum number of global summation instances achievable per unit of network resource. Addressing the problem of K nodes collaboratively computing a global sum, the work proposes a “aggregate-then-broadcast” strategy and establishes, for the first time, a general cut-set upper bound alongside a linear programming lower bound based on time–frequency resource sharing. Through information-theoretic analysis and optimized scheduling, the approach achieves tight or near-tight bounds on canonical topologies—including ring, complete graph, and hypercube—demonstrating that the derived upper bound is at most twice the lower bound, thereby attaining state-of-the-art performance.

Technology Category

• Multiagent Systems: Agent Communication
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

In the All-Reduce problem, each one of the K nodes holds an input and wishes to compute the sum of all K inputs through a communication network where each pair of nodes is connected by a parallel link with arbitrary bandwidth. The computation rate of All-Reduce is defined as the number of sum instances that can be computed over each network use. For the computation rate, we provide a cut-set upper bound and a linear programming lower bound based on time (bandwidth) sharing over all schemes that first perform Reduce (aggregating all inputs at one node) and then perform Broadcast (sending the sum from that node to all other nodes). Specializing the two general bounds gives us the optimal computation rate for a class of communication networks and the best-known rate bounds (where the upper bound is no more than twice of the lower bound) for cyclic, complete, and hypercube networks.

Problem

• All-Reduce
• computation rate
• communication network
• bandwidth
• distributed computing

Innovation

• computation rate
• All-Reduce
• cut-set bound
• linear programming
• time-sharing