Order Optimal Task Allocation in Distributed Computing via Interweaved Cliques

2026-04-20

AI Summary

This work addresses the challenge of simultaneously minimizing worst-case communication overhead and computational load for general functions admitting d-ary decompositions in distributed computing. The authors propose a deterministic Interweaved Clique (IC) assignment framework grounded in combinatorial design theory. This approach circumvents the restrictive existence conditions of Steiner systems, thereby revealing for the first time the fundamental scaling laws of the problem over a significantly broader range of parameters, while permitting modest heterogeneity in workers’ storage loads. The constructed IC scheme achieves communication cost within a constant factor of 4e from the information-theoretic lower bound and maintains order-wise optimal computation load.

Technology Category

• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Distributed Search
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We consider a distributed computing system in which a master node coordinates $N$ workers to evaluate a function over $n$ input files, where this function accepts general decomposition. In particular, we focus on the general case where the requested function admits a $d$-uniform decomposition, meaning that it can be decomposed into a set of subfunctions that each depends on a unique $d$-tuple of the $n$ files. Our objective is to design file and task allocations that minimize the worst-case communication from the master to any worker and the worst-case computational load across workers. We first show that the optimal file and task allocation with minimum communication and computation costs admits a natural characterization within combinatorial design theory: it corresponds to a Steiner system $S(t, k, v)$ with $t=d$, $v=n$, and $k \approx \frac{n}{N^{1/d}}$. However, Steiner systems are known to exist only for very restricted parameter regimes. To overcome this limitation, we propose the information-theoretic-inspired \emph{Interweaved Clique (IC) design}, a universal and deterministic allocation framework that relaxes the strict structure of Steiner systems by allowing slight variations in worker file loads. Although slightly suboptimal, the IC design achieves a communication cost within a constant factor $4e$ from our converse, while also maintaining an order-optimal computation cost, thus allowing this work to derive the fundamental scaling laws of this general distributed computing problem for a large range of parameters.

Problem

• distributed computing
• task allocation
• communication cost
• computation load
• function decomposition

Innovation

• Interweaved Clique
• distributed computing
• task allocation
• combinatorial design
• order-optimal