🤖 AI Summary
This paper addresses the problem of solving distributed/federated linear systems under data heterogeneity. To characterize the joint impact of inter-device and intra-device data distribution discrepancies on convergence, we introduce “angular heterogeneity”—a novel geometric metric that unifies their effects for the first time. We theoretically establish that the Averaged Projection Convergence (APC) method achieves optimal convergence rate in highly heterogeneous settings and derive an explicit quantitative relationship between heterogeneity level and convergence speed. Unlike optimization-based approaches such as Distributed Heavy-Ball Method (D-HBM), APC inherits both the robustness of projection-based algorithms and acceleration capabilities. Numerical experiments on real-world heterogeneous datasets demonstrate that APC significantly outperforms existing baselines, exhibiting faster and more stable convergence—especially under strong heterogeneity.
📝 Abstract
We consider the fundamental problem of solving a large-scale system of linear equations. In particular, we consider the setting where a taskmaster intends to solve the system in a distributed/federated fashion with the help of a set of machines, who each have a subset of the equations. Although there exist several approaches for solving this problem, missing is a rigorous comparison between the convergence rates of the projection-based methods and those of the optimization- based ones. In this paper, we analyze and compare these two classes of algorithms with a particular focus on the most efficient method from each class, namely, the recently proposed Accelerated Projection-Based Consensus (APC) [1] and the Distributed Heavy-Ball Method (D-HBM). To this end, we first propose a geometric notion of data heterogeneity called angular heterogeneity and discuss its generality. Using this notion, we bound and compare the convergence rates of the studied algorithms and capture the effects of both cross-machine and local data heterogeneity on these quantities. Our analysis results in a number of novel insights besides showing that APC is the most efficient method in realistic scenarios where there is a large data heterogeneity. Our numerical analyses validate our theoretical results.
