Jointly Computation- and Communication-Efficient Distributed Learning

2025-08-21

AI Summary

To address the joint optimization of computational and communication efficiency in distributed learning over undirected networks, this paper proposes a novel distributed optimization algorithm within the ADMM framework. The method uniquely integrates stochastic gradient estimation, multiple local updates, and gradient compression into a single ADMM formulation, while supporting synchronous communication. Theoretically, under strong convexity and smoothness of the objective function, the algorithm achieves exact linear convergence—a first such guarantee for ADMM-based methods under coupled gradient compression and local updates. Empirical evaluations on image classification tasks demonstrate that the proposed method significantly outperforms state-of-the-art distributed SGD, FedADMM, and communication-compression approaches, achieving superior trade-offs among convergence speed, communication overhead, and computational load.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We address distributed learning problems over undirected networks. Specifically, we focus on designing a novel ADMM-based algorithm that is jointly computation- and communication-efficient. Our design guarantees computational efficiency by allowing agents to use stochastic gradients during local training. Moreover, communication efficiency is achieved as follows: i) the agents perform multiple training epochs between communication rounds, and ii) compressed transmissions are used. We prove exact linear convergence of the algorithm in the strongly convex setting. We corroborate our theoretical results by numerical comparisons with state of the art techniques on a classification task.

Problem

• Designing efficient distributed learning over networks
• Achieving computation and communication efficiency jointly
• Using stochastic gradients and compressed transmissions

Innovation

• ADMM-based algorithm with stochastic gradients
• Multiple training epochs between communication rounds
• Compressed transmissions for communication efficiency