FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models

2024-03-14

arXiv.org

AI Summary

In federated learning, frequent gradient communication between heterogeneous clients and the server constitutes a major bottleneck. To address this, we propose FedComLoc—a novel framework built upon the Scaffnew optimizer that, for the first time, deeply integrates Top-K gradient sparsification with low-bit quantization, augmented by an error compensation mechanism to jointly optimize communication compression and convergence stability. FedComLoc supports multi-step local SGD training across heterogeneous devices, substantially reducing uplink communication overhead. Extensive experiments under realistic heterogeneous settings demonstrate over 70% reduction in communication volume, while maintaining convergence speed and model accuracy comparable to non-compressed baselines. This significantly enhances training efficiency and practical feasibility for edge deployment in federated learning.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Federated Learning (FL) has garnered increasing attention due to its unique characteristic of allowing heterogeneous clients to process their private data locally and interact with a central server, while being respectful of privacy. A critical bottleneck in FL is the communication cost. A pivotal strategy to mitigate this burden is emph{Local Training}, which involves running multiple local stochastic gradient descent iterations between communication phases. Our work is inspired by the innovative emph{Scaffnew} algorithm, which has considerably advanced the reduction of communication complexity in FL. We introduce FedComLoc (Federated Compressed and Local Training), integrating practical and effective compression into emph{Scaffnew} to further enhance communication efficiency. Extensive experiments, using the popular TopK compressor and quantization, demonstrate its prowess in substantially reducing communication overheads in heterogeneous settings.

Problem

• Reduce communication costs in federated learning
• Integrate compression with local training strategies
• Enhance efficiency in heterogeneous client settings

Innovation

• Integrating compression into Scaffnew algorithm
• Using TopK compressor and quantization techniques
• Reducing communication overhead in federated learning