FedBCGD: Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning

2024-10-28

ACM Multimedia

AI Summary

This work addresses the high communication overhead of large-scale models, such as Vision Transformers, in federated learning by proposing Federated Block Coordinate Gradient Descent (FedBCGD) and its accelerated variant, FedBCGD+. The method introduces, for the first time in federated learning, a block-wise parameter communication mechanism that uploads only a subset of parameter blocks per round, combined with stochastic variance reduction and client drift control strategies. Theoretical analysis shows that the communication complexity is reduced by a factor of 1/N compared to existing methods, where N denotes the number of blocks. Experimental results demonstrate that the proposed algorithms achieve faster convergence and higher communication efficiency than current state-of-the-art approaches.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Large Vision Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Blockchains and distributed ledgers

Abstract

Although Federated Learning has been widely studied in recent years, there are still high overhead expenses in each communication round for large-scale models such as Vision Transformer. To lower the communication complexity, we propose a novel Federated Block Coordinate Gradient Descent (FedBCGD) method for communication efficiency. The proposed method splits model parameters into several blocks including a shared block and enables uploading a specific parameter block by each client, which can significantly reduce communication overhead. Moreover, we also develop an accelerated FedBCGD algorithm (called FedBCGD+) with client drift control and stochastic variance reduction. To the best of our knowledge, this paper is the first work on parameter block communication for training large-scale deep models. We also provide the convergence analysis for the proposed algorithms. Our theoretical results show that the communication complexities of our algorithms are a factor 1 /N lower than those of existing methods, where N is the number of parameter blocks, and they enjoy much faster convergence than their counterparts. Empirical results indicate the superiority of the proposed algorithms compared to state-of-the-art algorithms.

Problem

• Federated Learning
• Communication Efficiency
• Large-scale Models
• Communication Overhead

Innovation

• Federated Learning
• Block Coordinate Descent
• Communication Efficiency
• Variance Reduction
• Client Drift Control