An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data

2025-09-05

AI Summary

To address client drift caused by data heterogeneity in federated learning, as well as high communication, computation, and memory overheads, this paper proposes FedSub. The method introduces low-dimensional subspace projection during local updates, constraining model updates to a shared low-dimensional subspace, and employs low-dimensional dual variables to jointly enforce global consistency. Theoretically, we analyze how step size and projection matrices affect convergence rate and provide rigorous convergence guarantees. Empirically, FedSub achieves comparable model accuracy to baseline methods on image and text benchmarks, while reducing communication volume by approximately 60%, decreasing training memory usage by 55%, and significantly lowering computational cost. The algorithm thus offers strong efficiency, provable convergence, and scalability—making it particularly suitable for resource-constrained federated environments.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

This work addresses the key challenges of applying federated learning to large-scale deep neural networks, particularly the issue of client drift due to data heterogeneity across clients and the high costs of communication, computation, and memory. We propose FedSub, an efficient subspace algorithm for federated learning on heterogeneous data. Specifically, FedSub utilizes subspace projection to guarantee local updates of each client within low-dimensional subspaces, thereby reducing communication, computation, and memory costs. Additionally, it incorporates low-dimensional dual variables to mitigate client drift. We provide convergence analysis that reveals the impact of key factors such as step size and subspace projection matrices on convergence. Experimental results demonstrate its efficiency.

Problem

• Addresses client drift from data heterogeneity
• Reduces communication, computation, and memory costs
• Mitigates client drift using low-dimensional dual variables

Innovation

• Subspace projection reduces communication computation costs
• Low-dimensional dual variables mitigate client drift
• Convergence analysis reveals impact of key factors