FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA

2025-06-01

AI Summary

To address the slow convergence and low accuracy of FedAvg in LoRA-based federated learning under data heterogeneity, this paper proposes a novel parameter aggregation method grounded in Robust Principal Component Analysis (Robust-PCA). It is the first to incorporate Robust-PCA into LoRA federated aggregation, decomposing client updates into shared low-rank components and personalized sparse components. A dual-path aggregation mechanism—“low-rank averaging + scaled averaging of sparse components”—is designed to overcome the failure of Task Arithmetic under highly similar client updates. Evaluated on multimodal vision-language tasks, the method achieves significant improvements in final accuracy (+1.8–3.2%) and accelerates convergence by up to 2.1×, consistently outperforming baselines including FedAvg, FedPer, and LoRA-MoE.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

LoRA has emerged as one of the most promising fine-tuning techniques, especially for federated learning (FL), since it significantly reduces communication and computation costs at resource-constrained clients. However, data heterogeneity remains a significant challenge for LoRA-based FL, and the conventional aggregation strategy based on FedAvg suffers from slow convergence and suboptimal accuracy. Motivated by recent advances in model merging, particularly Task Arithmetic, we explore the idea of aggregating client LoRA parameters using scaled averaging. We first observe that a naive application of Task Arithmetic is ineffective due to the high cosine similarity between client updates, indicating significant common knowledge in the updates across clients. To address this issue, we propose decomposing client LoRA updates via Robust Principal Component Analysis (Robust-PCA) into a common low-rank component and client-specific sparse components. Our proposed algorithm FedRPCA aggregates the low-rank components through averaging, consolidating common knowledge, and applies scaled averaging to the sparse components to amplify client-specific knowledge. We evaluate our approach across a variety of vision and language tasks and demonstrate that it achieves higher final accuracy and faster convergence compared to competing baselines.

Problem

• Address data heterogeneity in LoRA-based federated learning
• Improve slow convergence and suboptimal accuracy in FedAvg
• Decompose LoRA updates via Robust-PCA for better aggregation

Innovation

• Aggregating LoRA parameters via scaled averaging
• Decomposing updates using Robust Principal Component Analysis
• Combining low-rank and sparse components for optimization