FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation

2025-05-24

AI Summary

In federated learning, heterogeneous low-rank adaptation (LoRA) fine-tuning lacks theoretical convergence guarantees due to client-specific rank selection, which induces parameter truncation and gradient drift—key causes of performance degradation. To address this, we propose the first unbiased weighted aggregation mechanism: leveraging the full-rank global model as a calibration reference, we derive optimal aggregation weights analytically and rigorously establish an $O(1/sqrt{T})$ convergence rate. Our method jointly exploits LoRA’s structural properties and gradient drift analysis. Empirically, it achieves 1–3% accuracy gains over state-of-the-art methods across multiple real-world datasets. Crucially, this work delivers the first rigorous convergence proof—and accompanying empirical validation—for heterogeneous LoRA-based federated fine-tuning.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Federated Learning (FL) facilitates the fine-tuning of Foundation Models (FMs) using distributed data sources, with Low-Rank Adaptation (LoRA) gaining popularity due to its low communication costs and strong performance. While recent work acknowledges the benefits of heterogeneous LoRA in FL and introduces flexible algorithms to support its implementation, our theoretical analysis reveals a critical gap: existing methods lack formal convergence guarantees due to parameter truncation and biased gradient updates. Specifically, adapting client-specific LoRA ranks necessitates truncating global parameters, which introduces inherent truncation errors and leads to subsequent inaccurate gradient updates that accumulate over training rounds, ultimately degrading performance. To address the above issues, we propose extbf{FedHL}, a simple yet effective extbf{Fed}erated Learning framework tailored for extbf{H}eterogeneous extbf{L}oRA. By leveraging the full-rank global model as a calibrated aggregation basis, FedHL eliminates the direct truncation bias from initial alignment with client-specific ranks. Furthermore, we derive the theoretically optimal aggregation weights by minimizing the gradient drift term in the convergence upper bound. Our analysis shows that FedHL guarantees $mathcal{O}(1/sqrt{T})$ convergence rate, and experiments on multiple real-world datasets demonstrate a 1-3% improvement over several state-of-the-art methods.

Problem

• Addresses biased gradient updates in federated heterogeneous LoRA
• Eliminates truncation errors in global parameter aggregation
• Ensures formal convergence guarantees for heterogeneous LoRA adaptation

Innovation

• Unbiased aggregation via full-rank global model
• Optimal weights minimizing gradient drift
• Heterogeneous LoRA adaptation in FL