Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach

2026-08-10

AI Summary

This work addresses the ambiguity in selecting which LoRA factor—A or B—to share when fine-tuning large language models in federated learning. The authors propose FedAS-LoRA, which reveals for the first time that sharing factor A aligns input subspaces while sharing factor B aligns output subspaces. They introduce a Rank-Aware Shared-Subspace Sufficiency (RSS) metric to adaptively determine the superior sharing strategy based on local data distributions prior to training. FedAS-LoRA leverages frozen backbone representations and employs a least-squares proxy model combined with RSS to efficiently evaluate subspace sufficiency and guide the selection. Extensive experiments demonstrate that FedAS-LoRA consistently outperforms existing federated LoRA approaches across diverse tasks, non-IID settings, LoRA ranks, and client participation ratios.

Abstract

Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e., $A$ and $B$, providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether $A$ should be shared across clients while $B$ remains client-specific (Share-A/Local-B), or whether $B$ should instead be shared while $A$ remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-$r$ input-side space, whereas Share-B/Local-A requires a common rank-$r$ output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-$r$ input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.

Problem

• Federated Learning
• Low-Rank Adaptation
• Factor Sharing
• LoRA
• Large Language Models

Innovation

• Federated Learning
• Low-Rank Adaptation
• Factor Sharing
• Adaptive Strategy
• Subspace Sufficiency