Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity

2026-08-31

AI Summary

本文提出FedRoRA框架，通过分解共享全局方向和个人化秩幅度解决联邦学习中资源和数据异质性问题，提高个性化适应。

Abstract

Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismatches for aggregation but typically provide a unified global model for all clients sharing the same rank, failing to capture client-specific features in non-IID scenarios. In this paper, we propose FedRoRA (Federated Rank-wise Personalized LoRA), a novel framework that enables fine-grained personalization within rank-heterogeneous federations. FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales. On the server side, it extracts a global subspace via singular value decomposition (SVD) and redistributes client-specific initializations through a personalized projection and top-$k$ selection mechanism. Extensive experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.

Problem

• Federated Learning
• Low-Rank Adaptation
• Rank Heterogeneity
• Personalization

Innovation

• FedRoRA
• Rank Heterogeneity
• Personalization
• Singular Value Decomposition (SVD)
• Top-k Selection