PreLort: Prefix-Nested LoRA for Federated Fine-Tuning under Rank Heterogeneity

2026-06-14

AI Summary

This work addresses the challenges in federated fine-tuning caused by heterogeneous adapter ranks across clients, which lead to uncontrolled distribution of low-rank representations and suboptimal aggregation performance. To mitigate these issues, the authors propose a prefix-nested LoRA architecture integrated with a segmented aggregation rule and a multi-rank truncation training strategy. In this framework, the low-rank components prioritize learning task-critical information, while higher-rank components provide additional representational capacity. This design enables efficient and consistent information sharing and aggregation under heterogeneous rank configurations. Experimental results demonstrate that the proposed method significantly outperforms existing heterogeneous federated LoRA approaches across multiple foundation models, achieving higher accuracy and ROUGE-L scores while maintaining comparable or lower perplexity.

Abstract

Federated fine-tuning of large language models using parameter-efficient methods such as LoRA enables privacy-preserving adaptation of foundation models. Heterogeneous hardware resources introduce challenges, as clients with different adapter ranks cannot be directly aggregated. While existing methods enable aggregation under heterogeneous ranks, they fail to control how information is distributed across rank dimensions, leading to suboptimal use of shared low-rank representations. Instead, we propose PreLort: a nested low-rank formulation for federated LoRA that organizes adapter dimensions into a prefix hierarchy. Our approach ensures that lower-rank dimensions encode task-relevant information, while higher-rank dimensions capture additional capacity. Building on this, we introduce (i) a segment-wise aggregation rule that averages only over clients contributing to each rank segment, avoiding dilution from zero-padded lower-rank clients, and (ii) a prefix-nested training strategy that optimizes each adapter under multiple rank truncations, encouraging useful signal to concentrate in low-rank prefix dimensions. Together, these components encourage a consistent low-rank prefix capturing the most task-relevant information, while higher-rank dimensions learn additional capacity. This allows low-rank clients to benefit from richer information contributed by higher-rank clients, as prefix dimensions are consistently learned and aggregated. Experiments demonstrate that our method consistently outperforms prior heterogeneous federated LoRA methods in accuracy and ROUGE-L, while achieving lower or comparable perplexity across multiple base models.

Problem

• Federated Fine-Tuning
• Rank Heterogeneity
• LoRA
• Parameter-Efficient Adaptation
• Low-Rank Representation

Innovation

• Prefix-Nested LoRA
• Federated Fine-Tuning
• Rank Heterogeneity
• Low-Rank Adaptation
• Segment-wise Aggregation