FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

2026-02-27

AI Summary

This work addresses the limitations of existing federated LoRA fine-tuning approaches, which consider only statistical heterogeneity across client data while neglecting the functional heterogeneity inherent in different layers of large language models, thereby struggling to balance generalization and personalization. To overcome this, we propose FedTreeLoRA, a novel framework that treats statistical and functional heterogeneity as orthogonal yet coupled dimensions. FedTreeLoRA introduces a tree-structured hierarchical aggregation mechanism: shallow layers (“trunk”) share universal knowledge across clients, while deeper layers (“branches”) enable personalized adaptation, with each client dynamically adjusting its depth of parameter sharing. By integrating LoRA with federated learning, our method enables fine-grained, layer-wise parameter sharing. Experiments demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods on both NLU and NLG benchmarks, effectively enhancing both generalization and personalization in heterogeneous federated settings.

Abstract

Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side \textit{statistical heterogeneity} but treated the model as a monolithic block, ignoring the \textit{functional heterogeneity} across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are \textit{orthogonal in source yet coupled in interaction}, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose \textbf{FedTreeLoRA}, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow `trunks'while progressively specializing on deep `branches'. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization.

Problem

• statistical heterogeneity
• functional heterogeneity
• federated learning
• LoRA
• personalization

Innovation

• Federated Learning
• Low-Rank Adaptation
• Statistical Heterogeneity
• Functional Heterogeneity
• Tree-structured Aggregation