FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

2026-08-15

AI Summary

This study addresses the performance degradation in heterogeneous federated LoRA fine-tuning caused by factor aggregation discrepancies and initialization errors. We propose a Product Alignment framework that aggregates heterogeneous rank updates within the product space to reconstruct a global adapter, jointly mitigating aggregation bias and initialization variance while preserving local continuity and global consistency. This approach supports personalized computational budgets without requiring dense communication. We theoretically prove convergence under heterogeneous ranks. Empirical evaluations on NLU and NLG tasks demonstrate significant improvements over baselines, achieving a 6.82 percentage point increase in average GLUE accuracy under heterogeneous settings. These results confirm that our framework enables efficient and robust heterogeneous federated fine-tuning.

Abstract

Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a $6.82$ percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.

Problem

• Federated LoRA
• Aggregation Error
• Initialization Mismatch
• Data Heterogeneity
• Heterogeneous Ranks

Innovation

• Product-Aligned Framework
• Heterogeneous Federated LoRA
• Aggregation Mismatch Mitigation
• Rank-Constrained Global Adapter
• Factor Continuity Preservation