Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA

2026-05-07

AI Summary

This work addresses the semantic inconsistency in existing federated LoRA methods, which neglect gauge equivalence when aggregating low-rank updates. To resolve this, the authors propose GLoRA, a novel approach that estimates a consensus update subspace from clients’ projection matrices and performs gauge-aware low-rank aggregation within a shared reference coordinate system, yielding semantically coherent server-side representations. GLoRA enables direct readout of heterogeneous-rank adapters without requiring dense reconstruction, thereby balancing computational efficiency and model adaptability. Experimental results demonstrate that GLoRA significantly outperforms current methods on the GLUE and SuperNI benchmarks, exhibiting robust performance across diverse settings—including data heterogeneity, resource constraints, task heterogeneity, and even unseen tasks.

Abstract

Federated LoRA enables parameter-efficient adaptation of large language models under decentralized data and limited client resources.However, directly averaging LoRA factors is representation-dependent: the same intrinsic update admits infinitely many gauge-equivalent factorizations, so factor-level aggregation can change under arbitrary coordinate choices while the underlying update remains unchanged. This reveals a semantic mismatch in existing federated LoRA aggregation rules. We propose \textbf{GLoRA}, a gauge-aware server representation for federated LoRA.Instead of aggregating raw factors, GLoRA estimates a consensus update subspace from client projectors and aggregates client updates in shared reference coordinates, thereby representing semantic update aggregation entirely in low-rank form. To support heterogeneous client capacities, GLoRA further provides a rank-compatible readout that instantiates adapters of different ranks from the same server state without dense update reconstruction. Experiments on GLUE and SuperNI show that GLoRA consistently outperforms federated LoRA baselines under data, resource, and task heterogeneity, including heterogeneous client ranks, sparse participation, larger backbones, and unseen-task evaluation. GLoRA also achieves a favorable efficiency--performance trade-off, suggesting that effective federated LoRA requires not merely averaging low-rank factors, but defining a semantically meaningful server-side representation for aggregation.

Problem

• Federated LoRA
• gauge dependence
• low-rank representation
• semantic mismatch
• factor aggregation

Innovation

• Federated LoRA
• gauge invariance
• low-rank representation
• consensus subspace
• rank-compatible adaptation