Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

2026-07-15

AI Summary

In federated learning with heterogeneous clients fine-tuning large models via LoRA, data-parameter interference leads to unstable adapter aggregation. This work is the first to formulate this issue as a subspace alignment problem and introduces Dysco, a dynamic subspace augmentation method. Dysco extracts data-agnostic subspace bases from local activations each round, enabling the server to construct client-specific merged subspaces optimized for compatibility and mitigate representation drift through a multi-round augmentation mechanism. Theoretical analysis yields a closed-form solution for a fixed server-side merged subspace. Experiments demonstrate that Dysco reduces training loss by up to 9× over baselines on synthetic tasks and improves the performance of five federated algorithms by up to 4.3% on MIMIC-IV clinical text classification, with only a 0.9% increase in runtime overhead, significantly outperforming existing federated LoRA approaches.

Abstract

Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable. We identify the data-parameter interference as a geometric source of this instability. This interference is controlled by the alignment between LoRA update subspaces and client activations, suggesting that federated LoRA aggregation should be viewed not only as parameter averaging but also as subspace allocation. We propose Dynamic Subspace Boosting (Dysco), a plug-in method that allocates client-specific LoRA subspaces in a federated and dynamic manner. In each round, clients compute activation-insensitive subspaces from local representations and transmit only the resulting bases; the server then constructs client-specific merged subspaces through a closed-form solution that maximizes compatibility with other clients' insensitive directions. To handle representation drift, Dysco performs multi-round subspace boosting to preserve past update directions while adapting to future representations. We provide a convergence analysis that embeds the data-parameter interference as an aggregation-error term in a standard federated optimization bound, and prove that Dysco's server-fixed merged subspaces yield a tighter upper bound on this error. Experiments on controlled synthetic federated tasks and on MIMIC-IV clinical-note classification with Llama-3.2-1B show that Dysco substantially reduces interference, reduces the final-round synthetic training loss by up to 9 times relative to baselines under the orthogonal-subspace partition the theory identifies, improves all five tested FL algorithms by up to 4.3% on MIMIC, outperforms recent federated LoRA methods, and adds only 0.9% wall-clock overhead. Our code is available at https://github.com/illidanlab/Dysco.

Problem

• Federated Learning
• Low-Rank Adaptation
• Subspace Interference
• Client Heterogeneity
• Parameter Aggregation

Innovation

• Dynamic Subspace Boosting
• LoRA Interference
• Federated Learning
• Subspace Allocation
• Activation-Insensitive Subspaces