SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clients

2026-01-16

AI Summary

This work addresses the parameter aggregation bias and training instability in federated learning caused by heterogeneous LoRA ranks across clients. To mitigate these issues, the authors propose a selective dual-module LoRA architecture that decouples each client’s adapter into a global module and a local module. Only the global modules are aligned and aggregated across clients, while the local modules remain private. This design inherently supports personalization and seamlessly integrates with differential privacy mechanisms, avoiding semantic degradation and privacy leakage associated with enforcing uniform ranks or shared subspaces. By combining parameter-efficient fine-tuning, selective aggregation, and noise injection, the proposed method achieves superior performance on the GLUE benchmark compared to existing federated LoRA approaches, offering an improved trade-off between utility and privacy.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Privacy-enhancing technologies

Abstract

Federated learning (FL) for large language models (LLMs) has attracted increasing attention as a privacy-preserving approach for adapting models over distributed data, where parameter-efficient methods such as Low-Rank Adaptation (LoRA) are widely adopted to reduce communication and memory costs. However, practical deployments often exhibit rank and data heterogeneity: clients operate under different low-rank budgets and data distributions, making direct aggregation of LoRA updates biased and unstable. Existing approaches either enforce a unified rank or align heterogeneous updates into a single shared subspace, which tends to mix transferable and client-specific directions and consequently undermines personalization. Moreover, under differential privacy (DP), perturbing such structurally mixed updates injects noise into directions that should remain purely local, leading to unnecessary utility degradation. To address these issues, we propose Selective Decoupled Federated LoRA (SDFLoRA), a structure-aware LoRA framework that decouples each client update into a shared component for aggregation and a private component that preserves client-specific semantics. Only the shared component participates in subspace alignment, while the private component remains local and uncommunicated, making the training DP-compatible and stabilizing aggregation under rank heterogeneity. By injecting noise only into the aggregated shareable update, this approach avoids perturbations to local directions and improves the utility-privacy trade-off. Experiments on multiple benchmarks demonstrate that SDFLoRA outperforms federated LoRA baselines and achieves a strong utility-privacy trade-off.

Problem

• federated learning
• rank heterogeneity
• LoRA
• differential privacy
• personalization

Innovation

• LoRA
• federated learning
• rank heterogeneity
• dual-module decomposition
• differential privacy