Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models

2026-02-23

AI Summary

This work addresses the significant performance degradation observed when directly applying Low-Rank Adaptation (LoRA) in differentially private federated learning (DPFL), particularly for large vision models, where it introduces three critical challenges: gradient coupling, noise amplification, and increased model sharpness. To mitigate these issues, the authors propose LA-LoRA, a novel approach that, for the first time, explicitly identifies and resolves these problems through a local alternating update mechanism. This mechanism effectively decouples the gradient interactions between LoRA’s two low-rank matrices and aligns client update directions, thereby suppressing noise amplification and smoothing the parameter space. Under a stringent privacy budget of ε=1, LA-LoRA achieves a 16.83% improvement in test accuracy over the current best baseline, RoLoRA, on Tiny-ImageNet, and establishes state-of-the-art performance on large models such as Swin Transformer and RoBERTa, substantially enhancing training stability and convergence in DPFL.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Security and Privacy: Privacy-enhancing technologies

Abstract

Fine-tuning large vision models (LVMs) and large language models (LLMs) under differentially private federated learning (DPFL) is hindered by a fundamental privacy-utility trade-off. Low-Rank Adaptation (LoRA), a promising parameter-efficient fine-tuning (PEFT) method, reduces computational and communication costs by introducing two trainable low-rank matrices while freezing pre-trained weights. However, directly applying LoRA in DPFL settings leads to performance degradation, especially in LVMs. Our analysis reveals three previously underexplored challenges: (1) gradient coupling caused by the simultaneous update of two asymmetric low-rank matrices, (2) compounded noise amplification under differential privacy, and (3) sharpness of the global aggregated model in the parameter space. To address these issues, we propose LA-LoRA (\textbf{L}ocal \textbf{A}lternating \textbf{LoRA}), a novel approach that decouples gradient interactions and aligns update directions across clients to enhance robustness under stringent privacy constraints. Theoretically, LA-LoRA strengthens convergence guarantees in noisy federated environments. Extensive experiments demonstrate that LA-LoRA achieves state-of-the-art (SOTA) performance on Swin Transformer and RoBERTa models, showcasing robustness to DP noise and broad applicability across both LVMs and LLMs. For example, when fine-tuning the Swin-B model on the Tiny-ImageNet dataset under a strict privacy budget ($ε= 1$), LA-LoRA outperforms the best baseline, RoLoRA, by 16.83\% in test accuracy. Code is provided in \repolink.

Problem

• Federated Learning
• Differential Privacy
• LoRA
• Large Vision Models
• Large Language Models

Innovation

• LA-LoRA
• Differentially Private Federated Learning
• Low-Rank Adaptation
• Gradient Decoupling
• Parameter-Efficient Fine-Tuning