Improving Parameter-Efficient Federated Learning with Differentially Private Refactorization

2026-05-08

AI Summary

This work addresses the accuracy degradation in differentially private federated learning when using low-rank adaptation (LoRA), caused by global aggregation errors and noise overwhelming useful signals. To mitigate this, the authors propose FedPower, a novel framework that first reconstructs and truncates full-rank client updates on the server to control sensitivity, then projects them back into a low-rank subspace via a new differentially private low-rank decomposition mechanism, PowerDP. A key innovation lies in calibrating injected noise before orthogonalization in subspace iterations, which effectively preserves matrix orthogonality while simultaneously achieving both sample-level and client-level differential privacy. Experiments demonstrate that FedPower significantly improves the privacy-accuracy trade-off under stringent privacy budgets, incurs low computational overhead, and exhibits robustness and strong privacy guarantees across diverse language understanding tasks and against three types of membership inference attacks.

Abstract

Federated Learning (FL) with parameter-efficient fine-tuning, such as Low-Rank Adaptation (LoRA), enables scalable model training on distributed data. However, when combined with Differential Privacy (DP), LoRA often introduces errors during global aggregation and amplifies the negative effect of DP noise. Existing cross-silo FL approaches mitigate the aggregation error by freezing one LoRA module and applying output perturbation. However, in a restricted low-rank subspaces, this additive noise frequently overwhelms the signals of the weight matrices, leading to suboptimal accuracy. To address this vulnerability, we propose FedPower, a differentially private cross-silo FL framework that reshapes server-side aggregation. Instead of perturbing mismatched low-rank factors, FedPower explicitly reconstructs and clips full-rank client updates to bound the sensitivity. The server then projects the exact aggregated update back into a secure low-rank space using PowerDP, a novel differentially private low-rank factorization mechanism. Based on simultaneous subspace iteration, PowerDP injects calibrated DP noise prior to the final orthonormalization step, effectively mitigates the negative effect of DP noise by preserving matrix orthogonality. We provide rigorous theoretical analyses establishing sensitivity bounds for subspace projections, proving that FedPower achieves both sample-level and client-level DP. Extensive experiments on various language understanding tasks in cross-silo FL settings show that FedPower is robust against tight privacy budgets while adding negligible computational overheads. Additional empirical study on different DP noise injection schemes validates the effectiveness of PowerDP in improving the tradeoff in accuracy and privacy. Evaluation on three different membership inference attacks validates the robustness and privacy-preserving capability of the proposed framework.

Problem

• Federated Learning
• Differential Privacy
• Low-Rank Adaptation
• Parameter-Efficient Fine-Tuning
• Aggregation Error

Innovation

• Federated Learning
• Differential Privacy
• Low-Rank Adaptation
• PowerDP
• Parameter-Efficient Fine-Tuning