DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models

2025-09-10

AI Summary

To address the dual challenges of privacy preservation and communication efficiency in federated fine-tuning of large language models (LLMs) on edge devices, this paper proposes a privacy-enhanced low-rank adaptation framework. Our method tightly integrates LoRA adapters with ($varepsilon$, $delta$)-differential privacy: during local training, we apply gradient clipping and Gaussian noise injection to LoRA weight matrices, ensuring unbiased parameter updates. We derive the first theoretical upper bound on the resulting update variance and provide precise privacy budget calibration. Federated averaging is then employed to balance communication efficiency and end-device privacy guarantees. Extensive experiments on standard benchmarks demonstrate state-of-the-art fine-tuning performance while satisfying stringent differential privacy constraints. The method exhibits strong scalability and deployment feasibility in resource-constrained edge environments.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

As on-device large language model (LLM) systems become increasingly prevalent, federated fine-tuning enables advanced language understanding and generation directly on edge devices; however, it also involves processing sensitive, user-specific data, raising significant privacy concerns within the federated learning framework. To address these challenges, we propose DP-FedLoRA, a privacy-enhanced federated fine-tuning framework that integrates LoRA-based adaptation with differential privacy in a communication-efficient setting. Each client locally clips and perturbs its LoRA matrices using Gaussian noise to satisfy ($ε$, $δ$)-differential privacy. We further provide a theoretical analysis demonstrating the unbiased nature of the updates and deriving bounds on the variance introduced by noise, offering practical guidance for privacy-budget calibration. Experimental results across mainstream benchmarks show that DP-FedLoRA delivers competitive performance while offering strong privacy guarantees, paving the way for scalable and privacy-preserving LLM deployment in on-device environments.

Problem

• Privacy risks in federated fine-tuning of on-device LLMs
• Integrating differential privacy with LoRA adaptation efficiently
• Balancing competitive performance with strong privacy guarantees

Innovation

• LoRA-based adaptation with differential privacy
• Local clipping and perturbing LoRA matrices
• Theoretical analysis for unbiased updates and variance bounds