Decentralized Low-Rank Fine-Tuning of Large Language Models

2025-01-26

AI Summary

To address the challenges of fine-tuning large language models (LLMs) in decentralized, privacy-sensitive settings with scattered data, this paper proposes Dec-LoRA—the first serverless, decentralized low-rank adaptation framework. Dec-LoRA tightly integrates LoRA adapters with gossip-based consensus optimization and gradient compression, eliminating reliance on a central parameter server and enabling efficient collaboration via fully local training and direct peer-to-peer client communication. Experiments on BERT and LLaMA-2 demonstrate that Dec-LoRA incurs less than 1.5% accuracy degradation under Non-IID data while reducing communication overhead by 42%, significantly improving convergence and robustness under heterogeneity and quantization constraints. Its core contribution lies in establishing the first end-to-end decentralized LoRA fine-tuning paradigm, jointly optimizing privacy preservation, communication efficiency, and model performance.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Distributed Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems

Abstract

The emergence of Large Language Models (LLMs) such as GPT-4, LLaMA, and BERT has transformed artificial intelligence, enabling advanced capabilities across diverse applications. While parameter-efficient fine-tuning (PEFT) techniques like LoRA offer computationally efficient adaptations of these models, their practical deployment often assumes centralized data and training environments. However, real-world scenarios frequently involve distributed, privacy-sensitive datasets that require decentralized solutions. Federated learning (FL) addresses data privacy by coordinating model updates across clients, but it is typically based on centralized aggregation through a parameter server, which can introduce bottlenecks and communication constraints. Decentralized learning, in contrast, eliminates this dependency by enabling direct collaboration between clients, improving scalability and efficiency in distributed environments. Despite its advantages, decentralized LLM fine-tuning remains underexplored. In this work, we propose exttt{Dec-LoRA}, an algorithm for decentralized fine-tuning of LLMs based on low-rank adaptation (LoRA). Through extensive experiments on BERT and LLaMA-2 models, we evaluate exttt{Dec-LoRA}'s performance in handling data heterogeneity and quantization constraints, enabling scalable, privacy-preserving LLM fine-tuning in decentralized settings.

Problem

• Decentralized Fine-tuning
• Large Language Models
• Privacy Preservation

Innovation

• Dec-LoRA
• Decentralized Fine-tuning
• Privacy Protection