🤖 AI Summary
To address the convergence difficulties and high communication overhead of large language models (LLMs) in federated learning (FL) caused by data heterogeneity, this paper introduces FedLLM—the first unified analytical framework for LLMs in FL. It systematically surveys two dominant paradigms: federated fine-tuning and federated prompt learning, while rigorously analyzing core challenges including data heterogeneity, communication efficiency, and privacy preservation. The work identifies promising future directions—namely, federated pre-training and LLM-augmented FL—and fills a critical gap in systematic literature review. A multidimensional taxonomy and evaluation framework is established to clarify key technical bottlenecks. Integrating insights from FL, LLM adaptation, prompt engineering, distributed optimization, and privacy-preserving computation, this study delivers a practical, robust, and privacy-aware methodology for deploying LLMs in real-world federated settings. (149 words)
📝 Abstract
Large language models are rapidly gaining popularity and have been widely adopted in real-world applications. While the quality of training data is essential, privacy concerns arise during data collection. Federated learning offers a solution by allowing multiple clients to collaboratively train LLMs without sharing local data. However, FL introduces new challenges, such as model convergence issues due to heterogeneous data and high communication costs. A comprehensive study is required to address these challenges and guide future research. This paper surveys Federated learning for LLMs (FedLLM), highlighting recent advances and future directions. We focus on two key aspects: fine-tuning and prompt learning in a federated setting, discussing existing work and associated research challenges. We finally propose potential research directions for federated LLMs, including pre-training and how LLMs can further enhance federated learning.