Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning

2022-10-04

AI Summary

How can large pre-trained models be efficiently deployed in federated learning while balancing communication efficiency and model performance? This paper introduces FedPEFT, the first systematic integration of parameter-efficient fine-tuning (PEFT) into federated learning. In FedPEFT, clients update only a small set of trainable modules—e.g., LoRA or Adapters—while the server performs lightweight aggregation of these sparse updates. The framework natively accommodates practical constraints including Non-IID data distributions, client dropouts, and differential privacy requirements. Extensive experiments across multiple federated benchmarks demonstrate that FedPEFT reduces total communication overhead by up to 95% compared to standard baselines, while matching or surpassing the accuracy of FedAvg. These results significantly enhance the practical feasibility of deploying large language models in resource-constrained edge environments.

Abstract

Federated learning (FL) has emerged as a promising paradigm for enabling the collaborative training of models without centralized access to the raw data on local devices. In the typical FL paradigm (e.g., FedAvg), model weights are sent to and from the server each round to participating clients. Recently, the use of small pre-trained models has been shown to be effective in federated learning optimization and improving convergence. However, recent state-of-the-art pre-trained models are getting more capable but also have more parameters, known as the"Foundation Models."In conventional FL, sharing the enormous model weights can quickly put a massive communication burden on the system, especially if more capable models are employed. Can we find a solution to enable those strong and readily available pre-trained models in FL to achieve excellent performance while simultaneously reducing the communication burden? To this end, we investigate the use of parameter-efficient fine-tuning in federated learning and thus introduce a new framework: FedPEFT. Specifically, we systemically evaluate the performance of FedPEFT across a variety of client stability, data distribution, and differential privacy settings. By only locally tuning and globally sharing a small portion of the model weights, significant reductions in the total communication overhead can be achieved while maintaining competitive or even better performance in a wide range of federated learning scenarios, providing insight into a new paradigm for practical and effective federated systems.

Problem

• Federated Learning
• Pre-trained Models
• Communication Efficiency

Innovation

• FedPEFT
• Parameter-Efficient Fine-Tuning
• Communication Cost Reduction