FedBaF: Federated Learning Aggregation Biased by a Foundation Model

2024-10-24

arXiv.org

AI Summary

To address model security risks arising from foundational model weight leakage in federated learning (FL), this paper proposes a knowledge injection mechanism during aggregation that avoids distributing foundational model parameters. Methodologically, we design a bias-weighted aggregation strategy integrating gradient alignment and parameter regularization, enabling secure and dynamic incorporation of foundational model knowledge into Vision Transformers, Pre-ResNets, and Transformer-based language models. Our key contribution is the first implicit integration of pre-trained knowledge at the FL aggregation server—preserving model confidentiality while enhancing generalization. Experiments demonstrate significant improvements: image classification accuracy increases by 15.8% (non-IID) and 11.4% (IID); language modeling perplexity decreases by up to 39.2%. The approach notably improves robustness under non-IID data distributions and adversarial settings.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

Foundation models are now a major focus of leading technology organizations due to their ability to generalize across diverse tasks. Existing approaches for adapting foundation models to new applications often rely on Federated Learning (FL) and disclose the foundation model weights to clients when using it to initialize the global model. While these methods ensure client data privacy, they compromise model and information security. In this paper, we introduce Federated Learning Aggregation Biased by a Foundation Model (FedBaF), a novel method for dynamically integrating pre-trained foundation model weights during the FL aggregation phase. Unlike conventional methods, FedBaF preserves the confidentiality of the foundation model while still leveraging its power to train more accurate models, especially in non-IID and adversarial scenarios. Our comprehensive experiments use Pre-ResNet and foundation models like Vision Transformer to demonstrate that FedBaF not only matches, but often surpasses the test accuracy of traditional weight initialization methods by up to 11.4% in IID and up to 15.8% in non-IID settings. Additionally, FedBaF applied to a Transformer-based language model significantly reduced perplexity by up to 39.2%.

Problem

• Enhance FL model accuracy without disclosing foundation model weights
• Address non-IID and adversarial challenges in federated learning
• Improve privacy and security in foundation model adaptation

Innovation

• Federated Learning with foundation model bias
• Dynamic integration of pre-trained model weights
• Confidentiality-preserving FL aggregation method