Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models

2025-01-23

AI Summary

To address the challenge of simultaneously ensuring privacy preservation and personalized learning in multimodal large language models (MLLMs) for applications such as customer support, this paper proposes a differentially private federated prompt learning framework. Our method introduces a novel two-tier differential privacy mechanism: local differential privacy (LDP) is applied to the two low-rank components of locally trained prompts, while global differential privacy (GDP) is enforced on the aggregated global prompt. By integrating Low-Rank Adaptation (LoRA) with residual representation, we effectively decouple generalization capability from personalized modeling. Extensive experiments on multiple multimodal benchmarks demonstrate that our approach significantly outperforms existing federated prompt learning methods. Under privacy budgets ε = 2–8, it maintains high accuracy, improves personalization performance by 12.7%, and reduces privacy-induced accuracy degradation by 41%. To the best of our knowledge, this is the first work achieving synergistic optimization of privacy, personalization, and generalization in federated multimodal prompt learning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Multimodal Large Language Models (LLMs) are pivotal in revolutionizing customer support and operations by integrating multiple modalities such as text, images, and audio. Federated Prompt Learning (FPL) is a recently proposed approach that combines pre-trained multimodal LLMs such as vision-language models with federated learning to create personalized, privacy-preserving AI systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalization can lead to overfitting, reducing generalizability, while stringent privacy measures, such as differential privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank adaptation scheme to capture generalization while maintaining a residual term that preserves expressiveness for personalization. To ensure privacy, we introduce a novel method where we apply local differential privacy to the two low-rank components of the local prompt, and global differential privacy to the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks.

Problem

• Privacy Preservation
• Personalized Learning
• Adaptive Multimodal Information

Innovation

• Differential Privacy
• Federated Prompt Learning
• Personalization vs Generalization