Problem
Research questions and friction points this paper is trying to address.
Decentralized Federated Learning
Prompt Tuning
Non-IID Data
Imbalanced Data
Wasserstein-based Optimization
D-FROST: Decentralized Federated pRompt-tuning via Optimal tranSporT for Non-IID and Imbalanced Data
🤖 AI Summary
本文提出D-FROST算法，通过最优传输方法解决非独立同分布和不平衡数据下的去中心化联邦提示调优问题。
📝 Abstract
Prompt tuning provides a parameter-efficient way to adapt foundation models (FMs) by freezing the pretrained backbone and updating only a small set of learnable prompts. This property makes prompt tuning especially suitable for decentralized federated learning (DFL), where exchanging full-model updates can be prohibitively expensive. However, prompt tuning in DFL introduces new challenges. Prompt sets learned from heterogeneous local data may not be index-wise aligned, making standard decentralized averaging unsuitable. In addition, the algorithm should be theoretically guaranteed to achieve consensus and make progress toward the shared objective. In this work, we provide the first study of prompt tuning in DFL. We formulate decentralized prompt tuning as a Wasserstein-based optimization problem over prompt measures, which captures the set-valued structure of prompts. We then propose D-FROST, an optimal-transport-based (OT-based) decentralized prompt-tuning algorithm that merges neighborhood prompts into compact representative prompt sets through transportation-based matching. We further analyze D-FROST by bounding the Wasserstein consensus error across clients, and establishing convergence of the network-level prompt barycenter to a neighborhood of stationarity. Experiments under heterogeneous client data demonstrate the effectiveness of D-FROST for decentralized prompt tuning.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Decentralized Federated Learning
Prompt Tuning
Optimal Transport
Wasserstein-based Optimization
Non-IID Data
🔎 Similar Papers
No similar papers found.
