Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuning

2026-01-10

arXiv.org

AI Summary

This work identifies a novel membership inference attack (MIA) threat in federated prompt tuning, wherein a malicious server can exploit soft prompts to determine whether a client’s private dataset contains a specific sample. The paper introduces PromptMIA, the first method to expose this privacy vulnerability, formalizing it as a security game and establishing a theoretical lower bound on the attacker’s advantage. By injecting adversarial soft prompts and monitoring their updates during training, PromptMIA achieves high-accuracy membership inference across multiple benchmark datasets. Crucially, conventional defenses based on gradient perturbation or output sanitization prove ineffective against this attack, revealing significant limitations of existing privacy-preserving mechanisms in the emerging paradigm of federated prompt-based learning.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Prompt Engineering / Prompting
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Membership inference attack (MIA) poses a significant privacy threat in federated learning (FL) as it allows adversaries to determine whether a client's private dataset contains a specific data sample. While defenses against membership inference attacks in standard FL have been well studied, the recent shift toward federated fine-tuning has introduced new, largely unexplored attack surfaces. To highlight this vulnerability in the emerging FL paradigm, we demonstrate that federated prompt-tuning, which adapts pre-trained models with small input prefixes to improve efficiency, also exposes a new vector for privacy attacks. We propose PromptMIA, a membership inference attack tailored to federated prompt-tuning, in which a malicious server can insert adversarially crafted prompts and monitors their updates during collaborative training to accurately determine whether a target data point is in a client's private dataset. We formalize this threat as a security game and empirically show that PromptMIA consistently attains high advantage in this game across diverse benchmark datasets. Our theoretical analysis further establishes a lower bound on the attack's advantage which explains and supports the consistently high advantage observed in our empirical results. We also investigate the effectiveness of standard membership inference defenses originally developed for gradient or output based attacks and analyze their interaction with the distinct threat landscape posed by PromptMIA. The results highlight non-trivial challenges for current defenses and offer insights into their limitations, underscoring the need for defense strategies that are specifically tailored to prompt-tuning in federated settings.

Problem

• membership inference attack
• federated learning
• prompt tuning
• privacy attack
• soft prompts

Innovation

• federated prompt-tuning
• membership inference attack
• soft prompts
• privacy attack
• PromptMIA