DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models

2026-03-25

AI Summary

This work identifies a novel identity-linkage privacy threat wherein fine-tuning vision-language models (VLMs) on a small set of private user images can inadvertently leak sensitive identity information and associated attributes. To address this, we formalize the threat model for the first time, construct the first identity-linkage benchmark dataset, and propose DP²-VL—the first adversarial data poisoning method specifically designed to protect privacy in VLMs. DP²-VL injects imperceptible perturbations that induce dataset-level shifts in the encoder’s embedding space, causing attackers to overfit to spurious features during fine-tuning. Extensive experiments demonstrate that DP²-VL achieves strong generalization, robustness, and consistent protection across diverse mainstream VLMs—including LLaVA, Qwen-VL, and MiniGPT-v2—while remaining effective against cross-model transfer and post-processing defenses.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Privacy
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Recent advances in visual-language alignment have endowed vision-language models (VLMs) with fine-grained image understanding capabilities. However, this progress also introduces new privacy risks. This paper first proposes a novel privacy threat model named identity-affiliation learning: an attacker fine-tunes a VLM using only a few private photos of a target individual, thereby embedding associations between the target facial identity and their private property and social relationships into the model's internal representations. Once deployed via public APIs, this model enables unauthorized exposure of the target user's private information upon input of their photos. To benchmark VLMs' susceptibility to such identity-affiliation leakage, we introduce the first identity-affiliation dataset comprising seven typical scenarios appearing in private photos. Each scenario is instantiated with multiple identity-centered photo-description pairs. Experimental results demonstrate that mainstream VLMs like LLaVA, Qwen-VL, and MiniGPT-v2, can recognize facial identities and infer identity-affiliation relationships by fine-tuning on small-scale private photographic dataset, and even on synthetically generated datasets. To mitigate this privacy risk, we propose DP2-VL, the first Dataset Protection framework for private photos that leverages Data Poisoning. Though optimizing imperceptible perturbations by pushing the original representations toward an antithetical region, DP2-VL induces a dataset-level shift in the embedding space of VLMs'encoders. This shift separates protected images from clean inference images, causing fine-tuning on the protected set to overfit. Extensive experiments demonstrate that DP2-VL achieves strong generalization across models, robustness to diverse post-processing operations, and consistent effectiveness across varying protection ratios.

Problem

• privacy risk
• vision-language models
• identity-affiliation leakage
• data poisoning
• private photo protection

Innovation

• identity-affiliation learning
• data poisoning
• vision-language models
• privacy protection
• dataset-level embedding shift