DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction

2026-05-12

AI Summary

This work addresses the challenge of auditing training data privacy in black-box vision-language models (VLMs), which typically expose only textual outputs and withhold internal signals such as logits or probabilities. To this end, the authors propose DistractMIA, a novel membership inference framework that injects semantically meaningful distractors into input images and infers training set membership by analyzing shifts in the model’s textual responses. This approach pioneers a purely output-level membership inference mechanism based on semantic distraction, eliminating reliance on object-centric assumptions or access to internal model states. By incorporating reference-set calibration and repeated generation, DistractMIA enhances inference reliability. Empirical evaluations demonstrate that it consistently outperforms existing output-level and even higher-privilege baselines across multiple VLMs and benchmarks, while also exhibiting strong generalization to non-natural domains such as medical imaging.

Abstract

Vision-language models (VLMs) are trained on large-scale image-text corpora that may contain private, copyrighted, or otherwise sensitive data, motivating membership inference as a tool for training-data auditing. This is especially challenging for deployed VLMs, where auditors typically observe only generated textual responses. Existing VLM membership inference attacks either rely on probability-level signals unavailable in such settings, or use mask-based semantic prediction tasks whose effectiveness depends on object-centric visual assumptions. To address these limitations, we propose DistractMIA, an output-only black-box framework based on semantic distraction. Rather than removing visual evidence, DistractMIA preserves the original image, inserts a known semantic distractor, and measures how generated responses change. This design is motivated by the intuition that member samples remain more anchored to the original image semantics, while non-member samples are more easily redirected toward the distractor. To make this signal reliable, DistractMIA calibrates distractor configurations on a reference set and derives membership scores from repeated textual generations, capturing response stability and distractor uptake without accessing logits, probabilities, or hidden states. Experiments across multiple VLMs and benchmarks show that DistractMIA consistently outperforms both output-only and stronger-access baselines. Its performance on a medical benchmark further demonstrates applicability beyond object-centric natural images.

Problem

• Membership Inference
• Vision-Language Models
• Black-Box Attack
• Training Data Privacy
• Semantic Distraction

Innovation

• membership inference
• vision-language models
• semantic distraction
• black-box attack
• output-only auditing