Revealing Training Data Exposure in Vision Language Large Models via Parameter Gradients

2026-06-23

AI Summary

This work addresses the challenge of tracing training data in vision-language foundation models within high-privacy domains such as healthcare, where conventional auditing methods fall short. The authors propose GradAudit, a novel framework that leverages the stability of model parameter gradients as a discriminative signal: gradients from training samples exhibit consistent alignment during optimization, whereas those from non-training samples manifest pronounced noise. Departing from existing approaches that rely on black-box outputs or unimodal signals, GradAudit analyzes cross-modal gradient dynamics across both pretraining and fine-tuning stages to accurately detect image-text co-training relationships. Experiments demonstrate that GradAudit substantially outperforms state-of-the-art methods on both medical and general datasets, revealing that current auditing techniques significantly underestimate unauthorized data usage—a gap that widens with increasing model scale.

Abstract

Vision-Language Large Models (VLLMs) trained on massive crawled corpora raise pressing copyright and data-provenance concerns. These concerns are particularly acute in healthcare, where patient medical images paired with clinical reports demand rigorous privacy safeguards. However, existing training data detection methods either fail in cross-modal scenarios or rely on superficial output signals with insufficient discriminative power. We introduce GradAudit, a gradient-based auditing framework that examines internal optimization dynamics rather than treating VLLMs as black boxes. Our approach builds on a key observation: model parameters converge to regions where gradients on training samples become stable and well-aligned, whereas gradients on non-training samples remain noisy and inconsistent. By analyzing these gradient signatures, GradAudit achieves strong separability and detects genuine image-text associations learned during training, not merely individual modality membership. Empirically, across both medical and general-domain datasets, GradAudit substantially outperforms state-of-the-art baselines in both pretraining and fine-tuning VLLMs. In a case study employing copyrighted content, we show that existing training data detection methods not only underestimate the extent of unauthorized data usage, but that this underestimation becomes more pronounced as models become more recent and more advanced.

Problem

• training data exposure
• vision-language models
• data privacy
• copyright
• gradient analysis

Innovation

• gradient-based auditing
• training data detection
• vision-language models
• data privacy
• cross-modal alignment