When Privacy Isn't Synthetic: Hidden Data Leakage in Generative AI Models

2025-12-05

AI Summary

This paper uncovers a previously overlooked membership inference risk in generative AI–synthesized data: even without explicit memorization of training samples, structural overlap between the original and synthetic data manifolds can leak individual membership information. To exploit this vulnerability, we propose a clustering-center–based black-box inference attack—leveraging unsupervised clustering and density estimation to identify dense neighborhoods in the synthetic data distribution, constructing proxy representations of training samples, and enabling high-accuracy membership inference via black-box queries. Crucially, our approach attributes privacy leakage to distributional neighborhood alignment rather than pointwise sample memorization, thereby challenging conventional privacy evaluation paradigms. Experiments across sensitive domains—including healthcare and finance—demonstrate significant membership leakage in synthetically generated data, even when trained with differential privacy, revealing fundamental limitations of existing privacy-preserving mechanisms.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Generative models are increasingly used to produce privacy-preserving synthetic data as a safe alternative to sharing sensitive training datasets. However, we demonstrate that such synthetic releases can still leak information about the underlying training samples through structural overlap in the data manifold. We propose a black-box membership inference attack that exploits this vulnerability without requiring access to model internals or real data. The attacker repeatedly queries the generative model to obtain large numbers of synthetic samples, performs unsupervised clustering to identify dense regions of the synthetic distribution, and then analyzes cluster medoids and neighborhoods that correspond to high-density regions in the original training data. These neighborhoods act as proxies for training samples, enabling the adversary to infer membership or reconstruct approximate records. Our experiments across healthcare, finance, and other sensitive domains show that cluster overlap between real and synthetic data leads to measurable membership leakage-even when the generator is trained with differential privacy or other noise mechanisms. The results highlight an under-explored attack surface in synthetic data generation pipelines and call for stronger privacy guarantees that account for distributional neighborhood inference rather than sample-level memorization alone, underscoring its role in privacy-preserving data publishing. Implementation and evaluation code are publicly available at:github.com/Cluster-Medoid-Leakage-Attack.

Problem

• Detect hidden data leakage in synthetic data from generative models
• Propose black-box membership inference attack using synthetic samples clustering
• Evaluate privacy risks across sensitive domains despite noise mechanisms

Innovation

• Black-box membership inference attack exploiting structural overlap
• Unsupervised clustering to identify dense synthetic distribution regions
• Analyzing cluster medoids as proxies for original training data