Turning Black Box into White Box: Dataset Distillation Leaks

2026-03-01

AI Summary

This work systematically uncovers a critical privacy risk in existing dataset distillation methods: when compressing real data into synthetic data, these approaches may implicitly encode the training trajectory of models, thereby leaking sensitive information about the original dataset. To expose this vulnerability, the authors propose an Information Revelation Attack (IRA) that integrates model inversion and membership inference techniques to effectively infer the distillation algorithm, model architecture, membership status, and even reconstruct sensitive samples from the synthetic data alone. Experimental results demonstrate that IRA can accurately identify both the distillation method and model structure, and successfully recover original sensitive data with high fidelity. These findings fundamentally challenge the prevailing assumption that dataset distillation inherently preserves privacy, revealing instead a severe and previously underappreciated privacy leakage risk.

Technology Category

• Machine Learning: Privacy
• Computer Vision: Adversarial Attacks & Robustness
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Dataset distillation compresses a large real dataset into a small synthetic one, enabling models trained on the synthetic data to achieve performance comparable to those trained on the real data. Although synthetic datasets are assumed to be privacy-preserving, we show that existing distillation methods can cause severe privacy leakage because synthetic datasets implicitly encode the weight trajectories of the distilled model, they become over-informative and exploitable by adversaries. To expose this risk, we introduce the Information Revelation Attack (IRA) against state-of-the-art distillation techniques. Experiments show that IRA accurately predicts both the distillation algorithm and model architecture, and can successfully infer membership and recover sensitive samples from the real dataset.

Problem

• dataset distillation
• privacy leakage
• synthetic data
• membership inference
• information revelation

Innovation

• dataset distillation
• privacy leakage
• Information Revelation Attack
• membership inference
• synthetic data