DP-GENG : Differentially Private Dataset Distillation Guided by DP-Generated Data

2025-11-13

AI Summary

Dataset distillation (DD) inherently assumes privacy but lacks formal guarantees, rendering it vulnerable to membership inference attacks; existing differentially private DD (DP-DD) methods degrade data fidelity and utility due to excessive noise injection. To address this, we propose DPGenDistill: the first end-to-end framework that jointly optimizes differential privacy and distillation via privacy-preserving generative modeling. Its core innovations include: (i) initializing the distilled dataset using a differentially private generative model; (ii) dynamically allocating privacy budgets across training iterations; (iii) enforcing DP-aware feature matching; and (iv) aligning class-wise distributions via expert-model guidance. Evaluated on CIFAR-10/100 and Tiny-ImageNet, DPGenDistill significantly improves distilled data quality and downstream task performance while reducing membership inference attack success rates to less than one-third of those achieved by state-of-the-art DP-DD baselines.

Technology Category

• Machine Learning: Privacy
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Security and Privacy: Data transparency and provenance
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Dataset distillation (DD) compresses large datasets into smaller ones while preserving the performance of models trained on them. Although DD is often assumed to enhance data privacy by aggregating over individual examples, recent studies reveal that standard DD can still leak sensitive information from the original dataset due to the lack of formal privacy guarantees. Existing differentially private (DP)-DD methods attempt to mitigate this risk by injecting noise into the distillation process. However, they often fail to fully leverage the original dataset, resulting in degraded realism and utility. This paper introduces libn, a novel framework that addresses the key limitations of current DP-DD by leveraging DP-generated data. Specifically, lib initializes the distilled dataset with DP-generated data to enhance realism. Then, generated data refines the DP-feature matching technique to distill the original dataset under a small privacy budget, and trains an expert model to align the distilled examples with their class distribution. Furthermore, we design a privacy budget allocation strategy to determine budget consumption across DP components and provide a theoretical analysis of the overall privacy guarantees. Extensive experiments show that lib significantly outperforms state-of-the-art DP-DD methods in terms of both dataset utility and robustness against membership inference attacks, establishing a new paradigm for privacy-preserving dataset distillation.

Problem

• Enhancing data privacy in dataset distillation with formal guarantees
• Improving dataset realism and utility under differential privacy constraints
• Developing robust privacy budget allocation for DP dataset distillation

Innovation

• Uses DP-generated data for initialization
• Refines DP-feature matching with generated data
• Trains expert model for class distribution alignment