When Model Knowledge meets Diffusion Model: Diffusion-assisted Data-free Image Synthesis with Alignment of Domain and Class

2025-06-18

AI Summary

To address the challenge of reconstructing the image distribution learned by a pre-trained model in the absence of original training data, this paper proposes the first data-free image synthesis framework leveraging text-to-image diffusion models. Methodologically, it treats the diffusion model as a strong natural image prior and employs knowledge distillation to align features of the target pre-trained model. It introduces two key innovations: (i) a Domain Alignment Guidance (DAG) mechanism that steers synthesis toward the original training domain, and (ii) Class-Alignment Tokens (CATs) that enforce class-level distribution matching in the latent space. Experiments on PACS and ImageNet demonstrate substantial improvements over existing data-free image synthesis (DFIS) methods: synthesized images exhibit higher fidelity to the true training distribution, achieving state-of-the-art performance in data-free settings.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Open-source pre-trained models hold great potential for diverse applications, but their utility declines when their training data is unavailable. Data-Free Image Synthesis (DFIS) aims to generate images that approximate the learned data distribution of a pre-trained model without accessing the original data. However, existing DFIS meth ods produce samples that deviate from the training data distribution due to the lack of prior knowl edge about natural images. To overcome this limitation, we propose DDIS, the first Diffusion-assisted Data-free Image Synthesis method that leverages a text-to-image diffusion model as a powerful image prior, improving synthetic image quality. DDIS extracts knowledge about the learned distribution from the given model and uses it to guide the diffusion model, enabling the generation of images that accurately align with the training data distribution. To achieve this, we introduce Domain Alignment Guidance (DAG) that aligns the synthetic data domain with the training data domain during the diffusion sampling process. Furthermore, we optimize a single Class Alignment Token (CAT) embedding to effectively capture class-specific attributes in the training dataset. Experiments on PACS and Ima geNet demonstrate that DDIS outperforms prior DFIS methods by generating samples that better reflect the training data distribution, achieving SOTA performance in data-free applications.

Problem

• Generate images approximating pre-trained model data without original data
• Align synthetic images with training data distribution using diffusion models
• Improve synthetic image quality via domain and class alignment guidance

Innovation

• Leverages text-to-image diffusion model as image prior
• Introduces Domain Alignment Guidance for domain alignment
• Optimizes Class Alignment Token for class-specific attributes