Targeted Data Protection for Diffusion Model by Matching Training Trajectory

2025-12-11

AI Summary

Existing passive data-degradation methods for privacy-preserving personalization of diffusion models fail to deliver stable, controllable privacy protection against misuse of sensitive user data. Method: We propose an active target-data protection paradigm that enables controllable, full-training-trajectory-level redirection—achieved via a verifiable, dilution-resistant trajectory-matching mechanism to align optimization paths with user-specified concepts, and an adversarial-perturbation-driven fine-tuning framework (TAFAP) integrating dataset distillation, latent-space gradient constraints, and target-concept guidance. Contribution/Results: Under dual control of identity and visual patterns, our method achieves high-fidelity, lossless image generation with precise target-concept transfer—marking the first such capability. It significantly improves redirection robustness while ensuring end-to-end, verifiable privacy protection throughout the fine-tuning process.

Technology Category

• Machine Learning: Privacy
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Security and Privacy: Data transparency and provenance

Abstract

Recent advancements in diffusion models have made fine-tuning text-to-image models for personalization increasingly accessible, but have also raised significant concerns regarding unauthorized data usage and privacy infringement. Current protection methods are limited to passively degrading image quality, failing to achieve stable control. While Targeted Data Protection (TDP) offers a promising paradigm for active redirection toward user-specified target concepts, existing TDP attempts suffer from poor controllability due to snapshot-matching approaches that fail to account for complete learning dynamics. We introduce TAFAP (Trajectory Alignment via Fine-tuning with Adversarial Perturbations), the first method to successfully achieve effective TDP by controlling the entire training trajectory. Unlike snapshot-based methods whose protective influence is easily diluted as training progresses, TAFAP employs trajectory-matching inspired by dataset distillation to enforce persistent, verifiable transformations throughout fine-tuning. We validate our method through extensive experiments, demonstrating the first successful targeted transformation in diffusion models with simultaneous control over both identity and visual patterns. TAFAP significantly outperforms existing TDP attempts, achieving robust redirection toward target concepts while maintaining high image quality. This work enables verifiable safeguards and provides a new framework for controlling and tracing alterations in diffusion model outputs.

Problem

• Achieves targeted data protection in diffusion models
• Controls entire training trajectory for stable redirection
• Enables verifiable safeguards with maintained image quality

Innovation

• Trajectory alignment via adversarial perturbations for control
• Persistent transformation through full training trajectory matching
• Verifiable redirection of both identity and visual patterns