EnTruth: Enhancing the Traceability of Unauthorized Dataset Usage in Text-to-image Diffusion Models with Minimal and Robust Alterations

2024-06-20

arXiv.org

AI Summary

To address the challenge of tracing unauthorized use of training data in generative models, this paper proposes a lightweight and robust watermarking method based on a template memorization mechanism. The method pioneers the constructive exploitation of diffusion models’ “memory phenomenon” for copyright attribution: it injects learnable templates into training data with minimal perturbation (only 0.3% pixel-level alteration), inducing infringing models to produce verifiable, watermark-specific outputs. Integrated with adversarial-robust fine-tuning and generation-quality preservation constraints, the approach forms an end-to-end traceability framework. Evaluated across multiple text-to-image diffusion models and datasets, it achieves >98% detection accuracy, incurs negligible degradation in generation fidelity (FID increase <0.5), and demonstrates strong robustness against common post-processing attacks—including cropping, compression, and filtering.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Generative models, especially text-to-image diffusion models, have significantly advanced in their ability to generate images, benefiting from enhanced architectures, increased computational power, and large-scale datasets. While the datasets play an important role, their protection has remained as an unsolved issue. Current protection strategies, such as watermarks and membership inference, are either in high poison rate which is detrimental to image quality or suffer from low accuracy and robustness. In this work, we introduce a novel approach, EnTruth, which Enhances Traceability of unauthorized dataset usage utilizing template memorization. By strategically incorporating the template memorization, EnTruth can trigger the specific behavior in unauthorized models as the evidence of infringement. Our method is the first to investigate the positive application of memorization and use it for copyright protection, which turns a curse into a blessing and offers a pioneering perspective for unauthorized usage detection in generative models. Comprehensive experiments are provided to demonstrate its effectiveness in terms of data-alteration rate, accuracy, robustness and generation quality.

Problem

• Protecting datasets from unauthorized use in text-to-image diffusion models
• Overcoming limitations of current watermarking and membership inference methods
• Establishing traceable evidence of infringement through template memorization techniques

Innovation

• Uses template memorization for traceability enhancement
• Triggers specific behavior as infringement evidence
• Applies memorization positively for copyright protection