Data Attribution for Text-to-Image Models by Unlearning Synthesized Images

2024-06-13

Neural Information Processing Systems

AI Summary

Data attribution for text-to-image models—identifying the most influential training samples for a given generated image—faces prohibitive computational costs when relying on iterative retraining. This work proposes the first reverse attribution method that employs synthetically generated images as proxy triggers. Leveraging gradient-driven loss perturbation, selective parameter updates, and influence sensitivity analysis, our approach efficiently pinpoints high-impact training images without full-scale retraining. Crucially, it preserves model generalization while substantially improving attribution accuracy. We validate the method across multiple state-of-the-art text-to-image models, demonstrating superior attribution accuracy over existing baselines, two orders-of-magnitude reduction in computational overhead, and strong empirical agreement with retraining-based “gold-standard” attribution—a key indicator of reliability.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The goal of data attribution for text-to-image models is to identify the training images that most influence the generation of a new image. Influence is defined such that, for a given output, if a model is retrained from scratch without the most influential images, the model would fail to reproduce the same output. Unfortunately, directly searching for these influential images is computationally infeasible, since it would require repeatedly retraining models from scratch. In our work, we propose an efficient data attribution method by simulating unlearning the synthesized image. We achieve this by increasing the training loss on the output image, without catastrophic forgetting of other, unrelated concepts. We then identify training images with significant loss deviations after the unlearning process and label these as influential. We evaluate our method with a computationally intensive but"gold-standard"retraining from scratch and demonstrate our method's advantages over previous methods.

Problem

• Identify influential training images
• Simulate unlearning synthesized images
• Efficient data attribution method

Innovation

• Simulates unlearning synthesized images
• Increases loss on output image
• Identifies influential training images