Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection

2026-03-13

AI Summary

Text-to-image diffusion models are prone to memorizing and reproducing training data, raising significant copyright and privacy concerns. To address this issue, this work proposes Region-Aware Prompt Augmentation (RAPTA), which enhances generation diversity during training by introducing semantically consistent prompt variants. Additionally, the authors design Attention-Driven Multimodal Copy Detection (ADMCD), a lightweight method that fuses local, global, and textural cues for efficient copy detection without requiring large-scale annotated data. This study is the first to integrate region-aware prompt augmentation with a lightweight multimodal detection framework, effectively mitigating memorization while preserving strong image-prompt alignment. Experimental results demonstrate that RAPTA substantially reduces overfitting without compromising generation quality, and ADMCD outperforms existing unimodal metrics in copy detection performance.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

State-of-the-art text-to-image diffusion models can produce impressive visuals but may memorize and reproduce training images, creating copyright and privacy risks. Existing prompt perturbations applied at inference time, such as random token insertion or embedding noise, may lower copying but often harm image-prompt alignment and overall fidelity. To address this, we introduce two complementary methods. First, Region-Aware Prompt Augmentation (RAPTA) uses an object detector to find salient regions and turn them into semantically grounded prompt variants, which are randomly sampled during training to increase diversity, while maintaining semantic alignment. Second, Attention-Driven Multimodal Copy Detection (ADMCD) aggregates local patch, global semantic, and texture cues with a lightweight transformer to produce a fused representation, and applies simple thresholded decision rules to detect copying without training with large annotated datasets. Experiments show that RAPTA reduces overfitting while maintaining high synthesis quality, and that ADMCD reliably detects copying, outperforming single-modal metrics.

Problem

• memorization
• text-to-image diffusion
• copyright risk
• privacy risk
• image reproduction

Innovation

• Region-Aware Prompt Augmentation
• Multimodal Copy Detection
• Text-to-Image Diffusion
• Memorization Mitigation
• Attention-Driven Fusion