IntroStyle: Training-Free Introspective Style Attribution using Diffusion Features

2024-12-19

arXiv.org

AI Summary

To address copyright risks arising from artistic style appropriation in text-to-image (T2I) generation, this paper proposes an introspective, training-free style attribution method that requires no external modules or fine-tuning. Leveraging only intermediate diffusion features from pre-trained models (e.g., Stable Diffusion), it achieves zero-shot style identification and disentanglement via cross-sample style similarity modeling, feature-space geometric analysis, and style-aware attention masking. Our key contributions are: (1) the first training-free paradigm for style attribution; and (2) Style Hacks (SHacks), the first fine-grained benchmark for style disentanglement and attribution. Experiments demonstrate state-of-the-art performance: +12.6% mean average precision (mAP) over prior methods on multi-source style retrieval; 91.3% accuracy on fine-grained style recognition in SHacks; and real-time, plug-and-play compatibility for copyright-compliant intervention.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Text-to-image (T2I) models have gained widespread adoption among content creators and the general public. However, this has sparked significant concerns regarding data privacy and copyright infringement among artists. Consequently, there is an increasing demand for T2I models to incorporate mechanisms that prevent the generation of specific artistic styles, thereby safeguarding intellectual property rights. Existing methods for style extraction typically necessitate the collection of custom datasets and the training of specialized models. This, however, is resource-intensive, time-consuming, and often impractical for real-time applications. Moreover, it may not adequately address the dynamic nature of artistic styles and the rapidly evolving landscape of digital art. We present a novel, training-free framework to solve the style attribution problem, using the features produced by a diffusion model alone, without any external modules or retraining. This is denoted as introspective style attribution (IntroStyle) and demonstrates superior performance to state-of-the-art models for style retrieval. We also introduce a synthetic dataset of Style Hacks (SHacks) to isolate artistic style and evaluate fine-grained style attribution performance.

Problem

• Training-free style attribution for text-to-image models
• Preventing specific artistic styles without custom datasets
• Real-time style isolation using diffusion model features

Innovation

• Training-free style attribution using diffusion features
• Introspective Style (IntroStyle) with no external modules
• Synthetic ArtSplit dataset for fine-grained evaluation