CleanStyle: Plug-and-Play Style Conditioning Purification for Text-to-Image Stylization

2026-02-24

AI Summary

This work addresses the issue of semantic content leakage from style reference images in encoder-based diffusion models for text-to-image stylization, which often compromises prompt fidelity and stylistic consistency. To mitigate this without requiring model retraining, the authors propose a plug-and-play style-condition purification framework. The core innovation lies in identifying trailing components in style embeddings—via singular value decomposition (SVD)—as the primary source of content leakage, and introducing two key mechanisms: a dynamic suppression strategy (CS-SVD) and a timestep-aware, style-specific classifier-free guidance (SS-CFG). This approach significantly enhances style consistency, visual quality, and text-image alignment across diverse style and prompt combinations.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Style transfer in diffusion models enables controllable visual generation by injecting the style of a reference image. However, recent encoder-based methods, while efficient and tuning-free, often suffer from content leakage, where semantic elements from the style image undesirably appear in the output, impairing prompt fidelity and stylistic consistency. In this work, we introduce CleanStyle, a plug-and-play framework that filters out content-related noise from the style embedding without retraining. Motivated by empirical analysis, we observe that such leakage predominantly stems from the tail components of the style embedding, which are isolated via Singular Value Decomposition (SVD). To address this, we propose CleanStyleSVD (CS-SVD), which dynamically suppresses tail components using a time-aware exponential schedule, providing clean, style-preserving conditional embeddings throughout the denoising process. Furthermore, we present Style-Specific Classifier-Free Guidance (SS-CFG), which reuses the suppressed tail components to construct style-aware unconditional inputs. Unlike conventional methods that use generic negative embeddings (e.g., zero vectors), SS-CFG introduces targeted negative signals that reflect style-specific but prompt-irrelevant visual elements. This enables the model to effectively suppress these distracting patterns during generation, thereby improving prompt fidelity and enhancing the overall visual quality of stylized outputs. Our approach is lightweight, interpretable, and can be seamlessly integrated into existing encoder-based diffusion models without retraining. Extensive experiments demonstrate that CleanStyle substantially reduces content leakage, improves stylization quality and improves prompt alignment across a wide range of style references and prompts.

Problem

• content leakage
• text-to-image stylization
• style transfer
• diffusion models
• prompt fidelity

Innovation

• content leakage
• style transfer
• diffusion models
• Singular Value Decomposition
• classifier-free guidance