FBSDiff: Plug-and-Play Frequency Band Substitution of Diffusion Features for Highly Controllable Text-Driven Image Translation

2024-08-02

ACM Multimedia

AI Summary

Pretrained text-to-image diffusion models suffer from limited controllability and struggle to achieve precise, text-driven image editing. To address this, we propose a training-free, plug-and-play image-to-image translation method. Our core innovation is the first application of discrete cosine transform (DCT) to map intermediate diffusion features into the frequency domain, enabling a novel band-replacement layer that supports on-demand disentanglement and substitution of semantic, stylistic, and other guidance factors—thereby facilitating fine-grained, high-degree-of-freedom cross-modal editing. The method is architecture-agnostic and seamlessly integrates with mainstream diffusion models (e.g., Stable Diffusion) without fine-tuning. Extensive evaluations across multiple benchmarks demonstrate significant improvements in image quality, semantic fidelity, and editing controllability, while enabling real-time interactive editing. Our code and an interactive web demo are publicly released.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Large-scale text-to-image diffusion models have been a revolutionary milestone in the evolution of generative AI, allowing wonderful image generation with natural-language text prompt. However, the issue of lacking controllability of such models restricts their practical applicability for real-life content creation. Thus, attention has been focused on leveraging a reference image to control text-to-image synthesis, which is also regarded as manipulating (or editing) a reference image as per a text prompt, namely, text-driven image-to-image translation. This paper contributes a novel, concise, and efficient approach that adapts pre-trained large-scale text-to-image (T2I) diffusion model to the image-to-image (I2I) paradigm in a plug-and-play manner, realizing high-quality and versatile text-driven I2I translation without model training, fine-tuning, or online optimization process. To guide T2I generation with a reference image, we propose to decompose diverse guiding factors with different frequency bands of diffusion features in the DCT spectral space, and accordingly devise a novel frequency band substitution layer which realizes dynamic control of the reference image to the T2I generation result in a plug-and-play manner. We demonstrate that our method allows flexible control over both guiding factor and guiding intensity of the reference image simply by tuning the type and bandwidth of the substituted frequency band, respectively. Extensive qualitative and quantitative experiments verify superiority of our approach over related methods in I2I translation visual quality, versatility, and controllability. Our project is publicly available at: https://xianggao1102.github.io/FBSDiff_webpage/.

Problem

• Enhancing controllability of text-to-image diffusion models
• Enabling plug-and-play text-driven image translation
• Improving image quality and controllability via frequency band substitution

Innovation

• Plug-and-play frequency band substitution layer
• DCT spectral space decomposition for control
• No training or fine-tuning required