Control-CLIP: Decoupling Category and Style Guidance in CLIP for Specific-Domain Generation

2025-02-17

AI Summary

Current text-to-image diffusion models rely on the CLIP text encoder, which struggles to disambiguate semantic categories (e.g., “cat”) from artistic styles (e.g., “Pokémon style”), leading to imprecise fine-grained style generation. To address this, we propose the first category-style disentangled CLIP fine-tuning framework. Our method leverages lightweight style-annotated data and a redesigned cross-modal cross-attention mechanism to enable complementary learning of category and style representations. Crucially, it requires no architectural modification to the underlying diffusion model and supports plug-and-play integration for precise style control. Experiments across diverse stylistic domains demonstrate significant improvements over baselines: our approach enhances style fidelity, controllability, and generalization while preserving generation diversity and image quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Text-to-image diffusion models have shown remarkable capabilities of generating high-quality images closely aligned with textual inputs. However, the effectiveness of text guidance heavily relies on the CLIP text encoder, which is trained to pay more attention to general content but struggles to capture semantics in specific domains like styles. As a result, generation models tend to fail on prompts like"a photo of a cat in Pokemon style"in terms of simply producing images depicting"a photo of a cat". To fill this gap, we propose Control-CLIP, a novel decoupled CLIP fine-tuning framework that enables the CLIP model to learn the meaning of category and style in a complement manner. With specially designed fine-tuning tasks on minimal data and a modified cross-attention mechanism, Control-CLIP can precisely guide the diffusion model to a specific domain. Moreover, the parameters of the diffusion model remain unchanged at all, preserving the original generation performance and diversity. Experiments across multiple domains confirm the effectiveness of our approach, particularly highlighting its robust plug-and-play capability in generating content with various specific styles.

Problem

• Enhances CLIP for specific-domain image generation.
• Decouples category and style in CLIP guidance.
• Improves text-to-image diffusion model precision.

Innovation

• Decoupled CLIP fine-tuning framework
• Minimal data fine-tuning tasks
• Modified cross-attention mechanism