SPDiffusion: Semantic Protection Diffusion Models for Multi-concept Text-to-image Generation

2024-09-02

AI Summary

Current text-to-image diffusion models suffer from semantic entanglement in multi-concept generation (e.g., multiple characters/objects), manifesting as concept confusion and attribute misbinding, thereby degrading text–image alignment. To address this, we propose the Semantic Protection (SP) framework—a label-free approach introducing two novel components: SP-Extraction for unsupervised concept region localization and SP-Attention, a protection-aware cross-attention mechanism. Integrated into the UNet backbone, SP-Attention enables region-aware token masking and feature isolation, effectively disentangling concepts from their attributes. Our method preserves generation fidelity while significantly improving concept localization accuracy and attribute binding correctness. On established multi-concept generation benchmarks, SP achieves state-of-the-art performance, demonstrating robust semantic decoupling without architectural or training overhead.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Recent text-to-image models have achieved impressive results in generating high-quality images. However, when tasked with multi-concept generation creating images that contain multiple characters or objects, existing methods often suffer from semantic entanglement, including concept entanglement and improper attribute binding, leading to significant text-image inconsistency. We identify that semantic entanglement arises when certain regions of the latent features attend to incorrect concept and attribute tokens. In this work, we propose the Semantic Protection Diffusion Model (SPDiffusion) to address both concept entanglement and improper attribute binding using only a text prompt as input. The SPDiffusion framework introduces a novel concept region extraction method SP-Extraction to resolve region entanglement in cross-attention, along with SP-Attn, which protects concept regions from the influence of irrelevant attributes and concepts. To evaluate our method, we test it on existing benchmarks, where SPDiffusion achieves state-of-the-art results, demonstrating its effectiveness.

Problem

• Addresses semantic entanglement in multi-concept text-to-image generation.
• Resolves concept entanglement and improper attribute binding issues.
• Improves text-image consistency using novel concept region extraction.

Innovation

• SPDiffusion resolves semantic entanglement issues
• Introduces SP-Extraction for concept region extraction
• SP-Attn protects concept regions from irrelevant attributes