Boundary Attention Constrained Zero-Shot Layout-To-Image Generation

2024-11-15

arXiv.org

AI Summary

Current text-to-image diffusion models exhibit limited control over spatial composition and object counting. To enable zero-shot layout-to-image (L2I) generation, this paper proposes a boundary-aware attention constraint mechanism: without fine-tuning the pre-trained model or adding auxiliary modules, it imposes geometric boundary-aware constraints on cross-modal attention maps during the reverse diffusion process. The method integrates self-attention pixel correlations with a triplet loss to optimize latent representations. Crucially, it is the first to embed geometric boundary priors directly into the attention mechanism while keeping the base model frozen, thereby significantly improving layout fidelity. Evaluated on DrawBench and HRS benchmarks, our approach consistently outperforms existing zero-shot L2I methods. Quantitative metrics and qualitative analysis both confirm superior spatial controllability and semantic consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent text-to-image diffusion models excel at generating high-resolution images from text but struggle with precise control over spatial composition and object counting. To address these challenges, several studies developed layout-to-image (L2I) approaches that incorporate layout instructions into text-to-image models. However, existing L2I methods typically require either fine-tuning pretrained parameters or training additional control modules for the diffusion models. In this work, we propose a novel zero-shot L2I approach, BACON (Boundary Attention Constrained generation), which eliminates the need for additional modules or fine-tuning. Specifically, we use text-visual cross-attention feature maps to quantify inconsistencies between the layout of the generated images and the provided instructions, and then compute loss functions to optimize latent features during the diffusion reverse process. To enhance spatial controllability and mitigate semantic failures in complex layout instructions, we leverage pixel-to-pixel correlations in the self-attention feature maps to align cross-attention maps and combine three loss functions constrained by boundary attention to update latent features. Comprehensive experimental results on both L2I and non-L2I pretrained diffusion models demonstrate that our method outperforms existing zero-shot L2I techniuqes both quantitatively and qualitatively in terms of image composition on the DrawBench and HRS benchmarks.

Problem

• Achieving precise spatial composition control in text-to-image generation
• Eliminating the need for fine-tuning or additional control modules
• Improving object counting and layout accuracy in complex scenes

Innovation

• Training-free layout-to-image generation method
• Uses cross-attention maps to quantify layout inconsistencies
• Optimizes latent features with three loss functions