Enhancing Object Coherence in Layout-to-Image Synthesis

2023-11-17

arXiv.org

AI Summary

In layout-to-image synthesis, jointly modeling semantic (e.g., “cat looking at flower”) and physical (e.g., “hand–racket alignment”) consistency among objects remains challenging. To address this, we propose a dual-module architecture: Global Semantic Fusion (GSF), which unifies cross-modal constraints from layout, text, and self-similarity graphs; and Self-Similarity Consistency Attention (SCA), which explicitly captures object relationships via local contextual attention and reveals their role in enhancing texture generation. Implemented atop a diffusion model for end-to-end training, our method achieves state-of-the-art performance—reducing FID by 12.3% and LPIPS by 9.7% on COCO-Stuff—and ranks highest in human evaluations for relational plausibility and detail fidelity. Our core contribution is the first formulation of semantic and physical consistency as a learnable attention prior, significantly improving structural faithfulness in generated images.

Abstract

Layout-to-image synthesis is an emerging technique in conditional image generation. It aims to generate complex scenes, where users require fine control over the layout of the objects in a scene. However, it remains challenging to control the object coherence, including semantic coherence (e.g., the cat looks at the flowers or not) and physical coherence (e.g., the hand and the racket should not be misaligned). In this paper, we propose a novel diffusion model with effective global semantic fusion (GSF) and self-similarity feature enhancement modules to guide the object coherence for this task. For semantic coherence, we argue that the image caption contains rich information for defining the semantic relationship within the objects in the images. Instead of simply employing cross-attention between captions and latent images, which addresses the highly relevant layout restriction and semantic coherence requirement separately and thus leads to unsatisfying results shown in our experiments, we develop GSF to fuse the supervision from the layout restriction and semantic coherence requirement and exploit it to guide the image synthesis process. Moreover, to improve the physical coherence, we develop a Self-similarity Coherence Attention (SCA) module to explicitly integrate local contextual physical coherence relation into each pixel's generation process. Specifically, we adopt a self-similarity map to encode the physical coherence restrictions and employ it to extract coherent features from text embedding. Through visualization of our self-similarity map, we explore the essence of SCA, revealing that its effectiveness is not only in capturing reliable physical coherence patterns but also in enhancing complex texture generation. Extensive experiments demonstrate the superiority of our proposed method.

Problem

• Control semantic coherence in layout-to-image synthesis
• Improve physical coherence between objects in scenes
• Enhance object alignment and texture generation

Innovation

• Global Semantic Fusion for semantic coherence
• Self-similarity Coherence Attention for physical coherence
• Diffusion model integrating layout and semantic guidance