Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis

2025-03-28

AI Summary

Text-to-image (T2I) diffusion models frequently suffer from spatial misalignment—i.e., generated objects deviate from their intended positions specified in the text prompt—yet existing training-free methods lack effective mechanisms to model such spatial relationships. To address this, we propose STORM, a training-free framework that pioneers the integration of optimal transport theory into T2I generation. STORM defines a spatial transport cost function and dynamically recalibrates attention maps during early denoising steps to explicitly inject spatial priors. Crucially, it modifies neither model architecture nor parameters, operating solely at inference time via attention distribution optimization. Experiments demonstrate that STORM significantly improves spatial alignment across multiple benchmarks, achieving state-of-the-art performance. Moreover, it concurrently mitigates object omission and attribute misbinding, exhibiting strong generalizability and computational efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Diffusion-based text-to-image (T2I) models have recently excelled in high-quality image generation, particularly in a training-free manner, enabling cost-effective adaptability and generalization across diverse tasks. However, while the existing methods have been continuously focusing on several challenges, such as"missing objects"and"mismatched attributes,"another critical issue of"mislocated objects"remains where generated spatial positions fail to align with text prompts. Surprisingly, ensuring such seemingly basic functionality remains challenging in popular T2I models due to the inherent difficulty of imposing explicit spatial guidance via text forms. To address this, we propose STORM (Spatial Transport Optimization by Repositioning Attention Map), a novel training-free approach for spatially coherent T2I synthesis. STORM employs Spatial Transport Optimization (STO), rooted in optimal transport theory, to dynamically adjust object attention maps for precise spatial adherence, supported by a Spatial Transport (ST) Cost function that enhances spatial understanding. Our analysis shows that integrating spatial awareness is most effective in the early denoising stages, while later phases refine details. Extensive experiments demonstrate that STORM surpasses existing methods, effectively mitigating mislocated objects while improving missing and mismatched attributes, setting a new benchmark for spatial alignment in T2I synthesis.

Problem

• Addresses mislocated objects in text-to-image synthesis
• Enhances spatial alignment without model retraining
• Optimizes attention maps for precise object positioning

Innovation

• Training-free text-to-image synthesis method
• Dynamic adjustment of object attention maps
• Optimal transport theory for spatial adherence