Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformers

2025-04-14

AI Summary

DiT-based text-to-image models (e.g., FLUX, SD v3.5) lack fine-grained instance control in multi-instance synthesis (MIS), leading to poor spatial localization and attribute fidelity. Method: We propose a training-free, plug-and-play hierarchical step-layer attention refinement technique. First, we uncover the layer-wise response hierarchy of instance/background/attribute tokens in DiT; then, we design a token- and layer-wise joint analysis framework integrating instance-guided sketch layout, hierarchical attention masking, and cross-step dynamic suppression to achieve disentangled alignment of multiple instances and their attributes in the prompt. Contribution/Results: Evaluated on an upgraded T2I-CompBench and complex-scene benchmarks, our method significantly improves instance localization accuracy and attribute fidelity—boosting key metrics by 12.7%—enabling high-fidelity, multi-subject, compositionally complex image generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Text-to-image (T2I) generation models often struggle with multi-instance synthesis (MIS), where they must accurately depict multiple distinct instances in a single image based on complex prompts detailing individual features. Traditional MIS control methods for UNet architectures like SD v1.5/SDXL fail to adapt to DiT-based models like FLUX and SD v3.5, which rely on integrated attention between image and text tokens rather than text-image cross-attention. To enhance MIS in DiT, we first analyze the mixed attention mechanism in DiT. Our token-wise and layer-wise analysis of attention maps reveals a hierarchical response structure: instance tokens dominate early layers, background tokens in middle layers, and attribute tokens in later layers. Building on this observation, we propose a training-free approach for enhancing MIS in DiT-based models with hierarchical and step-layer-wise attention specialty tuning (AST). AST amplifies key regions while suppressing irrelevant areas in distinct attention maps across layers and steps, guided by the hierarchical structure. This optimizes multimodal interactions by hierarchically decoupling the complex prompts with instance-based sketches. We evaluate our approach using upgraded sketch-based layouts for the T2I-CompBench and customized complex scenes. Both quantitative and qualitative results confirm our method enhances complex layout generation, ensuring precise instance placement and attribute representation in MIS.

Problem

• Enhance multi-instance synthesis in DiT-based models
• Address hierarchical attention in diffusion transformers
• Optimize multimodal interactions for complex prompts

Innovation

• Hierarchical attention specialty tuning for DiT
• Step-layer-wise optimization of attention maps
• Training-free multi-instance synthesis enhancement