Exploring Multimodal Diffusion Transformers for Enhanced Prompt-based Image Editing

2025-08-10

AI Summary

Unified multimodal attention in MM-DiT architectures impedes prompt-guided image editing due to entangled cross-modal interactions. Method: We propose a general editing framework featuring bidirectional information flow, which for the first time decouples the MM-DiT attention matrix into four distinct modules: text-to-image, image-to-text, text self-attention, and image self-attention—thereby explicitly characterizing cross-modal dynamics. We further introduce a prompt-conditioned feature modulation strategy and a few-step adaptation mechanism to enable controllable editing from global semantics to local details. Contribution/Results: Our method is compatible with multiple MM-DiT models—including Stable Diffusion 3 and Flux.1—without fine-tuning. It maintains high fidelity and editing flexibility across arbitrary inference steps, significantly improving editing accuracy and generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Transformer-based diffusion models have recently superseded traditional U-Net architectures, with multimodal diffusion transformers (MM-DiT) emerging as the dominant approach in state-of-the-art models like Stable Diffusion 3 and Flux.1. Previous approaches have relied on unidirectional cross-attention mechanisms, with information flowing from text embeddings to image latents. In contrast, MMDiT introduces a unified attention mechanism that concatenates input projections from both modalities and performs a single full attention operation, allowing bidirectional information flow between text and image branches. This architectural shift presents significant challenges for existing editing techniques. In this paper, we systematically analyze MM-DiT's attention mechanism by decomposing attention matrices into four distinct blocks, revealing their inherent characteristics. Through these analyses, we propose a robust, prompt-based image editing method for MM-DiT that supports global to local edits across various MM-DiT variants, including few-step models. We believe our findings bridge the gap between existing U-Net-based methods and emerging architectures, offering deeper insights into MMDiT's behavioral patterns.

Problem

• Analyze MM-DiT's attention mechanism for image editing
• Develop prompt-based editing for bidirectional multimodal transformers
• Bridge U-Net and MM-DiT methods for global-local edits

Innovation

• Unified bidirectional attention mechanism for multimodal fusion
• Decomposed attention matrices into four distinct blocks
• Robust prompt-based editing for global to local changes