DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation

2024-12-24

AI Summary

To address abrupt prompt transitions, motion discontinuity, and training-data dependency in multi-prompt long-video generation, this paper proposes a training-free dynamic video synthesis method. We formulate multi-prompt generation as a temporally smooth editing task, introducing a 3D full-attention sharing mechanism and mask-guided cross-prompt attention modulation within the MM-DiT architecture to ensure semantic alignment and motion consistency. Our work is the first to achieve training-free multi-prompt video generation on MM-DiT; we propose a novel mask-guided inference-time control paradigm; and we introduce MPVBench—the first dedicated benchmark for evaluating multi-prompt video generation. Experiments demonstrate state-of-the-art performance in prompt adherence, natural scene transitions, and object motion stability, without any fine-tuning or additional training.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Sora-like video generation models have achieved remarkable progress with a Multi-Modal Diffusion Transformer MM-DiT architecture. However, the current video generation models predominantly focus on single-prompt, struggling to generate coherent scenes with multiple sequential prompts that better reflect real-world dynamic scenarios. While some pioneering works have explored multi-prompt video generation, they face significant challenges including strict training data requirements, weak prompt following, and unnatural transitions. To address these problems, we propose DiTCtrl, a training-free multi-prompt video generation method under MM-DiT architectures for the first time. Our key idea is to take the multi-prompt video generation task as temporal video editing with smooth transitions. To achieve this goal, we first analyze MM-DiT's attention mechanism, finding that the 3D full attention behaves similarly to that of the cross/self-attention blocks in the UNet-like diffusion models, enabling mask-guided precise semantic control across different prompts with attention sharing for multi-prompt video generation. Based on our careful design, the video generated by DiTCtrl achieves smooth transitions and consistent object motion given multiple sequential prompts without additional training. Besides, we also present MPVBench, a new benchmark specially designed for multi-prompt video generation to evaluate the performance of multi-prompt generation. Extensive experiments demonstrate that our method achieves state-of-the-art performance without additional training.

Problem

• Video Generation
• Dynamic Variation
• Prompt-based Transition

Innovation

• DiTCtrl
• MM-DiT
• MPVBench