Enhancing Multi-Text Long Video Generation Consistency without Tuning: Time-Frequency Analysis, Prompt Alignment, and Theory

2024-12-23

arXiv.org

AI Summary

To address poor temporal consistency and unnatural scene transitions in long-video generation, this paper proposes a training-free, plug-and-play multi-prompt framework for enhancing long-video coherence. Methodologically, it establishes the first theoretical guarantee for frequency-domain modeling in diffusion models and introduces TiARA—a time-frequency attention reweighting algorithm based on the short-time Fourier transform—alongside PromptBlend, an interpretable multi-prompt semantic alignment and interpolation mechanism. Key contributions include: (1) the first time-frequency attention reweighting scheme tailored for diffusion models; (2) the first theoretical analysis of STFT-based attention modulation; and (3) a smooth, multi-text-prompt interpolation strategy enabling seamless transitions. Extensive experiments on multiple long-video benchmarks demonstrate significant improvements over state-of-the-art methods in both temporal consistency and visual coherence, all without model fine-tuning.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite the considerable progress achieved in the long video generation problem, there is still significant room to improve the consistency of the videos, particularly in terms of smoothness and transitions between scenes. We address these issues to enhance the consistency and coherence of videos generated with either single or multiple prompts. We propose the Time-frequency based temporal Attention Reweighting Algorithm (TiARA), which meticulously edits the attention score matrix based on the Discrete Short-Time Fourier Transform. Our method is supported by a theoretical guarantee, the first-of-its-kind for frequency-based methods in diffusion models. For videos generated by multiple prompts, we further investigate key factors affecting prompt interpolation quality and propose PromptBlend, an advanced prompt interpolation pipeline. The efficacy of our proposed method is validated via extensive experimental results, exhibiting consistent and impressive improvements over baseline methods. The code will be released upon acceptance.

Problem

• Improving long video consistency and smoothness
• Enhancing multi-prompt video coherence and transitions
• Developing frequency-based attention editing for consistency

Innovation

• TiARA edits attention scores via Fourier Transform
• PromptBlend aligns prompts for better interpolation
• Frequency-based analysis ensures frame consistency