STIV: Scalable Text and Image Conditioned Video Generation

2024-12-10

arXiv.org

AI Summary

The video generation field lacks systematic, scalable modeling paradigms. This paper proposes STIV—a unified text-image joint-driven video generation framework built upon the Diffusion Transformer (DiT), supporting both text-to-video (T2V) and text-and-image-to-video (TI2V), and naturally extending to video prediction, frame interpolation, multi-view synthesis, and long-video generation. STIV introduces a novel frame-replacement-based image conditioning injection mechanism and joint text-image classifier-free guidance, enabling multi-task unification and strong generalization within a minimal architectural footprint. Coupled with rigorous data curation and training optimization, its 8.7B-parameter model achieves state-of-the-art performance on VBench at 512 resolution: 83.1 for T2V (surpassing CogVideoX-5B and Pika) and 90.1 for TI2V—setting a new SOTA. STIV establishes the first transparent, systematic, and reproducible large-scale video generation paradigm.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The field of video generation has made remarkable advancements, yet there remains a pressing need for a clear, systematic recipe that can guide the development of robust and scalable models. In this work, we present a comprehensive study that systematically explores the interplay of model architectures, training recipes, and data curation strategies, culminating in a simple and scalable text-image-conditioned video generation method, named STIV. Our framework integrates image condition into a Diffusion Transformer (DiT) through frame replacement, while incorporating text conditioning via a joint image-text conditional classifier-free guidance. This design enables STIV to perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks simultaneously. Additionally, STIV can be easily extended to various applications, such as video prediction, frame interpolation, multi-view generation, and long video generation, etc. With comprehensive ablation studies on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V, surpassing both leading open and closed-source models like CogVideoX-5B, Pika, Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and extensible recipe for building cutting-edge video generation models, we aim to empower future research and accelerate progress toward more versatile and reliable video generation solutions.

Problem

• Developing scalable text-image conditioned video generation models
• Systematically exploring model architectures and training strategies
• Creating transparent recipes for versatile video generation tasks

Innovation

• Integrates image conditioning via frame replacement
• Uses joint image-text classifier-free guidance
• Enables simultaneous T2V and TI2V generation tasks