A Survey of Interactive Generative Video

2025-04-30

AI Summary

This paper addresses core challenges in Interactive Generative Video (IGV)—including real-time responsiveness, open-domain controllability, long-range temporal coherence, physically accurate modeling, and causal reasoning—arising in gaming, embodied AI, and autonomous driving. To this end, we propose the first ideal five-module IGV framework: Generation, Control, Memory, Dynamics, and Intelligence. Our approach integrates diffusion/Transformer-based generative models, cross-modal control interfaces, sequential memory mechanisms, neural physical simulation, and causal representation learning to systematically resolve multimodal interaction and dynamic environmental response. We further construct the first structured IGV technology roadmap, explicitly identifying key technical bottlenecks and developmental trajectories for each module. This work establishes both a theoretical foundation and an engineering paradigm for next-generation interactive video systems.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Interactive Generative Video (IGV) has emerged as a crucial technology in response to the growing demand for high-quality, interactive video content across various domains. In this paper, we define IGV as a technology that combines generative capabilities to produce diverse high-quality video content with interactive features that enable user engagement through control signals and responsive feedback. We survey the current landscape of IGV applications, focusing on three major domains: 1) gaming, where IGV enables infinite exploration in virtual worlds; 2) embodied AI, where IGV serves as a physics-aware environment synthesizer for training agents in multimodal interaction with dynamically evolving scenes; and 3) autonomous driving, where IGV provides closed-loop simulation capabilities for safety-critical testing and validation. To guide future development, we propose a comprehensive framework that decomposes an ideal IGV system into five essential modules: Generation, Control, Memory, Dynamics, and Intelligence. Furthermore, we systematically analyze the technical challenges and future directions in realizing each component for an ideal IGV system, such as achieving real-time generation, enabling open-domain control, maintaining long-term coherence, simulating accurate physics, and integrating causal reasoning. We believe that this systematic analysis will facilitate future research and development in the field of IGV, ultimately advancing the technology toward more sophisticated and practical applications.

Problem

• Defining Interactive Generative Video (IGV) technology combining generative and interactive features
• Surveying IGV applications in gaming, embodied AI, and autonomous driving
• Proposing a framework and analyzing challenges for ideal IGV systems

Innovation

• Combines generative and interactive video capabilities
• Proposes five essential modules for IGV systems
• Addresses real-time generation and open-domain control