EVA: An Embodied World Model for Future Video Anticipation

2024-10-20

arXiv.org

AI Summary

Existing video generation models exhibit limited reasoning capabilities in embodied future prediction, struggling with multi-step inference and out-of-distribution (OOD) generalization. Method: We propose EVA, a unified embodied video anticipation framework that decouples complex dynamic intent modeling into four meta-tasks. Our approach introduces (1) EVA-Bench—a novel benchmark for embodied video prediction; (2) a quadruple meta-task decomposition mechanism; and (3) multi-stage LoRA-adaptive pretraining to jointly optimize vision-language understanding and high-fidelity video generation. EVA integrates a vision-language model (VLM), diffusion or autoregressive video generators, and a meta-task-decoupled architecture. Contribution/Results: On EVA-Bench, EVA significantly surpasses state-of-the-art methods. It achieves substantial improvements in long-horizon prediction accuracy and physical plausibility for both human and robotic action scenarios, demonstrating the effective transfer of large foundation models to real-world dynamic prediction tasks.

Technology Category

• Computer Vision: Large Vision Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Embodied AI

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

World models integrate raw data from various modalities, such as images and language to simulate comprehensive interactions in the world, thereby displaying crucial roles in fields like mixed reality and robotics. Yet, applying the world model for accurate video prediction is quite challenging due to the complex and dynamic intentions of the various scenes in practice. In this paper, inspired by the human rethinking process, we decompose the complex video prediction into four meta-tasks that enable the world model to handle this issue in a more fine-grained manner. Alongside these tasks, we introduce a new benchmark named Embodied Video Anticipation Benchmark (EVA-Bench) to provide a well-rounded evaluation. EVA-Bench focused on evaluating the video prediction ability of human and robot actions, presenting significant challenges for both the language model and the generation model. Targeting embodied video prediction, we propose the Embodied Video Anticipator (EVA), a unified framework aiming at video understanding and generation. EVA integrates a video generation model with a visual language model, effectively combining reasoning capabilities with high-quality generation. Moreover, to enhance the generalization of our framework, we tailor-designed a multi-stage pretraining paradigm that adaptatively ensembles LoRA to produce high-fidelity results. Extensive experiments on EVA-Bench highlight the potential of EVA to significantly improve performance in embodied scenes, paving the way for large-scale pre-trained models in real-world prediction tasks.

Problem

• Enhance video prediction with intermediate reasoning strategies
• Address limitations in multi-step and OOD scenario predictions
• Develop a benchmark for evaluating embodied world models

Innovation

• Reflection of Generation enhances video prediction.
• Combines vision-language and video generation models.
• Multistage training for high-fidelity video frames.