Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering

2024-06-02

arXiv.org

AI Summary

Modeling dynamic 3D physical attributes (e.g., velocity, acceleration, collisions) and inter-object interactions in videos remains challenging, hindering high-level temporal and action-oriented semantic reasoning. Method: We introduce DynSuperCLEVR—the first video question-answering benchmark explicitly designed for 4D dynamic physical reasoning—and propose NS-4DPhysics, a novel model integrating physics priors with explicit 4D scene representations. It infers latent world states via 3D generative modeling and unifies factual, predictive, and counterfactual reasoning through neural-symbolic inference. Contribution/Results: NS-4DPhysics achieves the first physics-driven, cross-frame 4D scene disentanglement and structured causal reasoning. On DynSuperCLEVR, it significantly outperforms state-of-the-art VideoQA methods and multimodal large language models, demonstrating that explicit physics-guided representation is critical for complex spatiotemporal causal reasoning.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Low Level & Physics-based Vision
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

For vision-language models (VLMs), understanding the dynamic properties of objects and their interactions in 3D scenes from videos is crucial for effective reasoning about high-level temporal and action semantics. Although humans are adept at understanding these properties by constructing 3D and temporal (4D) representations of the world, current video understanding models struggle to extract these dynamic semantics, arguably because these models use cross-frame reasoning without underlying knowledge of the 3D/4D scenes. In this work, we introduce DynSuperCLEVR, the first video question answering dataset that focuses on language understanding of the dynamic properties of 3D objects. We concentrate on three physical concepts -- velocity, acceleration, and collisions within 4D scenes. We further generate three types of questions, including factual queries, future predictions, and counterfactual reasoning that involve different aspects of reasoning about these 4D dynamic properties. To further demonstrate the importance of explicit scene representations in answering these 4D dynamics questions, we propose NS-4DPhysics, a Neural-Symbolic VideoQA model integrating Physics prior for 4D dynamic properties with explicit scene representation of videos. Instead of answering the questions directly from the video text input, our method first estimates the 4D world states with a 3D generative model powered by physical priors, and then uses neural symbolic reasoning to answer the questions based on the 4D world states. Our evaluation on all three types of questions in DynSuperCLEVR shows that previous video question answering models and large multimodal models struggle with questions about 4D dynamics, while our NS-4DPhysics significantly outperforms previous state-of-the-art models. Our code and data are released in https://xingruiwang.github.io/projects/DynSuperCLEVR/.

Problem

• Understanding dynamic 4D object properties in videos
• Improving video question answering with physics priors
• Addressing limitations in current 4D scene reasoning models

Innovation

• Uses 3D generative model with physics priors
• Integrates neural-symbolic reasoning for VideoQA
• Estimates 4D world states for dynamic understanding