VLM4D: Towards Spatiotemporal Awareness in Vision Language Models

2025-08-04

AI Summary

Current vision-language models (VLMs) exhibit significant deficiencies in dynamic spatiotemporal reasoning—e.g., object motion, rotation, viewpoint changes, and temporal continuity. To address this, we introduce VLM4D, the first benchmark explicitly designed to evaluate VLMs’ spatiotemporal perception capabilities, comprising both real-world and synthetic videos with structured question-answer pairs targeting translation, rotation, viewpoint transformation, and motion coherence. We conduct the first systematic evaluation of mainstream VLMs on dynamic scenes, revealing critical limitations in multi-cue integration and temporal consistency modeling. Building on these insights, we propose 4D feature field reconstruction and spatiotemporal supervised fine-tuning, which substantially improve dynamic understanding performance on VLM4D—yet still fall markedly short of human-level competence. This work establishes a new paradigm and scalable toolkit for assessing and enhancing spatiotemporal reasoning in VLMs.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Vision language models (VLMs) have shown remarkable capabilities in integrating linguistic and visual reasoning but remain fundamentally limited in understanding dynamic spatiotemporal interactions. Humans effortlessly track and reason about object movements, rotations, and perspective shifts-abilities essential for robust dynamic real-world understanding yet notably lacking in current VLMs. In this paper, we introduce VLM4D, the first benchmark specifically designed to evaluate the spatiotemporal reasoning capabilities of VLMs. Our benchmark comprises diverse real-world and synthetic videos accompanied by carefully curated question-answer pairs emphasizing translational and rotational motions, perspective awareness, and motion continuity. Through comprehensive evaluations of state-of-the-art open and closed-source VLMs, we identify significant performance gaps compared to human baselines, highlighting fundamental deficiencies in existing models. Extensive analysis reveals that VLMs struggle particularly with integrating multiple visual cues and maintaining temporal coherence. We further explore promising directions, such as leveraging 4D feature field reconstruction and targeted spatiotemporal supervised fine-tuning, demonstrating their effectiveness in enhancing spatiotemporal comprehension. Our work aims to encourage deeper exploration into improving VLMs' spatial and temporal grounding, paving the way towards more capable and reliable visual intelligence for dynamic environments.

Problem

• Evaluating VLMs' spatiotemporal reasoning with VLM4D benchmark
• Addressing VLMs' deficiencies in dynamic visual cue integration
• Enhancing VLMs' temporal coherence and motion understanding

Innovation

• 4D feature field reconstruction for spatiotemporal understanding
• Spatiotemporal supervised fine-tuning to enhance model performance
• Diverse real-world and synthetic video benchmark VLM4D