HumanMoveVQA: Can Video MLLMs reason about human movement in videos?

2026-06-26

AI Summary

Existing video-based multimodal large models struggle to capture the global spatiotemporal trajectories and orientation dynamics of human motion, limiting them to coarse-grained semantic understanding. To address this gap, this work introduces the first video question-answering benchmark specifically designed for egocentric-view global human motion reasoning. By anchoring to the world coordinate system defined in the first frame, the benchmark elevates 2D observations to world-consistent 3D trajectories and comprises over 10,000 structured question-answer pairs spanning seven task categories, including trajectory prediction, orientation reasoning, and temporal inference. The benchmark is built via a scalable, multi-stage pipeline integrating 2D-to-3D pose lifting, coordinate alignment, question generation, and targeted fine-tuning. Experiments reveal significant deficiencies of mainstream closed-source models on such deep reasoning tasks, while fine-tuned open-source baselines demonstrate substantial performance gains.

Abstract

Despite the rapid advance of Multimodal Large Language Models (MLLMs) in high-level video understanding, a fundamental bottleneck remains: these models collapse complex human motion into coarse semantic labels. Existing benchmarks mostly focus on scene-centric events or local joint articulations, failing to probe global human motion in space over time (trajectory and orientation changes). We introduce HumanMoveVQA, the first comprehensive benchmark designed to evaluate global trajectory and orientation reasoning from an exocentric perspective. Our benchmark utilizes a first-frame anchored world coordinate system, preserving translation and rotation relative to a fixed starting point. We propose a scalable, multi-stage pipeline that lifts 2D video observations into world-consistent 3D motion tracks to generate over 10K structured question-answer pairs across seven reasoning categories, including motion aggregation, sequential ordering, and trajectory-level inference. Our extensive evaluation reveals a critical capability gap in state-of-the-art proprietary models on deep human motion understanding. However, we demonstrate that this is a learnable problem; by fine-tuning an open-source baseline with our targeted, world-consistent supervision, we achieve a significant improvement.HumanMoveVQA establishes a rigorous geometric foundation for developing next-generation, movement-aware video understanding models.

Problem

• human motion
• video understanding
• trajectory reasoning
• orientation change
• Multimodal Large Language Models

Innovation

• HumanMoveVQA
• world-consistent 3D motion
• trajectory reasoning
• video MLLMs
• exocentric human motion understanding