4DVLT: Dynamic Scene Understanding with Worldline-Centered Vision-Language Tracking

2026-06-21

AI Summary

Existing approaches struggle to accurately align language instructions with persistent worldlines in dynamic scenes while maintaining consistent object identity, metric 3D motion, and multi-view 2D projection coherence. This work introduces, for the first time, a worldline-centric 4D visual-language tracking task, formulating an object-centric 4D state graph and proposing a graph-conditioned worldline inference framework that jointly reasons over multi-view videos and natural language instructions. To facilitate research in this direction, we present Instruct-4D, a large-scale benchmark encompassing diverse reasoning challenges. Experimental results demonstrate that our proposed model, 4DTrack-Qwen3.5-9B, achieves a TGA_Top1 score of 62.68 on this benchmark, outperforming the strongest baseline by 19.62 percentage points and significantly improving both target localization accuracy and worldline reconstruction quality.

Abstract

4D dynamic scene understanding requires grounding language to a persistent worldline that binds identity, metric 3D motion, and synchronized multi-view 2D projections. Existing paradigms capture only part of this structure: large multimodal models reason over rich visual evidence but rarely preserve metric topology, while vision-language tracking remains tied to fragmented 2D or 3D outputs and local continuation. We therefore introduce \textbf{4DVLT}, a worldline-centered task for instruction-conditioned 4D dynamic scene understanding in fully observed multi-view video, and \textbf{Instruct-4D}, a benchmark with 129.4K question-answer pairs, 64.7K target entities, 851 scenes, and 9 reasoning-oriented query types. To address this setting, we present \textbf{4DTrack}, which casts instruction-conditioned tracking as graph-conditioned worldline inference through an object-centric 4D state graph, metric-guided routing, bidirectional decoding, and kinematic calibration. On Instruct-4D, 4DTrack-Qwen3.5-9B reaches 62.68 $\mathrm{TGA}_{\mathrm{Top1}}$ and surpasses the best adapted VLT baseline by 19.62 points. These results show that worldline-centered modeling improves both target grounding and recovered worldline quality. The project page is available at https://github.com/mikubaka88/4DVLT.

Problem

• 4D dynamic scene understanding
• worldline
• vision-language tracking
• instruction grounding
• multi-view video

Innovation

• worldline-centered
• 4D dynamic scene understanding
• vision-language tracking
• metric 3D motion
• instruction-conditioned tracking