Structured 4D Latent Predictive Model for Robot Planning

2026-07-01

AI Summary

This work addresses the limitations of existing 2D video–based robotic prediction models, which lack 3D geometric understanding and consequently struggle to ensure spatial reasoning accuracy and physical consistency. To overcome these challenges, the authors propose a structured 4D implicit-space prediction model that explicitly captures the temporal evolution of 3D scene structure by fusing visual observations with textual instructions. The approach incorporates a goal-conditioned inverse dynamics module to generate executable actions and employs a unified, decodable 3D representation that enables temporally coherent, physically plausible long-horizon prediction and planning across multiple viewpoints. Experimental results demonstrate that the model significantly outperforms current methods on complex manipulation tasks, excelling in 3D consistency, multi-view coherence, and generalization across visual conditions, with successful validation on a real robotic platform.

Abstract

Video predictive models are emerging as a powerful paradigm in robotics, offering a promising path toward task generalization, long-horizon planning, and flexible decision-making. However, prevailing approaches often operate on 2D video sequences, inherently lacking the 3D geometric understanding necessary for precise spatial reasoning and physical consistency. We introduce a Structured 4D Latent Predictive Model, which predicts the evolution of a scene's 3D structure in a structured latent space conditioned on observations and textual instructions. Our representation encodes the scene holistically and can be decoded into diverse 3D formats, enabling a more complete and 3D consistent scene understanding. This structured 4D latent predictive model serves as a planner, generating future scenes that are translated into executable actions by a goal-conditioned inverse dynamics module. Experiments demonstrate that our model generates futures with strong visual quality, substantially better 3D consistency and multi-view coherence compared to state-of-the-art video-based planners. Consequently, our full planning pipeline achieves superior performance on complex manipulation tasks, exhibits robust generalization to novel visual conditions, and proves effective on real-world robotic platforms. Our website is available at https://structured-4d-model.github.io/.

Problem

• 3D geometric understanding
• video predictive models
• spatial reasoning
• physical consistency
• robot planning

Innovation

• 4D latent prediction
• structured 3D representation
• video predictive planning
• inverse dynamics module
• 3D consistency