modeling

This work addresses the lack of a unified predictive framework for world models in robotic manipulation, which has led to fragmented research and ambiguous design choices. Focusing on three core questions—what to predict, how to relate predictions to actions, and when to use predictions—the paper proposes a functional taxonomy that distinguishes between integrated prediction-action models and explicit predictive planners, positioning world models as foundational predictive infrastructure for robot learning. The study presents a systematic review covering latent dynamics models, action-conditioned video generation, 3D/4D scene prediction, physics simulators, and prediction modules in vision–language–action systems. It also consolidates evaluation protocols across 34 manipulation datasets, highlighting open challenges such as contact modeling and hallucination control through the lenses of prediction fidelity, task performance, and simulation reliability.

A fundamental ambiguity persists in world modeling research regarding whether its core objective is *understanding* underlying world mechanisms or *predicting* future dynamics. Method: This paper introduces the first “Understanding-oriented vs. Prediction-oriented” dichotomy framework to rigorously delineate their theoretical boundaries and synergistic relationships. We conduct cross-domain analysis—spanning autonomous driving, robotics, and social simulation—to characterize divergent application paradigms. Integrating multimodal foundation models (e.g., GPT-4), video generation (e.g., Sora), causal inference, neuro-symbolic modeling, and reinforcement learning, we construct a unified four-dimensional taxonomy covering representation, prediction, intervention, and evaluation. Contribution/Results: We establish the first comprehensive survey framework for world models, explicitly identifying key challenges—including interpretability, generalization, and causal validity—and propose a three-tier evolutionary roadmap toward AGI. This work provides both a theoretical benchmark and practical guidance for advancing world modeling research.

The “world model” concept in neural networks lacks an operational definition, hindering rigorous evaluation and comparison. Method: We propose a formal, testable criterion grounded in linear probing theory, defining a world model as a causal representation of the environment’s latent state space, augmented by a nontriviality condition that excludes superficial fitting to data or task structure. Contribution/Results: This work provides the first verifiable, reproducible operational definition of world models; explicitly separates representational content (latent states) from computational mechanism (generation process); and establishes a unified conceptual language and experimental benchmark for empirical investigation. By enforcing causal fidelity and nontriviality, our framework significantly improves both the precision and testability of assessing neural networks’ internal causal modeling capacity—enabling principled diagnosis of whether learned representations genuinely capture environment dynamics rather than spurious correlations.

This study addresses the limitations of traditional physics simulators in robotics—such as restricted expressiveness due to simplifying assumptions, high data costs, and difficulties in modeling complex physical interactions—by systematically reviewing video generation models as embodied world models. Integrating high-fidelity, multimodal-conditioned video synthesis with imitation learning, reinforcement learning, and visual planning frameworks, this work provides the first comprehensive analysis of their potential and limitations in tasks including action prediction, dynamics modeling, and policy evaluation. The review highlights breakthroughs in high-fidelity modeling of physical interactions while identifying key challenges in instruction following, physical consistency, and safety. These insights lay a theoretical foundation and outline future directions for replacing conventional simulators and enabling deployment in safety-critical scenarios.

This work addresses the limitations of traditional spatiotemporal physical system modeling, which relies on pixel-level next-frame prediction and suffers from error accumulation and high training costs, thereby hindering effective support for downstream scientific tasks such as physical parameter estimation. The study proposes evaluating general self-supervised learning methods based on their ability to yield physically meaningful representations, using downstream task performance—particularly physical parameter estimation—as a benchmark. By comparing objective functions rooted in pixel-space prediction versus those operating in latent spaces (e.g., Joint Embedding Predictive Architecture, JEPA), the authors demonstrate that latent-space modeling substantially enhances both the physical interpretability of learned representations and downstream task performance. Empirical results reveal that certain general-purpose self-supervised approaches outperform specialized physics-based models, highlighting their significant potential for scientific representation learning.

Existing state space models (SSMs) struggle to capture non-stationary, multi-scale causal dynamics prevalent in real-world systems. Method: We propose a scalable hierarchical world model that jointly integrates latent-parameter SSMs with multi-timescale SSMs, enabling— for the first time—graph-structured, exact probabilistic inference with end-to-end temporal learning. Leveraging probabilistic graphical models, belief propagation, and Bayesian inference, our approach explicitly represents uncertainty to better approximate inherent stochasticity. Contribution/Results: Evaluated on diverse real and simulated robotic tasks, the model matches or surpasses leading Transformer variants in long-horizon future prediction while substantially enhancing cross-temporal and cross-spatial causal reasoning capabilities.

This study addresses critical challenges in applying deep learning to scientific computing—namely, poor interpretability, heavy data dependency, and insufficient physical consistency—within physics-based simulation scenarios. We propose a physics-driven AI modeling framework integrating physics-informed loss functions, differentiable simulators, diffusion-based generative models, physics-guided reinforcement learning, and custom neural architectures, implemented via an interactive Jupyter-based experimental platform. Crucially, we pioneer the systematic embedding of physical priors across the entire deep learning pipeline—model formulation, training, and inference—enabling high-fidelity, data-efficient, and verifiable scientific modeling. The resulting methodology is modular, reusable, and immediately deployable, significantly enhancing model generalizability and interpretability. This work establishes a novel paradigm and technical foundation for next-generation scientific foundation models.

This work proposes an isomorphic world model that addresses the limitation of existing approaches, which disrupt the spatial topology of sensory inputs during visual compression and thereby hinder brain-inspired physical prediction. By integrating neural fields with a motion-gated mechanism, the model leverages local lateral connections modulated by motor commands to preserve sensory spatial structure while enabling geometrically propagative predictions of physical states. Without explicit supervision, it learns ballistic dynamics and generates intermediate trajectories through a purely imagination-based training strategy. In real-world transfer tasks, the model achieves nearly twice the success rate of conventional latent-space methods and spontaneously develops a body schema along with body-selective representations.

Existing world models struggle to infer the complete physical structure of scenes and interactions among objects from partially observed videos. This work proposes a novel probabilistic world model based on autoregressive sequence modeling, which enables efficient training and supports conditional estimation over arbitrary visual variables—such as appearance and dynamics. The model generates multimodal future states over multiple steps, automatically discovers objects and their subparts, and facilitates 3D manipulation and physical reasoning. Experiments demonstrate that the model successfully extracts hierarchical object structures in tasks such as Visual Jenga, significantly enhancing the understanding of complex physical interactions.

This work addresses the challenge of efficiently generating high-quality action candidates in world model–based continuous control. The authors propose PRISM, a framework that, building upon a JEPA-style latent world model, introduces a lightweight MLP to directly extract a state-conditional Gaussian prior from a frozen encoder. This prior is then incorporated into model predictive control (MPC) as a precision-weighted product of Gaussians to guide action sampling. Notably, PRISM requires neither additional visual encoders nor large models, resulting in a concise, task-agnostic architecture. Empirical results demonstrate significant performance gains, with success rates improved by 35% on the Cube task and 32% on PushT, all without introducing substantial inference overhead.

This work addresses the limited generalizability of current data-driven physics simulation methods to real-world scenarios, primarily due to the absence of particle-level state annotations in real videos. To overcome this challenge, we propose the first differentiable particle dynamics model that requires no particle-level supervision and can be trained end-to-end directly on unlabeled real videos. Our approach integrates a dense particle representation based on Gaussian splatting, neural dynamics modeling, and rendering-based supervision to jointly learn the evolution of particle positions and orientations, thereby eliminating the need for heuristic sampling strategies. Evaluated on a newly curated dataset comprising approximately 500 diverse real-world videos of object interactions, our method demonstrates robust motion prediction capabilities in complex, realistic settings.

This work addresses the lack of verifiability in learned world models when deployed in high-assurance systems by proposing a novel framework that integrates classical model order reduction (MOR) with modern world modeling. The approach combines proper orthogonal decomposition (POD) with an encoder–decoder architecture, incorporates physics-informed error bounds derived from physical priors, and employs measurement-driven action-conditioned modeling to ensure verifiable closed-loop predictions, exceptional data efficiency, and physical consistency. By systematically unifying MOR theory with contemporary world model paradigms, this study establishes a new modeling methodology that simultaneously achieves reliability and performance for safety-critical applications.