3D and 4D World Modeling: A Survey

2025-09-04

AI Summary

Existing world modeling research predominantly focuses on 2D image/video generation, neglecting large-scale scene modeling using native 3D/4D representations—such as RGB-D, occupancy grids, and LiDAR point clouds—and lacks a unified definition and systematic taxonomy. Method: This paper introduces, for the first time, a standardized definition and a structured classification framework for 3D/4D world models, systematically categorizing generative paradigms into VideoGen, OccGen, and LiDARGen. It integrates generative modeling, 3D perception, and spatiotemporal modeling techniques, and synthesizes evaluation metrics and benchmark datasets. Contribution/Results: As the first comprehensive survey in this emerging field, it establishes WorldBench—an open-source literature platform—thereby filling a critical theoretical gap and providing foundational guidance and standardization pathways for 3D/4D world modeling research.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

World modeling has become a cornerstone in AI research, enabling agents to understand, represent, and predict the dynamic environments they inhabit. While prior work largely emphasizes generative methods for 2D image and video data, they overlook the rapidly growing body of work that leverages native 3D and 4D representations such as RGB-D imagery, occupancy grids, and LiDAR point clouds for large-scale scene modeling. At the same time, the absence of a standardized definition and taxonomy for ``world models'' has led to fragmented and sometimes inconsistent claims in the literature. This survey addresses these gaps by presenting the first comprehensive review explicitly dedicated to 3D and 4D world modeling and generation. We establish precise definitions, introduce a structured taxonomy spanning video-based (VideoGen), occupancy-based (OccGen), and LiDAR-based (LiDARGen) approaches, and systematically summarize datasets and evaluation metrics tailored to 3D/4D settings. We further discuss practical applications, identify open challenges, and highlight promising research directions, aiming to provide a coherent and foundational reference for advancing the field. A systematic summary of existing literature is available at https://github.com/worldbench/survey

Problem

• Surveying 3D and 4D world modeling methods
• Addressing lack of standardized definitions and taxonomy
• Reviewing datasets and metrics for 3D/4D settings

Innovation

• Leveraging native 3D and 4D representations
• Introducing structured taxonomy for world models
• Systematically summarizing datasets and evaluation metrics