NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfolding

2025-09-29

AI Summary

This paper introduces a novel paradigm for constructing interactive 3D virtual worlds from a single image, addressing the core challenge of achieving photorealistic 3D rendering in user-explored regions while efficiently synthesizing non-interactive areas. Methodologically, it proposes a progressive 3D unfolding mechanism: foreground objects are modeled as full, object-centric 3D representations and rendered via differentiable rendering; the background is synthesized semantically coherently in 2D. Notably, it is the first method to support natural language–driven appearance editing of objects and physically plausible dynamic responses. The approach integrates representation learning, object-level 3D generation, and a hybrid 2D–3D scene architecture. Evaluated on the WorldScore benchmark, it significantly outperforms existing 2D and 2.5D methods, demonstrating superior visual fidelity, real-time interactive capability, and immersive, free-viewpoint navigation.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce NeoWorld, a deep learning framework for generating interactive 3D virtual worlds from a single input image. Inspired by the on-demand worldbuilding concept in the science fiction novel Simulacron-3 (1964), our system constructs expansive environments where only the regions actively explored by the user are rendered with high visual realism through object-centric 3D representations. Unlike previous approaches that rely on global world generation or 2D hallucination, NeoWorld models key foreground objects in full 3D, while synthesizing backgrounds and non-interacted regions in 2D to ensure efficiency. This hybrid scene structure, implemented with cutting-edge representation learning and object-to-3D techniques, enables flexible viewpoint manipulation and physically plausible scene animation, allowing users to control object appearance and dynamics using natural language commands. As users interact with the environment, the virtual world progressively unfolds with increasing 3D detail, delivering a dynamic, immersive, and visually coherent exploration experience. NeoWorld significantly outperforms existing 2D and depth-layered 2.5D methods on the WorldScore benchmark.

Problem

• Generating interactive 3D worlds from single images
• Progressively rendering explored regions with high realism
• Enabling viewpoint manipulation and animation via natural language

Innovation

• Generates 3D virtual worlds from single image
• Uses hybrid 3D foreground and 2D background synthesis
• Progressively unfolds 3D detail during user exploration