GenEx: Generating an Explorable World

2024-12-12

arXiv.org

AI Summary

Embodied agents face significant challenges in long-horizon exploration within real-world 3D environments, including poor visual coherence, inaccurate 3D localization, and difficulty predicting occluded regions. Method: This paper introduces GenEx—the first generative embodied exploration framework that synthesizes complete, 3D-consistent, and interactive panoramic environments from a single RGB image. GenEx integrates diffusion modeling with a large-scale 3D dataset built in Unreal Engine to enable single-image-driven, 360° continuous 3D world generation and closed-loop consistency modeling. It embeds generative prior depth estimates directly into the decision-making loop for belief updating and outcome pre-simulation, and combines GPT-enhanced embodied planning with active 3D semantic mapping. Contribution/Results: Experiments demonstrate that GenEx substantially improves trajectory consistency, real-time mapping fidelity, and decision robustness in both goal-free exploration and goal-directed navigation tasks.

Technology Category

• Natural Language Processing: Generation
• Intelligent Robots: Embodied AI
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Understanding, navigating, and exploring the 3D physical real world has long been a central challenge in the development of artificial intelligence. In this work, we take a step toward this goal by introducing GenEx, a system capable of planning complex embodied world exploration, guided by its generative imagination that forms priors (expectations) about the surrounding environments. GenEx generates an entire 3D-consistent imaginative environment from as little as a single RGB image, bringing it to life through panoramic video streams. Leveraging scalable 3D world data curated from Unreal Engine, our generative model is rounded in the physical world. It captures a continuous 360-degree environment with little effort, offering a boundless landscape for AI agents to explore and interact with. GenEx achieves high-quality world generation, robust loop consistency over long trajectories, and demonstrates strong 3D capabilities such as consistency and active 3D mapping. Powered by generative imagination of the world, GPT-assisted agents are equipped to perform complex embodied tasks, including both goal-agnostic exploration and goal-driven navigation. These agents utilize predictive expectation regarding unseen parts of the physical world to refine their beliefs, simulate different outcomes based on potential decisions, and make more informed choices. In summary, we demonstrate that GenEx provides a transformative platform for advancing embodied AI in imaginative spaces and brings potential for extending these capabilities to real-world exploration.

Problem

• 3D world understanding
• visual coherence
• mapping and localization

Innovation

• 3D Imaginative Worlds
• Game Engine Data Augmentation
• Unseen Area Prediction