Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator

2026-07-06

AI Summary

This work addresses the limitations of existing embodied navigation research, which suffers from a lack of scalable, high-fidelity, and physically plausible interactive environments due to the scarcity of real-world scan data and the substantial sim-to-real gap in synthetic simulators. The authors propose Image2Sim, a real-time neural simulation framework that decouples 3D spatial anchoring from photorealistic observation synthesis. It constructs scene representations using feedforward feature Gaussian models and employs geometry-aware, single-step pixel-stream rendering to generate panoramic RGB-D observations. This approach enables fully automatic conversion of static, pose-annotated RGB-D sequences into large-scale interactive navigation environments, yielding nearly 20,000 scenes and over 10 million navigation samples. Navigation models trained solely on this neural simulation data significantly outperform current methods on standard benchmarks and demonstrate strong zero-shot transfer performance in real-world settings.

Abstract

Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated embodied data engine that generates high-fidelity observations, executable actions, and diverse navigation instructions at scale. It converts large collections of videos and images into nearly 20K interactive scenes and synthesizes more than 10 million navigation training samples. Navigation models trained entirely in these neural environments achieve strong improvements on major benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for embodied navigation at scale.

Problem

• embodied navigation
• scalable simulation
• sim-to-real gap
• interactive environments
• neural simulation

Innovation

• neural simulation
• feature Gaussian representation
• geometry-aware rendering
• embodied navigation
• sim-to-real transfer