InfiniVerse: Occupancy Guided Unbounded Scene Generation for Autonomous Driving

2026-06-30

AI Summary

Generating large-scale urban scenes that are photorealistic, controllable, and temporally coherent remains a key challenge in autonomous driving. This work proposes a unified framework introducing a novel hierarchical sketch-refine paradigm: it first reconstructs a 3D occupancy grid from single-frame multi-view images, then autoregressively extends the scene along arbitrary trajectories while leveraging a video diffusion model to synthesize spatiotemporally consistent, realistic videos. A video reprojection feedback mechanism refines the 3D representation, enabling cross-modal alignment and mutual enhancement between visual and spatial domains. The method achieves state-of-the-art performance on Waymo and nuScenes benchmarks, with FID of 6.4 and FVD of 67.97, significantly outperforming existing approaches in generation duration and stability.

Abstract

Generating realistic, controllable, and temporally coherent urban environments is a critical yet unresolved challenge in the autonomous driving community. In this paper, we introduce InfiniVerse, a unified pipeline for long-range, 2D-3D-aligned, and controllable synthesis of dynamic urban scenes from a single frame. In practice, our approach first reconstructs a 3D occupancy representation from the input multi-view frame. This representation serves as a foundation for autoregressive scene extension along arbitrary trajectories. Subsequently, a video diffusion model translates the coarse occupancy grid into realistic, spatiotemporally consistent video sequences. Moreover, we propose a hierarchical sketch-and-refine paradigm, in which the generated videos are re-projected as image-conditioned feedback to enhance the 3D occupancy representation, establishing cross-modal alignment and mutual enhancement between the visual and spatial domains. Extensive evaluations on the Waymo Open Dataset and nuScenes demonstrate that InfiniVerse achieves state-of-the-art performance, with a FID of 6.4 and FVD of 67.97, significantly outperforming existing benchmarks in both duration and stability.

Problem

• autonomous driving
• scene generation
• temporal coherence
• urban environments
• controllability

Innovation

• occupancy representation
• autoregressive scene extension
• video diffusion model
• sketch-and-refine paradigm
• cross-modal alignment