Instant NuRec: Feed-Forward 3D Gaussian Reconstruction for Driving Scene Simulation

2026-07-15

AI Summary

This work addresses the inefficiency and scene-specific tuning requirements of existing neural simulation methods for autonomous driving, which hinder the rapid generation of simulatable 3D environments. The authors propose a feedforward neural reconstruction model that, for the first time, generates complete, simulatable 3D driving scenes in a single forward pass. The model incorporates static and dynamic Gaussian layers, a sky cubemap, and camera ISP correction, while supporting non-pinhole camera models. Built upon the 3D Gaussian Splatting (3DGS) framework and the 3DGUT architecture, it leverages multi-view geometry and hierarchical representation learning. Evaluated on the Waymo Open Dataset, the method achieves a PSNR improvement of 2.01 dB over the strongest baseline and reconstructs multi-camera scenes in approximately 1.5 seconds, enabling seamless integration into closed-loop simulation systems such as NuRec and AlpaSim.

Abstract

3D simulation platforms are critical for autonomous driving because they enable end-to-end policy evaluation, thereby reducing development costs and improving safety. In recent years, neural simulation has become predominant, with methods such as NuRec playing a central role; however, these methods remain relatively slow and typically require per-scene tuning. In this work, we present Instant NuRec, a feed-forward neural reconstruction model that turns a short multi-view driving log into a fully simulatable 3D Gaussian Splatting (3DGS) world in a single forward pass. The model accepts multi-view input from a calibrated camera rig and emits a layered output consisting of static and dynamic 3DGS layers, a sky cubemap, and per-camera ISP corrections, while providing native support for non-pinhole camera models via 3DGUT. It reconstructs a 10-20-second multi-camera scene in roughly 1.5 seconds and achieves a PSNR on the Waymo Open Dataset that is 2.01 dB above the strongest evaluated baseline. Instant NuRec is deeply integrated into NuRec and is compatible with AlpaSim for closed-loop simulation.

Problem

• 3D reconstruction
• autonomous driving simulation
• neural rendering
• 3D Gaussian Splatting
• real-time simulation

Innovation

• Feed-forward reconstruction
• 3D Gaussian Splatting
• Driving scene simulation
• Non-pinhole camera support
• Instant neural rendering