HybridWorldSim: A Scalable and Controllable High-fidelity Simulator for Autonomous Driving

2025-11-27

AI Summary

Existing autonomous driving simulation methods struggle to simultaneously achieve high-fidelity novel view synthesis and geometric consistency under large viewpoint variations. This paper proposes the first high-fidelity simulation framework integrating multi-pass neural reconstruction with generative modeling: Neural Radiance Fields (NeRF) enable geometrically accurate reconstruction of static scenes, while conditional diffusion models synthesize controllable, physically plausible dynamic agents; multi-view consistency constraints jointly optimize visual and spatial fidelity. This hybrid architecture enables, for the first time, differentiable co-modeling of static backgrounds and dynamic objects. Evaluated on complex urban driving scenarios, it significantly improves novel view synthesis quality (PSNR +2.1 dB) and geometric accuracy in depth and pose estimation (EPE −18.7%). We further release MIRROR, a large-scale, multi-pass real-world driving dataset, establishing a new benchmark for simulation evaluation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Realistic and controllable simulation is critical for advancing end-to-end autonomous driving, yet existing approaches often struggle to support novel view synthesis under large viewpoint changes or to ensure geometric consistency. We introduce HybridWorldSim, a hybrid simulation framework that integrates multi-traversal neural reconstruction for static backgrounds with generative modeling for dynamic agents. This unified design addresses key limitations of previous methods, enabling the creation of diverse and high-fidelity driving scenarios with reliable visual and spatial consistency. To facilitate robust benchmarking, we further release a new multi-traversal dataset MIRROR that captures a wide range of routes and environmental conditions across different cities. Extensive experiments demonstrate that HybridWorldSim surpasses previous state-of-the-art methods, providing a practical and scalable solution for high-fidelity simulation and a valuable resource for research and development in autonomous driving.

Problem

• Simulates autonomous driving with high-fidelity and control
• Ensures visual and geometric consistency in novel views
• Integrates static backgrounds and dynamic agents for diverse scenarios

Innovation

• Hybrid simulation framework integrates neural reconstruction and generative modeling
• Enables diverse high-fidelity driving scenarios with visual consistency
• Releases multi-traversal dataset for robust benchmarking across cities