RealEngine: Simulating Autonomous Driving in Realistic Context

2025-05-22

AI Summary

Existing driving simulators suffer from systematic limitations in multimodal perceptual fidelity, scene realism, closed-loop evaluation capability, traffic diversity, multi-agent collaboration, and computational efficiency. This paper proposes SimFusion, a high-fidelity closed-loop driving simulation framework that, for the first time, jointly models background/foreground-decoupled 3D scene reconstruction and multimodal view synthesis—powered by Neural Radiance Fields (NeRF)—to unify perceptual authenticity with geometric accuracy. It introduces a closed-loop simulation engine supporting non-reactive scenario generation, safety-critical testing, and multi-agent coordination. Leveraging real-world multimodal sensor data, SimFusion significantly narrows the perception gap between simulation and reality. Evaluated on multiple benchmarks, it enhances the reliability of driving policy assessment while enabling scalable, low-cost, and highly diverse closed-loop testing.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Driving simulation plays a crucial role in developing reliable driving agents by providing controlled, evaluative environments. To enable meaningful assessments, a high-quality driving simulator must satisfy several key requirements: multi-modal sensing capabilities (e.g., camera and LiDAR) with realistic scene rendering to minimize observational discrepancies; closed-loop evaluation to support free-form trajectory behaviors; highly diverse traffic scenarios for thorough evaluation; multi-agent cooperation to capture interaction dynamics; and high computational efficiency to ensure affordability and scalability. However, existing simulators and benchmarks fail to comprehensively meet these fundamental criteria. To bridge this gap, this paper introduces RealEngine, a novel driving simulation framework that holistically integrates 3D scene reconstruction and novel view synthesis techniques to achieve realistic and flexible closed-loop simulation in the driving context. By leveraging real-world multi-modal sensor data, RealEngine reconstructs background scenes and foreground traffic participants separately, allowing for highly diverse and realistic traffic scenarios through flexible scene composition. This synergistic fusion of scene reconstruction and view synthesis enables photorealistic rendering across multiple sensor modalities, ensuring both perceptual fidelity and geometric accuracy. Building upon this environment, RealEngine supports three essential driving simulation categories: non-reactive simulation, safety testing, and multi-agent interaction, collectively forming a reliable and comprehensive benchmark for evaluating the real-world performance of driving agents.

Problem

• Simulating autonomous driving with realistic multi-modal sensing and rendering
• Enabling closed-loop evaluation for diverse traffic scenarios and interactions
• Providing efficient, scalable simulation for comprehensive agent performance assessment

Innovation

• Integrates 3D scene reconstruction and view synthesis
• Reconstructs scenes and traffic participants separately
• Supports non-reactive, safety, and multi-agent simulation