SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restoration

2025-12-25

AI Summary

Existing 3D autonomous driving simulators struggle to simultaneously achieve photorealistic rendering and interactive traffic editing—particularly suffering from geometric distortions and illumination artifacts during large-baseline novel-view synthesis and dynamic asset insertion. To address this, we propose a diffusion-based symmetric autoregressive online inpainting framework. Our method introduces two key innovations: (i) a dual-view ground-truth-guided reconstruction mechanism, and (ii) a training-free, context-aware vehicle insertion technique. It jointly leverages symmetric view modeling, autoregressive lateral generation, and illumination-consistent, training-free image inpainting. On novel-view enhancement and 3D vehicle insertion benchmarks, our approach achieves state-of-the-art performance, significantly suppressing geometric distortion and lighting artifacts. The framework enables real-time, controllable, high-fidelity simulation and effectively mitigates data scarcity in long-tail driving scenarios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

High-fidelity and controllable 3D simulation is essential for addressing the long-tail data scarcity in Autonomous Driving (AD), yet existing methods struggle to simultaneously achieve photorealistic rendering and interactive traffic editing. Current approaches often falter in large-angle novel view synthesis and suffer from geometric or lighting artifacts during asset manipulation. To address these challenges, we propose SymDrive, a unified diffusion-based framework capable of joint high-quality rendering and scene editing. We introduce a Symmetric Auto-regressive Online Restoration paradigm, which constructs paired symmetric views to recover fine-grained details via a ground-truth-guided dual-view formulation and utilizes an auto-regressive strategy for consistent lateral view generation. Furthermore, we leverage this restoration capability to enable a training-free harmonization mechanism, treating vehicle insertion as context-aware inpainting to ensure seamless lighting and shadow consistency. Extensive experiments demonstrate that SymDrive achieves state-of-the-art performance in both novel-view enhancement and realistic 3D vehicle insertion.

Problem

• Enables photorealistic rendering and interactive traffic editing in driving simulations
• Addresses large-angle view synthesis and artifact issues in 3D simulation
• Proposes a diffusion-based framework for joint rendering and scene editing

Innovation

• Unified diffusion framework for joint rendering and editing
• Symmetric auto-regressive online restoration for detail recovery
• Training-free harmonization via context-aware inpainting