UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulation

2024-11-28

arXiv.org

AI Summary

In urban scene simulation and autonomous driving testing, existing 3D vehicle models struggle to simultaneously achieve controllability and photorealism. Method: This paper proposes a single-image-driven digital twin modeling approach that integrates open-source CAD models with hand-crafted materials to construct an editable CAD-NeRF hybrid representation. It introduces the first retrieval–optimization co-framework, enabling CAD-level part manipulation, material transfer, and relighting while achieving reconstruction-grade rendering quality. Innovatively, it incorporates fisheye-based ambient light estimation and 3D Gaussian Splatting (3DGS) for background reconstruction, facilitating realistic vehicle insertion. Contribution/Results: Experiments demonstrate significant improvements in photorealism over state-of-the-art reconstruction and retrieval baselines. The method successfully generates challenging long-tail scenarios, effectively exposing performance degradation of perception models under out-of-distribution conditions—establishing a novel paradigm for autonomous driving safety evaluation.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Photorealistic 3D vehicle models with high controllability are essential for autonomous driving simulation and data augmentation. While handcrafted CAD models provide flexible controllability, free CAD libraries often lack the high-quality materials necessary for photorealistic rendering. Conversely, reconstructed 3D models offer high-fidelity rendering but lack controllability. In this work, we introduce UrbanCAD, a framework that pushes the frontier of the photorealism-controllability trade-off by generating highly controllable and photorealistic 3D vehicle digital twins from a single urban image and a collection of free 3D CAD models and handcrafted materials. These digital twins enable realistic 360-degree rendering, vehicle insertion, material transfer, relighting, and component manipulation such as opening doors and rolling down windows, supporting the construction of long-tail scenarios. To achieve this, we propose a novel pipeline that operates in a retrieval-optimization manner, adapting to observational data while preserving flexible controllability and fine-grained handcrafted details. Furthermore, given multi-view background perspective and fisheye images, we approximate environment lighting using fisheye images and reconstruct the background with 3DGS, enabling the photorealistic insertion of optimized CAD models into rendered novel view backgrounds. Experimental results demonstrate that UrbanCAD outperforms baselines based on reconstruction and retrieval in terms of photorealism. Additionally, we show that various perception models maintain their accuracy when evaluated on UrbanCAD with in-distribution configurations but degrade when applied to realistic out-of-distribution data generated by our method. This suggests that UrbanCAD is a significant advancement in creating photorealistic, safety-critical driving scenarios for downstream applications.

Problem

• Generates controllable photorealistic 3D vehicle digital twins
• Balances CAD model controllability with photorealistic rendering quality
• Enables realistic vehicle rendering and manipulation for urban simulations

Innovation

• Generates photorealistic 3D vehicle digital twins
• Uses retrieval-optimization pipeline for controllability
• Approximates lighting and reconstructs background with 3DGS