DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation

2026-06-30

AI Summary

This work addresses the challenges of scaling foreground vehicle insertion in autonomous driving simulation, where conventional 3D asset–based methods suffer from lighting inconsistencies and high manual modeling costs. The authors propose a point cloud–conditioned video inpainting framework that eliminates explicit reliance on 3D assets by employing a global-to-local hierarchical generation strategy. Integrating 3D Gaussian representations with an urban scene reconstruction pipeline, the method synthesizes temporally coherent and visually realistic vehicles within masked regions while achieving seamless blending into the original scenes. Evaluated across multiple datasets, the approach significantly outperforms existing solutions, demonstrating superior visual realism and geometric consistency, and enabling efficient, scalable augmentation of autonomous driving scenarios.

Abstract

A pivotal step in autonomous driving simulation involves inserting foreground vehicles with predefined trajectories into simulated scenes. This process enhances scene diversity and facilitates the creation of various corner cases for testing and improving autonomous driving models. However, existing methods often rely on pre-reconstructed 3D assets, which frequently lead to lighting inconsistencies between the inserted foreground and the background. Moreover, the reliance on limited, manually-curated 3D assets hinders large-scale deployment. To address these challenges, we propose DriveWeaver, a novel framework for controllable vehicle insertion in autonomous driving simulation. Specifically, for a masked target insertion area, DriveWeaver performs video inpainting conditioned on vehicle point clouds to generate high-quality, temporally consistent vehicles. This video-inpainting-based approach ensures seamless blending between the foreground and background, while the readily available point cloud conditions enable superior generalization. To support long-term generation, we further design a global-to-local hierarchical inpainting strategy, ensuring the consistent identity and appearance of the inserted vehicles. Meanwhile, we extract explicit 3D Gaussian representations of the inserted vehicles through an urban reconstruction pipeline to enable real-time rendering for autonomous driving simulation. Extensive experiments across diverse datasets demonstrate that our method outperforms existing baselines in visual realism and geometric consistency, providing a robust tool for scalable autonomous driving scene augmentation.

Problem

• autonomous driving simulation
• vehicle insertion
• video inpainting
• lighting inconsistency
• 3D assets

Innovation

• video inpainting
• point cloud conditioning
• controllable vehicle insertion
• 3D Gaussian representation
• autonomous driving simulation