MObI: Multimodal Object Inpainting Using Diffusion Models
Addressing the challenges of acquiring real-world multimodal (RGB + LiDAR) data for safety-critical applications such as autonomous driving—and the limited realism and spatial controllability of synthetic alternatives—this paper introduces the first diffusion-based framework conditioned on 3D bounding boxes for camera-LiDAR co-located object insertion guided by a single RGB reference image. Our method features: (1) 3D bounding box-driven spatial conditioning, replacing conventional masks to eliminate geometric ambiguity; (2) cross-modal feature alignment and consistency constraints to ensure geometric fidelity and semantic coherence; and (3) a joint sensor-generation architecture enabling scale-adaptive and synchronized cross-modal control. Evaluated on real automotive datasets, our approach improves SSIM and cross-modal consistency metrics by over 27% versus baselines, significantly enhancing robustness testing coverage for perception models.