Gaussian Splatting is an Effective Data Generator for 3D Object Detection

2025-04-23

AI Summary

To address the limited effectiveness of existing 3D data augmentation methods for autonomous driving object detection, this paper proposes the first explicit 3D augmentation framework based on Gaussian Splatting. Our method reconstructs realistic 3D driving scenes and implants 3D objects with precise poses and rigid-body transformations, enforced via BEV and multi-view geometric alignment to ensure physical plausibility and geometric consistency. Unlike image-conditioned diffusion models, our approach operates directly in 3D space, prioritizing geometric diversity over appearance perturbation. Evaluated on nuScenes, injecting only a small number of external 3D objects yields significant detection accuracy gains—outperforming state-of-the-art diffusion-based 3D augmentation methods. We further show that hard samples (e.g., highly occluded or high-loss instances) provide marginal benefits for camera-centric detectors. Our core contributions are: (i) pioneering the use of Gaussian Splatting for controllable, interpretable 3D detection data generation; and (ii) empirically demonstrating that geometric fidelity—not just visual realism—is critical for performance improvement.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Security and Privacy: Data transparency and provenance

Abstract

We investigate data augmentation for 3D object detection in autonomous driving. We utilize recent advancements in 3D reconstruction based on Gaussian Splatting for 3D object placement in driving scenes. Unlike existing diffusion-based methods that synthesize images conditioned on BEV layouts, our approach places 3D objects directly in the reconstructed 3D space with explicitly imposed geometric transformations. This ensures both the physical plausibility of object placement and highly accurate 3D pose and position annotations. Our experiments demonstrate that even by integrating a limited number of external 3D objects into real scenes, the augmented data significantly enhances 3D object detection performance and outperforms existing diffusion-based 3D augmentation for object detection. Extensive testing on the nuScenes dataset reveals that imposing high geometric diversity in object placement has a greater impact compared to the appearance diversity of objects. Additionally, we show that generating hard examples, either by maximizing detection loss or imposing high visual occlusion in camera images, does not lead to more efficient 3D data augmentation for camera-based 3D object detection in autonomous driving.

Problem

• Enhancing 3D object detection via Gaussian Splatting data augmentation
• Ensuring physically plausible 3D object placement with accurate annotations
• Comparing geometric vs. appearance diversity impact on detection performance

Innovation

• Uses Gaussian Splatting for 3D object placement
• Ensures physical plausibility with geometric transformations
• Enhances detection via high geometric diversity