3D StreetUnveiler with Semantic-Aware 2DGS

2024-05-28

arXiv.org

AI Summary

Reconstructing occlusion-free 3D “empty-street” scenes from crowded in-vehicle videos remains challenging due to dynamic obstructions and temporal inconsistencies. Method: This paper proposes a semantic-aware spatiotemporal co-reconstruction framework. Its core components are: (1) a novel semantic hard-label-driven 2D Gaussian Splatting (2DGS) representation for precise static obstacle modeling; (2) a three-region decomposition strategy based on rendered alpha maps, explicitly separating empty street regions, dynamic objects, and background; and (3) a time-reversed pseudo-label generation and re-optimization framework that leverages subsequent frames to guide prior-frame refinement, enhancing temporal consistency and geometric completeness over long trajectories. Results: Evaluated on real-world street-scene datasets, our method achieves high-fidelity reconstruction of 3D Gaussian fields for empty streets and enables robust mesh extraction. It significantly outperforms baseline approaches in reconstruction completeness and structural fidelity.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Unveiling an empty street from crowded observations captured by in-car cameras is crucial for autonomous driving. However, removing all temporarily static objects, such as stopped vehicles and standing pedestrians, presents a significant challenge. Unlike object-centric 3D inpainting, which relies on thorough observation in a small scene, street scene cases involve long trajectories that differ from previous 3D inpainting tasks. The camera-centric moving environment of captured videos further complicates the task due to the limited degree and time duration of object observation. To address these obstacles, we introduce StreetUnveiler to reconstruct an empty street. StreetUnveiler learns a 3D representation of the empty street from crowded observations. Our representation is based on the hard-label semantic 2D Gaussian Splatting (2DGS) for its scalability and ability to identify Gaussians to be removed. We inpaint rendered image after removing unwanted Gaussians to provide pseudo-labels and subsequently re-optimize the 2DGS. Given its temporal continuous movement, we divide the empty street scene into observed, partial-observed, and unobserved regions, which we propose to locate through a rendered alpha map. This decomposition helps us to minimize the regions that need to be inpainted. To enhance the temporal consistency of the inpainting, we introduce a novel time-reversal framework to inpaint frames in reverse order and use later frames as references for earlier frames to fully utilize the long-trajectory observations. Our experiments conducted on the street scene dataset successfully reconstructed a 3D representation of the empty street. The mesh representation of the empty street can be extracted for further applications. The project page and more visualizations can be found at: https://streetunveiler.github.io

Problem

• Reconstructing empty streets from crowded in-car camera observations.
• Removing temporarily static objects like vehicles and pedestrians.
• Enhancing temporal consistency in 3D street scene inpainting.

Innovation

• Uses 2D Gaussian Splatting for scalable 3D representation
• Divides scene into observed, partial-observed, unobserved regions
• Implements time-reversal framework for temporal consistency