InstDrive: Instance-Aware 3D Gaussian Splatting for Driving Scenes

2025-08-16

AI Summary

Existing dynamic driving scene reconstruction methods struggle with fine-grained instance modeling and editing: their background representations are overly homogeneous, and they rely on pre-assigned instance IDs or complex mapping mechanisms—making them unsuitable for open-world outdoor scenes with limited viewpoints and intricate geometry. This paper introduces the first instance-aware 3D Gaussian splatting framework tailored for dynamic driving scenes, enabling the first unsupervised 3D instance segmentation in such open-world settings. Leveraging implicit instance encoding and voxel-level consistency constraints, it establishes robust correspondence between continuous features and discrete identities—without requiring instance ID priors. Integrated techniques—including SAM-generated pseudo-labels, contrastive learning, a lightweight static codebook, and voxel-wise regularization loss—ensure discriminative 2D feature learning and coherent 3D instance representation. Experiments demonstrate substantial improvements in instance separation accuracy and reconstruction fidelity, enabling high-fidelity interactive editing.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Unsupervised & Self-Supervised Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Reconstructing dynamic driving scenes from dashcam videos has attracted increasing attention due to its significance in autonomous driving and scene understanding. While recent advances have made impressive progress, most methods still unify all background elements into a single representation, hindering both instance-level understanding and flexible scene editing. Some approaches attempt to lift 2D segmentation into 3D space, but often rely on pre-processed instance IDs or complex pipelines to map continuous features to discrete identities. Moreover, these methods are typically designed for indoor scenes with rich viewpoints, making them less applicable to outdoor driving scenarios. In this paper, we present InstDrive, an instance-aware 3D Gaussian Splatting framework tailored for the interactive reconstruction of dynamic driving scene. We use masks generated by SAM as pseudo ground-truth to guide 2D feature learning via contrastive loss and pseudo-supervised objectives. At the 3D level, we introduce regularization to implicitly encode instance identities and enforce consistency through a voxel-based loss. A lightweight static codebook further bridges continuous features and discrete identities without requiring data pre-processing or complex optimization. Quantitative and qualitative experiments demonstrate the effectiveness of InstDrive, and to the best of our knowledge, it is the first framework to achieve 3D instance segmentation in dynamic, open-world driving scenes.More visualizations are available at our project page.

Problem

• Reconstruct dynamic driving scenes from dashcam videos
• Achieve instance-level understanding in 3D driving scenes
• Enable flexible scene editing without complex preprocessing

Innovation

• Uses SAM masks for 2D contrastive feature learning
• Regularizes 3D instance identity encoding implicitly
• Employs lightweight static codebook for identity bridging