Seg-Wild: Interactive Segmentation based on 3D Gaussian Splatting for Unconstrained Image Collections

2025-07-09

AI Summary

Scene reconstruction and segmentation from unconstrained Internet images—characterized by inconsistent illumination and frequent transient occlusions—remain challenging. Method: This paper proposes an interactive segmentation framework based on 3D Gaussian splatting. It (1) enables joint 2D–3D interactive segmentation via multi-dimensional feature embedding and feature similarity matching; (2) introduces the Spiky 3D Gaussian Cutter, a SAM-guided Gaussian pruning strategy robust to anomalous geometry and illumination variations; and (3) establishes WildSeg, the first 3D segmentation benchmark tailored for野外 (in-the-wild) scenes. Contribution/Results: Experiments demonstrate substantial improvements over state-of-the-art methods in both segmentation accuracy and reconstruction quality, particularly in recovering transient occlusions and modeling cross-illumination consistency. WildSeg provides a rigorous evaluation protocol for 3D segmentation under realistic, uncontrolled conditions.

Technology Category

• Computer Vision: Segmentation
• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Reconstructing and segmenting scenes from unconstrained photo collections obtained from the Internet is a novel but challenging task. Unconstrained photo collections are easier to get than well-captured photo collections. These unconstrained images suffer from inconsistent lighting and transient occlusions, which makes segmentation challenging. Previous segmentation methods cannot address transient occlusions or accurately restore the scene's lighting conditions. Therefore, we propose Seg-Wild, an interactive segmentation method based on 3D Gaussian Splatting for unconstrained image collections, suitable for in-the-wild scenes. We integrate multi-dimensional feature embeddings for each 3D Gaussian and calculate the feature similarity between the feature embeddings and the segmentation target to achieve interactive segmentation in the 3D scene. Additionally, we introduce the Spiky 3D Gaussian Cutter (SGC) to smooth abnormal 3D Gaussians. We project the 3D Gaussians onto a 2D plane and calculate the ratio of 3D Gaussians that need to be cut using the SAM mask. We also designed a benchmark to evaluate segmentation quality in in-the-wild scenes. Experimental results demonstrate that compared to previous methods, Seg-Wild achieves better segmentation results and reconstruction quality. Our code will be available at https://github.com/Sugar0725/Seg-Wild.

Problem

• Segmenting scenes from unconstrained Internet photo collections
• Addressing inconsistent lighting and transient occlusions in images
• Improving 3D reconstruction and segmentation quality in-the-wild

Innovation

• 3D Gaussian Splatting for unconstrained images
• Multi-dimensional feature embeddings integration
• Spiky 3D Gaussian Cutter for smoothing