InstaScene: Towards Complete 3D Instance Decomposition and Reconstruction from Cluttered Scenes

2025-07-11

AI Summary

Humans naturally complete occluded objects, yet robots struggle with robust 3D instance decomposition and completion in cluttered scenes: existing methods treat the scene holistically, limiting their ability to identify and reconstruct complete objects from partial observations. This paper proposes an occlusion-robust framework for 3D instance decomposition and completion. Its core innovations are: (1) spatial contrastive learning—enforcing cross-view semantic consistency via multi-view rasterization-based correspondence tracking; and (2) in-situ generation—jointly leveraging observed imagery and geometric priors to guide a generative model in producing geometrically complete and semantically coherent object completions directly in the original 3D space. Experiments on both synthetic and real-world complex scenes demonstrate that our method significantly outperforms state-of-the-art approaches in instance segmentation accuracy, geometric reconstruction fidelity, and visual completeness—thereby enhancing robots’ holistic 3D perception capability in unstructured environments.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Humans can naturally identify and mentally complete occluded objects in cluttered environments. However, imparting similar cognitive ability to robotics remains challenging even with advanced reconstruction techniques, which models scenes as undifferentiated wholes and fails to recognize complete object from partial observations. In this paper, we propose InstaScene, a new paradigm towards holistic 3D perception of complex scenes with a primary goal: decomposing arbitrary instances while ensuring complete reconstruction. To achieve precise decomposition, we develop a novel spatial contrastive learning by tracing rasterization of each instance across views, significantly enhancing semantic supervision in cluttered scenes. To overcome incompleteness from limited observations, we introduce in-situ generation that harnesses valuable observations and geometric cues, effectively guiding 3D generative models to reconstruct complete instances that seamlessly align with the real world. Experiments on scene decomposition and object completion across complex real-world and synthetic scenes demonstrate that our method achieves superior decomposition accuracy while producing geometrically faithful and visually intact objects.

Problem

• Decomposing arbitrary 3D instances in cluttered scenes
• Reconstructing complete objects from partial observations
• Enhancing semantic supervision via spatial contrastive learning

Innovation

• Spatial contrastive learning for precise decomposition
• In-situ generation for complete reconstruction
• Tracing rasterization across views for semantic supervision