ROODI: Reconstructing Occluded Objects with Denoising Inpainters

2025-03-13

AI Summary

Extracting occluded objects in complex 3D scenes remains challenging—particularly under 3D Gaussian Splatting representations—due to difficulties in isolating individual object primitives and compensating for observation gaps caused by occlusion. Method: We propose an object-centric framework integrating pruning and generative inpainting. Specifically, we introduce local K-nearest-neighbor structure-guided Gaussian primitive pruning, diffusion-model-driven scene-aware inpainting, and joint geometric-visibility modeling for occlusion reasoning. Contribution/Results: Our key insight is the synergistic performance gain achieved by co-optimizing pruning and inpainting. Experiments on both synthetic and real-world datasets demonstrate significant improvements over state-of-the-art methods: object segmentation accuracy and reconstruction completeness under occlusion increase markedly. The framework enables end-to-end, object-level 3D extraction without requiring per-object supervision or explicit segmentation masks.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Local Search
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While the quality of novel-view images has improved dramatically with 3D Gaussian Splatting, extracting specific objects from scenes remains challenging. Isolating individual 3D Gaussian primitives for each object and handling occlusions in scenes remain far from being solved. We propose a novel object extraction method based on two key principles: (1) being object-centric by pruning irrelevant primitives; and (2) leveraging generative inpainting to compensate for missing observations caused by occlusions. For pruning, we analyze the local structure of primitives using K-nearest neighbors, and retain only relevant ones. For inpainting, we employ an off-the-shelf diffusion-based inpainter combined with occlusion reasoning, utilizing the 3D representation of the entire scene. Our findings highlight the crucial synergy between pruning and inpainting, both of which significantly enhance extraction performance. We evaluate our method on a standard real-world dataset and introduce a synthetic dataset for quantitative analysis. Our approach outperforms the state-of-the-art, demonstrating its effectiveness in object extraction from complex scenes.

Problem

• Extracting specific objects from 3D scenes remains challenging.
• Handling occlusions in 3D scenes is not fully solved.
• Improving object extraction performance using pruning and inpainting techniques.

Innovation

• Object-centric pruning using K-nearest neighbors
• Generative inpainting for occlusion compensation
• Synergy between pruning and inpainting enhances extraction