Object Reconstruction under Occlusion with Generative Priors and Contact-induced Constraints

2025-12-04

AI Summary

To address incomplete geometric reconstruction of objects in robotic manipulation caused by occlusion, this paper proposes a contact-guided 3D generative framework. The method jointly optimizes across modalities via a drag-based editing mechanism, integrating generative priors—capturing distributions of common object shapes—with sparse physical contact constraints derived from video observations and interactive manipulation. Compared to purely generative models or contact-only optimization approaches, our framework significantly reduces geometric uncertainty and achieves higher reconstruction accuracy on both synthetic and real-world data, demonstrating robustness even under severe occlusion. The core contribution lies in incorporating physical interaction into the generative reconstruction pipeline through a differentiable contact constraint embedding mechanism, enabling end-to-end joint optimization of geometric priors and task-driven signals.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Object geometry is key information for robot manipulation. Yet, object reconstruction is a challenging task because cameras only capture partial observations of objects, especially when occlusion occurs. In this paper, we leverage two extra sources of information to reduce the ambiguity of vision signals. First, generative models learn priors of the shapes of commonly seen objects, allowing us to make reasonable guesses of the unseen part of geometry. Second, contact information, which can be obtained from videos and physical interactions, provides sparse constraints on the boundary of the geometry. We combine the two sources of information through contact-guided 3D generation. The guidance formulation is inspired by drag-based editing in generative models. Experiments on synthetic and real-world data show that our approach improves the reconstruction compared to pure 3D generation and contact-based optimization.

Problem

• Reconstructs occluded object geometry using generative shape priors
• Incorporates contact information from videos for boundary constraints
• Improves reconstruction accuracy over pure generation and contact-based methods

Innovation

• Uses generative priors for unseen geometry
• Integrates contact constraints from interactions
• Applies contact-guided 3D generation method