Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images

2025-03-17

AI Summary

To address the failure of 3D reconstruction under real-world occlusion, this paper introduces the first end-to-end conditional generative framework for amodal 3D reconstruction. Methodologically, we propose mask-weighted multi-head cross-modal attention and occlusion-aware attention mechanisms to explicitly model geometric and appearance correlations between visible and occluded regions. Crucially, our model is trained solely on synthetic data without requiring real-world occlusion annotations. Unlike conventional two-stage paradigms (occlusion completion followed by reconstruction), our approach directly generates complete, geometrically consistent, and visually plausible 3D models from single occluded images. Experiments demonstrate state-of-the-art performance across multiple benchmarks in occlusion-aware 3D reconstruction. Our work establishes a new paradigm for unsupervised and weakly supervised 3D understanding, significantly improving reconstruction completeness and plausibility under partial observability.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Most image-based 3D object reconstructors assume that objects are fully visible, ignoring occlusions that commonly occur in real-world scenarios. In this paper, we introduce Amodal3R, a conditional 3D generative model designed to reconstruct 3D objects from partial observations. We start from a"foundation"3D generative model and extend it to recover plausible 3D geometry and appearance from occluded objects. We introduce a mask-weighted multi-head cross-attention mechanism followed by an occlusion-aware attention layer that explicitly leverages occlusion priors to guide the reconstruction process. We demonstrate that, by training solely on synthetic data, Amodal3R learns to recover full 3D objects even in the presence of occlusions in real scenes. It substantially outperforms existing methods that independently perform 2D amodal completion followed by 3D reconstruction, thereby establishing a new benchmark for occlusion-aware 3D reconstruction.

Problem

• Reconstruct 3D objects from occluded 2D images
• Develop occlusion-aware 3D reconstruction model
• Improve 3D object recovery in real-world scenarios

Innovation

• Conditional 3D generative model for occluded objects
• Mask-weighted multi-head cross-attention mechanism
• Occlusion-aware attention layer leveraging occlusion priors