CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image

2025-02-18

AI Summary

Single-image 3D scene reconstruction faces longstanding challenges including occlusion, geometric interpenetration, and physically implausible configurations. To address these, we propose an end-to-end framework: (1) joint 2D semantic segmentation and relative depth estimation for initial geometry; (2) GPT-driven spatial relation modeling to explicitly reason about object pose and support relationships; (3) occlusion-aware, part-level 3D generation via MAE-based point cloud conditioning; and (4) joint optimization using Signed Distance Fields (SDFs) fused with physics-aware constraint graphs to resolve floating, interpenetration, and support failures. We introduce the novel “component alignment” paradigm, enforcing semantic, geometric, and physical consistency simultaneously. Our method generates high-fidelity, texture-coherent, and physically plausible complete 3D scenes from real-world images, significantly improving object localization accuracy and cross-object spatial coherence—enabling downstream applications such as robotic simulation.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recovering high-quality 3D scenes from a single RGB image is a challenging task in computer graphics. Current methods often struggle with domain-specific limitations or low-quality object generation. To address these, we propose CAST (Component-Aligned 3D Scene Reconstruction from a Single RGB Image), a novel method for 3D scene reconstruction and recovery. CAST starts by extracting object-level 2D segmentation and relative depth information from the input image, followed by using a GPT-based model to analyze inter-object spatial relationships. This enables the understanding of how objects relate to each other within the scene, ensuring more coherent reconstruction. CAST then employs an occlusion-aware large-scale 3D generation model to independently generate each object's full geometry, using MAE and point cloud conditioning to mitigate the effects of occlusions and partial object information, ensuring accurate alignment with the source image's geometry and texture. To align each object with the scene, the alignment generation model computes the necessary transformations, allowing the generated meshes to be accurately placed and integrated into the scene's point cloud. Finally, CAST incorporates a physics-aware correction step that leverages a fine-grained relation graph to generate a constraint graph. This graph guides the optimization of object poses, ensuring physical consistency and spatial coherence. By utilizing Signed Distance Fields (SDF), the model effectively addresses issues such as occlusions, object penetration, and floating objects, ensuring that the generated scene accurately reflects real-world physical interactions. CAST can be leveraged in robotics, enabling efficient real-to-simulation workflows and providing realistic, scalable simulation environments for robotic systems.

Problem

• Recovers high-quality 3D scenes from single RGB images.
• Addresses occlusion and object alignment challenges in 3D reconstruction.
• Ensures physical consistency and spatial coherence in generated scenes.

Innovation

• GPT-based spatial relationship analysis
• Occlusion-aware 3D generation model
• Physics-aware correction with SDF