$φ$-Scene: Physically Grounded Image-to-3D Scene Reconstruction

2026-06-19

AI Summary

Existing single-image 3D scene reconstruction methods often lack physical plausibility due to issues such as floating, interpenetration, or unstable object contacts, limiting their applicability in simulation and robotics. This work proposes a topology-driven physical assembly framework that, for the first time, integrates physical stability as a core constraint into an open-vocabulary, compositional reconstruction pipeline. By inferring inter-object support relationships and optimizing objects hierarchically according to topological order, the method leverages signed distance fields (SDFs) to eliminate penetrations and incorporates rigid-body physics simulation to ensure stable contacts. Experiments on 3D-Front demonstrate that the approach significantly outperforms existing out-of-domain methods, with both human evaluators and vision-language models consistently preferring its outputs; dedicated physical metrics also confirm reduced penetration rates and dynamic drift.

Abstract

Reconstructing compositional 3D scenes from a single image is a fundamental challenge in 3D world modeling. Recent methods can recover high-fidelity, complete 3D objects and predict plausible scene arrangements, but most still treat scene reconstruction primarily as a visual and geometric prediction problem. Their outputs may therefore contain floating objects, interpenetrations, or unstable-contact artifacts, limiting their physical validity and downstream usability in simulation, robotics, and interactive environments. We present $φ$-Scene, a physically grounded approach to open-vocabulary and compositional image-to-3D scene reconstruction. The key premise is that a reconstructed scene should not be treated merely as a set of objects with predicted poses, but as a stable physical system. Accordingly, $φ$-Scene formulates reconstruction as topology-driven physical assembly: it infers how objects support one another, orders them accordingly, and progressively settles each object against its already stabilized support context. For each object in topological order, SDF-based optimization first resolves penetrations against the pre-settled support context, and rigid-body simulation then settles the object into a stable contact configuration under real-world physical constraints. Experiments on 3D-Front show that $φ$-Scene achieves the strongest overall performance among out-of-domain methods and remains highly competitive with in-domain baselines. Human and VLM evaluations further show strong preference for $φ$-Scene in visual quality, reference alignment, and physical plausibility. Finally, dedicated physical plausibility metrics covering static contact quality and dynamic stability demonstrate that $φ$-Scene substantially reduces penetration artifacts while producing much lower post-simulation drift, indicating more stable and physically grounded 3D scenes.

Problem

• 3D scene reconstruction
• physical plausibility
• image-to-3D
• object stability
• scene composition

Innovation

• physically grounded reconstruction
• topology-driven assembly
• SDF-based optimization
• rigid-body simulation
• compositional 3D scene