SAM3D-Phys: Towards Multi-Object Interactive Simulation in Real World

2026-05-28

AI Summary

This work addresses the challenge of incomplete object geometry in real-world scene reconstruction, commonly caused by occlusions and insufficient observations. To infer complete and physically simulatable object geometries, the authors propose a method that integrates multi-view 3D reconstruction with SAM3D, a generative 3D prior. By leveraging physics-constrained spatial optimization and mask-guided appearance distillation, the approach simultaneously recovers the poses, textures, and complete shapes of multiple objects while preserving consistency with the original scene. This study represents the first effort to combine generative 3D priors with the requirements of physical simulation, significantly enhancing the completeness, realism, and physical plausibility of multi-object interaction simulations in real-world environments.

Abstract

This work addresses the problem of recovering complete, simulatable object geometry from reconstructed real-world scenes, enabling physics-based interaction with objects embedded in the scene. While modern multi-view reconstruction methods can produce visually accurate environments, objects are often incomplete due to occlusions and limited observations, making them unsuitable for physics simulation. To address this limitation, we propose SAM3D-Phys, a framework that integrates scene reconstruction with generative 3D priors of SAM3D to recover physically simulatable objects. Our approach first reconstructs the scene from multi-view images to obtain scene geometry and partial observations of objects. We then leverage SAM3D to infer complete object geometry from these partial observations. To ensure that the recovered objects remain consistent with the reconstructed scene, we restore scene-consistent object states through two complementary strategies: a physics-constrained spatial optimization algorithm that iteratively aligns the recovered object to its original location, and a mask-guided appearance distillation module that refines texture fidelity based on the observed images. By recovering complete object geometry and restoring its pose and appearance within the scene, SAM3D-Phys produces clean object representations suitable for physics-based simulation, enabling simultaneous and physically consistent interactive simulation of multiple objects within a reconstructed scene. Project page: https://chnxindong.github.io/sam3d-phys/

Problem

• physics simulation
• 3D reconstruction
• object completion
• scene consistency
• interactive simulation

Innovation

• physics-based simulation
• 3D object completion
• scene-consistent reconstruction
• generative 3D priors
• interactive simulation