Real-to-Sim for Highly Cluttered Environments via Physics-Consistent Inter-Object Reasoning
Application Category
📝 Abstract
Reconstructing physically valid 3D scenes from single-view observations is a prerequisite for bridging the gap between visual perception and robotic control. However, in scenarios requiring precise contact reasoning, such as robotic manipulation in highly cluttered environments, geometric fidelity alone is insufficient. Standard perception pipelines often neglect physical constraints, resulting in invalid states, e.g., floating objects or severe inter-penetration, rendering downstream simulation unreliable. To address these limitations, we propose a novel physics-constrained Real-to-Sim pipeline that reconstructs physically consistent 3D scenes from single-view RGB-D data. Central to our approach is a differentiable optimization pipeline that explicitly models spatial dependencies via a contact graph, jointly refining object poses and physical properties through differentiable rigid-body simulation. Extensive evaluations in both simulation and real-world settings demonstrate that our reconstructed scenes achieve high physical fidelity and faithfully replicate real-world contact dynamics, enabling stable and reliable contact-rich manipulation.
Problem
Research questions and friction points this paper is trying to address.
Real-to-Sim
cluttered environments
physical consistency
contact reasoning
3D scene reconstruction
Innovation
Methods, ideas, or system contributions that make the work stand out.
physics-consistent reconstruction
differentiable simulation
contact graph
real-to-sim
cluttered scene understanding
🔎 Similar Papers
No similar papers found.
