Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling

2026-02-08

AI Summary

This work addresses the challenge of physically implausible object configurations—such as interpenetration or instability—commonly produced by traditional scene reconstruction methods under occlusion and noise, despite their high geometric accuracy, which limits their utility in simulation. To overcome this, we propose a holistic multi-object reconstruction approach that, for the first time, integrates non-penetration constraints, physical plausibility, and geometric fitting within a unified joint optimization framework, augmented by a contact-graph-guided fast rejection sampling mechanism. We also introduce Picasso, the first open-source dataset featuring real-world contact annotations and dedicated metrics for evaluating physical plausibility. Experiments on both Picasso and YCB-V demonstrate that our method significantly outperforms existing approaches, generating digital twin scenes that better align with physical laws and human intuition.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Security and Privacy: Data transparency and provenance

Abstract

In the presence of occlusions and measurement noise, geometrically accurate scene reconstructions -- which fit the sensor data -- can still be physically incorrect. For instance, when estimating the poses and shapes of objects in the scene and importing the resulting estimates into a simulator, small errors might translate to implausible configurations including object interpenetration or unstable equilibrium. This makes it difficult to predict the dynamic behavior of the scene using a digital twin, an important step in simulation-based planning and control of contact-rich behaviors. In this paper, we posit that object pose and shape estimation requires reasoning holistically over the scene (instead of reasoning about each object in isolation), accounting for object interactions and physical plausibility. Towards this goal, our first contribution is Picasso, a physics-constrained reconstruction pipeline that builds multi-object scene reconstructions by considering geometry, non-penetration, and physics. Picasso relies on a fast rejection sampling method that reasons over multi-object interactions, leveraging an inferred object contact graph to guide samples. Second, we propose the Picasso dataset, a collection of 10 contact-rich real-world scenes with ground truth annotations, as well as a metric to quantify physical plausibility, which we open-source as part of our benchmark. Finally, we provide an extensive evaluation of Picasso on our newly introduced dataset and on the YCB-V dataset, and show it largely outperforms the state of the art while providing reconstructions that are both physically plausible and more aligned with human intuition.

Problem

• scene reconstruction
• physical plausibility
• object interaction
• occlusion
• digital twin

Innovation

• physics-constrained reconstruction
• holistic scene reasoning
• non-penetration sampling
• contact graph
• physically plausible reconstruction