Scalable Real2Sim: Physics-Aware Asset Generation Via Robotic Pick-and-Place Setups

2025-03-01

AI Summary

Manual measurement of geometric and physical properties of real-world objects hinders scalable simulation asset creation. Method: This paper proposes an end-to-end, fully automatic Real2Sim pipeline that generates simulation-ready assets solely from unlabeled real-world robotic grasping interactions, using only robot joint torque sensors and external multi-view cameras. Contribution/Results: Its core innovation is an object-centric transparent alpha training strategy—first enabling direct foreground object decoupling and high-fidelity estimation of collision geometry, mass, and inertia tensor directly from photometric reconstructions (e.g., NeRF or Gaussian Splatting), without environmental modification or human intervention. The method integrates force sensing, multi-view visual reconstruction, physical identification, and foreground-background segmentation. Evaluated on diverse physical objects, the generated assets enable plug-and-play, high-fidelity dynamic simulation, significantly improving scalability and automation in robotic simulation dataset construction.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Web Mining and Content Analysis: Web data generation and simulation
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Simulating object dynamics from real-world perception shows great promise for digital twins and robotic manipulation but often demands labor-intensive measurements and expertise. We present a fully automated Real2Sim pipeline that generates simulation-ready assets for real-world objects through robotic interaction. Using only a robot's joint torque sensors and an external camera, the pipeline identifies visual geometry, collision geometry, and physical properties such as inertial parameters. Our approach introduces a general method for extracting high-quality, object-centric meshes from photometric reconstruction techniques (e.g., NeRF, Gaussian Splatting) by employing alpha-transparent training while explicitly distinguishing foreground occlusions from background subtraction. We validate the full pipeline through extensive experiments, demonstrating its effectiveness across diverse objects. By eliminating the need for manual intervention or environment modifications, our pipeline can be integrated directly into existing pick-and-place setups, enabling scalable and efficient dataset creation.

Problem

• Automates generation of simulation-ready assets for real-world objects
• Extracts object geometry and physical properties using robotic interaction
• Enables scalable dataset creation without manual intervention

Innovation

• Automated Real2Sim pipeline for simulation-ready assets
• Extracts object properties using joint torque sensors and camera
• Alpha-transparent training for high-quality mesh extraction