SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

2026-06-26

AI Summary

Training and evaluating robotic policies in the real world is costly and difficult to scale. This work proposes the first method to generate editable, function-preserving digital twin simulation scenes from a single video in a zero-shot manner. By integrating video-to-3D reconstruction, modular modeling, and affordance-based scene variation, the approach enables flexible editing of objects, environments, and tasks, and automatically produces diverse “digital cousins” for policy training. Evaluated across seven manipulation tasks and five policy architectures, simulation performance shows a strong correlation with real-world outcomes (average Pearson correlation coefficient of 0.911). Policies trained on the generated variants achieve zero-shot real-world success rate improvements of 17% (object), 21% (scene), and 40% (task), substantially enhancing generalization and transfer capabilities.

Abstract

Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shot real-to-sim scene construction from a video. SimFoundry generates sim-ready digital twins and supports object, scene, and task editing, enabling the automated generation of diverse digital cousins: affordance-preserving variations of reconstructed real-world scenes. Policies trained on SimFoundry data transfer zero-shot to challenging real tasks involving multi-step manipulation, articulated object interaction, and bimanual interaction, and its digital cousins (variations of the original scene, objects, and tasks) facilitate generalization to new real-world conditions. Across 7 manipulation tasks and 5 policy architectures, SimFoundry simulation evaluations strongly predict real-world performance, with mean Pearson correlation 0.911 and mean maximum ranking violation 0.018. When evaluating sim-trained policies zero-shot in the real world, policies trained with object, scene, and task cousins in simulation show average task success rate improvements of 17%, 21%, and 40%, respectively. Additional details at https://research.nvidia.com/labs/gear/simfoundry/ .

Problem

• robot policy
• real-to-sim
• scene generation
• zero-shot transfer
• digital twin

Innovation

• zero-shot real-to-sim
• digital twins
• affordance-preserving variations
• modular scene generation
• policy transfer