EgoPhys: Learning Generalizable Physics Models of Deformable Objects from Egocentric Video

2026-06-15

AI Summary

This work proposes a method for learning generalizable physical models of deformable objects from first-person RGB videos to construct controllable digital twins without test-time optimization. By introducing a codebook-based inverse physics modeling framework, the approach extracts universal physical priors from human interaction videos and predicts dense spring stiffness fields for unseen objects. To the best of our knowledge, this is the first method capable of generating zero-shot generalizable deformable digital twins solely from single-view RGB footage, enabling downstream robotic planning tasks. Experiments demonstrate superior performance over existing baselines in reconstruction accuracy, future state prediction, and cross-object generalization. The method has been successfully deployed on an xArm6 robot, validating its effectiveness in initializing real-world digital twins directly from human manipulation videos.

Abstract

Humans naturally understand object physics through everyday interactions, but faithfully predicting complex deformable dynamics, such as elastic materials and fabrics, remains a major challenge for computer vision and robotics. We present EgoPhys, a framework that constructs deformable physical digital twins from egocentric RGB-only video using generalizable priors. EgoPhys overcomes the limitations of existing methods to enable controllable deformable digital twin generation from egocentric videos by distilling per-object inverse-physics solutions into a compact codebook, enabling prediction of dense spring stiffness fields for unseen objects without per-spring test-time optimization. Trained with generalizable priors from diverse egocentric interactions, EgoPhys outperforms baselines in reconstruction, future prediction, and zero-shot generalization. To support training and evaluation, we curate an egocentric interaction dataset covering diverse deformable objects, scenes, and manipulation styles. We deploy EgoPhys on a real xArm6 robot, demonstrating that a digital twin initialized from a single egocentric human play video can serve as an internal world representation to aid in deformable-object planning, highlighting egocentric RGB observations as a scalable path toward real-to-sim pipelines.

Problem

• deformable objects
• physics modeling
• egocentric video
• digital twins
• generalizable priors

Innovation

• deformable object modeling
• egocentric video
• digital twin
• inverse physics
• zero-shot generalization