🤖 AI Summary
This work addresses the challenge of dexterous manipulation of unseen articulated objects. Methodologically, it introduces a novel paradigm that actively constructs an explicit world model via single-step physical interaction: for the first time, it reconstructs a digital twin of an articulated object from a single real-world point cloud observation. The approach integrates affordance estimation, point cloud registration, and eigengrasp-based dimensionality reduction, coupled with a sampling-based model predictive control (MPC) policy to enable goal-directed manipulation using both a two-finger gripper and a 16-DoF dexterous hand—without human demonstrations or reinforcement learning. Its core innovation is an explicit-model-driven, self-supervised interactive learning framework enabling cross-object and cross-tool generalization. Evaluated in simulation and on real robotic platforms, the method achieves 92.3% success rate with the two-finger gripper and 86.7% with the dexterous hand on unseen-object tasks including drawer opening, knob turning, and lid flipping.
📝 Abstract
Articulated object manipulation is ubiquitous in daily life. In this paper, we present DexSim2Real$^{2}$, a novel robot learning framework for goal-conditioned articulated object manipulation using both two-finger grippers and multi-finger dexterous hands. The key of our framework is constructing an explicit world model of unseen articulated objects through active one-step interactions. This explicit world model enables sampling-based model predictive control to plan trajectories achieving different manipulation goals without needing human demonstrations or reinforcement learning. It first predicts an interaction motion using an affordance estimation network trained on self-supervised interaction data or videos of human manipulation from the internet. After executing this interaction on the real robot, the framework constructs a digital twin of the articulated object in simulation based on the two point clouds before and after the interaction. For dexterous multi-finger manipulation, we propose to utilize eigengrasp to reduce the high-dimensional action space, enabling more efficient trajectory searching. Extensive experiments validate the framework's effectiveness for precise articulated object manipulation in both simulation and the real world using a two-finger gripper and a 16-DoF dexterous hand. The robust generalizability of the explicit world model also enables advanced manipulation strategies, such as manipulating with different tools.