Problem
Research questions and friction points this paper is trying to address.
Compositional Dexterous Functional Object Manipulation
robotic manipulation
functional object use
dexterous grasping
semantic-physical integration
CoDex: Learning Compositional Dexterous Functional Manipulation without Demonstrations
🤖 AI Summary
This work addresses the challenge of enabling robots to perform compositional, dexterous functional manipulation—such as operating spray bottles or hot-glue guns—without human demonstrations. The authors propose CoDex, a framework that leverages vision-language models to infer task-level semantic constraints, which guide an analytical optimization process to generate functional grasp candidates. These candidates are then used to efficiently train transferable manipulation policies via reinforcement learning. CoDex is the first method to autonomously discover complex dexterous functional manipulation strategies without human demonstrations, effectively integrating semantic reasoning with physical dexterity. Evaluated on a 7-DoF robotic arm equipped with a 16-DoF multi-fingered hand, the system successfully executes functional tasks on six previously unseen object categories, demonstrating strong generalization across objects and real-world environments.
📝 Abstract
In this work, we study Compositional Dexterous Functional Object Manipulation (CD-FOM): tasks such as aiming and actuating a spray bottle on a plant or a glue gun on wood, which require both actuating an object's internal mechanism and controlling its pose to apply the object's function to the environment. These tasks pose significant challenges for robots due to the demanding integration of semantic understanding of the object's function, actuation mode, and application area with intricate physical dexterity to manage grasp stability, movement trajectory, and actuation. We introduce CoDex, a zero-demonstration framework that autonomously discovers CD-FOM manipulation strategies. CoDex uses vision-language models (VLMs) to infer semantic constraints from the task and scene. These constraints guide analytic constrained optimization to generate a short list of functional grasp candidates that can be efficiently refined with reinforcement learning to generate full grasp-move-actuate policies transferable from simulation to the real world. We evaluate CoDex on a 7-DoF robot arm with a 16-DoF multi-fingered hand across six CD-FOM tasks involving previously unseen objects with internal mechanisms, including spray bottles, hot glue guns, air dusters, flashlights, and pepper grinders, and their application to unseen target objects, showcasing its ability to autonomously discover and execute complex, physically viable dexterous behaviors without human demonstrations. More information at https://robin-lab.cs.utexas.edu/CoDex/.
Innovation
Methods, ideas, or system contributions that make the work stand out.
zero-demonstration learning
compositional dexterous manipulation
vision-language models
constrained optimization
sim-to-real transfer
🔎 Similar Papers
No similar papers found.
