Transferring Kinesthetic Demonstrations across Diverse Objects for Manipulation Planning

2025-03-13

AI Summary

This work addresses the challenge of generalizing imitation-learning-based manipulation policies across objects with significant geometric discrepancies—e.g., pouring liquid into unseen containers with novel shapes and poses. We propose the Motion Transfer Frame (MTF) framework, which automatically identifies geometry-agnostic key points and dynamic reference frames grounded in both object geometry and task semantics. MTF integrates geometric-aware keypoint localization, reference-frame binding, and kinematic constraint modeling, and supports closed-loop validation from simulation to real robots. Its core contribution is geometry-invariant trajectory transfer that simultaneously enforces critical pose constraints (e.g., cup upright orientation), collision-free motion, and task success. Experiments demonstrate >92% pouring success across diverse unseen container configurations, substantially improving cross-morphology generalization of manipulation skills.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Motion & Tracking

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Given a demonstration of a complex manipulation task such as pouring liquid from one container to another, we seek to generate a motion plan for a new task instance involving objects with different geometries. This is non-trivial since we need to simultaneously ensure that the implicit motion constraints are satisfied (glass held upright while moving), the motion is collision-free, and that the task is successful (e.g. liquid is poured into the target container). We solve this problem by identifying positions of critical locations and associating a reference frame (called motion transfer frames) on the manipulated object and the target, selected based on their geometries and the task at hand. By tracking and transferring the path of the motion transfer frames, we generate motion plans for arbitrary task instances with objects of different geometries and poses. We show results from simulation as well as robot experiments on physical objects to evaluate the effectiveness of our solution.

Problem

• Transferring kinesthetic demonstrations across diverse object geometries.
• Ensuring motion constraints, collision-free paths, and task success.
• Generating motion plans using motion transfer frames for arbitrary tasks.

Innovation

• Motion transfer frames for diverse object manipulation
• Collision-free motion planning across different geometries
• Simulation and robot experiments validate effectiveness