Shadow: Leveraging Segmentation Masks for Cross-Embodiment Policy Transfer

2025-03-02

AI Summary

This work addresses cross-morphology robotic policy transfer: learning a policy solely from expert demonstrations on a single source manipulator and achieving zero-shot adaptation to structurally dissimilar target manipulators. To overcome the generalization bottleneck caused by input distribution shift between source and target robots, we propose Shadow Data Editing—a method that aligns training and test input distributions via joint image-level segmentation masks of both source and target robots. Integrated with end-to-end visual imitation learning, the approach is validated concurrently in simulation and on real hardware. Experiments demonstrate that, without any data from the target robot, our method significantly improves policy generalization and data efficiency—achieving an average task success rate over twice that of the best baseline. This establishes a scalable, deployment-friendly paradigm for policy transfer across morphologically diverse robots.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Data collection in robotics is spread across diverse hardware, and this variation will increase as new hardware is developed. Effective use of this growing body of data requires methods capable of learning from diverse robot embodiments. We consider the setting of training a policy using expert trajectories from a single robot arm (the source), and evaluating on a different robot arm for which no data was collected (the target). We present a data editing scheme termed Shadow, in which the robot during training and evaluation is replaced with a composite segmentation mask of the source and target robots. In this way, the input data distribution at train and test time match closely, enabling robust policy transfer to the new unseen robot while being far more data efficient than approaches that require co-training on large amounts of data from diverse embodiments. We demonstrate that an approach as simple as Shadow is effective both in simulation on varying tasks and robots, and on real robot hardware, where Shadow demonstrates an average of over 2x improvement in success rate compared to the strongest baseline.

Problem

• Enables cross-embodiment policy transfer between robots.
• Uses segmentation masks to align training and evaluation data.
• Improves success rate on unseen robots without extensive data.

Innovation

• Uses segmentation masks for policy transfer
• Matches input data distribution across robots
• Improves success rate by over 2x