Keypose Exploration: Efficient Automatic Trajectory Labelling and Cross-Embodiment Policy Transfer

2026-06-27

AI Summary

Existing key pose extraction methods rely on task-specific heuristics or manual annotations, limiting their generalization. This work proposes an automatic trajectory annotation framework that leverages a vision-language model for semantic event detection, integrates classical trajectory analysis for precise temporal alignment, and employs reachability graphs to filter candidate key poses. Using the resulting annotations, a key-pose-conditioned diffusion policy is trained, enabling task automation from a single demonstration. The approach supports zero-shot transfer across robot embodiments, matching the performance of standard diffusion policies on the Robomimic benchmark. Moreover, in multimodal insertion tasks, when feasible key poses exist, reachability-based filtering substantially improves transfer success rates.

Abstract

Keypose-based manipulation decomposes tasks into critical waypoints to simplify policy learning for long-horizon tasks, but existing approaches rely on task-specific heuristics or manual annotation to extract keyposes from demonstrations. We present an automatic trajectory labelling pipeline for grasp-related tasks. This pipeline combines vision-language models (VLMs) for semantic event detection with classical trajectory analysis for precise temporal alignment, requiring VLM inference only on one single demo among repeating ones per task. Using the labelled data, we train a keypose-guided Diffusion Policy (DP) that exploits keypose conditioning to intervene demonstration distributions. We explore the possibility to apply this property for cross-embodiment transfer: candidate keyposes are sampled and filtered via a reachability map, steering the policy toward kinematically feasible keyposes for the target robot. As a preliminary feasibility study, experiments on two robomimic tasks show that the labelled data produces policies matching a standard DP baseline, and that reachability-filtered keypose conditioning may benefit zero-shot transfer on the multimodal insertion task when feasible candidates are available.

Problem

• keypose
• automatic trajectory labelling
• cross-embodiment transfer
• manipulation
• demonstration

Innovation

• keypose extraction
• automatic trajectory labelling
• vision-language models
• cross-embodiment transfer
• diffusion policy