Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning

2026-07-15

AI Summary

Existing approaches struggle to simultaneously achieve low-cost, large-scale, structured, and reusable data collection and modeling for embodied manipulation. This work proposes the first end-to-end open-source toolchain that integrates crowdsourced first-person smartphone video acquisition, atomic action annotation, MANO hand pose estimation, camera trajectory reconstruction, and temporal semantic segmentation. The framework further supports cross-embodiment retargeting and training of Vision-Language-Action (VLA), Whole-Body Action Mapping (WAM), and World Models. The project releases approximately 2,000 hours of egocentric manipulation videos contributed by over 500 participants using more than 400 distinct devices, substantially lowering the barrier to data acquisition and reuse, and providing a high-quality foundational resource for research in embodied intelligence and human-to-robot skill transfer.

Abstract

Egocentric videos of human manipulation provide scalable supervision for embodied intelligence, yet existing resources rarely combine low-cost continuous capture, manipulation-level structured annotations, and reusable tools for robot learning. We present Open-AoE, an open, community-oriented egocentric manipulation dataset and toolchain spanning the full pipeline from smartphone capture to model training. Its first release contains approximately 2,000 hours of manipulation video collected in natural environments by 500+ contributors using 400+ smartphones. The dataset provides text annotations, MANO-based hand poses, camera trajectories, and temporally localized atomic actions. Open-AoE further includes a data processing pipeline that transforms raw recordings into structured samples through temporal action segmentation, semantic annotation, hand reconstruction, and camera trajectory reconstruction. Meanwhile, we provide a separate downstream toolchain supports visualization, cross-embodiment retargeting, model-specific data conversion, and training recipes for VLA policies, WAMs, and World Models. By integrating scalable capture, structured processing, and downstream adaptation, Open-AoE reduces the barriers to both data contribution and reuse, providing practical open infrastructure for embodied model training, human-to-robot transfer, and world modeling.

Problem

• egocentric manipulation
• embodied learning
• structured annotation
• robot learning
• open dataset

Innovation

• egocentric manipulation
• structured annotation
• hand pose estimation
• embodied learning
• open dataset