EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks

2026-04-26

AI Summary

This work addresses the scarcity of large-scale, high-quality real-world data that limits robot learning by introducing the largest open-source first-person manipulation dataset to date, centered on unconstrained everyday human activities in authentic domestic and retail environments. High-fidelity videos were collected using custom head-mounted devices and enriched with precise multimodal annotations, substantially enhancing the dataset’s ecological validity and diversity. Entirely grounded in real-world observations, this resource provides critical support for developing and deploying generalizable robotic models capable of operating effectively in complex, unstructured settings.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Video Understanding & Activity Analysis
• Humans and AI: Interaction Techniques and Devices

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

The advancement of robot learning is currently hindered by the scarcity of large-scale, high-quality datasets. While established data collection methods such as teleoperation and universal manipulation interfaces dominate current datasets, they suffer from inherent limitations in scalability and real-world deployability. Human egocentric video collection, by contrast, has emerged as a promising approach to enable scalable, natural and in-the-wild data collection. As such, we present EgoLive, a large-scale, high-quality egocentric dataset designed explicitly for robot manipulation learning. EgoLive establishes three distinctive technical advantages over existing egocentric datasets: first, it represents the largest open-source annotated egocentric dataset focused on real-world task-oriented human routines to date; second, it delivers leading data quality via a customized head-mounted capture device and comprehensive high-precision multi-modal annotations; third, all data is collected exclusively in unconstrained real-world scenarios and encompasses vertical field human working data, including home service, retail, and other practical work scenarios, providing superior diversity and ecological validity. With the introduction of EgoLive, we aim to provide the research community with a scalable, high-quality dataset that accelerates breakthroughs in generalizable robotic models and facilitates the real-world deployment of robot systems.

Problem

• robot learning
• large-scale dataset
• egocentric video
• real-world deployment
• data scarcity

Innovation

• egocentric dataset
• robot manipulation learning
• real-world data collection
• multi-modal annotation
• scalable robotics