EgoMAGIC- An Egocentric Video Field Medicine Dataset for Training Perception Algorithms

2026-04-23

AI Summary

This work addresses the critical scarcity of first-person medical procedure video data, which has significantly hindered the development of perception algorithms for complex surgical tasks. To bridge this gap, we introduce EgoMAGIC—the first large-scale, multimodal (stereo video and audio) egocentric dataset for medical procedures, encompassing 50 distinct tasks and 3,355 annotated video clips with fine-grained labels for actions, objects, and procedural errors. Leveraging this dataset, we train 40 YOLO object detection models, producing 1.95 million annotations across 124 medical object categories. Our approach achieves a state-of-the-art average mAP of 0.526 across eight action detection benchmarks. Concurrently, we launch an action detection challenge and publicly release the dataset to catalyze research toward intelligent medical assistants deployable on augmented reality platforms.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Data transparency and provenance

Abstract

This paper introduces EgoMAGIC (Medical Assistance, Guidance, Instruction, and Correction), an egocentric medical activity dataset collected as part of DARPA's Perceptually-enabled Task Guidance (PTG) program. This dataset comprises 3,355 videos of 50 medical tasks, with at least 50 labeled videos per task. The primary objective of the PTG program was to develop virtual assistants integrated into augmented reality headsets to assist users in performing complex tasks. To encourage exploration and research using this dataset, the medical training data has been released along with an action detection challenge focused on eight medical tasks. The majority of the videos were recorded using a head-mounted stereo camera with integrated audio. From this dataset, 40 YOLO models were trained using 1.95 million labels to detect 124 medical objects, providing a robust starting point for developers working on medical AI applications. In addition to introducing the dataset, this paper presents baseline results on action detection for the eight selected medical tasks across three models, with the best-performing method achieving average mAP 0.526. Although this paper primarily addresses action detection as the benchmark, the EgoMAGIC dataset is equally suitable for action recognition, object identification and detection, error detection, and other challenging computer vision tasks. The dataset is accessible via zenodo.org (DOI: 10.5281/zenodo.19239154).

Problem

• egocentric video
• medical activity dataset
• perception algorithms
• action detection
• augmented reality assistance

Innovation

• egocentric vision
• medical activity dataset
• action detection
• object detection
• augmented reality assistance