IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants

2025-11-24

AI Summary

Existing industrial multimodal datasets lack synchronized first-person (egocentric) and third-person (allocentric) recordings of dual-human collaborative tasks involving both cognitive and physical components, and are deficient in fine-grained error annotations and reasoning-oriented question-answering benchmarks. To address this, we introduce the first dual-perspective collaborative industrial dataset covering assembly, logistics, and maintenance scenarios. It synchronously integrates eye-tracking, speech, gestures, motion capture, and hand pose estimation, employing semi-dense point-cloud reconstruction and cross-modal annotation. The dataset comprises 3,460 egocentric video clips (197 hours) and 1,092 allocentric clips (97 hours). We further release companion evaluation benchmarks for error detection and reasoning-based QA. Empirical results demonstrate that state-of-the-art multimodal models still exhibit significant limitations in collaborative understanding and anomaly identification.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

We introduce IndEgo, a multimodal egocentric and exocentric dataset addressing common industrial tasks, including assembly/disassembly, logistics and organisation, inspection and repair, woodworking, and others. The dataset contains 3,460 egocentric recordings (approximately 197 hours), along with 1,092 exocentric recordings (approximately 97 hours). A key focus of the dataset is collaborative work, where two workers jointly perform cognitively and physically intensive tasks. The egocentric recordings include rich multimodal data and added context via eye gaze, narration, sound, motion, and others. We provide detailed annotations (actions, summaries, mistake annotations, narrations), metadata, processed outputs (eye gaze, hand pose, semi-dense point cloud), and benchmarks on procedural and non-procedural task understanding, Mistake Detection, and reasoning-based Question Answering. Baseline evaluations for Mistake Detection, Question Answering and collaborative task understanding show that the dataset presents a challenge for the state-of-the-art multimodal models. Our dataset is available at: https://huggingface.co/datasets/FraunhoferIPK/IndEgo

Problem

• Creating multimodal dataset for industrial collaborative tasks
• Addressing procedural understanding and mistake detection challenges
• Providing benchmarks for egocentric assistant model evaluation

Innovation

• Multimodal dataset combining egocentric and exocentric recordings
• Rich annotations including gaze, motion, and mistake detection
• Benchmarks for collaborative task understanding and reasoning