IMPACT: A Dataset for Multi-Granularity Human Procedural Action Understanding in Industrial Assembly

2026-04-11

AI Summary

This work addresses the absence of benchmark datasets supporting fine-grained, procedural action understanding under realistic deployment conditions in industrial assembly scenarios. The authors introduce a synchronized five-view RGB-D dataset centered on authentic assembly and disassembly tasks using commercial angle grinders. For the first time within a single industrial workflow, the dataset jointly provides synchronized egocentric-exocentric recordings, decoupled hand annotations, compliance state tracking, and explicit supervision of anomaly-recovery behaviors. It comprises 112 trials (39.5 hours) from 13 participants, featuring hierarchical action–state–compliance annotations, partial-order workflow modeling, NASA-TLX cognitive load assessments, and cross-view empty-interval alignment. Baseline experiments reveal fundamental limitations of current methods in handling incomplete observations, flexible execution paths, and corrective actions, establishing a new benchmark and set of challenges for future research.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Data transparency and provenance
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We introduce IMPACT, a synchronized five-view RGB-D dataset for deployment-oriented industrial procedural understanding, built around real assembly and disassembly of a commercial angle grinder with professional-grade tools. To our knowledge, IMPACT is the first real industrial assembly benchmark that jointly provides synchronized ego-exo RGB-D capture, decoupled bimanual annotation, compliance-aware state tracking, and explicit anomaly--recovery supervision within a single real industrial workflow. It comprises 112 trials from 13 participants totaling 39.5 hours, with multi-route execution governed by a partial-order prerequisite graph, a six-category anomaly taxonomy, and operator cognitive load measured via NASA-TLX. The annotation hierarchy links hand-specific atomic actions to coarse procedural steps, component assembly states, and per-hand compliance phases, with synchronized null spans across views to decouple perceptual limitations from algorithmic failure. Systematic baselines reveal fundamental limitations that remain invisible to single-task benchmarks, particularly under realistic deployment conditions that involve incomplete observations, flexible execution paths, and corrective behavior. The full dataset, annotations, and evaluation code are available at https://github.com/Kratos-Wen/IMPACT.

Problem

• procedural action understanding
• industrial assembly
• multi-granularity
• anomaly recovery
• RGB-D dataset

Innovation

• multi-granularity procedural understanding
• synchronized ego-exo RGB-D
• decoupled bimanual annotation
• compliance-aware state tracking
• anomaly-recovery supervision