Learning to Recognize Correctly Completed Procedure Steps in Egocentric Assembly Videos through Spatio-Temporal Modeling

2025-10-14

AI Summary

Existing program step recognition (PSR) models rely solely on single-frame spatial features, neglecting temporal dynamics and thus exhibiting insufficient robustness and accuracy under partial occlusion. To address this, we propose STORM-PSR, a dual-stream spatiotemporal modeling framework: its spatial branch leverages weakly supervised pretraining to enhance local feature representation, while its temporal branch employs a Transformer architecture to explicitly model inter-frame state evolution and capture fine-grained spatiotemporal changes. The two streams jointly enable precise completion-state discrimination and sequential reasoning of assembly steps. Evaluated on MECCANO and IndustReal, STORM-PSR reduces prediction latency by 11.2% and 26.1%, respectively, and significantly improves robustness under occlusion. This work pioneers the deep integration of strong temporal modeling with weakly supervised spatial learning, establishing a novel paradigm for first-person assembly understanding.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Procedure step recognition (PSR) aims to identify all correctly completed steps and their sequential order in videos of procedural tasks. The existing state-of-the-art models rely solely on detecting assembly object states in individual video frames. By neglecting temporal features, model robustness and accuracy are limited, especially when objects are partially occluded. To overcome these limitations, we propose Spatio-Temporal Occlusion-Resilient Modeling for Procedure Step Recognition (STORM-PSR), a dual-stream framework for PSR that leverages both spatial and temporal features. The assembly state detection stream operates effectively with unobstructed views of the object, while the spatio-temporal stream captures both spatial and temporal features to recognize step completions even under partial occlusion. This stream includes a spatial encoder, pre-trained using a novel weakly supervised approach to capture meaningful spatial representations, and a transformer-based temporal encoder that learns how these spatial features relate over time. STORM-PSR is evaluated on the MECCANO and IndustReal datasets, reducing the average delay between actual and predicted assembly step completions by 11.2% and 26.1%, respectively, compared to prior methods. We demonstrate that this reduction in delay is driven by the spatio-temporal stream, which does not rely on unobstructed views of the object to infer completed steps. The code for STORM-PSR, along with the newly annotated MECCANO labels, is made publicly available at https://timschoonbeek.github.io/stormpsr .

Problem

• Recognizing correctly completed steps in egocentric assembly videos
• Overcoming limitations of object occlusion in step recognition
• Improving temporal modeling for accurate procedure step identification

Innovation

• Dual-stream framework combines spatial and temporal features
• Spatial encoder uses weakly supervised pre-training approach
• Transformer-based temporal encoder models feature relationships over time