The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection

2026-06-19

AI Summary

This work addresses the limitations of existing approaches to procedural error detection and temporal action segmentation, which typically rely on multi-stage pipelines and task-specific training data, thereby constraining generalization. The paper proposes ZeProM, a unified framework that, for the first time, enables zero-shot procedural error detection and temporal action segmentation without any task-specific training. Built upon pretrained video-language models, ZeProM leverages prompt engineering and tailored inference mechanisms to perform both tasks in an end-to-end manner. Evaluated on the EgoPER and CaptainCook4D benchmarks, ZeProM achieves performance comparable to or even surpassing fully supervised methods, with notable gains of 4.4 points in mean EDA and 2.0 points in F1@.5 on EgoPER, strongly demonstrating its effectiveness and generalization capability.

Abstract

Procedural mistake detection is important for quality control and user assistance across many disciplines. Recent work in this field has achieved significant gains by using the reasoning capabilities of Video-Language Models (VLMs) as components within multi-stage pipelines, which consist of separate modules for supervised temporal action segmentation, error detection, and explainability. Consequently, they remain dependent on tailored training datasets and require task-specific training, limiting their wider applicability. To remedy this, we introduce zero-shot procedural mistake detection and propose a unified Zero-shot Procedural Mistake detection (ZeProM) framework that jointly solves procedural mistake detection and temporal action segmentation with a single pre-trained VLM. By evaluating our framework on two canonical mistake detection benchmarks, EgoPER and CaptainCook4D, we find that ZeProM can perform these tasks successfully, while approaching, or even outperforming, the performance of fully supervised methods. For instance, we achieve a 4.4 point improvement in EDA and a 2.0 point improvement in F1@.5 on average over all five EgoPER tasks compared to the strongest supervised methods. Overall, our results show the potential of unified methods for procedural mistake detection, and we hope this will steer the field away from highly complex pipelines and toward more generally applicable solutions.

Problem

• procedural mistake detection
• zero-shot learning
• Video-Language Models
• temporal action segmentation
• generalizability

Innovation

• Zero-shot learning
• Video-Language Models
• Procedural mistake detection
• Temporal action segmentation
• Unified framework