EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent

2026-07-15

AI Summary

Current evaluations of first-person video understanding lack systematic assessment of procedural comprehension and reasoning about critical steps, hindering the development of wearable AI assistants. This work proposes EgoProceVQA, a novel task that evaluates multimodal large language models’ ability to understand procedural sequences through six question types focused on key steps, accompanied by a benchmark dataset comprising 3,600 questions. The study further introduces EgoProceAgent, an innovative self-skill-exploration agent framework that operates without ground-truth labels by leveraging a general-purpose tool library and a standardized sub-skill repository to autonomously compose strategies in an unsupervised manner. Experiments demonstrate that this approach achieves state-of-the-art performance among open-source models across multiple everyday procedural tasks, revealing substantial room for improvement in current models’ procedural understanding capabilities.

Abstract

Most daily activities are inherently procedural. However, existing evaluations for egocentric video understanding seldom address procedural understanding and largely overlook complex key-step-level reasoning under the widely used video question answering (VQA) paradigm for MLLMs. Such capabilities are crucial for building procedural AI assistants deployable on wearable devices. To bridge this gap, we introduce the Egocentric Procedural Understanding VQA task (EgoProceVQA), which systematically evaluates egocentric procedural reasoning abilities of current MLLMs and agents through six types of key-step-centric questions. Furthermore, we develop EgoProceGen, a data generation platform that efficiently constructs QA data tailored to different question types. Based on this platform, we build a benchmark with 3,600 questions, four common procedural scenarios, and 31 everyday procedural tasks. Evaluations on EgoProceVQA show that existing MLLMs and agents still have substantial room for improvement in procedural understanding. Therefore, we further propose EgoProceAgent, a self-skill-exploration agentic framework. We design a generic tool library for procedural understanding and a standardized sub-skill library shared across tools and models, enabling self-exploration without ground-truth supervision. By exploring how to compose and select sub-skills, the agent discovers effective skill strategies for diverse problems, and attains state-of-the-art performance among open-source models on multiple tasks. Together, our benchmark, generation platform, and agentic framework establish a unified foundation for EgoProceVQA. Project page: https://z1oong.github.io/EgoProceVQA/.

Problem

• egocentric video understanding
• procedural understanding
• video question answering
• key-step reasoning
• MLLMs

Innovation

• EgoProceVQA
• procedural understanding
• self-skill-exploration
• egocentric video
• agentic framework