GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video

2026-07-03

AI Summary

This work addresses the limitations of current multimodal large language models (MLLMs) in real-time, closed-loop interactive task guidance from streaming video, where they struggle to effectively detect and correct execution errors despite strong performance in offline video understanding. To bridge this gap, the authors introduce GuideMe, the first benchmark specifically designed for streaming-video-based, multi-domain task guidance, encompassing four domains, 2,458 videos, and 47,775 interactive samples. They further propose a three-component evaluation framework integrating temporal–semantic bipartite matching, action-classification-based intervention timing detection, and LLM-as-a-Judge content assessment. Experimental results reveal that while existing MLLMs can generate plausible instructions, they exhibit significant deficiencies in error identification and real-time corrective feedback, thereby highlighting critical challenges in this emerging research direction.

Abstract

While multimodal Large Language Models (MLLMs) excel at offline video understanding, an interesting question of how far they are from serving as a real-time procedural coach remains unknown. Such a role typically requires an MLLM to continuously monitor the execution, detect mistakes, and provide corrective guidance in a closed-loop interaction. In this paper, we construct GuideMe, the first multi-domain benchmark for streaming video that supports training and evaluation of MLLMs for closed-loop interactive task guidance. It comprises 2,458 videos spanning 223.7 hours across diverse domains (\eg, cooking, object manipulation, daily-life guidance, and fitness), with 47,775 interaction samples covering next-step instructions, completion feedback, error detection, and corrective guidance. To evaluate existing models on GuideMe, we design a three-component assessment framework to measure the capabilities of representative MLLMs, which consists of temporal-semantic bipartite matching for sequence-level alignment, behavioral classification for intervention timing, and LLM-as-a-Judge for content quality. Extensive experiments highlight a critical performance asymmetry: despite excelling at providing instructions, existing MLLMs consistently fail to identify execution errors and respond with corrective feedback. Code and data are released at https://fawnliu.github.io/project/guideme.

Problem

• streaming video
• multimodal large language models
• error detection
• corrective guidance
• interactive task guidance

Innovation

• streaming video
• multimodal LLM
• interactive task guidance
• error detection
• closed-loop intervention