RegulAR: Graph-Grounded Error Recognition and Assistance for Procedural Tasks in AR

2026-08-27

AI Summary

针对AR中程序任务错误识别与恢复问题，RegulAR通过结合层次依赖图和多模态大语言模型来解析执行过程中的第一人称视角观察，以提供有效的干预指导。

Abstract

Errors are inevitable in procedural tasks, yet most AR guidance systems focus on step-by-step instruction delivery rather than helping users recognize and recover from mistakes. We present RegulAR, an AR task assistant for procedural error recognition and recovery. RegulAR models task instructions as a hierarchical dependency graph and combines this structure with a Multimodal Large Language Model (MLLM) to interpret egocentric observations during execution. This enables RegulAR to track progress, identify deviations by error type, estimate their impact on later steps, and deliver appropriately salient interventions through an in-situ head-up display that visualizes task state and recovery guidance. By making procedural structure explicit, RegulAR supports not only next-step guidance, but also reasoning about what went wrong, why it matters, and how users can get back on track. In a within-subject study (N=12), participants reported better task-structure understanding and recovery support with RegulAR than the MLLM-only baseline.

Problem

• Error Recognition
• Augmented Reality
• Procedural Tasks
• Recovery Support

Innovation

• Hierarchical Dependency Graph
• Multimodal Large Language Model (MLLM)
• Egocentric Observations
• Error Recognition and Recovery
• In-situ Head-up Display