Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling

2024-10-22

arXiv.org

AI Summary

Existing AR assistance systems predominantly operate in a passive, reactive manner, lacking the capacity to infer user intent and dynamically integrate environmental context—thereby limiting generalizability and intelligent adaptability. To address this, we propose the first proactive AR assistant that synergistically integrates the Belief–Desire–Intention (BDI) cognitive architecture with multimodal large language models (MLLMs). Our approach explicitly models users’ beliefs, desires, and intentions alongside evolving environmental context, enabling real-time, rule-free, situation-aware guidance. Crucially, it departs from conventional passive paradigms by supporting generalized, goal-directed reasoning. In a user study with 16 participants, our system achieves performance on par with an expert-crafted Wizard-of-Oz baseline—demonstrating efficacy, robustness, and practical deployability. This work significantly advances AR assistance by enhancing reusability across tasks and adaptability to diverse, dynamic scenarios.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Augmented Reality assistance are increasingly popular for supporting users with tasks like assembly and cooking. However, current practice typically provide reactive responses initialized from user requests, lacking consideration of rich contextual and user-specific information. To address this limitation, we propose a novel AR assistance system, Satori, that models both user states and environmental contexts to deliver proactive guidance. Our system combines the Belief-Desire-Intention (BDI) model with a state-of-the-art multi-modal large language model (LLM) to infer contextually appropriate guidance. The design is informed by two formative studies involving twelve experts. A sixteen within-subject study find that Satori achieves performance comparable to an designer-created Wizard-of-Oz (WoZ) system without relying on manual configurations or heuristics, thereby enhancing generalizability, reusability and opening up new possibilities for AR assistance.

Problem

• Augmented Reality
• Assistance System
• Proactivity

Innovation

• Satori AR System
• Contextual Understanding
• Adaptive Assistance