CARING-AI: Towards Authoring Context-aware Augmented Reality INstruction through Generative Artificial Intelligence

2025-01-27

AI Summary

Current AR tutorials lack contextual adaptability and are constrained by device capabilities and technical expertise, hindering personalized instruction in dynamic environments. To address this, we propose CARING-AI: a novel system that introduces environment navigation as an implicit contextual input mechanism to drive generative AI—integrating large language models (LLMs) with 3D animation generation—for real-time synthesis of spatiotemporally aligned, humanoid virtual-avatar-guided AR instructions. We establish an AIGC design space tailored for AR pedagogy, encompassing asynchronous, remote, and impromptu instructional scenarios. The system integrates multimodal perception, lightweight AR rendering, and humanoid avatar motion synthesis. Two user studies (N=12) demonstrate that authoring time decreases by 62% on average, contextual fidelity of instructions significantly improves, and high-fidelity AR instructional content can be deployed without coding.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Context-aware AR instruction enables adaptive and in-situ learning experiences. However, hardware limitations and expertise requirements constrain the creation of such instructions. With recent developments in Generative Artificial Intelligence (Gen-AI), current research tries to tackle these constraints by deploying AI-generated content (AIGC) in AR applications. However, our preliminary study with six AR practitioners revealed that the current AIGC lacks contextual information to adapt to varying application scenarios and is therefore limited in authoring. To utilize the strong generative power of GenAI to ease the authoring of AR instruction while capturing the context, we developed CARING-AI, an AR system to author context-aware humanoid-avatar-based instructions with GenAI. By navigating in the environment, users naturally provide contextual information to generate humanoid-avatar animation as AR instructions that blend in the context spatially and temporally. We showcased three application scenarios of CARING-AI: Asynchronous Instructions, Remote Instructions, and Ad Hoc Instructions based on a design space of AIGC in AR Instructions. With two user studies (N=12), we assessed the system usability of CARING-AI and demonstrated the easiness and effectiveness of authoring with Gen-AI.

Problem

• Augmented Reality
• Adaptability
• Context Awareness

Innovation

• CARING-AI
• Generative AI (GenAI)
• Augmented Reality (AR) Tutorials