PILAR: Personalizing Augmented Reality Interactions with LLM-based Human-Centric and Trustworthy Explanations for Daily Use Cases

2025-12-18

AI Summary

Current eXplainable AI (XAI) methods for everyday augmented reality (AR) applications fail to simultaneously satisfy real-time responsiveness, contextual awareness, personalization, and coherent multi-dimensional explanations (i.e., *when*, *what*, and *how*). Method: We propose PILAR—the first unified explainability framework that deeply integrates large language models (LLMs) into the AR interaction closed loop. PILAR synergistically combines on-device real-time object detection, personalized recommendation, and fine-tuned open-source LLMs within an edge–cloud collaborative architecture to generate low-latency, context-aware, and human-understandable dynamic explanations. Contribution/Results: User studies demonstrate that PILAR improves task completion speed by 40%, while significantly enhancing user satisfaction, perceived usability, and transparency. These results validate the effectiveness and trustworthiness of a single-model-driven, multi-dimensional LLM-based explanation paradigm in real-world AR settings.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Transparent, Interpretable, Explainable ML
• Computer Vision: Interpretability, Explainability, and Transparency

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Artificial intelligence (AI)-driven augmented reality (AR) systems are becoming increasingly integrated into daily life, and with this growth comes a greater need for explainability in real-time user interactions. Traditional explainable AI (XAI) methods, which often rely on feature-based or example-based explanations, struggle to deliver dynamic, context-specific, personalized, and human-centric insights for everyday AR users. These methods typically address separate explainability dimensions (e.g., when, what, how) with different explanation techniques, resulting in unrealistic and fragmented experiences for seamless AR interactions. To address this challenge, we propose PILAR, a novel framework that leverages a pre-trained large language model (LLM) to generate context-aware, personalized explanations, offering a more intuitive and trustworthy experience in real-time AI-powered AR systems. Unlike traditional methods, which rely on multiple techniques for different aspects of explanation, PILAR employs a unified LLM-based approach that dynamically adapts explanations to the user's needs, fostering greater trust and engagement. We implement the PILAR concept in a real-world AR application (e.g., personalized recipe recommendations), an open-source prototype that integrates real-time object detection, recipe recommendation, and LLM-based personalized explanations of the recommended recipes based on users' dietary preferences. We evaluate the effectiveness of PILAR through a user study with 16 participants performing AR-based recipe recommendation tasks, comparing an LLM-based explanation interface to a traditional template-based one. Results show that the LLM-based interface significantly enhances user performance and experience, with participants completing tasks 40% faster and reporting greater satisfaction, ease of use, and perceived transparency.

Problem

• Enhancing explainability in AI-driven AR systems
• Providing personalized and context-aware explanations for users
• Unifying explanation techniques to improve user trust and engagement

Innovation

• Uses LLM for context-aware personalized AR explanations
• Unified approach adapts to user needs dynamically
• Integrates real-time detection with LLM-based explanations