V-SHiNE: A Virtual Smart Home Framework for Explainability Evaluation

2026-02-12

AI Summary

Existing approaches struggle to scalably and authentically evaluate the quality of explanations in smart home systems and their impact on users. To address this gap, this work proposes a browser-based virtual smart home simulation framework that supports environment configuration, behavioral simulation, plug-in integration of explanation engines, and multimodal explanation delivery. The framework enables reproducible, user-centered evaluation through fine-grained logging of user interactions. It constitutes the first lightweight yet high-fidelity experimental platform specifically designed for evaluating explainable intelligent systems. A user study involving 159 participants demonstrates the feasibility and effectiveness of the framework, establishing a standardized infrastructure for assessing explainable AI in smart home contexts.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Intelligent Robots: Multi-Robot Systems

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Explanations are essential for helping users interpret and trust autonomous smart-home decisions, yet evaluating their quality and impact remains methodologically difficult in this domain. V-SHiNE addresses this gap: a browser-based smarthome simulation framework for scalable and realistic assessment of explanations. It allows researchers to configure environments, simulate behaviors, and plug in custom explanation engines, with flexible delivery modes and rich interaction logging. A study with 159 participants demonstrates its feasibility. V-SHiNE provides a lightweight, reproducible platform for advancing user-centered evaluation of explainable intelligent systems

Problem

• explainability evaluation
• smart home
• user-centered evaluation
• autonomous systems
• explanation quality

Innovation

• explainable AI
• smart home simulation
• user-centered evaluation
• virtual framework
• interaction logging