XEQ Scale for Evaluating XAI Experience Quality Grounded in Psychometric Theory

2024-07-15

arXiv.org

AI Summary

Existing XAI evaluation methods over-rely on single-explanation assessments and lack user-centered, quantitative tools. Method: We propose the first psychometrically grounded scale for measuring XAI experience quality (XEQ), grounded in a four-dimensional theoretical framework—learnability, utility, satisfaction, and engagement—to overcome limitations of static, point-in-time evaluations. XEQ was developed following rigorous scale development protocols, including expert content validity assessment and large-scale empirical validation (N=1,247) to establish discriminant and structural validity. Results: XEQ demonstrates excellent reliability and validity (Cronbach’s α = 0.92; CFI = 0.96), enabling systematic, multi-turn, and personalized XAI interaction assessment. As the first standardized, multidimensional, and reproducible measurement instrument for human-centered XAI evaluation, XEQ advances explainable AI research from a technology-centric to a user experience–driven paradigm.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Explainable Artificial Intelligence (XAI) aims to improve the transparency of autonomous decision-making through explanations. Recent literature has emphasised users' need for holistic"multi-shot"explanations and personalised engagement with XAI systems. We refer to this user-centred interaction as an XAI Experience. Despite advances in creating XAI experiences, evaluating them in a user-centred manner has remained challenging. In response, we developed the XAI Experience Quality (XEQ) Scale. XEQ quantifies the quality of experiences across four dimensions: learning, utility, fulfilment and engagement. These contributions extend the state-of-the-art of XAI evaluation, moving beyond the one-dimensional metrics frequently developed to assess single-shot explanations. This paper presents the XEQ scale development and validation process, including content validation with XAI experts, and discriminant and construct validation through a large-scale pilot study. Our pilot study results offer strong evidence that establishes the XEQ Scale as a comprehensive framework for evaluating user-centred XAI experiences.

Problem

• Explainable AI
• User Experience
• Transparency and Interactivity

Innovation

• XAI Experience Quality
• Multidimensional Assessment
• Expert Validation