Quality of explanation of xAI from the prespective of Italian end-users: Italian version of System Causability Scale (SCS)

2025-04-22

AI Summary

This study addresses the lack of culturally adapted, explainable artificial intelligence (xAI) explanation quality assessment tools for Italian end users. To bridge this gap, we conducted the first cross-cultural adaptation and empirical validation of the System Causality Scale (SCS) into Italian. Employing a mixed-methods approach—including forward-backward translation, content validity ratio analysis (CVR ≥ 0.49), and cognitive interviews with representative Italian users—we refined the original scale, eliminating non-applicable items and yielding a 9-item Italian System Causality Scale (I-SCS). The I-SCS demonstrates strong reliability and validity, and is fully comprehensible to Italian users. It constitutes the first rigorously validated, culturally adapted xAI explanation quality instrument for the Italian context, filling a critical methodological void in human-centered xAI evaluation. The scale is immediately applicable in both academic research and industrial practice for assessing causal interpretability of AI systems.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability
• Reasoning under Uncertainty: Causality

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Background and aim: Considering the scope of the application of artificial intelligence beyond the field of computer science, one of the concerns of researchers is to provide quality explanations about the functioning of algorithms based on artificial intelligence and the data extracted from it. The purpose of the present study is to validate the Italian version of system causability scale (I-SCS) to measure the quality of explanations provided in a xAI. Method: For this purpose, the English version, initially provided in 2020 in coordination with the main developer, was utilized. The forward-backward translation method was applied to ensure accuracy. Finally, these nine steps were completed by calculating the content validity index/ratio and conducting cognitive interviews with representative end users. Results: The original version of the questionnaire consisted of 10 questions. However, based on the obtained indexes (CVR below 0.49), one question (Question 8) was entirely removed. After completing the aforementioned steps, the Italian version contained 9 questions. The representative sample of Italian end users fully comprehended the meaning and content of the questions in the Italian version. Conclusion: The Italian version obtained in this study can be used in future research studies as well as in the field by xAI developers. This tool can be used to measure the quality of explanations provided for an xAI system in Italian culture.

Problem

• Validate Italian version of System Causability Scale (I-SCS)
• Measure quality of explanations in xAI for Italian users
• Ensure cultural relevance of xAI explanation assessment tool

Innovation

• Used forward-backward translation method
• Validated Italian System Causability Scale
• Removed low validity question (CVR<0.49)