Towards a Unified Multidimensional Explainability Metric: Evaluating Trustworthiness in AI Models

2026-07-15

AI Summary

This work addresses the lack of a unified, multidimensional standard for evaluating explainability, which hinders reliable comparisons across models, datasets, and explainable AI (XAI) methods. The authors propose a comprehensive evaluation framework encompassing fidelity, conciseness, and stability, integrating diverse metrics into a unified explainability scoring system for the first time. Through systematic benchmarking of mainstream XAI approaches—including LIME and SHAP—on multiple open-source datasets, they construct an explainability meta-knowledge base. This knowledge base enables predictive scoring of explainability for new models and data contexts, significantly enhancing contextual adaptability. Experimental results validate the framework’s effectiveness and uncover patterns in how explainability varies with model architecture, data characteristics, and user backgrounds, offering a practical and scalable tool for trustworthy AI development.

Abstract

In this paper, we present a comprehensive framework for assessing the explainability of various XAI methods, such as LIME and SHAP, across multiple datasets and machine learning models, with the ultimate goal of creating a unified multidimensional explainability score. Our methodology focuses on three key aspects of explainability: fidelity, simplicity, and stability. We leverage benchmarking experiments to systematically evaluate these aspects and use the insights gained to construct an offline knowledge base. This knowledge base captures the explainability scores for each registered model and serves as a valuable resource for context-dependent evaluation of explainability. By analyzing the complementary characteristics and metadata of AI models, datasets, and XAI methods, the knowledge base will enable the estimation of explainability scores for previously unseen datasets and models. Properties like fidelity, simplicity, and stability may vary significantly based on the dataset, underlying model, and domain expertise of the end user. We demonstrate our framework by applying it to three open-source datasets, discussing the implications of the obtained results in relation to the characteristics of the datasets. Our work contributes to the growing field of XAI by providing a robust and versatile tool for evaluating and comparing the explainability of various XAI methods, ultimately supporting the development of more transparent and trustworthy AI systems.

Problem

• explainability
• trustworthiness
• XAI
• evaluation metric
• multidimensional

Innovation

• explainability metric
• XAI evaluation
• multidimensional framework
• knowledge base
• trustworthy AI