Exploring SAIG Methods for an Objective Evaluation of XAI

2026-02-09

AI Summary

This study addresses the lack of unified objective evaluation criteria in explainable artificial intelligence (XAI), primarily due to the absence of widely accepted “ground truth” labels for explanations. To tackle this challenge, the paper presents a systematic review and analysis of Synthetic Artificial Intelligence Ground Truth (SAIG) methods, which construct artificial ground truths to enable direct assessment of XAI techniques. It introduces, for the first time, a seven-dimensional classification framework that clarifies the distinctions and underlying logic among existing SAIG approaches. Through comprehensive literature review, comparative analysis, and categorization modeling, the work elucidates the core mechanisms of these methods. The findings reveal a significant lack of consensus in XAI evaluation and underscore the urgent need for standardization, offering both a theoretical foundation and a structured analytical perspective for future benchmark development.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Transparent, Interpretable, Explainable ML
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

The evaluation of eXplainable Artificial Intelligence (XAI) methods is a rapidly growing field, characterized by a wide variety of approaches. This diversity highlights the complexity of the XAI evaluation, which, unlike traditional AI assessment, lacks a universally correct ground truth for the explanation, making objective evaluation challenging. One promising direction to address this issue involves the use of what we term Synthetic Artificial Intelligence Ground truth (SAIG) methods, which generate artificial ground truths to enable the direct evaluation of XAI techniques. This paper presents the first review and analysis of SAIG methods. We introduce a novel taxonomy to classify these approaches, identifying seven key features that distinguish different SAIG methods. Our comparative study reveals a concerning lack of consensus on the most effective XAI evaluation techniques, underscoring the need for further research and standardization in this area.

Problem

• Explainable AI
• XAI evaluation
• objective evaluation
• ground truth
• Synthetic Artificial Intelligence Ground truth

Innovation

• Synthetic Artificial Intelligence Ground truth
• XAI evaluation
• taxonomy
• explainable AI
• objective evaluation