Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness

2026-05-26

AI Summary

This work addresses the problem of unfaithful explanations generated by large language model (LLM)-based explainable AI (XAI) agents, which often produce plausible-sounding but inaccurate rationales that misalign with the target model’s actual behavior. To mitigate this, the authors propose FAX, a novel framework that introduces an explicit verification mechanism: it decomposes explanations into verifiable claims, cross-validates them using intrinsic faithfulness tools, filters out contradictory content, and then synthesizes a final explanation. Additionally, they introduce CRAFTER-XAI-Bench, the first faithfulness evaluation benchmark tailored for open-world reinforcement learning. Experiments show that FAX improves simulated faithfulness from 0.20 to 0.46 on this benchmark while preserving high informativeness, relevance, and fluency; on tabular datasets, it matches existing methods’ performance, highlighting the limitations of conventional benchmarks that conflate task accuracy with explanation faithfulness.

Abstract

Explainable AI (XAI) helps users interpret model behavior and identify potential faults. Agentic XAI systems use Large Language Models (LLMs) to make explanations more accessible through natural-language interaction, but they can also produce plausible yet unfaithful explanations. This risk arises because unreliable XAI outputs for complex models can be amplified by LLMs and mislead users. We propose Faithful Agentic XAI (FAX), a framework that improves explanation faithfulness through explicit verification. FAX decomposes draft explanations into claims and cross-checks them against inherently faithful tools, filtering unsupported or contradictory claims before final generation. We also introduce CRAFTER-XAI-Bench, an open-world reinforcement learning benchmark with complex policies, diverse goals, and challenging scenarios for assessing model-specific faithfulness. On CRAFTER-XAI-Bench, FAX improves simulation faithfulness from 0.20 for the strongest baseline to 0.46 while maintaining high informativeness, relevance, and fluency. On three tabular benchmarks, FAX performs competitively with prior Agentic XAI baselines, but our analysis shows that these settings can conflate task accuracy with model-specific faithfulness. These findings show that explicit verification is essential for faithful Agentic XAI and that that faithfulness benchmarks must be designed to test explanations against the behavior of the target model itself.

Problem

• Explainable AI
• Faithfulness
• Agentic XAI
• Verification
• Benchmark

Innovation

• Faithful Agentic XAI
• explanation verification
• claim decomposition
• CRAFTER-XAI-Bench
• model-specific faithfulness