Uncovering Bugs in Formal Explainers: A Case Study with PyXAI

2025-11-05

AI Summary

This work addresses the frequent generation of erroneous explanations by formal explainable artificial intelligence (XAI) tools—such as PyXAI—in practical implementations. We propose, for the first time, a systematic methodology for verifying the correctness of formal explanation generators. Our approach integrates formal verification theory with empirical testing frameworks to construct an automated detection pipeline that cross-validates both logical consistency and semantic correctness of explanations across multiple datasets. Experimental evaluation reveals that PyXAI produces logically inconsistent explanations on most benchmark datasets, exposing critical implementation flaws in state-of-the-art formal XAI tools. Beyond identifying a key reliability bottleneck in deploying formal XAI, our study establishes both the necessity and feasibility of implementation-level correctness verification. It thereby introduces a novel paradigm for developing and evaluating trustworthy XAI systems, grounded in rigorous, automation-supported validation of explanation logic.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

Formal explainable artificial intelligence (XAI) offers unique theoretical guarantees of rigor when compared to other non-formal methods of explainability. However, little attention has been given to the validation of practical implementations of formal explainers. This paper develops a novel methodology for validating formal explainers and reports on the assessment of the publicly available formal explainer PyXAI. The paper documents the existence of incorrect explanations computed by PyXAI on most of the datasets analyzed in the experiments, thereby confirming the importance of the proposed novel methodology for the validation of formal explainers.

Problem

• Validating practical implementations of formal explainable AI methods
• Developing a novel methodology to assess formal explainer tools
• Identifying incorrect explanations produced by PyXAI explainer system

Innovation

• Developed methodology for validating formal explainers
• Assessed PyXAI implementation through case study
• Detected incorrect explanations in formal explainer outputs