Evaluation of Black-Box XAI Approaches for Predictors of Values of Boolean Formulae

2025-09-12

AI Summary

To address the lack of objective evaluation criteria for variable importance explanations in black-box XAI applied to Boolean function predictors, this paper formalizes variable importance based on actual causality and introduces B-ReX, a novel interpreter integrating causal analysis with the ReX framework. B-ReX quantifies explanation fidelity using Jensen–Shannon divergence, achieving a mean divergence of 0.072 ± 0.012 on large-scale Boolean formula benchmarks—significantly outperforming existing black-box XAI methods. Its core contributions are threefold: (1) the first systematic application of actual causality theory to assess interpretability in Boolean logic; (2) a computationally tractable, theoretically grounded measure of variable importance; and (3) an open-source, efficient, and scalable B-ReX toolkit, establishing a new paradigm for logic-based explainable AI.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Reasoning under Uncertainty: Causality
• Machine Learning: Causal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Evaluating explainable AI (XAI) approaches is a challenging task in general, due to the subjectivity of explanations. In this paper, we focus on tabular data and the specific use case of AI models predicting the values of Boolean functions. We extend the previous work in this domain by proposing a formal and precise measure of importance of variables based on actual causality, and we evaluate state-of-the-art XAI tools against this measure. We also present a novel XAI tool B-ReX, based on the existing tool ReX, and demonstrate that it is superior to other black-box XAI tools on a large-scale benchmark. Specifically, B-ReX achieves a Jensen-Shannon divergence of 0.072 $pm$ 0.012 on random 10-valued Boolean formulae

Problem

• Evaluating black-box XAI approaches for Boolean function predictors
• Proposing formal variable importance measure using actual causality
• Benchmarking state-of-the-art XAI tools against novel B-ReX

Innovation

• Proposed formal causality-based variable importance measure
• Introduced novel XAI tool B-ReX for Boolean functions
• Demonstrated superior performance on large-scale benchmark evaluation