Are We Merely Justifying Results ex Post Facto? Quantifying Explanatory Inversion in Post-Hoc Model Explanations

2025-04-11

AI Summary

This paper identifies the “explanation inversion” problem in posteriori explanation methods (e.g., LIME, SHAP): such methods often rationalize model predictions *backward* from outputs rather than faithfully reconstructing the true input-to-output decision pathway—especially under spurious correlations. To address this, the authors propose Inversion Quantification (IQ), the first formal, quantifiable framework for measuring explanation inversion. They further introduce Reproduce-by-Poking (RBP), a model-agnostic enhancement strategy that applies forward perturbations to inputs and theoretically guarantees inversion mitigation. Extensive evaluation across tabular, image, and text modalities on synthetic benchmarks demonstrates that mainstream explanation methods consistently exhibit inversion; RBP reduces inversion by an average of 1.8% while offering both empirical efficacy and theoretical provability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Post-hoc explanation methods provide interpretation by attributing predictions to input features. Natural explanations are expected to interpret how the inputs lead to the predictions. Thus, a fundamental question arises: Do these explanations unintentionally reverse the natural relationship between inputs and outputs? Specifically, are the explanations rationalizing predictions from the output rather than reflecting the true decision process? To investigate such explanatory inversion, we propose Inversion Quantification (IQ), a framework that quantifies the degree to which explanations rely on outputs and deviate from faithful input-output relationships. Using the framework, we demonstrate on synthetic datasets that widely used methods such as LIME and SHAP are prone to such inversion, particularly in the presence of spurious correlations, across tabular, image, and text domains. Finally, we propose Reproduce-by-Poking (RBP), a simple and model-agnostic enhancement to post-hoc explanation methods that integrates forward perturbation checks. We further show that under the IQ framework, RBP theoretically guarantees the mitigation of explanatory inversion. Empirically, for example, on the synthesized data, RBP can reduce the inversion by 1.8% on average across iconic post-hoc explanation approaches and domains.

Problem

• Investigates if post-hoc explanations reverse input-output relationships
• Quantifies how explanations rely on outputs versus true decision processes
• Proposes method to mitigate explanatory inversion in popular explanation techniques

Innovation

• Proposes Inversion Quantification (IQ) framework
• Introduces Reproduce-by-Poking (RBP) enhancement
• Mitigates explanatory inversion in post-hoc methods