Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models

2026-06-28

AI Summary

This study addresses a critical epistemological gap in scientific machine learning: while post-hoc interpretability methods—such as feature importance and counterfactual explanations—enhance model transparency, they remain insufficient for substantiating claims about the true causal structures or mechanistic underpinnings of natural phenomena. Through an integrated philosophical analysis and critique of scientific modeling practices, the paper systematically demonstrates that even when a model’s predictions are reliable and its explanations faithfully reflect its internal logic, this does not warrant inferences about the actual mechanisms governing the target phenomenon. The work exposes fundamental cognitive limitations of explainable AI in scientific discovery, arguing that external validation is indispensable for formulating credible scientific hypotheses. On this basis, it proposes a novel epistemological framework to delineate the appropriate boundaries for employing model-based explanations within scientific reasoning.

Abstract

Post-hoc explanation methods are routinely used to interpret scientific machine learning models, with the deliverable understood to be insight into the phenomenon the model has been trained on. The transition may be taken to be secured once the model is reliable enough and the explanation faithful enough. We argue it is not. Reliability checks that the model's predictions match the phenomenon's outcomes, and faithfulness checks that the explanation matches the model, but neither checks whether the model works as the phenomenon works, which is what a claim about structure requires. The chain can support candidate hypotheses under external corroboration, but it cannot, on its own, support claims about how the phenomenon is in fact structured.

Problem

• reliability
• faithfulness
• post-hoc explanations
• scientific models
• model interpretability

Innovation

• post-hoc explanation
• reliability
• faithfulness
• scientific models
• model interpretability