🤖 AI Summary
Permutation feature importance (PFI) suffers from unreliable interpretations under feature correlations, as it generates unrealistic out-of-distribution samples that induce model extrapolation. To address this, we propose TRIP—a nonparametric hypothesis testing framework for diagnosing whether PFI scores are compromised by extrapolation. TRIP imposes no strong distributional assumptions, scales to high dimensions, and ensures robustness via theoretical guarantees and an adaptive resampling strategy. Extensive experiments on synthetic and real-world datasets demonstrate that TRIP accurately identifies misleading PFI estimates, thereby substantially improving the credibility of feature importance assessments. This work constitutes the first systematic approach to validating PFI reliability in extrapolatory regimes, providing a critical diagnostic tool for interpretable machine learning.
📝 Abstract
Along with accurate prediction, understanding the contribution of each feature to the making of the prediction, i.e., the importance of the feature, is a desirable and arguably necessary component of a machine learning model. For a complex model such as a random forest, such importances are not innate -- as they are, e.g., with linear regression. Efficient methods have been created to provide such capabilities, with one of the most popular among them being permutation feature importance due to its efficiency, model-agnostic nature, and perceived intuitiveness. However, permutation feature importance has been shown to be misleading in the presence of dependent features as a result of the creation of unrealistic observations when permuting the dependent features. In this work, we develop TRIP (Test for Reliable Interpretation via Permutation), a test requiring minimal assumptions that is able to detect unreliable permutation feature importance scores that are the result of model extrapolation. To build on this, we demonstrate how the test can be complemented in order to allow its use in high dimensional settings. Through testing on simulated data and applications, our results show that the test can be used to reliably detect when permutation feature importance scores are unreliable.