Integrated Gradient Correlation: a Dataset-wise Attribution Method

2024-04-22

arXiv.org

AI Summary

Existing attribution methods operate at the single-sample level, limiting their ability to support task-level or dataset-scale interpretability—particularly when identifying critical input regions that must exhibit cross-sample stability. To address this, we propose Integrated Gradient Correlation (IGC), the first dataset-wise attribution framework. IGC models cross-sample correlations between feature attributions and model predictions, enabling region-specific statistical aggregation. It is compatible with both scalar prediction tasks (e.g., fMRI decoding) and classification tasks (e.g., MNIST). Crucially, IGC supports direct summation and statistical inference over cross-sample attribution maps. We validate IGC on the Natural Scenes Dataset (NSD) and MNIST, demonstrating its effectiveness in uncovering spatially selective decision patterns of deep models. By enabling large-scale, reproducible attribution analysis, IGC advances the frontier of interpretable deep learning.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Attribution methods are primarily designed to study the distribution of input component contributions to individual model predictions. However, some research applications require a summary of attribution patterns across the entire dataset to facilitate the interpretability of the scrutinized models. In this paper, we present a new method called Integrated Gradient Correlation (IGC) that relates dataset-wise attributions to a model prediction score and enables region-specific analysis by a direct summation over associated components. We demonstrate our method on scalar predictions with the study of image feature representation in the brain from fMRI neural signals and the estimation of neural population receptive fields (NSD dataset), as well as on categorical predictions with the investigation of handwritten digit recognition (MNIST dataset). The resulting IGC attributions show selective patterns, revealing underlying model strategies coherent with their respective objectives.

Problem

• Summarize dataset-wide attribution patterns for model interpretability
• Localize stable important input regions across numerous components
• Enable region-specific analysis and relate attributions to predictions

Innovation

• Dataset-wise attribution method
• Direct summation over components
• Correlates attributions with prediction scores