LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivity

2024-04-04

arXiv.org

AI Summary

Weak interpretability and low fidelity of attribution maps plague Vision Transformers (ViTs). To address this, we propose LeGrad—the first method to leverage gradients of layer-wise attention maps as the core explanatory signal. LeGrad generates high-fidelity attribution maps via weighted aggregation of multi-layer attention gradients, jointly modeling token responses from intermediate and final layers. It requires no architectural modification or additional training, effectively balancing low-level feature sensitivity with high-level semantic consistency. Evaluated on three challenging tasks—segmentation-based localization, perturbation robustness, and open-vocabulary explanation—LeGrad significantly outperforms state-of-the-art methods: it improves spatial localization accuracy by 12.6%, enhances noise robustness by 3.2×, and establishes an efficient, plug-and-play paradigm for ViT interpretability.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Machine Learning: Transparent, Interpretable, Explainable ML
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Vision Transformers (ViTs), with their ability to model long-range dependencies through self-attention mechanisms, have become a standard architecture in computer vision. However, the interpretability of these models remains a challenge. To address this, we propose LeGrad, an explainability method specifically designed for ViTs. LeGrad computes the gradient with respect to the attention maps of ViT layers, considering the gradient itself as the explainability signal. We aggregate the signal over all layers, combining the activations of the last as well as intermediate tokens to produce the merged explainability map. This makes LeGrad a conceptually simple and an easy-to-implement tool for enhancing the transparency of ViTs. We evaluate LeGrad in challenging segmentation, perturbation, and open-vocabulary settings, showcasing its versatility compared to other SotA explainability methods demonstrating its superior spatial fidelity and robustness to perturbations. A demo and the code is available at https://github.com/WalBouss/LeGrad.

Problem

• Visual Transformers
• ViT Model
• Interpretability

Innovation

• LeGrad
• Visual Transformer (ViT)
• Robustness and Accuracy