Gradient-Skipping Relevance Propagation for Efficient Explainability of Vision Transformers

2026-07-11

AI Summary

This work addresses a critical limitation in existing interpretability methods for Vision Transformers, which often overlook the heterogeneous importance of attention heads and treat residual connections as identity mappings, leading to attribution bias. To remedy this, the authors propose GradSkip, a novel approach that explicitly models the varying significance of attention heads and introduces a skip-connection-aware relevance propagation mechanism. By leveraging gradient-driven adaptive head weighting and dynamic allocation of attribution across residual pathways, GradSkip optimizes explanation fidelity in a data-dependent manner. The method achieves state-of-the-art attribution accuracy on ImageNet-1K and BloodMNIST while reducing computational overhead by over 14× compared to prior approaches, and further demonstrates superior localization capability in segmentation tasks.

Abstract

Vision Transformers (ViTs) are difficult to interpret because current methods of relevance propagation and attention flow do not fully consider some key architectural features, such as the uneven importance of attention heads and residual connections. Prior approaches typically assume uniform importance across attention heads; furthermore, they model skip connections as identity paths, leading to inaccurate relevance attribution. To address these issues, we introduce GradSkip, a novel relevance propagation method for ViTs based on adaptive head weighting and skip-aware propagation. GradSkip models the different importance of the attention heads and dynamically distributes relevance between the attention and residual paths. Experiments on ImageNet1K and BloodMNIST demonstrate a state-of-the-art faithfulness of GradSkip while requiring over 14 times fewer GFLOPs than the best-performing existing approaches. Additional evaluations using transformer-based segmentation confirm improved localization and alignment with ground-truth regions.

Problem

• Vision Transformers
• relevance propagation
• attention heads
• residual connections
• explainability

Innovation

• Gradient-Skipping
• Vision Transformers
• Relevance Propagation
• Adaptive Head Weighting
• Skip-Aware Propagation