Pairwise Matching of Intermediate Representations for Fine-grained Explainability

2025-03-28

AI Summary

In fine-grained image recognition, existing interpretability methods yield coarse localizations and fail to distinguish subtle intra-class variations. To address this, we propose PAIR-X, a pairwise interpretability framework that achieves high-precision local alignment visualization by matching intermediate-layer representations with Layer-wise Relevance Propagation (LRP) relevance scores. Our key contributions include: (i) the first pairwise intermediate representation matching mechanism; (ii) the first quantifiable evaluation metric tailored for fine-grained matching interpretability; and (iii) the first explanation outputs validated by domain experts as practically usable in animal and architectural re-identification tasks. PAIR-X jointly leverages intermediate activation features and relevance scores, incorporating similarity-driven local region alignment and confidence-weighted visualization. It outperforms state-of-the-art methods across 35 public re-ID benchmarks; achieves 100% expert usability endorsement; and quantitatively demonstrates superior discriminative rationality—its saliency maps better distinguish correct matches even when model similarity scores are identical.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

The differences between images belonging to fine-grained categories are often subtle and highly localized, and existing explainability techniques for deep learning models are often too diffuse to provide useful and interpretable explanations. We propose a new explainability method (PAIR-X) that leverages both intermediate model activations and backpropagated relevance scores to generate fine-grained, highly-localized pairwise visual explanations. We use animal and building re-identification (re-ID) as a primary case study of our method, and we demonstrate qualitatively improved results over a diverse set of explainability baselines on 35 public re-ID datasets. In interviews, animal re-ID experts were in unanimous agreement that PAIR-X was an improvement over existing baselines for deep model explainability, and suggested that its visualizations would be directly applicable to their work. We also propose a novel quantitative evaluation metric for our method, and demonstrate that PAIR-X visualizations appear more plausible for correct image matches than incorrect ones even when the model similarity score for the pairs is the same. By improving interpretability, PAIR-X enables humans to better distinguish correct and incorrect matches. Our code is available at: https://github.com/pairx-explains/pairx

Problem

• Generates fine-grained visual explanations for subtle image differences
• Improves interpretability of deep learning models in fine-grained categories
• Enables better human distinction between correct and incorrect matches

Innovation

• Leverages intermediate activations and relevance scores
• Generates fine-grained pairwise visual explanations
• Proposes novel quantitative evaluation metric