H-Sets: Hessian-Guided Discovery of Set-Level Feature Interactions in Image Classifiers

2026-04-23

AI Summary

This work addresses the limitation of existing image attribution methods, which predominantly focus on marginal effects of individual features and struggle to capture semantically meaningful interactions among pixels. To overcome this, the authors propose the H-Sets framework, which leverages the input Hessian matrix—used here for the first time—to detect local high-order feature interactions and recursively aggregates them into semantically coherent sets. Integrating Segment Anything as a spatial prior, the method employs IDG-Vis, an attribution technique grounded in Integrated Directional Gradients and Harsanyi dividends that satisfies established interpretability axioms, to produce set-level explanations. Experiments across multiple state-of-the-art models on ImageNet and CUB demonstrate that H-Sets generates saliency maps that are sparser, more faithful, and more interpretable, thereby advancing beyond current approaches in both modeling feature interactions and providing theoretical guarantees.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Feature attribution methods explain the predictions of deep neural networks by assigning importance scores to individual input features. However, most existing methods focus solely on marginal effects, overlooking feature interactions, where groups of features jointly influence model output. Such interactions are especially important in image classification tasks, where semantic meaning often arises from pixel interdependencies rather than isolated features. Existing interaction-based methods for images are either coarse (e.g., superpixel-only) or, fail to satisfy core interpretability axioms. In this work, we introduce H-Sets, a novel two-stage framework for discovering and attributing higher-order feature interactions in image classifiers. First, we detect locally interacting pairs via input Hessians and recursively merge them into semantically coherent sets; segmentation from Segment Anything (SAM) is used as a spatial grouping prior but can be replaced by other segmentations. Second, we attribute each set with IDG-Vis, a set-level extension of Integrated Directional Gradients that integrates directional gradients along pixel-space paths and aggregates them with Harsanyi dividends. While Hessians introduce additional compute at the detection stage, this targeted cost consistently yields saliency maps that are sparser and more faithful. Evaluations across VGG, ResNet, DenseNet and MobileNet models on ImageNet and CUB datasets show that H-Sets generate more interpretable and faithful saliency maps compared to existing methods.

Problem

• feature interactions
• image classifiers
• feature attribution
• interpretability
• Hessian

Innovation

• Hessian
• feature interactions
• saliency maps
• IDG-Vis
• interpretability