PhaseWin: An Efficient Search Algorithm for Faithful Visual Attribution

2026-06-16

AI Summary

This work addresses the high computational complexity of exhaustive search in faithful visual attribution, where existing greedy methods still require O(n²) forward passes, struggling to balance efficiency and faithfulness. The authors formulate faithful visual attribution as an ordered subset search problem and propose PhaseWin, an algorithm that leverages phased window search, monotonic evidence accumulation, adaptive candidate pruning, and local refinement. Under mild structural assumptions on features, PhaseWin achieves near-greedy faithfulness guarantees while maintaining controllable, linear evaluation complexity. Experiments across image classification, object detection, visual grounding, and image captioning demonstrate that PhaseWin attains the highest faithfulness with the fewest forward passes, successfully reducing complexity from O(n²) to O(n).

Abstract

Visual attribution is a fundamental tool for interpreting modern vision and vision-language models, particularly when their decisions must be inspected, diagnosed, or audited. Its goal is to explain how a model's decision depends on local regions of the visual input, typically by assigning an importance ordering over candidate image regions. Given an image partitioned into $n$ regions, faithful attribution can be cast as an ordered subset-search problem, in which progressively inserting the selected regions should recover the target model response as early as possible. Exhaustive search over region subsets incurs exponential cost, while the widely used greedy search still requires a quadratic number of model evaluations, because every selection step rescores all remaining candidates. We propose PhaseWin, an efficient subset-search algorithm for faithful visual attribution. PhaseWin reorganizes greedy region selection into a phased window-search procedure: rather than re-evaluating the full candidate set at every step, it alternates between global candidate screening, adaptive pruning, and localized window refinement, while preserving the essential region-ranking behavior of greedy search. We analyze PhaseWin under monotone evidence-accumulation conditions and show that, under feature-level structural assumptions, it attains controllable linear evaluation complexity together with near-greedy faithfulness guarantees. Extensive experiments on image classification, object detection, visual grounding, and image captioning show that, among all compared attribution methods, PhaseWin reaches high faithfulness with the fewest forward passes, empirically realizing the predicted reduction from $O(n^2)$ to $O(n)$. The code is available at https://github.com/Qihuai27/phasewin-va.

Problem

• visual attribution
• subset search
• faithfulness
• model interpretation
• efficiency

Innovation

• PhaseWin
• visual attribution
• subset search
• linear complexity
• faithfulness