🤖 AI Summary
This work addresses the insufficient interpretability of vision-language models (VLMs) in high-stakes applications. We propose PixelSHAP—the first model-agnostic, pixel-level attribution framework specifically designed for VLMs. PixelSHAP extends Shapley value theory to structured visual entities by introducing zero-shot object masking and recomposition, embedding-space similarity measurement, and an efficient sampling strategy to quantify each pixel’s contribution to VLM outputs—thereby jointly supporting semantic object-level understanding and precise pixel-level localization. Crucially, it operates without access to model parameters or gradients, enabling black-box interpretation of commercial VLMs. Evaluated on real-world high-risk scenarios—including autonomous driving—PixelSHAP successfully identifies critical visual drivers underlying model decisions, substantially enhancing transparency and trust. An open-source implementation demonstrates strong cross-model robustness and practical utility across diverse VLM architectures.
📝 Abstract
Interpretability in Vision-Language Models (VLMs) is crucial for trust, debugging, and decision-making in high-stakes applications. We introduce PixelSHAP, a model-agnostic framework extending Shapley-based analysis to structured visual entities. Unlike previous methods focusing on text prompts, PixelSHAP applies to vision-based reasoning by systematically perturbing image objects and quantifying their influence on a VLM's response. PixelSHAP requires no model internals, operating solely on input-output pairs, making it compatible with open-source and commercial models. It supports diverse embedding-based similarity metrics and scales efficiently using optimization techniques inspired by Shapley-based methods. We validate PixelSHAP in autonomous driving, highlighting its ability to enhance interpretability. Key challenges include segmentation sensitivity and object occlusion. Our open-source implementation facilitates further research.