VISTA: Auditing Semantic Divergence in Vision-Language Models

2026-07-03

AI Summary

This work addresses the challenge that vision-language models may exhibit abnormally consistent responses to images containing demographic attributes, corporate logos, or ideological symbols—behaviors often undetectable through textual audits alone. To tackle this, the authors propose VISTA, a black-box, cross-model auditing framework that leverages semantic entropy and distributional divergence analysis to identify model-specific bias behaviors triggered by visual concepts without requiring internal model access. Applying VISTA across six state-of-the-art models and 19 thematic categories, the study uncovers 142 high-suspicion cases (1.2%), revealing a novel semantic disagreement pattern termed “selective refusal.” Notably, the method successfully reproduces concept-conditioned stances deliberately implanted via fine-tuning, demonstrating its efficacy in exposing visually induced biases.

Abstract

Vision-language models can exhibit visual concept-conditioned divergence: given images containing demographic features, corporate logos, or ideological symbols, some models produce unusually uniform responses that differ from what peer models say about the same input. These behaviors evade text-only audits because visual concepts cannot be isolated or substituted the way text tokens can. We present VISTA (Visual Inconsistency Screening Through Analysis), a black-box cross-model audit that couples semantic entropy with distribution-based divergence to flag model-specific anomalies. In a controlled study, we implant concept-conditioned stances in three VLMs via fine-tuning on small biased datasets and confirm that VISTA detects them. Auditing six VLMs across 19 topics, VISTA surfaces 142 high-suspicion cases (1.2%) and identifies selective refusal as a previously unreported divergence pattern, where models refuse demographic queries at rates varying from 0 to 65% across groups.

Problem

• vision-language models
• semantic divergence
• visual concept-conditioned divergence
• model auditing
• selective refusal

Innovation

• vision-language models
• semantic divergence
• black-box auditing
• semantic entropy
• selective refusal